JP2010081562A - Imaging device, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To determine an appropriate standard to initiate photographing, based on the facial expression of a shooting object. <P>SOLUTION: An imaging device determines a photographing scene, based on the facial expression included in a facial-expression information stored in the RAM of a memory unit 119. The determined photographing scene is stored in the RAM of the memory unit 119. It is determined whether or not an evaluated value corresponding to one shooting object (current shooting object) arbitrarily selected by a CPU 112, in each of the facial-expression evaluated values corresponding to each of the shooting objects, exceeds a standard value (smile threshold value) selected or established by the CPU 112 as corresponding to the determined scene, in plural kinds of thresholds preliminarily stored in the memory unit 119. The current shooting object is determined as the shooting object of the facial expression in the scene, if exceeded. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging apparatus, a method, and a program for determining a shooting situation.

  In Patent Literature 1, the CPU extracts the face image of the person to be photographed from each frame image of the captured moving image, performs smile evaluation and neatness evaluation for each face image, and for each frame image, An overall smile evaluation value and an overall correct evaluation value for each subject are calculated. Further, the CPU analyzes one of the frame images, determines the scene of the captured image, and determines that the shooting is a casual scene shooting, sets the priority evaluation value as a smile evaluation value, If it is determined that the shooting is a formal scene, the priority evaluation value is set as a neat evaluation value, and the rank is set for each frame image in descending order of the priority evaluation value. Then, each frame image is displayed on the monitor based on the order.

  In Patent Document 2, in a period until an image signal obtained by imaging is recorded on a recording medium, the face detection unit detects a human face from the image signal, and the facial expression evaluation unit detects the detected face. The facial expression is evaluated, and a facial expression evaluation value indicating the degree of closeness to one facial expression between different facial expressions such as a smile and a normal facial expression is calculated. The notification control unit generates notification information corresponding to the calculated facial expression evaluation value, and performs control so as to notify the subject through, for example, a display, an LED light emitting unit, an audio output unit, or the like. Further, the facial expression evaluation unit requests the recording operation control unit to record the image signal on the recording medium when the facial expression evaluation value exceeds a predetermined threshold value.

  In Patent Document 3, when there are a plurality of subjects, an image is taken a plurality of times, a face with a high smile degree of each subject is extracted, and the images are combined into a single image.

  In Patent Document 4, a shape representing a facial expression of a person's subject is repeatedly extracted from an image photographed by a camera, and the degree of coincidence between the extracted shape of the subject and a predetermined shape is calculated. The subject is photographed when a predetermined value is exceeded.

  In Patent Document 5, feature points that form facial expressions are set in the extracted face image, and the feature amount of each face image is calculated by detecting the interval and position of the set feature points. For example, by identifying specific parts that make up a person's face in the image and determining the face area, after identifying the face area, the eyebrows, eyes, and mouth that form the facial expression of the person are identified, Feature points are set for each part of the eyebrows, both eyes, and the mouth.

In Patent Document 6, an attribute inside a face area having an eye area determined to have eyes closed is changed, a video signal of the face area with the changed attribute is generated, and this is combined with a recording image. Display on LCD. Further, it is assumed that a “cut-out face area” of a desired image selected from the recording images is pasted on the face area where the blinking is detected.
JP 2004-46591 A JP 2008-42319 A JP 2007-299297 A JP 2004-294498 A JP 2005-31423 A JP 2008-131094 A

  In Patent Document 1, a single subject can be evaluated, but a plurality of subjects cannot be evaluated comprehensively.

  In Patent Document 2, shooting is performed at the moment when a predetermined number or more of subjects are smiling, but there is a problem that a person who is not smiling is included.

  In Patent Document 3, the face portion of the photographed image is replaced with a smile face of each person photographed separately, but there is a problem that the extracted face and body are not synthesized accurately.

  In Patent Document 4, a picture is automatically taken when a subject has a predetermined expression, but there is no mention of a plurality of subjects.

  In Patent Document 5, it is not determined whether the facial expression is suitable for shooting.

  An object of the present invention is to determine an appropriate shooting start reference based on the facial expression of each subject.

  An imaging apparatus according to the present invention includes an imaging unit that sequentially converts an electrical signal obtained by continuously photoelectrically converting a subject image received through an imaging lens into image data and outputs the image data, and a first output from the imaging unit. A first evaluation value calculation unit that calculates an evaluation value indicating the facial expression of each subject based on one image data, and a shooting that determines a shooting scene based on the evaluation value of each subject calculated by the first evaluation value calculation unit A scene determination unit; a second evaluation value calculation unit that calculates an evaluation value indicating the facial expression of each subject based on second image data output after the first image data output by the imaging unit; A threshold setting unit that sets a scene determination threshold corresponding to the shooting scene determined by the determination unit, and whether or not the evaluation value of each subject calculated by the second evaluation value calculation unit exceeds the scene determination threshold set by the threshold setting unit Each subject is shot depending on Comprises a photographing state finalization determination unit that determines whether the undetermined object not in a state suitable for either photographic scene is determined subject a state suitable for over emissions, and.

  An imaging control unit that starts or permits main imaging by the imaging unit when the imaging state determination determination unit determines that each subject is a determined subject.

  When the predetermined time has passed without the evaluation value of the undetermined subject reaching the scene decision threshold, the shooting state decision determination unit determines the undetermined subject as the decided subject.

  When the predetermined time corresponding to the photographic scene determined by the photographic scene determination unit has passed without the evaluation value of the undetermined subject reaching the scene determination threshold, the shooting state determination determination unit determines the undetermined subject as the determination subject.

  A change determination unit configured to determine whether the position or background of the confirmed subject has changed based on the second image data, and the shooting state determination determination unit is configured so that the change determination unit is a fixed subject before the main imaging by the imaging unit; The determined subject is determined to be an undetermined subject in accordance with the determination that the position or background has changed.

  A finalized image storage unit that stores a finalized image showing the expression of the finalized subject in a predetermined storage medium, and a finalized image stored in the predetermined storage medium are combined with a through image that is an image sequentially output from the imaging unit. A display control unit for outputting to the display device.

  A mode setting unit that accepts the setting of the shooting mode is provided, and the threshold setting unit sets a scene determination threshold corresponding to the shooting mode set by the shooting mode setting unit and the shooting scene determined by the shooting scene determination unit.

  An image pickup method according to the present invention includes an image pickup apparatus including an image pickup unit that sequentially converts electric signals obtained by continuously photoelectrically converting a subject image received through an image pickup lens into image data and outputs the image data. A step of calculating an evaluation value indicating the facial expression of each subject based on the first image data output by the unit, a step of determining a shooting scene based on the calculated evaluation value of each subject, and an output of the imaging unit Calculating an evaluation value indicating the expression of each subject based on second image data output after the first image data, setting a scene determination threshold corresponding to the determined shooting scene, and calculating Depending on whether or not the evaluation value of each subject exceeds the scene decision threshold set by the threshold setting unit, each subject is a confirmed subject suitable for the shooting scene or a state suitable for the shooting scene And determining whether the no undetermined object, to run.

  An imaging program for causing the imaging apparatus to execute this imaging method is also included in the present invention.

  According to the present invention, the shooting scene is determined based on the evaluation value of the facial expression of each subject, and then each subject is suitable for shooting depending on whether or not the evaluation value of each subject exceeds the threshold corresponding to the shooting scene. It is determined whether or not a state has been reached. That is, by evaluating the expression of the subject as a whole, it is possible to appropriately set a reference for what kind of expression image is to be taken.

<First Embodiment>
FIG. 1 is a functional block diagram of a digital camera 10 according to a preferred embodiment of the present invention. First, in the digital camera 10 shown in the figure, an information processing apparatus (in addition to the CPU, peripheral devices necessary for information processing of the CPU, such as ROM, RAM, I / O, etc. are included. These are collectively referred to as a CPU for convenience). 112 controls each circuit in the digital camera 10 based on inputs from various buttons and keys of the operation unit 113 including a release switch, a mode dial, and the like. A program executed by the CPU 112 is stored in the ROM of the storage unit 119.

  The image pickup unit 102 receives an image pickup lens including a zoom lens and a focus lens, a motor for driving the image pickup lens, a motor driver for driving the motor in accordance with a command from the CPU 112, receives a subject image formed by the image pickup lens, performs photoelectric conversion, and performs analog conversion. It includes a solid-state imaging device such as a CCD or CMOS that outputs as an image signal.

  Now, when the still image shooting mode is set by the mode dial, the CPU 112 displays the frame images (through images) sequentially obtained from the imaging unit 102 on the display unit 110 so that the shooting angle of view can be confirmed.

  The imaging unit 102 includes an A / D converter, a gain adjustment circuit, an image signal processing circuit, a white balance correction circuit, a gamma correction circuit, a YC processing circuit, a luminance / color difference signal generation circuit, a sharpness correction circuit, a contrast correction circuit, and a captured image. An image processing unit including an outline processing unit that performs image processing including contour correction, a noise reduction processing unit that performs image noise reduction processing, and the like is processed, and an image signal is processed according to a command from the CPU 112 and converted into digital image data.

  The image data output from the imaging unit 102 is converted into a luminance signal (Y signal) and a color difference signal (Cr, Cl signal), and subjected to predetermined processing such as gamma correction, and then the VRAM in the storage unit 119. Stored in

  On the other hand, only the G pixel component is extracted from the image data by the high pass filter (HPF) 125. Then, the G pixel component extracted by the integration unit 126 is integrated and sent to the CPU 112. The CPU 112 calculates an average value of areas near the subject center (for example, four areas) arbitrarily set with the cross key among the image data of the divided areas (for example, 64 divisions) for one screen sent from the integrating unit 126. This is used as an autofocus (AF) evaluation value. The AF evaluation value is calculated every elapse of a predetermined period, and updated and stored in the storage unit 119 each time it is calculated. The CPU 112 determines the lens position at which the subject is focused according to the AF evaluation value. The determination method is arbitrary. For example, it is determined whether or not the AF evaluation value has substantially reached the vicinity of the maximum point. If the AF evaluation value has substantially reached the maximum point, the AF evaluation value is determined to be in focus. In this case, it is determined that the subject is not in focus (contrast AF). Alternatively, if the AF evaluation value exceeds a predetermined threshold, it can be determined that the subject is in focus.

  When the captured image is output to the display unit 110 on the monitor, the YC signal is read from the VRAM and sent to the video encoder 134. The video encoder 134 converts the input YC signal into a predetermined display signal (for example, an NTSC color composite video signal) and outputs the converted signal to the display unit 110.

  The YC signal of each frame processed at a predetermined frame rate is sequentially written in the VRAM, and a video signal generated from the YC signal is supplied to the display unit 110, so that the image being captured is displayed in real time. 110. The user can check the shooting angle of view from the video (through) displayed on the display unit 110.

  Here, when the release switch is half-pressed, the AE and AF processes are started. The AE / AF process is performed, and the recording operation is started when the release switch is fully pressed. The image data for recording acquired in response to the full press of the release switch is stored in the image storage unit 132. In the case of capturing a moving image, the above-described images are continuously captured, and each image constitutes a moving image as a frame image (frame image). Movie capture starts when the release switch is fully pressed for the first time and ends when the release switch is fully pressed for the second time. Alternatively, the video is captured only while the full press continues. The sound collected from the microphone 17 can also be captured synchronously with the acquisition of the moving image.

  The Y / C signal stored in the image storage unit 132 is compressed by the CPU 112 according to a predetermined format (for example, Exif (Exchangeable Image File Format) for a still image, MPEG4 for a moving image), and then stored as a file on an external recording medium. To be recorded. The image is recorded in the data part of the Exif file. The CPU 112 records shooting date / time information and the like in a predetermined tag (such as an image description tag) in the header portion of the Exif file. The recorded image can be transmitted to various electronic devices 200 such as a personal computer 200a, a printer 200b, and a PDA connected via the communication interface 76.

  The printer 200b prints a still image received from the camera 10 or the personal computer 200a on a mount made of a paper medium such as plain paper or photographic paper in a predetermined arrangement. Still images arranged in a predetermined arrangement can be output as electronic media such as “PDF” of Adobe Systems Inc. to a display, a recording medium, a personal computer connected via the Internet, and the like. In short, the output destination of the created image print is not limited to a specific one. In this specification, an electronic medium creation form on which a desired image can be arranged at a desired position is also called a mount.

  When the moving image shooting mode is set by the mode dial, the moving image recording operation starts in conjunction with the full pressing operation of the release switch, and the moving image recording operation stops when the release switch is pressed again. The recording operation may be performed while the release switch is continuously pressed, and the recording may be stopped by releasing the press. The moving image data is stored in the image storage unit 132 in the MPEG4 format, for example.

  When the playback mode is selected by the mode dial, the compressed data of the last image file (last recorded file) stored in the image storage unit is read out. The read image compressed data is decompressed into an uncompressed YC signal by the CPU 112 and added to the video encoder 134. The video encoder 134 creates an NTSC RGB color composite video signal from the input YC signal and outputs it to the display unit 110. Thereby, an image is displayed on the display unit 110.

  The display unit 110 corresponds to an external display device connected to an LCD, a finder, a video output terminal, or the like built in the camera 10. The video encoder 134 includes an OSD signal generation circuit 135. The OSD signal generation circuit 135 displays characters such as a shutter speed, an aperture value, the number of images that can be photographed, a photographing date and time, a warning instruction, and a symbol such as an icon in accordance with a command from the CPU 112. Generate a signal to display. The video encoder 134 mixes the signal output from the OSD signal generation circuit 135 with an image signal as necessary, and supplies it to the LCD 114. As a result, a combined image in which other images such as characters and icons are combined with the through image and the reproduced image is displayed.

  The image storage unit 132 is means for holding image data (still images, moving images) obtained by shooting, and for example, a memory card called smart media is used. The form of the recording medium is not limited to this, and it may be a PC card, a compact flash (registered trademark), a magnetic disk, an optical disk, a magneto-optical disk, a memory stick, etc., electronic, magnetic, optical, or a combination thereof A readable / writable medium can be used in accordance with the above-described method. A configuration in which a plurality of media can be mounted regardless of different types or the same type of recording media may be adopted. The means for storing the image file is not limited to a removable medium that can be attached to and detached from the camera body, but may be a recording medium (built-in memory) built in the camera.

  The CPU 112 is a control circuit that performs overall control of each circuit of the camera 10 and includes an operation unit including a release switch, a cross key, a power switch, a mode dial, an information position designation key, a strobe button (not shown), a zoom key, a menu / execution button, and the like. Based on the signal received from 113, the operation of the corresponding circuit is controlled, and display control in the display unit 110, strobe light emission control, AF auxiliary light emission control, autofocus (AF) control, automatic exposure (AE) control, etc. I do.

  Each time the power supply of the camera 10 is turned on by the power switch, power is supplied to each circuit of the digital camera 10 from a main power supply (not shown) composed of a battery mounted inside the camera body.

  The communication interface 76 can be connected to various information processing devices such as a personal computer and a printer, and transmits and receives data in accordance with commands from the CPU 112.

  The motion detection circuit 21 includes an acceleration sensor, a potentiometer, a gravity sensor, a tilt sensor, a posture sensor, an electronic compass, and the like. Specifically, the motion detection circuit 21 outputs a left-right motion information indicating the left-right motion direction and the motion amount of the camera 100, and a vertical motion indicating the up-down motion direction and the motion amount of the camera 100. A vertical motion detection unit that outputs information, and a tilt motion detection unit that outputs tilt motion information indicating the tilt direction and the tilt amount of the housing of the camera 100 with respect to the horizontal direction are provided. The motion information includes left and right motion information, up and down motion information, tilt motion information, a time when detection of each motion is started, a time when detection of each motion is finished, and a time when detection of each motion is continued. The time information output from the RTC 15 is used.

  For example, the distance measuring unit 81 emits light from a light emitting unit to a subject within the shooting range of the imaging unit 102, receives light returned from the subject and returns the subject from an angle formed by the light emitting unit and the light receiving unit. A triangulation distance sensor that calculates the distance up to can be used. Alternatively, various known techniques can be adopted as long as the object is a means for measuring a distance such as a sensor applying the TOF (Time of Flight) principle or a distance image (3D image) of a compound eye camera.

  A touch panel 114 is stacked on the display unit 110. When the user presses a corresponding part of the display unit 110 with a finger or a pen, information indicating the pressed position is output to the CPU 112.

  A speaker 85 is connected to the CPU 112 via an audio processing circuit 84. The audio processing circuit 84 drives the speaker 85 based on the notification sound data (shutter sound, warning sound, etc.) sent from the CPU 112, thereby outputting the notification sound and the sound recorded together with the moving image.

  In the video shooting mode, the analog audio signal acquired from the microphone 17 is converted into digital data after being amplified by the audio processing circuit 84 and is subjected to data processing such as being compressed in a predetermined compression recording format (such as MPEG4). The image is stored in the image storage unit 132 together with the acquired image. Even in the still image shooting mode, it is possible to record audio before and after the main shooting instruction. In addition to the sound stored in the image storage unit 132, various notification sound data such as operation sound, shutter sound, and AF in-focus sound stored in advance in the storage unit 119 are decoded and analog sound by the sound processing circuit 84. After data processing such as signal conversion and amplification is performed, the data is output to the speaker 85. The speaker 85 outputs sound according to the analog sound signal output from the sound processing circuit 84.

  An RTC (real time clock) 15 outputs current date and time information. This current time information can be recorded together with the image as shooting date / time information of the recording image by the imaging unit 102.

  The GPS receiver 120 obtains current position information of the camera 10 by receiving radio waves from GPS satellites and measuring the distance from the orbit. This current position information can be stored in the image storage unit 132 together with an image (for example, as meta information or header information) as shooting position information of an image for recording by the imaging unit 102.

  A microphone 17 is installed on the rear surface of the camera casing, and these various devices are provided around the display unit 110.

  The image analysis unit 145 analyzes the image output from the imaging unit 102 or the image storage unit 132 and creates and outputs facial expression information, composition information, subject information, image quality information, and the like according to the result.

  For example, facial expression information is information that identifies the facial expressions of individual subjects such as anger, disgust, fear, sadness (sadness), happiness (laughter), surprise, no expression (serious), is there.

  The composition information includes the presence / absence of a person and the position of a main subject (typically a focused subject, but may be a subject with face detection). The subject information includes the position of the face area, subject size, person name, presence / absence of subject motion, direction of motion, and speed of motion. Note that various known techniques may be used for composition analysis, subject extraction, face detection, and image quality analysis (for example, Japanese Patent Application Laid-Open Nos. 2007-97090 and 2005-354606 by the present applicant). Such information may be recorded as supplementary information of the image.

  The image quality information includes image information indicating the presence / absence of blurring / blurring and the presence / absence of eye-meditation. As for shake detection, as in the case of conventional camera shake detection, it is determined that there is camera shake when the size of the acceleration vector detected by the motion detection circuit 21 exceeds a predetermined threshold. The blur detection can be performed based on, for example, edge detection (for example, Japanese Patent Application Laid-Open No. 2005-122721 by the present applicant). A known technique can also be used for determining whether or not eyes are closed (for example, Japanese Patent Application Laid-Open No. 2007-49631 by the present applicant).

  The digital camera 10 may be any device that can acquire a subject image as image data, and may include a mobile phone with a camera, a personal computer with a camera, a digital video camera, a digital still camera, and the like.

  The storage unit 132 stores character information and template data to be combined with image data, and the image processing unit 144 adds desired data to a desired image in accordance with an instruction from the CPU 112 or a selection operation from the operation unit 113. Is synthesized.

  FIG. 2 is a flowchart of the photographing process. This process starts according to the setting of the shooting mode. Alternatively, it may be started in response to an instruction for photographing preparation operation (half-pressing the shutter button) being input to the operation unit 113. A program that causes the CPU 112 to execute this processing is stored in the storage unit 119. The “CPU 112” is not built in the camera 10 itself, but may include another electronic device, for example, a built-in computer 200a. That is, the execution subject of this processing is the CPU of the personal computer 200a, and the control target can be the camera 10 separate from the personal computer 200a.

  Since this process includes a process (S3) for determining a scene to be photographed from now on, it is better that the timing is closer to the main imaging timing. Therefore, this process starts immediately before the main imaging, for example, when the shutter button is half-pressed. Good.

  In S1, acquisition of a through image is started. The through image is periodically acquired at predetermined intervals.

  In S <b> 2, the image analysis unit 145 analyzes the acquired latest through image, creates facial expression information corresponding to each subject included in the through image, and outputs it to the RAM of the storage unit 119. Discrimination of the facial expression of the subject can be performed as described in Patent Documents 2 and 5.

  In S3, a shooting scene is determined based on the facial expression included in the facial expression information stored in the RAM of the storage unit 119. For example, the expression of each subject is replaced with a numerical value proportional to various types of scenes (the larger the numerical value is, the closer the laughing face is to the angry face, etc.), and the scene corresponding to the average value of these numerical values is determined. Or, if the smile expression of each subject is majority, a smile scene, if the serious face is majority, a serious scene, if an angry facial expression is majority, an angry face scene, a sad face is majority. If there is a sad scene. The determined shooting scene is stored in the RAM of the storage unit 119.

  In S4, it is determined whether or not the scene determined in S3 is a smile scene. In the case of Yes, the process proceeds to S5, and in the case of No, the process proceeds to a serious scene photographing subroutine described later.

  In S5, the image analysis unit 145 is controlled to analyze the latest through image acquired after S3, create facial expression information corresponding to each subject included in the through image, and output to the RAM of the storage unit 119. . Further, based on the facial expression information corresponding to each subject stored in the RAM, a smile degree evaluation value indicating the degree of smile of the facial expression of each subject is calculated and stored in the RAM of the storage unit 119. For example, as in Patent Document 2, whether a certain subject's facial expression is close to a smile or a facial expression other than a smile (no facial expression, angry facial expression, sad facial expression, surprised facial expression, disgusting facial expression) is a numerical value. To make the smile evaluation value of the subject.

  Of the smile degree evaluation values corresponding to each subject, smile degree evaluation values corresponding to one subject (current subject) arbitrarily selected by the CPU 112 are stored in the storage unit 119 in advance. It is determined whether or not the reference value (smile threshold) selected and set by the CPU 112 as corresponding to the smile scene determined in S4 is exceeded. For example, identification information is given in order from the subject in the through image at the upper left corner to the lower right corner, and the current subject is selected one by one along the order of the identification information. If the smile level evaluation value of the current subject> the smile threshold value, the current subject is determined to be a smile subject and the process proceeds to S9. If smile level evaluation value ≦ the smile threshold value, the current subject is determined to be a non-smile subject and the process proceeds to S6. move on.

  In S6, it is determined whether the position of the current subject has changed, or whether the background has changed (including the case where the photographer himself moves to change the background, and so on). This is because, for example, the existing area of the current subject detected in the latest through image acquired is stored, the image in the area corresponding to the existing area is extracted from the current through image, and the current subject's If difference information with the image is created and the difference information becomes 0 (no color data), it is determined that the position of the current subject has not changed. Conversely, if the difference information is not 0, it is determined that the position of the current subject has changed.

  Alternatively, an area (background area) other than the existence area of all subjects detected in the previously acquired through image is stored, and an image in an area corresponding to the background area is extracted from the current through image, Difference information with the background area of the through image acquired last time is created, and if the difference information becomes 0 (no color data), it is determined that the background has not changed. Conversely, if the difference information is not 0, it is determined that the background has changed.

  Alternatively, it is not necessary to determine the change in the position of the subject and the change in the background based on only the image information. If the distance to the subject measured by the distance measuring unit 81 changes, it can be determined that the position of the subject has changed, or GPS reception can be performed. If the current position information of the machine 120, the azimuth information output by the electronic compass included in the motion detection circuit 21 or the pan / tilt direction of the camera 10 detected by the tilt detection circuit change, it can be determined that the background has changed.

  If it is determined that the position of the subject has changed or the background has changed, the process proceeds to S7, and if it is determined that both have not changed, the process proceeds to S8. Note that “change” refers to a substantial change, and changes within a predetermined error range are not determined to be changes.

  In S7, the smile expression information and smile expression evaluation values of all the subjects in the RAM are cleared, and all the subjects are determined as non-smile subjects. Then, the process returns to S5. Or, when the background changes, the smile expression information and smile expression evaluation value of all subjects are cleared, and when the subject position changes, the smile expression information and smile expression evaluation values of only the subject whose position has changed are cleared. The subject may be determined as a non-smile subject. By doing this, it is possible to prevent the use of an old fixed image (described later) before the position and background are largely changed.

  In S8, a predetermined facial expression standby time (for example, 5 seconds) elapses after it is determined in S5 that the smile facial expression evaluation value of the current subject has fallen below the smile threshold (non-smiling subject) or the scene is determined in S3. Determine whether or not. If Yes, the current subject is determined to be a smiling subject and the process proceeds to S9. If No, the process returns to S5. The predetermined facial expression standby time may be a length corresponding to the scene determined in S4.

  In S9, the face image of the current subject is extracted from the latest through image, and is stored in the storage unit 119 as a confirmed face image associated with the identification information of the current subject. The face image can be extracted by a known technique. If the current subject is a smiling subject (Yes in S5), the confirmed face image is a face image with a smiling expression. If the current subject is not a smiling subject (No in S5), the confirmed face image is a facial image with an expression other than smile.

  After the confirmed face image is stored, the display of the confirmed face image is locked regardless of the output of the through image. That is, the CPU 112 instructs the video encoder 134 to synthesize a confirmed face image with new through images sequentially input thereafter. The composition position of the confirmed face image to the new through image is set to the same coordinates as the coordinates in the through image from which the confirmed face image has been extracted.

  In S10, it is determined whether or not the saving of the confirmed face image has been completed for all subjects including the current subject. If yes, go to S11, if no, go to S12.

  In S <b> 11, the imaging unit 102 is controlled to perform main shooting (acquisition of an image for recording). This may be directly instructed to start shooting, or a message suitable for the user is displayed by displaying on the display unit 110 a message that a photo opportunity has arrived or a message such as “It is photo opportunity” from the speaker 85. It is also possible to notify that the chance has arrived and to start shooting while allowing a shooting instruction by pressing the shutter button of the operation unit 113.

  A recording image (which may be a still image or a moving image) acquired by actual photographing under the control of the imaging unit 102 is subjected to various processing by the image processing unit 144 and then compressed and stored in the image storage unit 132. Even if a subject corresponding to a face area that has not reached the smile threshold is included in the recording image, if the confirmed face image of the subject that has reached the smile threshold is stored, the confirmed face image is used for recording. Correction may be made by replacing the subject in the image. This is the same as a known face image replacement / correction technique such as Patent Document 6.

  In S12, one subject is arbitrarily selected from subjects that have not become current subjects since the scene determination (S3), and this is set as a new current subject, and the processing from S5 is continued. Instead of switching the current subjects one by one in S12, assuming that all of the plurality of subjects are current subjects, S5 to S11 may be performed in parallel for all the current subjects. In this case, S12 is not necessary. The processing of S5 to S10 may be repeated every time a new through image is supplied, or may be repeated every frame interval or time interval of a predetermined through image.

  FIG. 3 shows a serious scene shooting subroutine that proceeds when it is determined No in S4.

  In S24, it is determined whether or not the scene determined in S3 is a serious scene. If Yes, the process proceeds to S25, and if No, the process proceeds to a subroutine described later.

  In S25, the image analysis unit 145 is controlled to analyze the latest through image acquired after S3, and facial expression information corresponding to each subject included in the through image is created and output to the RAM of the storage unit 119. . Then, based on the facial expression information corresponding to each subject stored in the RAM, a seriousness evaluation value indicating the degree of seriousness of the facial expression of each subject is calculated and stored in the RAM of the storage unit 119. For example, as in Patent Document 2, whether an expression of a certain subject is close to an expression or whether it is close to an expression other than an expression (a smile, an angry expression, a sad expression, a surprised expression, an expression of disgust). It is digitized and used as the seriousness evaluation value of the subject.

  Then, out of each of the seriousness evaluation values corresponding to each subject, the seriousness evaluation values corresponding to one subject (current subject) arbitrarily selected by the CPU 112 are a plurality of types of threshold values stored in the storage unit 119 in advance. It is determined whether or not a reference value (serious face threshold) selected and set by the CPU 112 as corresponding to the serious scene determined in S24 is exceeded. For example, identification information is given in order from the subject in the through image at the upper left corner to the lower right corner, and the current subject is selected one by one along the order of the identification information. If the seriousness evaluation value of the current subject is greater than the seriousness face threshold, the current subject is determined to be a serious face subject and the process proceeds to S29. Otherwise, the current subject is determined to be a non-serious face subject and the process proceeds to S26. .

  In S26, it is determined whether the position of the current subject has changed or the background has changed. This can be done in the same manner as S6 described above.

  If it is determined that the position of the subject has changed or the background has changed, the process proceeds to S27. If it is determined that both have not changed, the process proceeds to S28.

  In S27, as in S7, the expression information of all or some of the subjects in the RAM and the serious facial expression evaluation value are cleared. Then, the process returns to S5.

  In S28, whether or not a predetermined facial expression waiting time (for example, 5 seconds) has elapsed since it was determined in S25 that the serious facial expression evaluation value of the current subject has fallen below the serious face threshold, or since the scene was determined in S3. Judging. In the case of Yes, the current subject is determined to be a serious face subject and the process proceeds to S29, and in the case of No, the process returns to S25. The predetermined facial expression waiting time may be a length corresponding to the scene determined in S24. For example, since a serious expression is easier to make than a natural smile, the expression standby time corresponding to a serious face scene is 5 seconds, and the expression standby time corresponding to a smile scene is 8 seconds.

  In S29, the face image of the current subject is extracted from the latest through image, and stored in the storage unit 119 as a confirmed face image associated with the identification information of the current subject. The face image can be extracted by a known technique. If the current subject is a serious face subject, the confirmed face image is a face image with a serious expression. If the current subject is not a serious face subject, the confirmed face image is a face image with an expression other than expressionless.

  In S30, it is determined whether or not the saving of the confirmed face image has been completed for all subjects including the current subject. If yes, go to S31, if no, go to S32.

  In S31, the main photographing is performed as in S11. This may be to automatically start shooting, or to allow shooting instruction by pressing the shutter button and start shooting.

  In S32, one subject is arbitrarily selected from subjects that have not become the current subject so far after the scene determination (S3), and this is set as a new current subject, and the processing from S25 onward is continued. Instead of switching the current subjects one by one in S32, assuming that all of the plurality of subjects are current subjects, S25 to S31 may be performed in parallel for all the current subjects. In this case, S32 is not necessary.

  The subroutine executed after the determination of No in S24 performs the same processing as S5 to S12 for facial expressions other than smiles and serious faces, such as angry faces and sad faces. That is, a threshold corresponding to each scene is prepared in the process corresponding to S5. However, this subroutine can be omitted if it is not necessary to save the confirmed face area for expressions other than smiles and serious faces. In short, S5 to S12 are subroutines that can be applied to any scene as long as a threshold corresponding to each scene and expression is used.

  Note that the evaluation value threshold values in S5 and S25 or the time threshold values in S8 and S28 may be changed according to the shooting mode. For example, when the indoor shooting mode or the portrait shooting mode is set from the operation unit 113 in S5 and S25, the CPU 112 sets the evaluation value thresholds used in S5 or S25 to V1-1 and V1-2, respectively. When a shooting mode related to outdoor shooting from 113, for example, a sports mode, is set, a threshold value V2-1 smaller than the evaluation value threshold value V1-1 and a threshold value V2-2 smaller than the evaluation value threshold value V1-2 are set. Although the evaluation value threshold values V1-1 to V2-2 are stored in the storage unit 119, the CPU 112 selects a threshold value corresponding to the shooting mode. In this way, in the case of indoor shooting, it is possible to wait more strictly for a facial expression suitable for shooting.

  Alternatively, when the indoor shooting mode or the portrait shooting mode is set from the operation unit 113 in S8 and S28, the CPU 112 sets the time thresholds T1-1 and T1-2 in S8 and S28, respectively, and the outdoor shooting is performed from the operation unit 113. In S8 and S28, the CPU 112 sets the threshold T2-1 smaller than the time threshold T1-1 and the threshold T2-2 smaller than the time threshold T1-2, respectively. . In this way, in the case of indoor shooting, it is possible to lengthen the time for waiting for a facial expression suitable for shooting.

  For example, a through image as shown in FIG. 5 is obtained from a shooting scene including three subjects SB1, SB2, and SB3 as shown in FIG. 4, and as shown in FIG. 6, the subjects SB1, SB2, and SB3 Smile level evaluation values V (SB1), V (SB2), and V (SB3) corresponding to the subjects SB1, SB2, and SB3 are calculated based on the face areas F1, F2, and F3. An image with a border around the recognized face area may be generated by the OSD signal generation circuit 135 and displayed on the display unit 110.

  In the above processing, if the current subject is SB1 and V (SB1)> smile threshold, the process proceeds to S9, otherwise the process proceeds to S6. Alternatively, if the current subject is SB2 and V (SB2)> smile threshold, the process proceeds to S9, otherwise the process proceeds to S6. Alternatively, if the current subject is SB3 and V (SB3)> smile threshold, the process proceeds to S9, otherwise the process proceeds to S6.

  In FIG. 6, subjects SB1 and SB3 are smiling subjects. Although SB2 is not a smiling subject, a confirmed face image is secured in the storage unit 119 for all of SB1 to SB3.

  As shown in FIG. 7, the OSD signal generation circuit 135 generates an image in which a predetermined color is applied to the edge of the face area of the smiling subject and displays it on the display unit 110, indicating that the saving of the confirmed face image is completed. May be notified.

  After the confirmed face images are stored for all the subjects, the face area of each subject is locked as the confirmed face image. That is, even after the confirmed face image is stored for all the subjects, even if the expression of the subject is different from that when the confirmed face image is stored, it is not displayed on the display unit 110.

  For example, the subject SB1 is a smile as shown in FIG. 4, and the confirmed face image is stored as a smile face image. After that, the subject SB1 smiles for some reason as shown in FIG. Suppose that Even in this case, as shown in FIG. 9, a smile fixed face image of the stored subject SB1 is synthesized in the face area of the subject SB1 of the subsequent through image, and the expression is fixed as the smile fixed image. Let Also, the coloring of the frame F1 is continued.

  In addition, for a subject whose confirmed face image is not saved, if the reference value corresponding to each scene is exceeded before the expression change waiting time elapses, the face area of the subject is also saved as a confirmed face image, and is confirmed in subsequent through images. The face image is fixed.

  Further, even when the scene of FIG. 4 is shifted to the scene of FIG. 8 and the facial expression of the subject SB1 changes from a smile to no expression, the confirmed face image of SB1 is synthesized and displayed on the through image as shown in FIG. The expression of is fixed to the confirmed face image. After that, when the scene of FIG. 8 transitions to a scene as shown in FIG. 10 and the subject SB2 is not saved for the subject SB2 whose face image is not saved, the face area of the subject SB2 is It is stored as a confirmed face image, and a through image as shown in FIG. 11 is displayed. Thereafter, the confirmed face image of SB2 on the display unit 110 is fixed. For the subjects SB1 and SB3, after the determined smile face image is stored, even if the smile changes to no expression, it is not displayed.

  Also, as shown in FIG. 12, when a certain time has passed without the smile face image being stored for the subject SB2 and the subject SB2 remains expressionless, the confirmed face image of the subject SB2 is as shown in FIG. The face area is expressionless.

  In this way, once the facial expressions suitable for the scene have been prepared, the photographer does not have to be confused by the change in the facial expression of the subject. Press the button to perform shooting.

  The confirmed face image may include not only the face area but also a body part connected to the face. For example, it is assumed that the shooting scene is a group photo requesting a serious face, and there are four human subjects in the image I in FIG. 14, and the facial expression evaluation value of one of these subjects corresponds to the shooting scene. Suppose you exceed the standard value. An area having a predetermined positional relationship with the face area of the subject exceeding the reference value, for example, an area obtained by enlarging the face area 3 × 4 times obliquely up, down, left and right around the face area is defined as the confirmed subject area X, and the confirmed subject The image in the area X is stored as a substitute (confirmed area) for the confirmed face image. After saving the confirmed area, even if the subject in the confirmed subject area X changes its facial expression, the saved confirmed area remains displayed unless the position or background changes. Thus, even if the confirmed area larger than the face area is stored, the sense of discomfort with the surroundings is small even if the confirmed area is displayed as long as the background and the position of the person in the group photo do not change greatly from those determined once.

<Second Embodiment>
In the first embodiment, after saving the fixed face images (or alternatives thereof, the same applies hereinafter) corresponding to all subjects, the saved fixed face images are displayed as they are even if the facial expression of the subject changes. However, when the position of the subject changes greatly, it is meaningless to continue to display the confirmed face image as it is at the same position as long as the subject is in another position.

  Therefore, the following position change determination process is added to the flowchart of FIG. That is, as shown in FIG. 15, even when the confirmed face image is stored in S9, before proceeding to S10, it is determined whether or not there is a determination similar to S6, that is, whether or not the position of the subject has changed ( S9 ′) If it is determined that there is no position change, the process proceeds to S10, and if it is determined that there is a position change, the determined face image of the subject stored in the storage unit 119 is discarded (together other than the subject is determined). The face image may be discarded), and the composition of the confirmed image is canceled, and the process returns to S5 to display the through image. This additional processing can be added between S30 and S31, and although not shown, it can also be added to subroutines other than smiles and serious faces.

  As shown in FIG. 16, for a subject SB1, 2, 3 for which the confirmed face image is not stored, if the subject SB1,3 becomes a smile before the expression change waiting time elapses, the face area of the subject SB1,3 becomes a confirmed face image. The determined face image is synthesized in the subsequent through images as shown in FIG. Thereafter, for example, when the position of the subject SB2 in which the confirmed face image is stored changes as shown in FIG. 18, the confirmed face image of the subjects SB1 and 3 is discarded as shown in FIG. 19, and the frames around the confirmed face image are also deleted. The through image is displayed as it is.

  In S <b> 9 ′, it may be determined whether the background has changed instead of the subject position. If the background has changed, the process returns to S6, and if not, the process proceeds to S10. This is effective when there is a high possibility that the shooting position and the background will change as in outdoor shooting. This additional processing can be added to subroutines other than smiles and serious faces between S30 and S31.

Functional block diagram of digital camera Flow chart of shooting process Flow chart of shooting subroutine Diagram showing an example of a shooting scene Diagram showing an example of a through image The figure which shows an example of the image | video which shows the circumference | surroundings of the recognized face area | region The figure which shows an example of the image | video which shows the face area of a smiling subject The figure which shows a mode that the expression of the preserved subject of the fixed face image has changed The figure which shows an example of the through image by which the fixed face image was synthesize | combined The figure which shows another example of a photography scene The figure which shows another example of the through image by which the fixed face image was synthesize | combined. The figure which shows another example of a photography scene The figure which shows another example of the through image by which the fixed face image was synthesize | combined. A figure showing an example of a fixed area Flow chart of position change determination processing The figure which shows another example of a photography scene The figure which shows another example of the through image by which the fixed face image was synthesize | combined. The figure which shows another example of the imaging | photography scene where the to-be-photographed object position changed. The figure which shows an example of the through image which canceled the synthesis | combination of the fixed face image

Explanation of symbols

10: Camera, 112: CPU, 200: Electronic device

Claims (9)

An imaging unit that sequentially converts electrical signals obtained by continuously photoelectrically converting a subject image received through an imaging lens into image data and outputs the image data;
A first evaluation value calculation unit that calculates an evaluation value indicating the facial expression of each subject based on the first image data output by the imaging unit;
A shooting scene determination unit that determines a shooting scene based on the evaluation value of each subject calculated by the first evaluation value calculation unit;
A second evaluation value calculation unit for calculating an evaluation value indicating the expression of each subject based on second image data output after the first image data output by the imaging unit;
A threshold setting unit for setting a scene determination threshold corresponding to the shooting scene determined by the shooting scene determination unit;
A determined subject in which each subject is in a state suitable for the shooting scene, depending on whether or not the evaluation value of each subject calculated by the second evaluation value calculation unit exceeds a scene determination threshold set by the threshold setting unit A shooting state determination determination unit that determines whether the subject is an unconfirmed subject that is not suitable for the shooting scene;
An imaging apparatus comprising:   The imaging apparatus according to claim 1, further comprising: an imaging control unit that starts or permits main imaging by the imaging unit when the imaging state determination determination unit determines that each subject is a determined subject.   3. The shooting state confirmation determination unit according to claim 1 or 2, wherein when a predetermined time has passed without an evaluation value of the unconfirmed subject reaching the scene confirmation threshold, the unconfirmed subject is determined as the confirmed subject. Imaging device.   When the predetermined time corresponding to the shooting scene determined by the shooting scene determination unit has elapsed without the evaluation value of the unfixed subject reaching the scene determination threshold, the shooting state determination determination unit determines the undetermined subject as the The imaging device according to claim 3, wherein the imaging device is determined to be a confirmed subject. A change determination unit that determines whether the position or background of the determined subject has changed based on the second image data;
The imaging state determination determination unit determines the determination subject as an undetermined subject when the change determination unit determines that the position or background of the determination subject has changed before the main imaging by the imaging unit. Item 5. The imaging device according to any one of Items 1 to 4. A confirmed image storage unit that stores a confirmed image indicating the expression of the confirmed subject in a predetermined storage medium;
A display control unit that synthesizes a confirmed image stored in the predetermined storage medium with a through image that is an image sequentially output from the imaging unit and outputs the synthesized image to a predetermined display device;
An imaging apparatus according to any one of claims 1 to 5. It has a mode setting unit that accepts shooting mode settings,
The imaging apparatus according to claim 1, wherein the threshold setting unit sets a scene determination threshold corresponding to a shooting mode set by the shooting mode setting unit and a shooting scene determined by the shooting scene determination unit. An imaging apparatus including an imaging unit that sequentially converts an electrical signal obtained by continuously photoelectrically converting a subject image received through an imaging lens into image data and outputs the image data;
Calculating an evaluation value indicating the facial expression of each subject based on the first image data output by the imaging unit;
Determining a shooting scene based on the calculated evaluation value of each subject;
Calculating an evaluation value indicating the facial expression of each subject based on second image data output after the first image data output by the imaging unit;
Setting a scene determination threshold value corresponding to the determined shooting scene;
Depending on whether or not the calculated evaluation value of each subject exceeds a scene decision threshold set by the threshold setting unit, whether or not each subject is a confirmed subject in a state suitable for the shooting scene. Determining whether the subject is an indeterminate subject that is not suitable;
An imaging method for executing.   An imaging program for causing an imaging apparatus to execute the imaging method according to claim 8.

Images