JP2013131970A - Imaging device, imaging method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable appropriate determination of a scene in which best shot photographing should be performed.SOLUTION: A face is detected from a preview image, and further, it is determined whether the face is a smiling face. Then, when the face is not a smiling face, a past history is traced back, and when a predetermined time period has not elapsed since a smiling face became undetected and a face is detected by face detection, best shot photographing is performed, thereby enabling prevention of selection of an inappropriate image by best shot photographing in a state where a face cannot be detected.

Description

  The present invention particularly relates to an imaging apparatus, an imaging method, and a program suitable for use in performing best shot imaging.

  Conventionally, a technique for detecting a human face from a photographed image and detecting whether the detected face is a specific expression such as a smile is known. Further, in response to a single shooting instruction, images of a person including a face image are continuously shot, and one image having the best shooting state is selected from a plurality of shot images and recorded. Imaging devices have been proposed so far. For example, in Patent Document 1, the position, angle, size, and the like of a face are normalized based on the detection position of a facial organ, and a specific facial expression is detected using the obtained normalized face image. Techniques to do this are disclosed. Patent Document 2 discloses a technique capable of recording a best shot image in which a specific facial expression is photographed from continuously photographed images.

  Conventionally, in a function called best shot shooting, first, a face of a person is detected in a time series from a preview image before a recording image is shot, and the detected face is a specific facial expression (hereinafter referred to as a facial expression). , “Smile” will be described as an example). Next, when the user gives a shooting instruction in a state where the detected face is determined to be a smile, the images are continuously shot, the smile level of the shot images, the eye-meditation result, The degree of blurring is evaluated, and an optimal single image is selected based on the evaluation results.

  At this time, even if the preview image does not determine that the facial expression of the subject is a smile, if the user gives a shooting instruction before a predetermined time elapses after the smile cannot be detected, the best shot shooting is performed. Has been implemented. Since it takes some time to evaluate smiles, it takes time until a smile is determined again when it is temporarily determined that it is not a smile even though it is originally a smile. This is a proposed technique. In such a scene where smiles and non-smiles are frequently switched, even if the user returns to smile immediately after it is not determined to be a smile, if the user gives a shooting instruction, the best shot is immediately taken. It can be carried out.

JP 2007-213378 A JP 2009-89077 A

  However, when the best shot is taken using the technique described above, the following problems occur. This problem will be described with reference to FIG. In FIG. 3, a face surrounded by a dotted line indicates that the face is detected by face detection.

FIG. 3A is a diagram illustrating an operation example of switching between the shooting method of the best shot shooting and the single shooting when a smile is first detected, but a smile is not detected due to a straight face in the middle. It is.
As shown in FIG. 3A, since a smile is detected in the frames 301 to 304 of the preview image, the best shot shooting is performed when a shooting instruction is issued from the user during that time. Also, although the face has changed from the frame 305 to a true face, as described above, when the user gives a shooting instruction until a predetermined time elapses after the smile cannot be detected, the best shot shooting is performed. Is done. Therefore, the best shot shooting is performed in the frames 305 to 307 within the predetermined time range. Then, after the frame 308 after a predetermined time has elapsed, single shooting is performed when a shooting instruction is issued from the user.

On the other hand, FIG. 3 (b) shows the best shot shooting and single shooting when a smile is first detected, but the face turns sideways and the face itself is no longer detected. It is a figure which shows the operation example of switching of the imaging | photography system.
As shown in FIG. 3B, since smiles are detected in the frames 311 to 314 of the preview image, the best shot shooting is performed when a shooting instruction is issued from the user during that time. However, since no face is detected from the frame 315, no smile is detected. In this case, in the conventional technique, in the frames 315 to 317 until a predetermined time elapses, the best shot shooting is performed when a shooting instruction is issued from the user.

  In this way, in the conventional technology, even when the face cannot be detected because the subject is facing sideways or hidden behind the object, if a shooting instruction is given until a certain time has passed since the smile could not be detected, Take the best shot. When the best shot shooting is performed, the shooting time increases due to continuous shooting, and the processing time for selecting an optimal image increases. However, since a face is not detected, it is not possible to evaluate smiles and eye meditation, an optimal image cannot be selected, and there are problems that extra shooting time and processing time are required.

  The present invention has been made in view of the above-described problems, and it is an object of the present invention to appropriately determine a scene on which best shot shooting should be performed.

  An imaging apparatus according to the present invention is detected by an imaging unit that images a subject to generate a preview image, a face detection unit that detects the face of the subject from the preview image generated by the imaging unit, and the face detection unit. A facial expression detection means for detecting whether or not the face is a specific facial expression, and while the specific facial expression is detected by the facial expression detection means, a still image is continuously photographed in accordance with a photographing instruction from a user. Control means for controlling the image pickup means, and a control means for selecting an image having the highest evaluation value as a recording image from among the plurality of still images taken continuously, and the specifying by the expression detection means When the facial expression changes from a state in which the specific facial expression is detected to a state in which the specific facial expression is not detected, the control means does not detect the specific facial expression. Depending on the detection result by the detecting means, the timing for switching the control to shoot one still image is characterized by the different by the imaging means.

  According to the present invention, it is possible to appropriately determine a scene to be shot with the best shot, and it is possible to prevent an inappropriate image from being obtained or taking extra processing time.

1 is a block diagram illustrating a schematic configuration example of an imaging apparatus according to an embodiment. It is a flowchart which shows an example of the imaging | photography process sequence in embodiment. It is a figure explaining the timing which switches the conventional best shot imaging | photography and single imaging | photography. It is a figure explaining the timing which switches the best shot imaging | photography and single shooting imaging | photography in embodiment. It is a figure explaining the timing which switches best shot photography and single photography in the case of reducing the number of continuous shots of best shot photography.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
FIG. 1 is a block diagram illustrating a schematic configuration example of an imaging apparatus 100 according to the embodiment.
1, an imaging apparatus 100 according to the embodiment includes an optical system 1 including a lens and a diaphragm, a mechanical shutter (mechanical shutter) 2, an imaging element 3, a CDS circuit 4, an A / D converter 5, a timing signal generation circuit 6, And a drive circuit 7. Further, a signal processing circuit 8, an image memory 9, an image recording medium 10, a recording circuit 11, an image display device 12, a display circuit 13, a face detection circuit 14, a facial expression detection circuit 15, and a system control unit 16 are provided.

  The CDS circuit 4 performs analog signal processing on an image signal obtained via the optical system 1, the mechanical shutter 2, and the image sensor 3. The A / D converter 5 converts an analog signal into a digital signal. The timing signal generation circuit 6 is a circuit that generates a signal for operating the imaging device 3, the CDS circuit 4, and the A / D converter 5. The drive circuit 7 is a circuit that drives the optical system 1, the mechanical shutter 2, and the image sensor 3.

  The signal processing circuit 8 performs signal processing necessary for the image data, and stores the signal processed image data in the image memory 9. The recording circuit 11 records the image-processed image data on the image recording medium 10 that is removable from the imaging apparatus 100. The display circuit 13 displays an image based on the signal processed image data on the image display device 12.

  The face detection circuit 14 is a circuit for detecting a person's face from image data, and the expression detection circuit 15 is a circuit for detecting whether the face detected by the face detection circuit 14 is a smile. . Then, the system control unit 16 controls the entire imaging apparatus 100.

  Hereinafter, a photographing operation using the mechanical shutter 2 in the imaging apparatus 100 configured as described above will be described. First, the drive circuit 7 drives the aperture and the lens of the optical system 1 by a control signal from the system control unit 16, and the optical system 1 forms an object image set to an appropriate brightness on the image sensor 3. Let Next, the mechanical shutter 2 is driven by the drive circuit 7 in accordance with a control signal from the system control unit 16 so as to shield the image sensor 3 in accordance with the operation of the image sensor 3 so as to have a necessary exposure time. At this time, when the image pickup device 3 has an electronic shutter function, it may be used together with the mechanical shutter 2 to ensure a necessary exposure time.

  The image pickup device 3 is driven by a drive circuit 7 by a drive pulse based on an operation pulse generated by a timing signal generation circuit 6 controlled by a system control unit 16, and converts an object image into an electrical signal by photoelectric conversion to generate an analog image signal. Output as. The analog image signal output from the image pickup device 3 is subjected to an operation pulse generated by the timing signal generation circuit 6 controlled by the system control unit 16, and the clock synchronization noise is removed by the CDS circuit 4. 5 is converted into a digital image signal.

  Next, the signal processing circuit 8 performs image processing such as color conversion, white balance, and gamma correction, resolution conversion processing, and image compression processing on the digital image signal. The signal processing circuit 8 may output the digital image signal as it is to the image memory 9 or the recording circuit 11 as it is without performing signal processing by the control signal from the system control unit 16. Further, in response to a request from the system control unit 16, the signal processing circuit 8, for example, information on the image data generated in the process of signal processing such as the spatial frequency of the image, the average value of the designated area, the data amount of the compressed image, or the like The information extracted from the information is output to the system control unit 16. Further, the recording circuit 11 outputs information such as the type and free capacity of the image recording medium 10 to the system control unit 16 when requested by the system control unit 16.

  Next, the reproduction operation when image data is recorded on the image recording medium 10 will be described. First, the recording circuit 11 reads image data from the image recording medium 10 by a control signal from the system control unit 16. Then, when the image data is compressed data, the signal processing circuit 8 performs an image expansion process according to a control signal from the system control unit 16 and stores it in the image memory 9. The image data stored in the image memory 9 is subjected to resolution conversion processing in the signal processing circuit 8 and then converted into a signal suitable for the image display device 12 in the display circuit 13, and the image is displayed on the image display device 12. Is displayed.

  The face detection circuit 14 performs a known face detection process on the image data from the signal processing circuit 8 to detect a face area of a person included in a captured image of the image sensor 3. As a known face detection process, for example, a skin color region is extracted from the gradation color of each pixel represented by image data, and a face is detected with a matching degree with a face contour plate prepared in advance. is there. Also disclosed is a method of performing face detection by extracting facial feature points such as eyes, nose and mouth using a well-known pattern recognition technique. Note that the face detection processing technique is not limited to the technique described above, and various known techniques can be used.

  The facial expression detection circuit 15 determines a facial expression such as a smile for the face detected by the face detection circuit 14. Specifically, features necessary for calculating facial expressions, such as the contours of facial components such as eyes, nose, and mouth that make up the face, and the positions of facial components such as the positions of the eyes, corners of the eyes, nose, mouth corners, and lips Get the quantity. As a method for acquiring the feature amount, a discriminator for each face component obtained by template matching using a template of each face component or machine learning learning using a large number of sample images of the face component It can be obtained by a technique using. Then, the expression detection circuit 15 calculates an expression level indicating the degree to which the face is a specific expression based on the feature amount, and determines whether or not the face is a specific expression. The method for discriminating the facial expression is not limited to this, and various known methods can be used.

  Next, the smile detection performed by the system control unit 16 and the best shot shooting control associated therewith will be described with reference to the flowchart shown in FIG. The smile detection and the best shot shooting associated therewith are executed according to a computer program stored in a memory (not shown) in the system control unit 16. The smile detection control is sequentially performed in time series on the preview images sequentially captured at a predetermined frame rate by the image sensor 03. The interval at which smile detection control is performed depends on the processing capability of the face detection circuit 14, but may be about once every several frames.

  In a function called best shot shooting, first, a face of a person is detected in a time series from a preview image before a recording image is shot, and whether or not the detected face is a smile is determined. To do. Next, when the user gives a shooting instruction in a state where the detected face is determined to be a smile, the images are continuously shot, the smile level of the shot images, the eye-meditation result, In addition, the degree of blur is evaluated. Then, based on those evaluation results, an optimal single image is selected as a recording image.

FIG. 2 is a flowchart illustrating an example of a shooting processing procedure in the present embodiment.
First, in step S201, a face image is detected by the face detection circuit 14 from the preview image generated by the above-described procedure to detect a face image, and a smile detection process is performed based on the detected face image. This is performed by the facial expression detection circuit 15.

  In the smile detection process, first, an edge is detected from the face area, and the face area is divided into a plurality of areas. Then, an evaluation is made as to how closely the direction and strength of the edge in each region matches the direction and strength of the edge for smile determination in that region, and if the evaluation value exceeds a predetermined value, it is determined to be a smile To do. For example, in the case of a smile, the frying line is emphasized, the corners of the eyes are lowered, and the corners of the mouth are raised. In this way, it is only necessary to evaluate whether or not an edge that matches such a condition has occurred in the corresponding area of the face area.

  Further, smile evaluation requires more processing time than detection of a face area. For this reason, a low evaluation value is calculated due to the influence of the face orientation, etc., even though it is actually a smile, and if it is determined that it is not a smile, it will be a little before it is determined that it is a smile again. take time.

  Next, in step S202, it is determined whether or not the face of the subject is a smile as a result of the smile detection process in step S201. If the result of this determination is that the subject's face is smiling, the process proceeds to step S206, and setting is made such that best shot shooting is performed when a shooting instruction is issued from the user. At this time, an icon or the like indicating that the determination is made with a smile may be displayed on the image display device 12 by the display circuit 13 to notify the user.

  If the result of determination in step S202 is that the subject's face is not smiling by the smile detection process in step S201, the process proceeds to step S203. In step S203, it is determined whether there is a history in which a smile has been detected in the past frame. The system control unit 16 includes a buffer that temporarily stores the result of smile detection for each frame detected by the facial expression detection circuit 15. Then, by reading the smile detection result history in the past frame from a predetermined storage area secured in the buffer, it is determined whether or not there is a history of smile detection in the past frame.

  As a result of the determination in step S203, if there is no history of smile detection in the past frame, the process proceeds to step S207, where normal single shooting is set when a shooting instruction is issued from the user.

  On the other hand, if the result of determination in step S203 is that there is a history of smile detection in the past frame, the process proceeds to step S204 to determine whether or not a predetermined time has passed since the smile was not detected. To do. As a result of this determination, if a predetermined time has elapsed from when the smile is no longer detected until now, the process proceeds to step S207, and when a shooting instruction is issued from the user, normal single shooting is performed.

  On the other hand, as a result of the determination in step S204, if a predetermined time has not elapsed since the smile was no longer detected, the process proceeds to step S205, where it is determined whether the face is detected by the face detection circuit 14. . If the result of this determination is that a face has been detected, the process proceeds to step S206, where it is set to perform best shot shooting when a shooting instruction is issued from the user.

  When a face is detected when it is determined that it is not a smile from a state where it is determined that it is a smile, continuous shooting is performed if there is a shooting instruction from the user within a predetermined time as in the conventional case. Perform image evaluation and selection.

  On the other hand, if the face is not detected as a result of the determination in step S205, the process proceeds to step S207, where normal single shooting is set when a shooting instruction is issued from the user.

  If it is determined that it is not a smile from a state where it is determined that it is a smile, and if a face is not detected, if there is a shooting instruction from the user within a predetermined time, instead of continuous shooting, a single still Take a picture. If no face is detected in this way, it is considered unlikely that the smile determination has failed even though the face is actually a smile. Therefore, only one still image shooting is performed in order to suppress unnecessary image shooting and image evaluation load.

  Here, an operation example in the present embodiment will be described with reference to FIG. In FIG. 4, the face surrounded by a dotted line indicates that the face has been detected.

FIG. 4A is a diagram illustrating an operation example of switching between the shooting method between the best shot shooting and the single shooting when a smile is detected at first, but a smile is not detected due to becoming a true face in the middle. It is.
As shown in FIG. 4A, since smiles are detected in the frames 401 to 404 of the preview image, the best shot shooting is performed when a shooting instruction is issued from the user during that time. In addition, although the face has changed from the frame 405 to a true face, as described above, when the user gives a shooting instruction until a predetermined time elapses after the smile cannot be detected, the best shot shooting is performed. It is the mechanism that is done. For this reason, the best shot shooting is performed in the frames 405 to 407 within the predetermined time range.

  For example, in a scene where smiles and non-smiles are frequently switched, the best shot can be taken immediately when the user instructs to take a picture even if it returns to smile immediately after it is no longer determined to be a smile. Can do. After frames 408 and 409 after a predetermined time has elapsed, single shooting is performed when a shooting instruction is issued from the user.

On the other hand, in FIG. 4B, the smile was initially detected, but the face turned sideways from the middle and the face itself was not detected. It is a figure which shows the operation example of switching of the imaging | photography system with a copy.
As shown in FIG. 4B, since smiles are detected in the frames 411 to 414 of the preview image, the best shot shooting is performed when a shooting instruction is issued from the user during that time. Further, no faces are detected in the frames 415 to 419, and therefore no smile is detected. In the present embodiment, since no face is detected, when a shooting instruction is issued from the user without waiting for the elapse of a predetermined time, the camera immediately switches to single shooting.

  Conventionally, as in the example shown in FIG. 4B, even when the face turns sideways and the face itself is no longer detected, the user takes a picture until a predetermined time elapses after the smile cannot be detected. When an instruction is issued, the best shot shooting is performed. As described above, although the face is not detected, the shooting time increases due to continuous shooting, or the processing time increases in order to select an optimal image. In addition, since no face is detected, it is impossible to evaluate smiles and eye meditation, and an optimal image cannot be selected.

  On the other hand, in the present embodiment, if a face is not detected within a predetermined time after no smile is detected, single shooting is performed when a shooting instruction is issued from the user. For this reason, it is possible to prevent an increase in processing time due to the best shot shooting and to select an image with low accuracy.

(Other embodiments)
In addition, embodiment mentioned above shows an example and is not limited to this. In the embodiment described above, when a shooting instruction is issued from the user until a predetermined time elapses after the smile cannot be detected, the same best shot shooting as when a smile is detected is performed. It is configured. Furthermore, the configuration may be such that the number of continuous shots for best shot shooting is reduced as time passes.

  For example, as shown in FIG. 5, when the preview image is frames 501 to 504 and a smile is detected, five shots are taken continuously. When the smile is no longer detected, while the preview image is in the frames 505 to 507, the number of continuous shots is reduced to four, three, two, etc. as time elapses. It may be configured. When the preview images become frames 508 and 509, the mode is switched to single shooting. Furthermore, in the present embodiment, a smile is taken as an example in the case of detecting a specific facial expression, but other facial expressions may of course be used. As described above, the configuration and operation in the above-described embodiment can be appropriately changed.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed. As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

DESCRIPTION OF SYMBOLS 1 Optical system 2 Mechanical shutter 3 Image pick-up element 7 Drive circuit 14 Face detection circuit 15 Facial expression detection circuit 16 System control part

Claims (8)

Imaging means for imaging a subject and generating a preview image;
Face detection means for detecting the face of the subject from the preview image generated by the imaging means;
Facial expression detection means for detecting whether the face detected by the face detection means is a specific facial expression;
While the specific facial expression is detected by the facial expression detection means, the imaging means is controlled to continuously shoot still images according to a shooting instruction from a user, and the plurality of continuously shot images Control means for selecting an image with the highest evaluation value among the still images as a recording image,
When the specific facial expression is detected by the facial expression detection unit, the control unit detects the specific facial expression in a state where the specific facial expression is not detected. An imaging apparatus characterized in that the timing of switching to control for capturing a single still image by the imaging means varies depending on the result.   When the face is changed from a state in which the specific facial expression is detected to a state in which the specific facial expression is not detected, and the face of the subject is detected by the face detection unit, the control unit includes: The imaging apparatus according to claim 1, wherein the imaging unit is controlled to capture a single still image after a predetermined time has elapsed since a specific facial expression is no longer detected.   3. The control unit according to claim 2, wherein the control unit controls the imaging unit to reduce the number of still images to be continuously captured as time elapses until the predetermined time elapses. Imaging device.   When the state is changed from a state where the specific facial expression is detected to a state where the specific facial expression is not detected, and the face of the subject is not detected by the face detection unit, the control unit includes: The imaging apparatus according to any one of claims 1 to 3, wherein the imaging unit is controlled so as to capture a single still image.   5. The image pickup apparatus according to claim 1, wherein the evaluation value is a value based on a facial expression of the subject, a result of eye-browsing, and a blur amount of a still image.   The imaging apparatus according to claim 1, wherein the specific facial expression is a smile. An imaging method of an imaging apparatus including an imaging unit that images a subject and generates a preview image,
A face detection step of detecting the face of the subject from the preview image generated by the imaging means;
A facial expression detection step for detecting whether the face detected in the face detection step is a specific facial expression;
While the specific facial expression is detected in the facial expression detection step, the imaging unit is controlled so as to continuously capture still images in accordance with a capturing instruction from a user, and the plurality of continuously captured images A control step of selecting the image with the highest evaluation value among the still images as a recording image,
In the facial expression detection step, when the specific facial expression is detected to change to a state in which the specific facial expression is not detected, the control step includes the face detection step in the state in which the specific facial expression is not detected. According to a detection result, the timing for switching to control for capturing one still image by the imaging unit is made different. A program for controlling an image pickup apparatus including an image pickup unit that picks up a subject and generates a preview image,
A face detection step of detecting the face of the subject from the preview image generated by the imaging means;
A facial expression detection step for detecting whether the face detected in the face detection step is a specific facial expression;
While the specific facial expression is detected in the facial expression detection step, the imaging unit is controlled so as to continuously capture still images in accordance with a capturing instruction from a user, and the plurality of continuously captured images A control process for selecting an image with the highest evaluation value among the still images as a recording image is executed by a computer,
In the facial expression detection step, when the specific facial expression is detected to change to a state in which the specific facial expression is not detected, the control step includes the face detection step in the state in which the specific facial expression is not detected. According to a detection result, the timing for switching to control for capturing a single still image by the imaging unit is made different.

Images