JP5930703B2 - Image processing apparatus, imaging apparatus, and image processing method - Google Patents

Description

The present invention relates to an image processing device, an imaging device, and an image processing method , and more particularly to a technique suitable for use in recording the best image.

  Conventionally, there has been a demand for an image processing technique in which an image in which a face with a good expression that the photographer wants to leave is automatically recorded from images continuously shot. A good facial expression is, for example, a facial expression that does not meditate, or a good facial expression.

  As a technique for recording the best image, for example, in (Patent Document 1), there is an imaging unit that obtains a plurality of images in succession, and ranking of captured images is performed based on the state of a person's face in these images. This method has been proposed.

  Further, a face that is detected as a predetermined expression according to the evaluation value of the face in the evaluation means that detects each face based on the sequentially output images, obtains an evaluation value of the face, and evaluates whether the expression is a predetermined expression Change the evaluation value of. And the method of determining whether to acquire this image using the evaluation value is proposed (for example, patent document 2).

JP 2009-272740 A JP 2010-220201 A

  However, since the facial state and facial expression evaluation value are calculated with respect to the captured image in the facial region detected from the image, the facial state and facial expression evaluation value are calculated. There is a problem that the calculated facial state and facial expression evaluation value also vary.

For this reason, in the prior art disclosed in the above-mentioned patent document, it is difficult to appropriately rank and determine whether to acquire the main image if there are variations in the state of the human face and the expression evaluation value. It was.
SUMMARY OF THE INVENTION In view of the above-described problems, an object of the present invention is to suppress variations in facial expression evaluation values so that a photographer can select the best image that the photographer wants to leave.

An image processing apparatus according to the present invention sets a face detection unit for detecting a face from image data, and a plurality of regions whose sizes are changed based on the size of the face at the position of the face detected by the face detection unit. setting means for, from the image data of each region of the plurality of areas set by the setting means, calculates an evaluation value based on the facial expressions, the evaluation value calculated for the image data of each area Evaluation value calculating means for obtaining the facial expression evaluation value based on the image data, and selecting one of the image data based on the expression evaluation value obtained for each of the plurality of image data from a plurality of image data And selecting means to be stored in the storage means .

  According to the present invention, it is possible to suppress variation in facial expression evaluation values in each image, and it is possible to select the best image in which a face with a good facial expression that the photographer wants to leave is captured.

It is a block diagram showing an example of composition of an imaging device of an embodiment. It is a flowchart explaining the whole image selection process of the imaging device of an embodiment. It is a flowchart explaining the procedure of the image expression evaluation value setting process of embodiment. It is a flowchart explaining the procedure of the image selection process of the imaging device of embodiment. It is a figure which shows embodiment and demonstrates a mode that facial expression evaluation value calculation from a picked-up image.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus 100 to which the present invention is applied.
In FIG. 1, reference numeral 101 denotes a photographing lens including a zoom mechanism, a focus lens mechanism, and an aperture shutter mechanism. Reference numeral 102 denotes an image sensor that performs photoelectric conversion for converting reflected light from a subject into an electrical signal and outputs the electrical signal. Reference numeral 103 denotes an A / D conversion unit including a CDS circuit for removing output noise of the image sensor 102 and a non-linear amplification circuit performed before A / D conversion.

  Reference numeral 104 denotes an image processing unit. The image processing unit 104 can extract a signal component in a specific frequency band related to luminance and detect the focus state of the subject. Reference numeral 105 denotes a format conversion unit. Reference numeral 106 denotes a high-speed built-in memory (for example, a random access memory or the like, hereinafter referred to as DRAM). An image recording unit 107 includes a recording medium such as a memory card and its interface.

  Reference numeral 108 denotes a system control unit (CPU) that controls the system such as an imaging sequence. Reference numeral 109 denotes an operation unit for operating the camera from the outside. Reference numeral 110 denotes a photographing mode switch for performing settings such as switching the face detection mode to ON or OFF. Reference numeral 111 denotes a main switch for turning on the power to the system, and reference numeral 112 denotes a switch (hereinafter referred to as SW1) for performing a shooting standby operation such as AF or AE. Reference numeral 113 denotes a photographing switch (hereinafter referred to as SW2) for actually photographing after the operation of SW1.

  A face detection module 114 detects a face using the image signal processed by the image processing unit 104 and sends one or more pieces of detected face information (position / size / reliability) to the system control unit 108. .

  The face detection module 114 performs a known face detection process on the image data sent from the image processing unit 104 and detects a human face area included in the captured image of the image sensor 102. As a known face detection process, for example, a skin color region is extracted from the gradation color of each pixel represented by image data, and a face is detected with a matching degree with a face contour plate prepared in advance. Are known. Also disclosed is a method of performing face detection by extracting facial feature points such as eyes, nose and mouth using a well-known pattern recognition technique. The face detection module 114 can detect the face area and the face position in the image.

  The smile detection module 115 calculates a face smile level for the face area detected by the face detection module 114. Specifically, it is necessary to calculate the degree of smile, such as the contours of facial components such as eyes, nose and mouth that make up the face, and the positions of facial components such as the positions of the eyes, corners of the eyes, nose, mouth corners and lips Get feature values.

  Here, as a technique for acquiring the feature amount, a technique based on template matching using a template of each face component can be used. Alternatively, a method using a discriminator for each face component obtained by machine learning learning using a large number of sample images of the face component can be used. The smile detection module 115 calculates a smile level indicating the degree to which the face is smiling based on the feature amount acquired using the method described above.

  The eye area detection module 116 detects an eye area where an eye exists from the face area detected by the face detection module 114. The eye opening detection module 117 detects the area of the luminance region corresponding to the black eye from the histogram in the eye region detected by the eye region detection module 116. Then, according to the ratio between the area of the luminance area corresponding to the black eye and the size of the eye area, the degree of opening indicating the degree of opening is calculated.

  Note that the methods of face detection, smile level calculation, and opening level calculation are not limited to the above-described methods, and various known methods can be used. Further, the smile degree and the degree of opening calculated by the smile detection module 115 and the opening detection module 117 have different values due to variations in the face area detected by the face detection module 114. In the present embodiment, the processing described later is performed to suppress variations in the values of the smile level and the degree of openness in each image, and a face with a good expression that the photographer wants to leave is selected.

  The DRAM 106 is used as a high-speed buffer as temporary image storage means or a working memory for image compression / decompression. The operation unit 109 includes the following, for example. These include a menu switch for performing various settings such as a shooting function of the image pickup apparatus 100 and a setting at the time of image reproduction, a zoom lever for instructing a zoom operation of the photographing lens, and an operation mode switching switch between a photographing mode and a reproduction mode.

(Overall operation)
Next, an image selection operation performed by the imaging apparatus 100 according to the embodiment of the present invention will be described with reference to the flowcharts of FIGS. 2 to 4 and FIG. Each process of the flowcharts in FIGS. 2 to 4 is realized by developing a program recorded in a ROM (not shown) in a RAM (not shown) that functions as a work memory and executing the program by the system control unit 108.
First, in S201, face detection processing is executed from an image captured during shooting standby.
In S202, the degree of opening and the degree of smile are detected from the face detected in S201. That is, in S202, facial expression evaluation value detection processing is performed.
In S203, standard face parameters are calculated. In this embodiment, the standard face parameter calculation process is performed by averaging the detected face size and face position.

  In S204, standard facial expression evaluation value calculation processing is performed. The standard facial expression evaluation value is calculated by averaging the values within a predetermined range from the value obtained by averaging the facial expression evaluation values. As a result, it is possible to acquire a standard facial expression evaluation value when a normal face, that is, eyes are open and not smiling, and the acquired facial expression evaluation value is stored. In this embodiment, the standard opening degree is defined as the standard opening degree, and the standard smile degree is defined as the standard smile degree.

In S205, a process is performed to determine whether the shooting start process has been performed. If photographing start processing has been performed, the process proceeds to S206. If not, the process returns to S201 face detection.
In S206, image shooting is performed. In S207, it is determined whether a predetermined number of images have been shot. In the present embodiment, the predetermined number is three.

  The number of shots may be a predetermined value or may be changed depending on the shooting situation. For example, when the detected face is moving within the screen, the change depending on the shooting situation includes a method in which shooting is continuously performed until the face stops. When the predetermined number of shots are completed, face detection is performed from the shot image in S208.

Next, in S209, the face area detected from the photographed image is compared with the face area stored during photographing standby. In the present embodiment, the areas are compared, but the positions may be compared in addition to the comparison of the areas.
If the difference between the face areas is within a predetermined range, in S210, the face standard parameters stored during shooting standby are set to be used. Next, in S211, facial expression evaluation is performed in the set face area. Detect value. Next, in S216, it is determined whether or not a predetermined number of processes have been completed.

  As a result of the comparison in S209, if the difference between the face areas is greater than or equal to a predetermined value, the face area detected from the photographed image is set to be used in S212. In the present embodiment, the predetermined amount when compared with the standard face is set as the variation amount of the face detection size. The amount of variation in the detected size of the face may be determined in advance, or may be obtained from the variation in the face area detected during shooting standby.

  In S213, a facial expression evaluation value is detected in the set face area. Next, in step S214, the set face area is multiplied by a predetermined number, and in step S215, it is confirmed whether or not a predetermined number of detections to be processed have been completed. If the predetermined number of times has not been completed, the process returns to S213, and the facial expression evaluation value in the face region multiplied by the predetermined number is detected. In the present embodiment, the predetermined number of times is three.

  Here, the manner in which the facial expression evaluation value calculation process is performed a plurality of times on an image having the same facial expression evaluation value will be described with reference to FIG. Reference numeral 500 denotes a photographed image. Reference numeral 501 denotes a detected face area. First, the detected face area 501 is multiplied by a predetermined value. In this embodiment, 0.9 times is applied. This is the area indicated by 502. In this area 502, a facial expression evaluation value is calculated from the captured image 500. Next, a facial expression evaluation value is calculated in a region 503 that is the same size as the detected face region 501. Finally, a facial expression evaluation value is calculated from the photographed image 500 in an area 504 obtained by multiplying the face area by 1.1. Each of these facial expression evaluation values is stored. In addition, when the area | region used for facial expression evaluation value calculation exceeds a detectable area | region, it will be set to a detectable area | region.

  As a result of the confirmation in S215, if the predetermined number of floors has been completed, it is determined in S216 whether or not the processing of the predetermined number of shots has been completed. If the predetermined number of processes has not been completed, the process returns to S208, and face detection is performed on the next photographed image.

  If the predetermined number of processes have been completed, in step S217, the stored facial expression evaluation value is used to set an image facial expression evaluation value to be assigned to each image. The expression evaluation value assigned to the image is set as the image expression evaluation value. The image expression evaluation value setting process to be given to each image will be described later with reference to FIG.

Next, in S218, an image to be recorded is selected using the image expression evaluation value assigned to each image. A method for selecting an image to be recorded will be described later with reference to FIG.
Next, the image selected in S219 is recorded on the recording medium, and the process is terminated.

(Overview of image facial expression evaluation value setting process)
FIG. 3 is a flowchart for explaining the procedure of the image expression evaluation value setting process performed in S217 of FIG.
First, in S301, the standard opening degree is compared with the opening degree detected from the image. As for the degree of opening, there are two values for the left and right eyes.

  Next, in S302, it is determined whether or not the calculation of all the opening degrees has been completed. If the calculation has been completed, the process proceeds to S303. If the calculation has not been completed, the process returns to S301, and the difference is calculated. Continue the calculation. The degree of opening of all eyes is the degree of opening detected while changing the face area for each image.

  When the calculation of the difference is completed, whether or not the degree of opening of the image is distributed only to the higher or lower value of the standard opening degree using the difference in S303. Make a decision. Only when the numerical value is high or low is distributed only when the expression is considered to have changed.

  As a result of the determination in S303, if it is determined that the numerical value is distributed only to the higher one or the lower one, the median of the degree of opening detected from the image is selected in S305. The median is a numerical value located in the middle of a plurality of numerical values. If it is determined that the numerical value is not distributed only to the higher one or the lower one, a standard value is selected in S304. When the processing of S305 or S304 is completed, the standard smile level is compared with the smile level detected from the image in S306.

  Next, in S307, it is determined whether or not all smile levels have been calculated. If calculation has been completed, the process proceeds to S308, and if calculation has not been completed, the process returns to S306 to calculate the difference. Continue. All smile levels are the smile levels detected while changing the face area for each image.

  When the calculation of the difference is completed, it is determined in step S308 whether or not the smile level of the image is distributed to one side with respect to the standard smile level. “Distributed to one side” means that the numerical values of the differences are all positive values or negative values, and the expression is considered to have changed.

If it is determined that the image is distributed on one side, in S310, the median smile level detected from the image is selected. The median is a numerical value located in the middle of a plurality of numerical values.
If it is determined that the numerical value is not distributed only to the higher one or the lower one, a standard value is selected in S309.

In this embodiment, when the standard value is not used, the median value of the evaluation values detected from the image is taken. However, the standard value when the facial expression changes in advance during shooting standby is also stored, and those values are stored. A method of selecting by comparing with a numerical value is also conceivable.
The standard value when the facial expression changes is the average value of the eye meditation change that was memorized by averaging the degree of eye opening when the eyes were meditated while blinking during shooting standby, and the degree of smile when smiling It is the average value of smile change averaged and memorized.

  The degree of opening detected from the image is compared with the standard value of the opening of the eyes. If there is around, use the standard value of eye meditation change. In addition, the smile level detected from the image is compared with the standard smile level, and the smile level, that is, the higher value is distributed, and the smile level detected from the image is compared with the smile change standard value. If it exists, the smile change standard value is used.

(Overview of image selection process)
Next, the procedure of the image selection process performed in S218 of FIG. 2 will be described with reference to the flowchart of FIG.
First, in S401, it is determined whether the evaluation target image is an eye-meditation image. To determine whether or not the image is an eye-meditation image, the eye-meditation determination reference value is created by referring to the degree of opening in all the captured images. If the degree of opening of the eyes is lower than a predetermined value by the eye meditation determination reference value, it is determined that the image is an eye meditation image. However, if it is determined that the image determined to be an eye-meditation image is a smile image, it is not determined to be an eye-meditation image.

  Whether or not the image is a smile image is determined by referring to the smile level in all the captured images and creating a smile standard value. If the smile level is higher than the smile standard value, it is determined that the image is a smile image. If it is determined that the image is an eye-meditation image, the image to be evaluated is removed from the storage candidates in S402. If it is determined that the image is not an eye-meditation image, the process proceeds to S403.

  In S403, it is determined whether or not all of the captured images have been determined. If the confirmation of all images has not been completed, the process returns to S401. If all the images have been confirmed, it is next determined whether or not the image is a candidate for storage in S404. Specifically, in S404, it is determined whether or not the smile level of the evaluation target image is the maximum value. If the smile level of the evaluation target image is maximum, the image is set as a final storage candidate in S405. On the other hand, if it is determined in S404 that the smile level is not the maximum, the process proceeds to S406.

  Next, in S406, it is determined whether or not all the storage candidate images have been determined. If the determination of all the storage candidate images has not been completed, the process returns to S404 and smile determination is performed. If all the storage candidate images have been determined, the process is completed.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (computer program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various computer-readable storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program.

DESCRIPTION OF SYMBOLS 100 Image pick-up device, 101 Shooting lens, 102 Image pick-up element, 103 A / D conversion part, 104 Image processing part, 105 Format conversion part, 106 DRAM, 107 Image recording part, 108 System control part, 109 Operation part, 110 Shooting mode switch , 111 main switch, 112 shooting standby switch (SW1), 113 shooting switch (SW2), 114 face detection module, 115 smile detection module, 116 eye area detection module, 117 eye opening detection module

Claims (9)

Face detection means for detecting a face from image data;
Setting means for setting a plurality of regions whose sizes have been changed based on the size of the face at the position of the face detected by the face detection means;
From each area image data of a plurality of areas set by the setting means, calculates an evaluation value based on the facial expression, the face based on the evaluation value calculated for the image data of each area An evaluation value calculating means for obtaining a facial expression evaluation value ;
Selecting means for selecting any image data from a plurality of image data based on the facial expression evaluation value obtained for each of the plurality of image data, and storing the selected image data in a storage means ; Image processing device.   In the case where the evaluation value calculated with respect to the image data of each of the plurality of regions is distributed only to a higher one or a lower one than a predetermined standard value. The image processing apparatus according to claim 1, wherein the facial expression evaluation value is obtained based on an evaluation value calculated for the image data of each region.   3. The evaluation value calculation unit according to claim 1, wherein the evaluation value calculation unit sets a median value of the evaluation values calculated for the image data of each of the plurality of regions as the facial expression evaluation value. Image processing apparatus.   The image processing apparatus according to claim 1, wherein the facial expression evaluation value is an eye opening degree indicating a degree of eye opening.   The image processing apparatus according to claim 1, wherein the facial expression evaluation value is a smile degree indicating a frequency of smile. The image processing apparatus according to any one of claims 1 to 5,
Imaging apparatus characterized by having an imaging means for outputting the image data. A face detection step of detecting a face from image data;
A setting step of setting a plurality of regions whose sizes have been changed based on the size of the face at the position of the face detected in the face detection step;
From each area image data of a plurality of areas set in the setting step, calculating an evaluation value based on the facial expression, the face based on the evaluation value calculated for the image data of each area An evaluation value calculating step for obtaining a facial expression evaluation value ;
A selection step of selecting any image data from a plurality of image data based on the expression evaluation value obtained for each of the plurality of image data, and storing the selected image data in a storage step. Image processing method. A program for causing a computer to execute each step of the control method of the image processing apparatus,
A face detection step of detecting a face from image data;
A setting step of setting a plurality of regions whose sizes have been changed based on the size of the face at the position of the face detected in the face detection step;
From each area image data of a plurality of areas set in the setting step, calculating an evaluation value based on the facial expression, the face based on the evaluation value calculated for the image data of each area An evaluation value calculating step for obtaining a facial expression evaluation value ;
Causing a computer to execute a selection step of selecting any image data from a plurality of image data based on the expression evaluation value obtained for each of the plurality of image data and storing the selected image data in a storage step. A featured program.   A computer-readable storage medium storing the program according to claim 8.

Images