JP4149301B2 - Method for extracting feature parts of continuous image, program thereof, and digital camera - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a method for extracting a subject feature portion such as a face from a continuous image such as a continuously shot image or a bracketed image, a program thereof, and a digital camera equipped with the program.
[0002]
[Prior art]
For example, as described in Patent Document 1 below, the digital camera is equipped with an autofocus device that extracts the face part of the subject from the screen and automatically focuses the digital camera on the extracted face. There is something that has been. However, this Patent Document 1 only discloses a focusing technique, and does not describe how to extract a face image of a subject so that a face image can be extracted at high speed.
[0003]
When extracting a face part from the screen, template matching is used. This is because the similarity between each partial image sequentially extracted from the subject image in the search window and the face template is sequentially determined, and the face of the subject is located at the search window position that matches the face template with a similarity equal to or greater than a threshold. It is determined that it exists.
[0004]
Conventionally, when performing this template matching process, it is not known how big the subject's face is in the screen, so the size of the face template varies from a small template to a full-screen template. A large number of templates are prepared, and template matching processing is performed using all templates to extract face images.
[0005]
[Patent Document 1]
Japanese Patent Laid-Open No. 2001-215403
[Problems to be solved by the invention]
If features such as the subject's face can be extracted before shooting, etc., the subject's face can always be automatically focused and shot, and white balance can be taken to match the skin color of the face. There are many advantages. In addition, when the captured image data is taken into a personal computer or the like and manually processed by the user, if the control device extracts the position of the subject's face in the image, it can be controlled by adjusting the skin color, for example. The device can provide an appropriate guide to the user.
[0007]
However, the conventional technique has a problem that it takes time to extract a face because a large number of face templates are prepared and matching processing using each template is required.
[0008]
In particular, when multiple images are shot continuously or taken by auto bracket shooting, continuous shooting, etc., face extraction processing using a large number of templates for each shot image takes a lot of time and is highly functional. There is a problem that a control device equipped with a special arithmetic processing function is required.
[0009]
An object of the present invention is to provide a subject feature portion extraction method of a continuous image, a program thereof, and a digital camera that can extract subject feature portions from a plurality of consecutive captured images at high speed and with high accuracy.
[0010]
[Means for Solving the Problems]
The subject feature portion extraction method for continuous images of the present invention is the subject feature portion extraction method for continuous images in which a feature portion image of a subject is extracted from each of a plurality of consecutive photographed images by matching processing with a template. The upper and lower limits of the size of the feature portion image with respect to the size of the image are limited based on distance information between the camera that captured the captured image and the subject having the feature portion, and the limited upper and lower limits The feature portion image is searched by matching processing while changing the size of the template within the range, and the feature portion image is not included in the captured image when the feature portion image cannot be extracted within the range. It is characterized by determining .
[0011]
With this configuration, since the subject feature portion image is searched in a highly probable limited range where the subject feature portion image exists, the feature portion can be extracted at high speed. Further, by limiting the search range, erroneous detection can be prevented. That is, it is possible to prevent erroneous detection of features that appear to be characteristic portions (for example, faces) as characteristic portions even if they are of different sizes.
[0012]
The subject feature portion extraction method for continuous images of the present invention is the subject feature portion extraction method for continuous images in which a feature portion image of a subject is extracted from each of a plurality of consecutive photographed images by matching processing with a template. The upper and lower limits of the size of the feature portion image with respect to the size of the image are limited based on distance information between the camera that has captured the captured image and the subject having the feature portion, and the size of the feature portion image In the matching process between the image cut out from the resized photographed image and the fixed size template while changing the size of the resized photographed image so that the size is within the limited upper limit and lower limit. The feature portion image is searched, and when the feature portion image cannot be extracted within the range, the feature portion image is not included in the captured image. And judging a.
[0013]
With this configuration, in order to perform matching processing with a template of a fixed size while resizing the matching processing target image within the limited range, or with a matching processing target image of a fixed size while resizing the template size Therefore, the number of templates used in the matching process or the number of matching target images to be cut out from the captured image can be limited, and the processing time of the matching process can be shortened. In addition, since the search range is limited, erroneous detection can be prevented. That is, it is possible to prevent erroneous detection of features that appear to be characteristic portions (for example, faces) as characteristic portions even if they are of different sizes.
[0014]
The subject feature portion extraction program for continuous images of the present invention is characterized by executing the subject feature portion extraction method for continuous images.
[0015]
With this configuration, since the subject feature portion image is searched in a highly probable limited range where the subject feature portion image exists, the feature portion can be extracted at high speed.
[0016]
The digital camera of the present invention is characterized in that the above-described subject feature portion extraction program for continuous images is installed.
[0017]
With this configuration, since the subject feature portion image is searched in a highly probable limited range where the subject feature portion image exists, the feature portion can be extracted at high speed. Further, by limiting the search range, erroneous detection can be prevented. That is, it is possible to prevent erroneous detection of features that appear to be characteristic portions (for example, faces) as characteristic portions even if they are of different sizes.
[0018]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0019]
(First embodiment)
FIG. 1 is a configuration diagram of a digital still camera according to the first embodiment of the present invention. This digital still camera includes a solid-state imaging device 1 such as a CCD or a CMOS, a lens 2 and a diaphragm 3 placed in front of the solid-state imaging device 1, and correlated double sampling with respect to an image signal output from the solid-state imaging device 1. An analog signal processing unit 4 that performs processing, an A / D conversion unit 5 that converts an analog signal-processed image signal into a digital signal, gamma correction, synchronization processing, and the like for the image signal converted into a digital signal A digital signal processing unit 6 to be applied, an image memory 7 for storing an image signal processed by the digital signal processing unit 6, and an image signal (shooting data) stored in the image memory 7 when the user presses the shutter button. Is recorded on an external memory or the like, and a display unit 9 is provided on the back of the camera or the like and displays the stored contents of the image memory 7 through.
[0020]
The digital still camera further includes a control circuit 10 including a CPU, a ROM, and a RAM, an operation unit 11 that receives an instruction input from a user and performs an on-demand display process on the display unit 9, and an output from the image sensor 1. Then, the image signal processed by the digital signal processing unit 6 is taken in. Based on an instruction from the control circuit 10, as will be described in detail later, a feature extraction portion of the subject, in this example, a face extraction processing portion 12 that extracts a face portion, and the lens 2 A lens driving unit 13 that performs focusing and magnification control based on an instruction signal from the control circuit 10, a diaphragm driving unit 14 that controls an aperture amount of the diaphragm 3 based on an instruction signal from the control circuit 10, and solid-state imaging An image sensor control unit 15 that controls the driving of the element 1 based on an instruction signal from the control circuit 10 and a subject based on the instruction signal from the control circuit 10. And a distance measuring sensor 16 to measure the distance.
[0021]
FIG. 2 is a flowchart showing a processing procedure of face extraction processing performed by the face extraction processing unit 12 in accordance with the face extraction program. The face extraction program is stored in the ROM of the control circuit 10 shown in FIG. 1, and the face extraction processing unit 12 functions when the CPU reads the face extraction program into the RAM and executes it.
[0022]
The imaging device 1 of the digital still camera always outputs an image signal at a predetermined cycle even before the user presses the shutter button, and the digital signal processing unit 6 performs digital signal processing on each image signal. The face extraction processing unit 12 sequentially captures this image signal and performs the following processing on each input image (captured image).
[0023]
In step S1, face search processing is performed. FIG. 3 is a flowchart showing a detailed processing procedure of the face search processing. In FIG. 3, the face extraction processing unit 12 first acquires the size of the input image (step S11). For example, in the case of a camera in which the size of the input image used for the face extraction process is different depending on whether the user is shooting with 640 × 480 pixels or 1280 × 960 pixels, this size information is acquired. . When the size of the input image is fixed, this step S11 is not necessary. Next, distance information to the subject measured by the distance measuring sensor 16 is acquired from the control circuit 10 (step S12).
[0024]
In the next step S13, it is determined whether or not the zoom lens is used. If the zoom lens is used, the zoom position information is acquired from the control circuit 10 (step S14), and then the focus of the lens is obtained. Distance information is acquired from the control circuit 10 (step S15). If it is determined in step S13 that the zoom lens is not used, the process skips step S14 and proceeds to step S15.
[0025]
Based on the input image size information and the lens focal length information acquired by the above processing steps, it is possible to determine the size of the face of the person who is the subject in the input image. Therefore, in the next step S16, an upper limit and a lower limit range of the search window size according to the face size are determined.
[0026]
As shown in FIG. 4, the search window is a window 23 having the same size as the face image for the processed image 21 to be subjected to the template matching process, that is, the size of the template 22 shown in FIG. The normalized cross-correlation coefficient between the image cut out by the search window 23 and the template 22 is obtained by the following processing steps, the matching degree is calculated, and when the matching degree, that is, the similarity does not reach the threshold value, the search window 23 is shifted in the scanning direction 24 by a certain amount of pixels on the processed image 21, for example, one pixel, and an image for the next matching processing is cut out.
[0027]
Here, the processed image 21 is an image obtained by resizing the input image. For example, rather than performing a matching process using a high-definition input image of 1280 × 960 pixels as a processed image, an image obtained by resizing this input image to, for example, 200 × 150 pixels is used as a processed image, and a template (of course, a face on the template side) The image is not a high-definition face image, but a face image having a small number of pixels, for example, 20 × 20 pixels is used.) It is easier to detect a general “face” by ignoring individual differences. Become.
[0028]
In the next step S17, it is determined whether or not the size of the search window is within the range, that is, whether or not it is within the upper and lower limits of the face size in the processed image 21. Next, it is determined whether or not there is a template 22 having a size that matches the size of the search window 23 (step S18). If it exists, the corresponding template is selected (step S19). If it does not exist, the template is resized to generate a template that matches the size of the search window 23 (step S20), and the process proceeds to the next step 21. .
[0029]
In step S21, template matching processing is performed while scanning the search window 23 along the scanning direction 24 (FIG. 4), and it is determined whether or not there is an image part having a similarity equal to or greater than the threshold value α.
[0030]
If there is no image portion whose similarity is greater than or equal to the threshold value α, the process proceeds to step S22, the size of the search window 23 is changed as shown in FIG. 6, and then the size of the search window 23 to be used is changed. After the determination, the process proceeds to step S17. Thereafter, step S17 →... → step S21 → step S22 → step S17 is repeated.
[0031]
As described above, in this embodiment, the size of the template is changed as shown in FIG. 7 while changing the size of the search window 23 from the upper limit value to the lower limit value (or from the lower limit value to the upper limit value) as shown in FIG. Also change the template matching process.
[0032]
If an image portion showing a degree of similarity equal to or greater than the threshold value α is detected in step S21, the process proceeds to the face detection determination process in step S2 in FIG. 2, the face position is specified, and the position information is output to the control circuit 10. The leverage detection process ends.
[0033]
If step S17 →... → step S21 → step S22 → step S17 is repeated and the size of the search window 23 falls outside the upper and lower limits, the determination result in step S17 is negative (N). . In this case, the process proceeds to the face detection determination process in step S2 of FIG.
[0034]
As described above, according to the present embodiment, a plurality of types of templates used in the template matching process are prepared, and the matching process using each template is performed. The upper and lower limits of the template to be used based on the distance information to the subject. Therefore, the number of template matching processes can be reduced, and the face extraction process can be performed with high accuracy and high speed.
[0035]
In FIG. 2, when the position of the “face” is extracted in step S2 or it is determined that there is no face, the process proceeds to step S3, where it is determined whether there is a continuous input image. If there is no continuous image, the process returns to the face extraction process (step S1) in FIG. That is, if the newly input image is an image of a scene different from the previous frame (previous input image), face search processing is performed in step S1.
[0036]
When successive images are captured one after another, the determination result in step S3 is affirmative (Y). In this case, in step S4, it is determined whether or not the face of the subject has been extracted in the previous frame. If the determination result is negative (N), the process returns to step S1, and the face extraction process of FIG. 3 is performed.
[0037]
If consecutive images are captured one after another and the face of the subject is extracted in the previous frame, the determination result in step S4 is affirmative (Y), and the process proceeds to step S5. In step S5, the search range of the search window 23 is limited. In the face search process of FIG. 3, the search range of the search window 23 is the entire range of the processed image 21, but when the face position is detected in the previous frame, the input image (2) in FIG. As shown, the search range is limited to a highly probable range 21a where a face exists.
[0038]
Then, in the next step S6, a face image is searched in the limited search range 21a. Since the search range is limited, a face image can be extracted at high speed.
[0039]
After step S6, the process returns to step S3 and proceeds to the face search process for the next input image. In the case of auto bracket shooting, the subject is often stationary and does not move. Therefore, when an instruction for auto bracket shooting is input from the operation unit 11, the search range of the face in the input image (2) in FIG. Can be further limited.
[0040]
In addition, when shooting a moving subject continuously, the speed and direction of the subject can be known from the position of the face image extracted from the input image (1) and the input image (2) in FIG. In the next frame input image (3), the face search range can be further limited.
[0041]
As described above, in this embodiment, when a face image is extracted from each of a plurality of continuously input images, the search range in the next frame is limited by the position of the face extracted in the previous frame. Extraction can be performed at high speed. The face extraction process in step S6 is not limited to the template matching process, but may be another method.
[0042]
(Second Embodiment)
FIG. 9 is a flowchart showing the processing procedure of the face extraction program according to the second embodiment of the present invention. The configuration of a digital still camera equipped with this face extraction processing program is the same as that in FIG. 1, the processing procedure in FIG. 2 is also the same, and the processing procedure in FIG. 9 is used instead of the processing procedure in FIG.
[0043]
In the process of FIG. 3 in the first embodiment described above, the template matching process is performed while changing the size of the search window and the template. However, in this embodiment, the size of the search window and the template is fixed and the process is performed. Template matching processing is performed while resizing the size of the image 21.
[0044]
Steps S11 to S15 are the same as those in the first embodiment. After step S15, in the present embodiment, the upper limit and lower limit ranges of the size of the processed image 21 are determined (step S26). Then, in the next step S27, it is determined whether or not the size of the processed image 21 is within a range commensurate with the upper and lower limits of the face size.
[0045]
If it is determined in step S27 that the size of the processed image 21 is within the upper and lower limits, the process proceeds to step S21, where template matching processing is performed, and an image whose similarity is greater than or equal to the threshold α. Determine if the part exists. If an image part having a similarity equal to or greater than the threshold value α cannot be detected, the process returns from step S21 to step S28 to resize the processed image 21, and the template matching process is repeated. If an image portion equal to or greater than the threshold value α is detected, the process proceeds from step S21 to the face detection determination process of step S2 in FIG. 2, the face position is specified, and the position information is output to the control circuit 10 to output the face detection process. Exit.
[0046]
After the size of the processed image changes from the upper limit value to the lower limit value (or after the change from the lower limit value to the upper limit value) due to resizing of the processed image 21, the determination result of step S27 becomes negative (N). In this case, the process proceeds to step S2 in FIG.
[0047]
In this way, in this embodiment, the size of the subject's face relative to the input image is limited based on the distance information to the subject, so the number of template matching processes can be reduced, and face extraction can be performed with high accuracy and high speed. Can be done. Moreover, since only one template is prepared in advance, the storage capacity of the template can be reduced.
[0048]
In each of the above-described embodiments, the digital still camera has been described as an example. However, the present invention can also be applied to other digital cameras such as a digital camera mounted on a mobile phone or a digital video camera that performs moving image shooting. . In addition, the distance information to the subject is not limited to the case of using a measurement value or a known value of a distance measuring sensor, the distance information acquisition method may be any method, and the extraction target is not limited to the face, but other features. The present invention can also be applied to portions.
[0049]
The feature extraction program of each embodiment described above is not limited to being installed in a digital camera. For example, the feature extraction program can be installed in a photographic printer or image processing apparatus to extract a feature portion of a subject with high accuracy and high speed. Is possible. In this case, distance information and zoom information are required to limit the size of the template or the size of the processed image to the upper and lower limits of the feature partial image. It is good to use what was added as tag information to imaging | photography data. Further, it is preferable to use tag information added to the shooting data to determine whether the image is an auto bracket shooting or continuous shooting.
[0050]
【The invention's effect】
According to the present invention, when a feature portion of a subject is extracted from a continuous input image, the search range is limited using information on the feature portion extracted in the previous frame, so that high speed and high accuracy are possible. Become.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of a digital still camera according to a first embodiment of the present invention.
FIG. 2 is a flowchart showing a processing procedure of a face extraction program installed in the digital still camera shown in FIG. 1;
FIG. 3 is a flowchart showing a detailed processing procedure of step S1 shown in FIG. 1;
FIG. 4 is an explanatory diagram of scanning by a search window.
FIG. 5 is a diagram illustrating an example of a face template.
FIG. 6 is an explanatory diagram of an example of changing the size of a search window.
FIG. 7 is an explanatory diagram of an example of changing the size of a template.
FIG. 8 is an explanatory diagram of continuous input images and a search range.
FIG. 9 is a flowchart showing a processing procedure of a face extraction program according to the second embodiment of the present invention.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Image sensor 2 Lens 3 Aperture 6 Digital signal processing part 10 Control circuit 12 Face extraction processing part 16 Distance sensor 21 Processed image 22 Face template 23 Search window 24 Scanning direction

Claims (4)

In a method for extracting feature portions of a continuous image by extracting a feature portion image of a subject from each of a plurality of continuous captured images by matching processing with a template, the size of the feature portion image with respect to the size of the captured image is increased. The upper and lower limit ranges are limited based on distance information between the camera that captured the captured image and the subject having the characteristic part, and the template size is changed within the limited upper and lower limit ranges. A feature image of a continuous image characterized by searching for a feature partial image by matching processing and determining that the feature partial image is not present in the captured image when the feature partial image cannot be extracted within the range. Partial extraction method. In a method for extracting feature portions of a continuous image by extracting a feature portion image of a subject from each of a plurality of continuous captured images by matching processing with a template, the size of the feature portion image with respect to the size of the captured image is increased. The upper and lower limit ranges are limited based on distance information between the camera that captured the captured image and the subject having the characteristic portion, and the size of the characteristic portion image is within the limited upper and lower limit ranges. As described above, the feature image is searched by matching processing between the image cut out from the resized photographed image and the template having a fixed size while changing the size to resize the photographed image. subject of successive images and determining that the characteristic part image is not within the captured image when the feature partial image can not be extracted FEATURES part extraction method.   3. A continuous image subject feature portion extraction program that executes the continuous image subject feature portion extraction method according to claim 1 or 2.   A digital camera, comprising the subject image feature extraction program for continuous images according to claim 3.

Images