JP2010141847A - Image processor and method of processing image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve precision in subject detection by a simple method, while suppressing computational complexity in a device and a method for processing images for detecting subjects from the images. <P>SOLUTION: The image region of the same subject is identified in image regions detected as predetermined subject image regions. A history of detected reliability is recorded for the image region identified as the image region of the same subject. An image region, which meets a criterion decided according to a reliability level is determined to be the predetermined subject image region in the image regions identified as an image region of the same subject. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and an image processing method for detecting a subject included in a moving image.

  An image processing technique for detecting a specific subject (a person, an animal, a specific object, etc.) from an image is very useful. For example, an image processing technique for detecting a human face as a subject can be used in many fields such as video conferencing, a man-machine interface, security, a monitor system for tracking a human face, and image compression.

  Digital cameras and digital video cameras already detect human faces from captured images and realize exposure control and focus detection control based on face detection results.

  As such an image processing technique for detecting a specific subject from an image, various techniques have been proposed, but most of them are based on pattern matching. For example, there is a method of extracting a partial image at a plurality of different positions on the image, determining whether the partial image is a face area image, and detecting the face area on the image. Whether or not the partial image is a face region can be determined by a template matching method or a method using a classifier that learns facial features by a learning method such as a neural network.

  In either method, the reliability indicating the certainty that the partial image is the image of the subject area is calculated based on the image pattern of the partial image, and the partial image whose reliability exceeds a predetermined threshold is calculated as the image of the subject area. It is common to detect as

  For example, in Patent Document 1, the reliability is calculated from partial images having a plurality of different resolution patterns, and the image of the subject region is detected (subject detection) based on the sum of the reliability of each partial image.

  In Patent Document 2, a person's face is detected as a subject, the position of the face in the image is detected based on the detection result, the face is focused, and automatic focusing is performed so that the face is photographed with an optimal exposure. An imaging device that detects and automatically exposes is disclosed.

JP 2008-033424 A JP 2005-318554 A

  Subject detection by pattern matching is based on the degree of possibility that the subject is included in the image, and not only the subject to be detected is detected. For example, when a human face is detected as a subject, even if it is an area that does not meet the face, the face-like reliability based on pattern matching satisfies a predetermined threshold, the face is detected.

  In order to reduce such false detection (excessive detection), if the reliability threshold value for considering the subject area is set strictly, an area that does not satisfy the threshold value occurs despite the subject area, and the detection rate of the subject decreases. (Detection leak). Moreover, if the threshold value is set to be loose, erroneous detection in which a region different from the subject to be detected is detected as the subject frequently occurs. In other words, there is a trade-off relationship between detection failure and erroneous detection, and it is not easy to set optimal detection conditions.

  Although it is conceivable to increase the calculation amount of pattern matching and increase the reliability of detection, it is not practical in an apparatus such as a digital video camera or a digital camera that has limited calculation resources and requires real-time detection.

  The present invention has been made in view of the above-described problems of the prior art. In an image processing apparatus and an image processing method for performing subject detection from an image, subject detection accuracy can be improved by a simple method while suppressing the amount of computation. The purpose is to improve.

  The above-described object is to detect a predetermined subject area from images supplied in time series and subject detection means for detecting the reliability of the detected area, and a region detected by the subject detection means from a different image. A specifying unit for specifying the same subject region, a recording unit for recording subject data including a history of reliability detected by the subject detecting unit for the region specified by the specifying unit as the same subject region, and a specifying unit Judgment to determine, as a predetermined subject region, a region in which the reliability history recorded by the recording unit among the regions identified as the same subject region satisfies the determination criterion determined according to the reliability level And an image processing apparatus.

  Further, the above object is to provide a lens for forming a subject optical image, an imaging means for sequentially capturing the subject optical image formed by the lens, and outputting an image supplied in time series, and the present invention. It is also achieved by an imaging apparatus comprising such an image processing apparatus and a control unit that performs control of imaging conditions using information on a region determined by the determination unit as a predetermined subject area.

  In addition, the above-described object is an imaging apparatus that sequentially captures a lens for forming a subject optical image and a subject optical image formed by the lens, and outputs an image supplied in time series. The present invention has a change detection means for calculating a change amount of an image adjacent to the means in time series and deleting all subject data recorded by the recording means when the change amount exceeds a predetermined change amount. The image processing apparatus according to the present invention, and a control unit that controls the imaging conditions using information on the area determined by the determination unit as a predetermined subject area. When the information regarding the change in the angle of view is acquired and at least one of the information exceeds a predetermined threshold, the subject data recorded by the recording unit is assumed to be a large change in the image output by the imaging unit. Also achieved by an imaging apparatus characterized by having a deviation detection means for deleting all.

  In addition, the above-described object is to detect a subject area from a time-sequentially supplied image while detecting a predetermined subject area and detecting the reliability of the detected area in a subject detection step from a different image. In the recording unit, the identification step including the reliability step detected in the subject detection step for the identification step of identifying the region of the same subject in the identified region and the region identified as the same subject region in the identification step is recorded in the recording unit. Of the recording step to record and the region identified as the same subject region in the specific step, the reliability history recorded in the recording means satisfies the determination criteria determined according to the level of reliability, It is also achieved by an image processing method including a determination step of determining a predetermined subject area.

  With such a configuration, according to the present invention, in the image processing apparatus and the image processing method for detecting a subject from an image, the subject detection accuracy can be improved by a simple method while suppressing the amount of calculation.

Hereinafter, preferred and exemplary embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus as an example of an image processing apparatus according to an embodiment of the present invention.

  An optical object image is formed on the imaging surface of the imaging element 102 such as a CCD image sensor or a CMOS image sensor by the lens 101. The image sensor 102 outputs an electrical signal corresponding to the intensity of the incident light beam in units of pixels. This electrical signal is a video signal. The video signal output from the image sensor 102 is subjected to analog signal processing such as correlated double sampling (CDS) in the analog signal processing unit 103.

  The video signal output from the analog signal processing unit 103 is converted into a digital data format by the A / D conversion unit 104 and input to the imaging control unit 105 and the image processing unit 106. The image processing unit 106 performs image processing such as gamma correction and white balance processing. In addition to these normal image processing, the image processing unit 106 also performs image processing using information relating to the face area detected in the image supplied from the face detection unit 109, as will be described later.

  The video signal output from the image processing unit 106 is sent to the display unit 107. The display unit 107 is, for example, an LCD or an organic EL display, and displays a video signal. By sequentially displaying images taken continuously in time series on the display unit 107, the display unit 107 can function as an electronic viewfinder (EVF). The video signal is recorded on a recording medium 108, for example, a removable memory card. The recording destination may be a built-in memory of the camera or a connected external device capable of communication.

  The video signal output from the image processing unit 106 is also supplied to the face detection unit 109. The face detection unit 109 detects the face of a person in the image, and specifies the number of subjects and the face area. A known face detection method is used as the detection method. For example, there are a method using face knowledge (skin color information, parts such as eyes, nose, mouth) and a method of configuring a classifier for face detection by a learning algorithm represented by a neural network. In order to improve the recognition rate, face recognition is generally performed by combining a plurality of methods. Specifically, a face detection method using wavelet transform and image feature amount described in JP-A-2002-251380 may be used.

  Examples of the face area information output as a detection result by the face detection unit 109 include the position, size, inclination, reliability, and the like of face areas for the number of detected persons. Here, the reliability is a value representing the likelihood of the face detection result, and is determined in the face detection process.

  There are various methods for calculating the reliability. For example, the feature of the face image stored in advance and the feature of the image of the face region detected by the face detection unit 109 are compared, and the probability that the detected image of the face region is the image of the subject is obtained. There is a method for calculating the reliability. There is also a method of calculating the difference between the feature of the face image stored in advance and the feature of the image of the face area detected by the face detection unit 109 and calculating the reliability from the magnitude of the difference. Regardless of the reliability calculated by any method, if the reliability is high, the possibility of erroneous detection is low, and if the reliability is low, the possibility of erroneous detection is high.

  Information of the face area as a detection result of the face detection unit 109 is sent to the subject specifying unit 110. The subject specifying unit 110 specifies the same subject from the face detection results continuous in time series, and sends the subject information to the face determination unit 111. The face determination unit 111 determines a highly reliable face region as a face based on the time-series face detection results supplied from the face detection unit 109. Information of the detection result determined as the face (position, size, etc. of the face area in the captured image) is supplied to the image processing unit 106 and the imaging control unit 105. And it can use for control of imaging conditions, such as automatic focus detection control and automatic exposure control using information on a face area.

  The imaging control unit 105 controls a focus control mechanism and an exposure control mechanism (not shown) of the imaging lens based on the video signal output from the A / D conversion unit 104. The imaging control unit 105 can use the detection result information supplied from the face determination unit 111 to control the focus control mechanism and the exposure control mechanism. Therefore, the imaging apparatus according to the present embodiment can realize a function of performing an imaging process in consideration of information on a face area in a captured image. Specifically, exposure control, focus detection control, flash control, and the like based on the face area can be realized. The imaging control unit 105 controls the output timing and output pixels of the image sensor 102.

(Subject identification process)
Information on the face area (position, size, reliability, etc. in the captured image) is supplied from the face detection unit 109 to the subject specifying unit 110 as a face detection result. The face detection unit 109 performs face detection processing without holding or using past face detection results. Therefore, the subject specifying unit 110 holds the face detection result of the face detection unit 109 in time series, and specifies the face area of the same subject based on the time series face detection result. Thereby, subject tracking becomes possible.

Processing for specifying the same subject using the position and size of the face area will be described with reference to FIG.
FIG. 2 shows two frames that are sequentially captured in time series, where (b) shows the current captured image, and (a) shows the image one frame before (b). Here, face detection is performed for each frame, but face detection may be performed every several frames. The face areas 201, 202, and 203 detected in each frame by the face detection unit 109 are indicated by a display (face frame) indicating the face area.

For the face region detected in the previous frame shown in FIG. 2 (a), position, using the coordinates in the image _{(x t-1 (i)} , y t-1 (i)), the size s _{t -1} (i). For the face area detected in the current frame, the position is represented as (x _t (j), y _t (j)) and the size is represented as _st (j). Here, the values of i and j are integers that take values from 1 to n, and are assigned to each face area detected in the same frame. n indicates the total number of detected face regions.

  The subject specifying unit 110 determines, as the same subject, face regions whose positions and sizes are equal to or greater than 0 in the following expression 1 in face detection results that are continuous in time series.

  In other words, for a face area having the same size between consecutive frames, if the face area of the current frame is included in the area adjacent to the face area around the face area of the previous frame, The face area of the same subject is determined.

  Further, regarding a face area having a different size between successive frames, if the coordinate difference between the two face areas is included in the adjacent range set with the smaller face as a reference, it is determined as the same person. When there are a plurality of detection results of the previous frame in which the value of Expression 1 is 0 or more with reference to the result of the current frame, the smaller value of the expression is determined as the same person. On the other hand, if there is no result in which the detection result of the previous frame is 0 or more with the value of Expression 1 on the basis of the result of the current frame, the subject of the result is regarded as a new subject. That is, in FIG. 2, the face area 201 in the frame t-1 and the face area 202 in the frame t are determined as the same subject, and the face area 203 in the frame t is determined as a newly appearing subject.

  The subject specifying method described here is an example, and the same subject may be specified using other methods. For example, if information regarding the tilt and orientation of the face can be obtained from the face detection unit 109, they may be used as the subject specifying conditions in the subject specifying unit 110. It is also possible to acquire information of the imaging device such as the zoom rate (focal length) of the lens 101, the moving amount of the video camera, and ON / OFF of camera shake correction, and use these as conditions for specifying the subject.

  The subject specifying unit 110 holds a subject list 112 as shown in FIG. 6 and updates the subject list 112 every time a detection result is received from the face detection unit 109. Although details of the subject list 112 will be described later, position and size information, reliability level history, and the like are stored for each face area.

(Face determination process)
The face determination unit 111 determines information that can be effectively used by the image processing unit 106 and the imaging control unit 105 from the face detection result output by the face detection unit 109, and supplies the information to the image processing unit 106 and the imaging control unit 105. . That is, the face determination unit 111 extracts a highly reliable detection result from the face detection results output by the face detection unit 109 and supplies the detection result to the image processing unit 106 and the imaging control unit 105.

  Specifically, the face determination unit 111 holds the number of continuous detections or the continuous detection time corresponding to the reliability level (height) in advance as a determination criterion. Then, for the face area determined by the subject specifying unit 110 as the same subject, the face area that satisfies the determination criterion is finally determined as a highly reliable face area as a face.

  In other words, the face determination unit 111 determines, from among the face regions determined to be the same subject, the face region that has been detected continuously for a number of times or for a time corresponding to the reliability level calculated by the face detection unit 109. It is determined that the face area is high. Here, the level of reliability is a value obtained by performing a predetermined process such as normalization on the reliability obtained from the face detection unit 109 even if the reliability itself supplied from the face detection unit 109 is used. May be. In practice, the face determination unit 111 refers to the subject list 112 (FIG. 6) held and managed by the subject specifying unit 110 and determines a face region that satisfies the determination criterion from the history of reliability levels as a face. Then, the face determination unit 111 reads information on the position and size of the face area determined as a face from the subject list 112 and supplies the information to the image processing unit 106 and the imaging control unit 105.

FIG. 3 shows an example of the face area detected by the face detection unit 109 and the corresponding reliability level.
In the present embodiment, there are five reliability levels from 1 to 5, with 5 being the most reliable. In the example shown in FIG. 3, many face areas (401, 403, 404, 406, 408, 410) in which the face is correctly detected have a high reliability level, and a face in which an area other than the face is erroneously detected Regions (402, 405, 407, 409) often have low reliability levels. However, a correctly detected face area includes a low reliability level (face areas 403 and 408), and a misdetected face area includes a high reliability level (face areas 402 and 407). .

Therefore, it is difficult to accurately distinguish between correct detection and erroneous detection based on the reliability level obtained from only one frame.
Therefore, in the present embodiment, the reliability level of the face area of the same subject is tracked over time, and it is determined whether or not the face area of each subject is truly a face area. Specifically, as described above, for a face area determined to be the face area of the same subject, a determination criterion corresponding to the reliability level is provided, and a face area that satisfies the determination criterion is determined to be a true face region. . The determination criterion may be, for example, the number of times or time that a certain reliability level is continuously detected. In the present embodiment, the subject specifying unit 110 performs tracking and recording of the reliability level for each face area, and the face determination unit 111 performs determination based on the determination criterion.

FIG. 4 is a diagram schematically illustrating an example in which the determination criterion is applied in FIG.
In FIG. 4, a face area whose face frame is indicated by a solid line is determined as a face by the face determination unit 111, and a face area whose face frame is indicated by a dotted line is determined as a face by the face determination unit 111. It has not been shown.

In FIG. 4, for the sake of simplicity of understanding and explanation, it is assumed that a determination criterion is set that a face is determined when the reliability level of 4 or more continues for two frames, and no other face is determined.
At the stage of frame t-2 (FIG. 4 (a)), since no face area satisfies the determination criteria, none of the face areas 401 to 403 is determined as a face, and the face frame is indicated by a dotted line. ing.

  In the next frame t-1 (FIG. 4B), the face area 404 continues to have a reliability level of 5 from the face area 401 of the previous frame determined to be the same subject. It is determined. The erroneously detected face area 405 has a reliability level 4 in the corresponding face area 402 in the previous frame, but is not determined to be a face because the reliability level has dropped to 1 in the current frame. Further, the face area 406 detected correctly is not determined to be a face at this time because the corresponding face area 403 in the previous frame has a reliability level 2.

  In frame t (FIG. 4C), the reliability level of the face area 408 has decreased to 2, but since it has already been determined to be a face, it is still determined to be a face at this point. The face area 410 continues to have a reliability level 4 from the face area 406 of the previous frame determined to be the same subject, and thus is determined to be a face at this point. The erroneously detected face area 409 is still not determined to be a face because the reliability level is 3. The face area 407 erroneously detected in the frame t-1 is not determined to be a face because there is no face area that is determined to be the same subject in the frame t.

The face determination process in the imaging apparatus of the present embodiment will be further described using the flowchart shown in FIG.
First, the face detection unit 109 detects a human face from the image, and obtains the position (coordinates), size, reliability, and the like in the image for each detected face area (S601). In this embodiment, it is assumed that the face detection unit 109 detects a face for each image obtained continuously in time series.

  Next, for one of the detected face areas, the subject specifying unit 110 compares the detection result of the current frame with the detection result of the previous frame by the above-described subject specifying method to specify the face area of the same subject ( S602).

  Based on this determination, the subject specifying unit 110 updates or registers the subject list 112 as shown in FIG. That is, the subject specifying unit 110 updates the data registered in the subject list 112 for the face area detected in the current frame and determined to be the same subject as the face area detected in the previous frame. (S603). Also, the subject specifying unit 110 deletes from the subject list 112 subject data relating to subjects that were detected in the previous frame but not detected in the current frame.

  The subject specifying unit 110 rewrites and updates the position and size of the face area in the subject data with the detection result in the current frame. The number of continuous detections of the reliability level is rewritten and updated by a method according to the reliability level in the current frame and the determination criterion.

  For example, if the determination criterion is the number of continuous detections of a certain reliability level, the number of continuous detections of the corresponding reliability level is set to 1 only when the reliability level in the current frame is equal to the reliability level in the previous frame. increase. If the reliability level in the current frame is different from the reliability level in the previous frame, the continuous detection count of the reliability level in the current frame is set to “1”, and the continuous detection count of other reliability levels is set to “1”. “0”.

  On the other hand, if the criterion is the number of continuous detections at or above a certain reliability level, increase the reliability level at the current frame and the number of continuous detections at a reliability level lower than that, and the reliability level at the current frame The number of continuous detections with a high reliability level is all zero.

  For example, if the reliability level in the current frame is 5, the number of continuous detections of all reliability levels is increased. If the reliability level in the current frame is 3, the number of continuous detections at reliability levels 1 to 3 is increased, and the number of continuous detections at reliability levels 4 to 5 is set to 0. By such an updating method, it is possible to record the number of times that reliability levels equal to or higher than the individual reliability levels are continuously detected.

  On the other hand, the subject specifying unit 110 assigns a new subject ID to the subject list 112 for a subject different from the face region detected in the previous frame among the face regions detected in the current frame. Registration as subject data (S604).

  The subject specifying unit 110 updates and registers the data of the subject list 112 in FIG. 6 based on the position / size of the face area, the reliability level, and the subject specifying result supplied from the face detecting unit 109. .

The subject list 112 includes, for each subject ID for identifying the subject (face region), the position and size of the face region, the number of times of continuous detection for each reliability level, and a face determination flag indicating whether or not a face is determined. An update flag indicating registration or update in the current frame is associated.
The update flag is cleared to 0 for each frame, and the update flag becomes 1 by updating in S603 or registering in S604.

  During processing in the initial frame, since there is no data in the subject list 112, the subject specifying unit 110 registers all detected face area information as new subject data in the subject list 112. From the second frame onward, the subject specifying unit 110 performs subject determination processing using subject data in the subject list 112 and information on the face detection result of the current frame.

  In the face determination unit 111, based on the reliability level history recorded in the subject list 112, specifically, the number of continuous detections or the continuous detection time corresponding to the reliability level, Among the face areas, a face area that is likely to be a face is determined.

  Further, in the present embodiment, for a subject determined as a face by the face determination unit 111, if the face area specified as the same subject is detected regardless of the level of reliability in the subsequent detection result, Judged to be a face.

  Therefore, the face determination unit 111 first determines whether the face area detected in the current frame has already been determined as a face (S605). The face determination unit 111 determines that the face is already determined if the face determination flag in the subject list 112 illustrated in FIG. 6 is 1, and determines that the face is not determined if it is 0. Note that the value of the face determination flag at the time of registration of the subject data is 0.

  Then, the face determination unit 111 determines whether or not a predetermined determination criterion is satisfied with reference to the reliability level history with respect to the face region whose face determination flag is 0 (S606). Here, the reference value f (L) of the number of times of continuous detection according to the reliability level L is determined as the determination criterion, and it is determined whether or not this determination criterion is satisfied.

  FIG. 7 shows an example of the reference value f (L) of the number of times of continuous detection necessary for determining a face, which is set according to the reliability level. In FIG. 7, the horizontal axis indicates the reliability level L, and the vertical axis indicates the reference value f (L) of the number of times of continuous detection. The higher the reliability level, the lower the reference value f (L) is set. Yes. When the continuous detection time is used instead of the continuous detection count, the reference value f (L) is set to be shorter as the reliability level is higher.

  The reliability level L and the reference value f (L) for the number of continuous detections may be linear or non-linear. Further, the reference value f (L) for the number of times of continuous detection may be the number of times for each reliability level or the number of times of each reliability level or more.

  The face determination unit 111 determines that the face area included in the subject list 112 shown in FIG. 6 satisfies the reference value shown in FIG. The value of the face determination flag in 112 is set to 1 (S607). Then, the face determination unit 111 reads information on the face area determined to be a face from the subject list 112 and supplies the information to the image processing unit 106 and the imaging control unit 105.

  On the other hand, the face determination unit 111 does not determine that the face area has the number of continuous detections not satisfying the reference value, and leaves the face determination flag at 0. Then, the face determination unit 111 does not supply information on the face area that is not determined to be a face to the image processing unit 106 or the imaging control unit 105.

  Since the processing from S602 to S607 described above is performed for each face area detected by the face detection in S601, the face determination unit 111 has processed all face areas detected in the current frame. Is determined (S608).

  Then, when an unprocessed face area remains, the face determination unit 111 returns the process to S602 with one of them as a processing target. On the other hand, if all face areas detected in the current frame have been processed, the face determination unit 111 moves the process to S609.

  Then, if there is unupdated data in the subject data in the subject list 112, the face determination unit 111 is data of a subject that was detected in the previous frame but not detected in the current frame. It is deleted (S609).

  As described above, according to the present embodiment, when determining whether or not the face area detected by the face detection process is a face, not only the reliability in the detected frame but also the reliability history is used. By performing the determination, the accuracy of face detection can be increased by a simple method.

(Modification 1)
In the present embodiment, it has been described that data of a subject that was detected in the previous frame but not detected in the current frame is deleted from the subject list 112 in S609.

  However, even though the subject actually exists, there is a possibility that the subject has not been detected due to a problem of the face detection unit 109. In order to prevent the detection accuracy from being lowered due to such detection omission, the subject data is retained until it is confirmed that the subject once determined as a face is not continuously detected over a predetermined number of frames. It may be configured.

The face determination process in the imaging apparatus when such a configuration is employed will be further described with reference to the flowchart shown in FIG.
S901 to S908 in FIG. 8 are the same as S601 to S608 in FIG. 5 described above. By the processing from S901 to S908, subject identification processing and face determination processing are performed on the face area detected by the face detection unit 109 in the current frame.

FIG. 9 shows an example of the subject list 112 ′ used in this modification.
The subject list 112 ′ in FIG. 9 has information on the number of frames that hold subject data that has not been detected and the number of frames that have been held so far in addition to the information of the subject list 112 in FIG. . The number of held frames is cleared to 0 when the subject identifying unit 110 determines that the same subject has been detected by the face detection unit 109.

  When the face determination processing for all face areas detected in the current frame is completed, the face determination unit 111 determines whether the face determination flag is 1 (determined as a face) and there is unupdated subject data. Determination is made (S909).

  When the face determination flag is 1 and unupdated subject data exists, the face determination unit 111 refers to the corresponding “number of retained frames” and determines whether it is equal to or less than the “number of retained frames” (S910).

  If “the number of retained frames” is equal to or less than “the number of retained frames”, the face determination unit 111 updates the corresponding “number of retained frames” by increasing the corresponding “number of retained frames” without deleting the unupdated subject data ( S911).

  On the other hand, the face determination unit 111 does not update the unupdated subject data in which “the number of retained frames” exceeds the “number of retained frames”. In step S909, the face determination unit 111 determines whether or not the “number of retained frames” is equal to or less than the “number of retained frames” for all unupdated subject data that has been determined to be a face. Until then, the processing of S909 to S911 is repeatedly executed.

  Thereafter, the face determination unit 111 deletes the unupdated subject data from the subject list 112 ′ (S 912). Accordingly, the subject data to be deleted in S912 is subject data that has been determined to be a face, subject data for a face area that has not been detected for a predetermined number of frames continuously, and a face that has not been determined to be a face. This is subject data for a face area that is not detected in.

  The values such as the position and size in the subject data for the face area not detected in the current frame held in this modification are the values before the face area is not detected even if the values are just before the face area is not detected. It may be an estimated value from a value in a frame.

  Further, the value of “the number of frames to be held” can be determined according to the reliability level continuously detected immediately before the face area is not detected and the number of times of continuous detection. For example, if the high reliability level is continuously detected many times, the value of “the number of frames to hold” can be set large. In addition, when the high reliability level is detected continuously a few times or when the low reliability level is detected, the value of “the number of frames to be held” can be set small.

  Further, the value of the number of frames to be held can be determined based on other conditions in addition to or instead of the reliability level and the number of continuous detections. Other conditions may be, for example, the position and size of the face area in the subject list. For example, if the position of the face is the edge of the image, there is a high possibility that the face protrudes from the angle of view as a factor that is no longer detected. Therefore, if the face position is near the center of the image, the number of frames to be held can be increased, and if it is near the edge of the image, the number of frames to be held can be reduced.

(Modification 2)
In the present embodiment, the case where the number of continuous detections or the continuous detection time set according to the level of reliability is used as a determination criterion used for face determination in the face determination unit 111 has been described. However, the number of continuous detections or the continuous detection time may be changed according to the number of detected face areas (subjects), the amount of movement of the face area, and the like.

  The face determination process in the imaging apparatus when changing the number of continuous detections or the continuous detection time according to the number of detected face regions (subjects) will be further described with reference to the flowchart shown in FIG.

  First, the face detection unit 109 detects a human face from the image (S1101). Then, the face detection unit 109 counts the number of detected people (S1102). Note that the count of the detected number of people may be an accumulated value of the detected number of people in each frame included in a certain period, not the number of detected people in each frame. For example, if there are cases where a plurality of people pass at once, the number of detected people temporarily increases, and the determination criteria greatly fluctuate. For this reason, there is a possibility that a human face that has been detected until then is not detected. In addition, each time there is a detection failure of the face detection unit 109, the determination criterion may change. Therefore, by making the count of the detected number of people the cumulative value of the detected number of persons over a certain period, it is possible to suppress fluctuations in the determination criteria.

  Next, the subject specifying unit 110 compares the detection results of the previous frame with reference to the detection result of the current frame, and determines the face area of the same subject (S1103). If a face area of the same subject as the previous frame is detected in the current frame, the subject specifying unit 110 updates corresponding subject data in the subject list (S1104).

  On the other hand, when a face area of a subject not detected in the previous frame is detected in the current frame, the subject specifying unit 110 registers information about the face area as new subject data in the subject list (S1105). .

  FIG. 11 shows a subject list 112 ″ used in this modification. The subject list 112 ″ in this modification has information on the number of detected people in addition to the information on the subject list 112 in FIG. The number of detected persons is supplied from the face detection unit 109 to the subject specifying unit 110, and the subject specifying unit 110 registers in the subject list 112 "when registering the subject data.

  The face determination unit 111 determines whether the face area detected in the current frame is a face area of a subject that has already been determined to be a face (S1106). Then, the face determination unit 111 determines whether or not the number of continuous detections satisfies a reference value f (L, n) corresponding to the reliability level L and the detected number n regarding a face region that is not determined as a face ( S1107).

The face determination unit 111 determines a face area whose number of continuous detections satisfies the reference value as a face, and updates the subject list (S1108).
Here, in this modified example, if the number of detected persons counted in S1102 is small, the detection frequency should be increased. Therefore, the reference value for the number of continuous detections is not reduced, and the face criterion is relaxed. On the other hand, when the number of detected people is large, suppression of false detection is desired rather than increasing the detection frequency. Therefore, the reference value for the number of times of continuous detection is increased to make the determination criteria stricter.

An example of setting a reference value for the number of times of continuous detection according to the reliability level and the number of detected persons is shown in FIG.
In FIG. 12, the horizontal axis represents the reliability level, the depth axis represents the number of detected persons, and the reference value for the number of times of continuous detection corresponding thereto is indicated on the vertical axis. In the example of FIG. 12, the number of detected people is classified into two, that is, a large number and a small number according to a predetermined threshold value, but may be classified more finely. Further, the relationship between the reliability level and the number of detected persons and the number of continuous detections may be a linear relationship or a non-linear relationship.

The face determination unit 111 determines whether the above-described processing from S1103 to S1108 has been executed for each of the face areas detected by the face detection in S1101 (S1109).
If there is an unprocessed face area, the process from S1103 is repeated for one of the face areas. If all face areas have been processed, the face determination unit 111 deletes unupdated data from the subject data in the subject list in S1110.

  Instead of the number of people detected by the face detection unit 109, the number of face areas determined as faces by the face determination unit 111 may be reflected in the determination criterion. Needless to say, the continuous detection time may be used instead of the continuous detection count.

  Next, with reference to the flowchart shown in FIG. 13, the face determination process in the imaging apparatus in the case where the number of continuous detections or the continuous detection time used as the determination criterion is changed according to the number of detected face regions (subjects). explain.

  First, the face detection unit 109 detects a human face from the image (S1401). Next, the subject specifying unit 110 compares the detection results of the previous frame with reference to the detection result of the current frame, and determines the face area of the same subject (S1402).

  When a face area that is determined to be the same subject as the previous frame is detected in the current frame, the subject specifying unit 110 calculates the amount of movement of the subject from the difference in detection position of the face area (S1403). The movement amount of the subject may be an accumulated value obtained by adding the movement amount between two adjacent frames to a plurality of frames included in a certain period, not the movement amount between two adjacent frames. By setting the cumulative value, even if the amount of movement of the subject temporarily increases due to erroneous determination of the same subject, it is possible to prevent the judgment criterion from fluctuating greatly. Then, the subject specifying unit 110 updates the corresponding subject data in the subject list including the movement amount (S1404).

  On the other hand, when a face area of a subject not detected in the previous frame is detected in the current frame, the subject specifying unit 110 registers information about the face area as new subject data in the subject list (S1405). . The movement amount at the time of newly registering is 0.

  FIG. 14 shows a subject list 113 used in this modification. The subject list 113 in this modification has information on the amount of movement in addition to the information on the subject list 112 in FIG. For example, the movement amount is registered (updated) for the subject data corresponding to the face area determined by the subject specifying unit 110 as the same subject.

  The face determination unit 111 determines whether the face area detected in the current frame is a face area of a subject that has already been determined to be a face (S1406). Then, the face determination unit 111 determines whether or not the number of continuation detections satisfies a reference value f (L, w) corresponding to the reliability level L and the movement amount w of each subject for a face region that is not determined as a face. Determination is made (S1407).

The face determination unit 111 determines a face area whose number of continuous detections satisfies the reference value as a face, and updates the subject list (S1408).
In normal photography, a subject with a large amount of movement is unlikely to be a main subject, and a subject with a small amount of movement is likely to be a main subject. For this reason, in the present modification, the reference value for the number of times of continuous detection is reduced for a face region with a small amount of movement, and the determination criterion for the face is relaxed. On the other hand, for a face area with a large amount of movement, the reference value for the number of times of continuous detection is increased to tighten the criteria for determining the face.

FIG. 15 shows an example of setting a reference value for the number of times of continuous detection according to the reliability level and the amount of movement of the subject.
In FIG. 15, the horizontal axis indicates the reliability level, the depth axis indicates the magnitude of the movement amount, and the reference value of the number of times of continuous detection corresponding to them is indicated on the vertical axis. In the example of FIG. 15, the movement amount is classified into two types according to a predetermined threshold, but it may be classified more finely. Further, the relationship between the reliability level and the movement amount and the number of times of continuous detection may be a linear relationship or a non-linear relationship.

The face determination unit 111 determines whether the processing from S1402 to S1408 described above has been executed for each of the face areas detected by the face detection in S1401 (S1409).
If there is an unprocessed face area, the process from S1402 is repeatedly executed on one of the face areas. If the processing of all the face areas has been completed, the face determination unit 111 deletes unupdated data from the subject data in the subject list in S1410.

  Note that the method of reflecting the amount of movement of the subject in the determination criterion is not limited to the method of changing the number of continuous detections according to the amount of movement between successive frames or the added value of the amount of movement of each subject within a certain period. . For example, when the amount of movement between consecutive frames exceeds a certain threshold value, the reference value for the number of times of continuous detection may be changed.

(Modification 3)
In the present embodiment, any subject data is kept in the subject list until it is determined that the subject data has not been updated. Therefore, when the angle of view changes greatly due to a change in the imaging mode between consecutive frames, or when the amount of movement of the camera is large, the subject specifying unit 110 specifies different subject face regions as the same subject face region. There is a risk. If the subject is specified incorrectly, the reliability of the final face determination result is greatly reduced.

  To cope with this, in this modification, when it is detected that the entire image has changed significantly (such as a large change in the angle of view or shooting direction), all the subject data in the subject list is deleted, and the next After the frame, new subject data is registered.

FIG. 16 is a flowchart for explaining the face determination processing in the present modification.
First, for example, the image processing unit 106 serving as a change detection unit calculates the amount of change in the image between successive frames, and detects whether or not there has been a large change in the entire image (S1701). The amount of change can be calculated from the luminance component, color component, or edge component between successive frames. The image processing unit 106 determines that the image has changed significantly when the amount of change exceeds a predetermined amount of change. When the image processing apparatus according to the embodiment is applied to an imaging apparatus, a change in a shooting mode (for example, zoom magnification) is detected, or information is acquired from, for example, a gyro sensor that the imaging apparatus has for camera shake correction. Alternatively, it may be determined whether the entire image has changed significantly.

If there is a large change in the entire image, the image processing unit 106 deletes all subject data in the subject list (S1702).
In step S1703, the face detection unit 109 performs face detection processing. At this time, all face detection results are registered in the subject list as new subject data. On the other hand, if there is no significant change in the entire image, the process moves to S1703. Since S1703 to S1711 in FIG. 16 are the same as S601 to S609 in FIG.

  Instead of the image processing unit 106, the face detection unit 109 supplies image data to the subject specifying unit 110 without performing face detection, and the subject specifying unit 110 detects a large change in the entire image. Also good.

  In this modification, the “large change in the entire image” is, for example, a camera movement or a change in the angle of view that changes more than half of the scene, or can be determined as a scene change by a general scene change detection technique. It may be a change like this.

(Other embodiments)
The above-described embodiment and its modification have been described by applying the image processing apparatus according to the present invention to the imaging apparatus and paying attention to the face determination process during imaging. However, it will be understood that an image used for face determination is not limited to an image captured in real time, and may be a recorded image. Therefore, the above-described embodiment and its modification examples can be similarly applied to the face determination process at the time of moving image reproduction. Moreover, the above-described modification examples can be implemented in combination.

  Further, although face detection and face determination have been described as an example of subject detection and subject determination, the subject is not limited to a human face, and any object or organism that can be detected from an image by applying a known image recognition technique. It will also be understood that it may be.

Furthermore, the above-described embodiments (including modifications, the same applies to the following) can also be realized as software by a computer (or CPU, MPU, etc.) of a system or apparatus.
Therefore, the computer program itself supplied to the computer in order to implement the above-described embodiment by the computer also realizes the present invention. That is, the computer program itself for realizing the functions of the above-described embodiments is also one aspect of the present invention.

  The computer program for realizing the above-described embodiment may be in any form as long as it can be read by a computer. For example, it can be composed of object code, a program executed by an interpreter, script data supplied to the OS, but is not limited thereto.

  A computer program for realizing the above-described embodiment is supplied to a computer via a storage medium or wired / wireless communication. Examples of the storage medium for supplying the program include a magnetic storage medium such as a flexible disk, a hard disk, and a magnetic tape, an optical / magneto-optical storage medium such as an MO, CD, and DVD, and a nonvolatile semiconductor memory.

  As a computer program supply method using wired / wireless communication, there is a method of using a server on a computer network. In this case, a data file (program file) that can be a computer program forming the present invention is stored in the server. The program file may be an executable format or a source code.

Then, the program file is supplied by downloading to a client computer that has accessed the server. In this case, the program file can be divided into a plurality of segment files, and the segment files can be distributed and arranged on different servers.
That is, a server apparatus that provides a client computer with a program file for realizing the above-described embodiment is also one aspect of the present invention.

  In addition, a storage medium in which the computer program for realizing the above-described embodiment is encrypted and distributed is distributed, and key information for decrypting is supplied to a user who satisfies a predetermined condition, and the user's computer Installation may be allowed. The key information can be supplied by being downloaded from a homepage via the Internet, for example.

Further, the computer program for realizing the above-described embodiment may use an OS function already running on the computer.
Further, a part of the computer program for realizing the above-described embodiment may be configured by firmware such as an expansion board attached to the computer, or may be executed by a CPU provided in the expansion board. Good.

1 is a block diagram illustrating a configuration example of an imaging apparatus as an example of an image processing apparatus according to an embodiment of the present invention. It is a figure explaining the process which specifies the same subject using the position and size of a face area in the imaging device according to the embodiment of the present invention. It is a figure which shows the example of the face area detected by the face detection part, and the corresponding reliability level in the imaging device which concerns on embodiment of this invention. It is the figure which showed typically the example which applied the determination standard to the image shown in FIG. It is a flowchart for demonstrating the face determination process in the imaging device which concerns on embodiment of this invention. It is a figure showing an example of a photographic subject list in an imaging device concerning an embodiment of the present invention. . It is a figure which shows the example of a setting of the reference value of a reliability level and the continuous detection frequency in the imaging device which concerns on embodiment of this invention. It is a flowchart for demonstrating the face determination process in the imaging device which concerns on the modification 1 of embodiment of this invention. It is a figure which shows the example of the to-be-photographed object list | wrist in the imaging device which concerns on the modification 1 of embodiment of this invention. It is a flowchart for demonstrating the face determination process in the imaging device which concerns on the modification 2 of embodiment of this invention. It is a figure which shows the example of the to-be-photographed object list | wrist in the imaging device which concerns on the modification 2 of embodiment of this invention. It is a figure which shows the example of a setting of the reference value of the continuous detection frequency | count according to a reliability level and the detection number in the imaging device which concerns on the modification 2 of embodiment of this invention. It is a flowchart for demonstrating another face determination process in the imaging device which concerns on the modification 2 of embodiment of this invention. It is a figure which shows the example of another to-be-photographed object list | wrist in the imaging device which concerns on the modification 2 of embodiment of this invention. It is a figure which shows the example of a setting of the reference value of the continuous detection frequency | count according to the reliability level and subject moving amount in the imaging device which concerns on the modification 2 of embodiment of this invention. It is a flowchart for demonstrating the face determination process in the imaging device which concerns on the modification 3 of embodiment of this invention.

Claims (11)

Subject detection means for detecting a predetermined subject region from images supplied in time series and detecting the reliability of the detected region;
Of the areas detected by the subject detection means from different images, the specifying means for specifying the same subject area;
Recording means for recording subject data including a history of reliability detected by the subject detection means for an area specified by the specifying means as an area of the same subject;
Of the areas identified by the identifying means as the same subject area, the area in which the reliability history recorded by the recording means satisfies the determination criteria determined according to the reliability level is determined in advance. An image processing apparatus comprising: determination means for determining an area of a subject.   The determination criterion is the number of times or the time that the reliability is continuously detected by the subject detection unit according to the level of reliability, and the number of times or the time is shorter as the reliability level is higher. The image processing apparatus according to claim 1.   As for the area determined as the predetermined subject area, as long as the area is detected by the subject detection means, the determination means uses the predetermined subject area even if the reliability decreases. The image processing apparatus according to claim 1, wherein the image processing apparatus is determined to be present.   2. The identification unit is configured to delete the subject data corresponding to a region that is not detected by the subject detection unit from a region where the reliability history is recorded in the recording unit. The image processing apparatus according to claim 1.   The identification means includes the subject corresponding to a region that is not continuously detected for a predetermined number or time of images in the subject detection unit in the region where the history of reliability is recorded in the recording unit. The image processing apparatus according to claim 4, wherein the data is deleted.   The determination criterion is determined according to the level of reliability and the number of the areas detected by the subject detection unit, and when the number of the areas is smaller than when the number of the areas is large, 6. The image processing apparatus according to claim 1, wherein the image processing apparatus is determined so as to be easily determined as the predetermined subject area.   The determination criterion is determined according to the level of reliability and the amount of movement of the region identified by the specifying unit as the same subject region, and the region with the smaller amount of movement than the region with the larger amount of movement. The image processing apparatus according to claim 1, wherein the image processing apparatus is determined so as to be easily determined as the predetermined subject area.   A change detecting unit that calculates a change amount of the adjacent images in time series and deletes all the subject data recorded by the recording unit when the change amount exceeds a predetermined change amount. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. A lens for forming a subject optical image;
Imaging means for sequentially capturing an optical image of a subject formed by the lens and outputting an image supplied in time series;
An image processing apparatus according to any one of claims 1 to 8,
An imaging apparatus comprising: control means for controlling imaging conditions using information on an area determined by the determination means as the predetermined subject area. An imaging device,
A lens for forming a subject optical image;
Imaging means for sequentially capturing an optical image of a subject formed by the lens and outputting an image supplied in time series;
An image processing apparatus according to claim 8,
Control means for controlling imaging conditions using information of the area determined by the determination means as the predetermined subject area;
The image processing apparatus is
While acquiring information regarding the movement of the imaging device or the change in the angle of view of the lens, and when at least one of these information exceeds a predetermined threshold, the image output by the imaging means is assumed to change significantly. An image pickup apparatus comprising: a change detection unit that deletes all the subject data recorded by the recording unit. A subject detection step for detecting a predetermined region of the subject from an image supplied in time series and detecting the reliability of the detected region;
A specifying step for specifying a region of the same subject among the regions detected in the subject detection step from different images;
A recording step for recording subject data including a history of the reliability detected in the subject detection step in a recording unit for the region identified as the same subject region in the identifying step;
Among the areas identified as the same subject area in the identifying step, an area in which the reliability history recorded in the recording unit satisfies a determination criterion determined in accordance with the reliability level is determined in advance. An image processing method comprising: a determination step of determining an area of a defined subject.

Images