JP5383308B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus capable of detecting a face.

  In a digital camera having a face detection function, a blind eye detection function is known as a technique for reducing a failed image caused by the subject's eyes closing.

  The procedure of the eyebrows detection function is as follows. First, a face area is detected from a captured image by a face detection function. When the face area is determined by the face detection function, it is detected whether or not each face is closed by the eye detection function. The following two methods are generally used as means for preventing a failed image when a blindness is found.

  When it is detected that the subject has closed eyes, continuous shooting is automatically performed to obtain an image without closing eyes (see Patent Document 1).

  When it is detected that the subject has closed eyes, a warning display to the effect that the eyes are closed is given to the user to encourage re-shooting (see Patent Document 2).

  In this way, the camera can assist so that an unbroken image can be taken.

JP 2003-287784 A JP2007-49631

  However, in both methods, when there is a subject in the image that is different from the subject that the user wants to shoot, as shown in FIG. There was a problem that continuous shooting and eyebrows warnings were displayed. This is an annoying display for the user. In addition, since eyebrows detection processing is performed for an extra person, it takes more processing time than necessary.

  On the other hand, as shown in FIG. 10B, the problem is solved by treating only a face existing within a certain range of the center of the object scene or a face having a certain face size or more as a subject. A way to do this is considered. However, even if the area for detecting eyebrows is limited in this way, when taking a group photo as shown in FIG. 10C, it is necessary to focus on people who are out of a certain range or who have a small face size in the image. There was a problem that the pick detection function stopped working.

  This can be said to be the same problem not only in the case of detecting eyebrows but also in the case of performing red-eye correction.

  In addition, the personal authentication that analyzes the detected face and identifies who the face is is complicated, and has a processing time. Therefore, in order to improve the processing time, it is conceivable that persons who do not seem to be related in advance are excluded from the objects of personal authentication. In this case as well, it is required to extract only the target that the user wants to shoot from among the detected faces, as in the above-described eye-brow detection and red-eye correction.

  It is an object of the present invention to solve the problem that image analysis or display of image processing results is performed on a person who is not a subject of shooting, and to provide an image processing apparatus that is easy to use for the user. And

  Another problem of the present invention is that even when the user changes an arbitrary subject to the main subject, it is possible to determine which extra subject is based on the main subject newly selected by the user. . Therefore, it is to improve the probability that the image analysis or the display of the image processing result can be performed only on the subject that the user intends to shoot.

In order to solve the above-described problems, an image processing apparatus according to the present invention includes a face detection unit that detects a face from image data, and a plurality of faces detected from the face detection unit. And selecting a face as a related subject based on a correlation with the face selected as the main subject from the main subject selection means selected as the main subject and the face not selected as the main subject by the main subject selection means A related subject selection unit; an image processing unit that performs predetermined image processing on the face detected by the face detection unit; and an operation unit , wherein the main subject selection unit includes a first operation in the operation unit. And the related subject selecting means selects the face as the related subject in accordance with a second operation in the operation unit after the first operation, Previous The image processing means performs the predetermined image processing on the face selected by the main subject selection means and the related subject selection means, and does not perform the predetermined image processing on the other faces. It is characterized by that.
Similarly, in order to solve the above-described problems, an image processing apparatus according to the present invention detects a face from image data, and detects a face when a plurality of faces are detected from the face detection means. Based on the correlation between the main subject selecting means for selecting as the main subject and the face not selected as the main subject by the main subject selecting means, the face as the related subject is selected. A related subject selection unit to be selected; an image processing unit that performs predetermined image processing on the face detected by the face detection unit; and an operation unit, wherein the main subject selection unit is a first unit in the operation unit. The related subject selection means selects the face as the related subject according to a second operation in the operation unit after the first operation. The image processing means displays the result of the predetermined image processing for the face selected by the main subject selecting means and the related subject selecting means, and for the other faces, the predetermined image processing result is displayed. The result of the image processing is not displayed.

  According to the present invention, the convenience of the image processing apparatus can be improved.

  Another effect of the present invention is that a blinding warning display is not performed on an extra subject, so that it is possible to prevent the display from being annoying for the user.

  Another advantage of the present invention is that it does not perform time-consuming image analysis such as personal authentication for a person that the user does not intend as a subject, so that the processing time can be shortened compared to processing that does not distinguish the subject. It has become possible.

1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment of the present invention. The block diagram which shows the function of the image process part which concerns on embodiment of this invention 1 is a flowchart showing the overall operation of an imaging apparatus according to Embodiment 1 of the present invention. The flowchart which shows the algorithm of the face detection which concerns on embodiment of this invention 6 is a flowchart showing an algorithm for main subject selection according to the embodiment of the present invention. 6 is a flowchart showing an algorithm for selecting a related subject according to an embodiment of the present invention. The flowchart which shows the algorithm of the eyelid detection which concerns on embodiment of this invention 7 is a flowchart showing the overall operation of the imaging apparatus according to the second embodiment of the present invention. 7 is a flowchart showing the overall operation of an imaging apparatus according to Embodiment 3 of the present invention. Explanatory drawing about the subject of the prior art which this invention solves

(Embodiment 1)
In the present embodiment, an embodiment will be described in which eyebrows detection is performed for the main subject and related subjects, and eyebrows detection is not performed for other faces.

[Configuration of imaging device]
FIG. 1 shows the configuration of an image processing apparatus according to an embodiment of the present invention.

  In the present embodiment, a digital camera will be described as an example of an image processing apparatus.

  Reference numeral 101 denotes an optical system that includes a photographic lens and a diaphragm device, and adjusts the amount of light and the focal position of the optical image reflected from the subject.

  Reference numeral 102 denotes an image pickup device composed of a CCD, a CMOS sensor or the like that converts an optical image into an electric signal.

  Reference numeral 103 denotes an A / D (analog / digital) converter that converts image data of an analog signal output from the image sensor 102 into image data of a digital signal. Image data output from the A / D conversion unit 103 is written to the memory 106 via the image processing unit 104 and the memory control unit 105 or directly via the memory control unit 105.

  An image processing unit 104 performs image analysis processing and image processing such as face detection, face selection for performing image analysis, eye-browsing detection, red-eye correction, and personal authentication on the image data output from the A / D conversion unit. It is.

  A memory control unit 105 controls writing to the memory 106 or reading from the memory 106.

  In the memory 106, image data obtained from the image sensor 102 and the A / D conversion unit 103, image data subjected to image processing by the image processing unit 104, information including thumbnails of the image data, and images taken The number of image data, the number of face detections, etc. are written.

  A system control unit 107 controls the entire system of the digital camera. The system control unit 107 performs image analysis processing and image processing control on the image processing unit 104, and also converts the image data written in the memory 106 into a file and records it via the I / F (interface) unit 108. Record on media. The I / F unit 108 is an interface with a memory card or a recording medium.

  Reference numeral 109 denotes a D / A (digital / analog) converter, which converts image data written in the memory 106 into image data of an analog signal.

  Reference numeral 110 denotes a display unit that displays the image data converted by the D / A conversion unit 109 or displays the image data generated by the image sensor 102 through. The through display means that image data is continuously generated by the image sensor 102 and the state of the subject is displayed on the display unit 110 in real time as a moving image. The display unit 110 displays the image data processed by the image processing unit 104, the image data recorded in the memory 106 and the recording medium, and displays the results of image analysis processing and image processing.

  The operation unit 111 inputs various operation instructions to the system control unit 107. Examples of the operation unit include a release button 112 for instructing imaging, a direction instruction button 113 for operating a cursor and a menu, a touch panel 114, and the like.

  The release button 112 included in the operation unit 111 generates the first shutter switch signal SW1 when pressed halfway. The system control unit 107 detects the first shutter switch signal SW1, thereby performing AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, EF (flash pre-flash) processing, Operations such as face detection processing are started. AF processing means that the optical system 101 adjusts the position of the focal point of the optical image.

  When the release button 112 is fully pressed, the second shutter switch signal SW2 is generated. By detecting the second shutter switch signal SW2, the system control unit 107 starts an imaging process operation in which the image data read from the image sensor 102 is written and stored in a recording medium.

[Configuration of image processing unit]
FIG. 2 is a block diagram illustrating a configuration of the image processing unit 104.

  The image processing unit 104 according to the present exemplary embodiment includes a face detection unit 201, a main subject selection unit 202, a related subject selection unit 203, and a blindness detection unit 204. Each processing unit executes image analysis processing and image processing under the control of the system control unit 107.

  The face detection unit 201 detects a face area for the image data input to the image processing unit 104. The face image data detected by the face detection unit 201 and face detection information such as the eye position and mouth position of each face are written in the memory 106. The face detection algorithm will be described later.

  When a plurality of faces are detected by the face detection unit 201, the main subject selection unit 202 selects a face to be a main subject among a plurality of faces existing in the image data based on the face detection information. If only one face is detected, that face becomes the main subject.

  As a method of selecting a main subject, a method of selecting according to individual priority order registered in advance using a personal authentication algorithm or the like, a method of selecting according to the position of the detected face area, and the size of the face Etc. are common. For example, in Japanese Patent Application Laid-Open No. 2008-005438, the closer to the center of the screen, the larger the size, and the higher the degree of satisfying the face condition, the larger the weight value of the face, and the face with the largest weight value is defined as the main subject. How to do is described. Here, the reliability coefficient is a value indicating the degree to which the face satisfies the condition as a face. That is, the face reliability coefficient determined after detecting all the shapes of both eyes, nose, and mouth is determined to be the face area after only the shapes of both eyes and mouth are detected. The value is larger than the reliability coefficient of the face.

  Alternatively, the user may select the main subject by operating the direction instruction button 113 or the touch panel 114 included in the operation unit 111 from a plurality of detected faces. The face selected as the main subject is written in the memory 106 as face detection information to which information that can be identified as the main subject is added.

  Based on the face detection information, the related subject selection unit 203 determines whether the user is subject to image analysis processing or image processing based on the correlation with the face selected as the main subject. . If the subject is worth performing image analysis processing or image processing, the subject is selected as a related subject related to the main subject. The face selected as the related subject is written in the memory 106 as face detection information in which information that can be identified as the related subject is disabled. In the following, description will be given by taking eyebrows detection as an example of image analysis processing.

  The eyebrows detection unit 204 performs eyebrows detection on the face selected by the main subject selection unit 202 and the related subject selection unit 203 based on the face detection information obtained by the face detection unit 201. As a method for detecting eye-opening, for example, an area estimated to be around the eyes is extracted from the face. A method for determining whether or not the eyes are sufficiently open by template matching between the image data of the extracted area and the eye image stored in the memory in advance is known (see Japanese Patent Application Laid-Open No. 2006-070883). . The eyebrows detection algorithm of the present invention will be described later.

  In FIG. 2, the face detection unit 201, the main subject selection unit 202, the related subject selection unit 203, and the eyelid detection unit 204 have been described as being configured in separate circuits. It may be configured by a single circuit having all functions.

  The operation of the embodiment having the above configuration will be described.

[Overall operation of digital camera]
FIG. 3 is a flowchart showing the overall operation of the imaging apparatus according to the present invention.

  When the still image shooting mode for recording a still image is set, the system control unit 107 starts through display for displaying the image data generated by the image sensor 102 in step S301.

  In step S302, the system control unit 107 causes the face detection unit 201 to perform face detection to detect whether a human face exists in the through-displayed image data. When a human face is detected, the system control unit 107 detects the size of the detected face (width, height, number of pixels occupied by the face, etc.), position information, detected number, reliability coefficient, etc. in the image data. The information is stored in the memory 106 as information.

  Subsequently, in step S303, the system control unit 107 determines whether or not the first shutter switch signal SW1 is ON. If the first shutter switch signal is OFF, the processing step returns to S302. If the first shutter switch signal is ON, the process proceeds to step S304.

  In step S <b> 304, the system control unit 107 causes the main subject selection unit 202 to select a main subject from the faces detected by the face detection process, and stores the main subject in the memory 106. Details of the main subject selection method will be described later.

  When the main subject is selected by the main subject selection unit 202, the display unit 110 superimposes a frame indicating that on the face selected as the main subject so that it can be seen which face is selected as the main subject. Are displayed on the display unit 110. If frames are already displayed for a plurality of faces detected by the face detection unit 201, only the frame displayed on the face selected as the main subject is displayed in a different color so that the user can see at a glance. Make it possible to recognize the face as the main subject. The user sees this display, and if a face different from the intended face is selected as the main subject, the main subject is again turned off by turning the first shutter switch signal SW1 off and then on again. You can redo the selection. Alternatively, the user may select another face as the main subject by operating the touch panel 114 or the direction instruction button 113 included in the operation unit 111.

  In step S305, the system control unit 107 performs AF processing to focus the optical system 101 on the main subject selected in step S304, and performs AE processing to determine an aperture value and a shutter time (shutter speed). And stored in the memory 106.

  Next, in steps S306 and S307, the system control unit 107 determines whether or not the first shutter switch signal SW1 and the second shutter switch signal SW2 are ON. If the second shutter switch signal SW2 is turned on while the first shutter switch signal SW is turned on, the process proceeds from step S306 to step S308. When the first shutter switch signal SW1 is turned off, the process returns to S302. When the first shutter switch signal SW1 is ON and the second shutter switch signal SW2 is OFF, the processes of steps S306 and S307 are repeated.

  In step S308, the system control unit 107 starts an exposure process on the image sensor 102 according to the exposure amount obtained from the image data obtained so far according to the timing at which the second shutter switch signal SW2 is detected. To do. A photographing process for performing a developing process on the image data generated by the exposure process is executed, and the result is stored in the memory 106.

  In step S309, the system control unit 107 causes the face detection unit 201 to perform face detection again on the image data generated in step S308. Then, the main subject selection unit 202 selects, from the plurality of detected faces, the face position information selected as the main subject in step S304 and the face having the closest correlation with the size as the main subject, Store in memory 106.

  In step S <b> 310, the system control unit 107 determines the main subject based on the correlation between the face position information, the face size, the reliability coefficient, and the like between the face selected as the main subject and the other face stored in the memory 106. A face deemed highly relevant is selected as a related subject. Then, the result is stored in the memory 106. Of course, the related subject may be selected using only one of the correlations of the face position information, the size, and the reliability coefficient, or the related subject may be selected in consideration of other color signal and luminance signal correlations. May be.

  Next, in step S311, the system control unit 107 determines whether or not the eyes of the main subject and the related subject's face are closed in the image data captured in step S308. I do. The details of the detection of eyebrows will be described later.

  Next, in step S312, the system control unit 107 displays on the display unit 110 a REC review display of the image data obtained by the imaging processing in step S308 on the display unit 110. The REC review is a process of displaying image data on the display unit 110 for a predetermined time before recording on a recording medium after shooting a subject in order to confirm a shot image. At this time, if there is a subject whose eyes are detected in step S311, the display unit 110 displays that the subject knows that the eyes are eye-catching. For example, by displaying a frame over the image data so as to surround the eye of the subject that is pinching the eyes, it is appealing that the eyes are pinching, or a comment or image indicating that the eyes are pinching is overlaid on the image data Or display. By viewing this display, the user can recognize that the subject in the photographed image data has closed his eyes, and can perform the photographing again if necessary.

  After the REC review display, in step S313, the system control unit 107 executes a recording process in which the image data obtained by the photographing process is written as an image file on the recording medium.

  When the recording process of step S313 ends, in step S314, the system control unit 107 determines whether the second shutter switch signal SW2 is ON or OFF. If the second shutter switch signal SW2 is ON, the determination in step S314 is repeated, and the process waits for the second shutter switch signal SW2 to be OFF. During this time, the REC review display is continued. That is, when the recording process in step S312 is completed, the REC review display on the display unit 110 is continued until the second shutter switch signal SW2 is released. By configuring in this way, the user can carefully check the blindness of the captured image data using the REC review, for example, by continuously pressing the release button.

  Next, for example, after the user performs shooting with the release button 112 fully pressed, when the user releases the release button to release the fully pressed state, the process proceeds from step S314 to step S315. In step S314, the system control unit 107 returns the display state of the display unit 110 from the REC review display to the through display state. With this processing, after confirming the captured image data by the REC review display, the display state of the display unit 110 is automatically switched to a through display state in which image data from the image sensor 102 is sequentially displayed for the next photographing. Become.

  In step S316, the system control unit 107 determines whether the first shutter switch signal SW1 is ON or OFF. If the first shutter switch signal SW1 is ON, the process proceeds to step S306. If it is OFF, the process proceeds to step S302. Return processing. That is, when the release button 112 is being pressed halfway (the first shutter switch signal SW1 is ON), the system control unit 107 prepares for the next shooting (step S302). On the other hand, if the release button has been released (the first shutter switch signal SW1 is OFF), the system control unit 107 finishes a series of shooting operations and returns to the shooting standby state (step S306).

[Face detection algorithm]
Next, an algorithm for performing the above-described face detection will be described in detail with reference to FIG.

  The face detection unit 201 executes each process described below under the control of the system control unit 107.

  First, in step S <b> 401, the system control unit 107 sends the image data to be detected to the face detection unit 201 in the image processing unit 104.

  In step S402, the face detection unit 201 calculates a horizontal band-pass filter on the sent image data.

  In step S403, the face detection unit 201 calculates a vertical band-pass filter on the image data processed in step S402. By these arithmetic processes, a face edge component is detected from the image data.

  In step S <b> 404, the face detection unit 201 performs pattern matching on the detected edge component, and extracts a candidate group of parts constituting the eyes, nose, mouth, and ear faces.

  In step S405, the face detection unit 201 estimates a face candidate region from the distances and positions of the eye, nose, mouth, and ear candidate groups narrowed down in step S404. The face is finally detected by calculating a preset condition filter for these face candidate regions. Here, even if all of the candidate groups for eyes, nose, mouth, and ears are not complete, if these correlation distances and arrangements satisfy the conditions as a face, it is determined that the area is a face area. .

  In step S406, the face detection unit 201 outputs the face detection information according to the face detection result in step S405, and ends the face detection process. At this time, the system control unit 107 identifies the number of detected faces of the output face detection information, the face position information, the face size, a reliability coefficient indicating the degree of satisfying the face condition, and the detected face. Are stored in the memory 106. The reliability coefficient here is determined based on, for example, whether there are two eyes, how far the two eyes are, and whether a nose exists between the two eyes.

  As face detection methods other than the face detection algorithm described above, there are a method in which a face detection area is set in advance and a face in the area is detected, and a face is detected from the hue of the face.

[Main subject selection algorithm]
Next, the above-described main subject selection algorithm will be described in detail with reference to FIG.

  The main subject selection unit 202 executes each process described below under the control of the system control unit 107. At this time, the face detection information stored in the memory 106 in the face detection process is used.

  In step S501, the main subject selection unit 202 stores the number of detected faces detected in step S302 in the memory 106 and sets the counter N of the memory 106 that counts the faces to be subjected to the main subject selection process to 0. Initialize to (zero).

  In step S <b> 502, the main subject selection unit 202 reads the face detection results such as the reliability coefficient, the face size, and the face position information obtained by the face detection process from the memory 106.

  In step S503, the main subject selection unit 202 determines whether the number of detected faces detected by the face detection process is zero. When the face detection number is 0, there is no subject that seems to be a person, and the main subject process is terminated. If one or more faces are detected in step S503, the process proceeds to step S504.

  In step S504, the main subject selection unit 202 increments the value of the counter N by 1.

  In step S505, the main subject selection unit 202 determines whether or not the number of face detections detected by the face detection process is one. If there is only one detected face, the person who appears to be the main subject is identified as the person whose face is detected, so the detected face is selected as the main subject (step S512), and the main subject selection process is terminated. To do. At this time, the main subject selection unit 202 writes the face detection information by adding information capable of determining that the selected face is the main subject to the memory 106.

  If the number of detected faces is not one in step S505, that is, if there are a plurality of detected faces, the process proceeds to step S506.

  In step S506, the main subject selection unit 202 calculates weight 1 calculated based on the face reliability coefficient. In step S507, the main subject selection unit 202 calculates a weight 2 calculated based on the detected face size. In step S508, the main subject selection unit 202 calculates weight 3 based on the detected face position information. The weight here is a coefficient for calculating the final weight for selecting the main subject. For example, when the face size is 0 (pixel), the weight is set to 0, when the face size is 20 (pixel), the weight is set to 0.2, and when the face size is 30 (pixel), the weight is set to 0.8. .

  An example in which the weight is calculated from the distance between the center coordinates of the object scene and the face position where the face is detected will be described. The size of the display image data of the digital camera in this embodiment is 320 (pixels) × 240 (pixels). Therefore, the distance is calculated from the center coordinates (160, 120) of the object scene and the position information (X, Y) of the face read from the memory 106 by using the three square theorem. When the distance is 10 (pixels), the weight is set to 1.0 and 80 (pixels), and when the distance is 80 (pixels) or more, the weight is set to 0.6.

  As for the face reliability coefficient, when the eyes exist in pairs and the nose and mouth parts are extracted by face detection processing, the weight is 1.0, and when only one eye is extracted, the weight is Is set to 0.9, and 0.8 is set when only the nose and mouth are extracted.

  Next, in step S509, a value obtained by calculating the sum of the calculated weights 1 to 3 is written in the memory 106, and the process proceeds to step S510. The final weight calculated here may be a multiplication of each weight.

  In step S510, the main subject selection unit 202 selects the face having the largest final weight value as the main subject. At this time, information capable of determining that the selected face is the main subject is added to the memory 106, the face detection information is written, and the process proceeds to step S511.

  In step S <b> 511, the main subject selection unit 202 compares the value of the counter N with the face detection number previously written in the memory 106. If the value of the counter N is the same as the number of face detections, the main subject selection process has been executed for all faces whose faces have been detected, so the main subject selection process ends.

  When the value of the counter N is not the same as the face detection number, the process returns to step S502 and the process is repeated.

  The method for detecting the main subject using the face size, the position information of the face, and the reliability coefficient indicating the degree to which the condition as the face is satisfied has been described above, but the face estimated as a person registered in advance using personal authentication May be selected as the main subject.

  Personal authentication refers to who is the face in the image data by comparing the feature points of each person registered in a memory not shown in advance with the feature points extracted from the face detected by the face detection process. This is a process for identifying the above. Specifically, matching is performed between the arrangement information of each feature point such as eyes, nose and mouth extracted by face detection and the arrangement information of the facial feature points registered in the memory in advance. Calculate If there is a subject with the highest similarity and the similarity exceeds the threshold, it is determined that the subject is the same person as the individual registered in advance (see Japanese Patent Laid-Open No. 09-251534). . By storing the personal priorities in correspondence with the feature point arrangement information in the memory 106 in advance, when a plurality of faces are detected, the faces are successfully authenticated, and The face with the highest personal priority stored in the memory can be selected as the main subject.

[Related subject selection algorithm]
Next, the related subject selection algorithm described above will be described in detail with reference to FIGS. 6 (A) and 6 (B).

  The related subject selection unit 203 executes each process described below under the control of the system control unit 107. At this time, the face detection information stored in the memory 106 in the face detection process is used.

  In step S601 of FIG. 6A, the related subject selection unit 203 initializes a counter N that counts the face to be subjected to the main subject selection process to 0 (zero).

  In step S602, the related subject selection unit 203 reads the face detection result of the face size and face position information stored in the memory 106 and the selection result of the main subject, and proceeds to step S603.

  In step S603, the related subject selection unit 203 increments the value of the counter N, and selects a face detection result for any face from the plurality of read face detection results.

  In step S604, the related subject selection unit 203 determines whether the face detection result of the face selected in step S603 is the face selected by the main subject selection unit 202. If the face being referred to is not the main subject, the process proceeds to step S605. If the face being referred to is the main subject, the process proceeds to step S609.

  In step S605, the related subject selection unit 203 calculates a correlation value R between the face detection information of the main subject detected by the face detection unit 201 and the face detection information of the referenced face. The correlation value R is defined by, for example, the larger value of the ratio A / B or B / A between the face size A of the main subject and the reference face size B. The correlation value R may be defined by the absolute value of the difference A−B between the main subject and the referenced face size. In either case, the correlation value R becomes smaller as the face size A and face size B are closer. As an example of the face size mentioned here, there are the length or width of the face, or the number of pixels occupied by the face.

  In step S606, the related subject selection unit 203 compares the correlation value R with a preset threshold value TH. If the correlation value R is smaller than the threshold value TH as a result of the comparison, it is determined that the referenced face is a size close to the face of the main subject, and the process proceeds to step S607. If the correlation value R is greater than the threshold value TH, the process proceeds to step S609. The threshold value TH may be determined experimentally.

  In step S <b> 607, the related subject selection unit 203 determines that the referenced face is set in advance based on the position information (X, Y) of the face determined to be close in size to the face of the main subject. It is determined whether or not it exists inside the predetermined area D. Here, the predetermined area D set in advance can be arbitrarily set, for example, a vertical and horizontal 80% area in the shooting field and an area ratio of 64%. If it is determined that the object exists within the predetermined area D, the process proceeds to step S608 and is selected as a related subject. If it is determined in step S607 that the face being referred to does not exist within the predetermined area D, the process proceeds to step S609.

  In step S <b> 609, the related subject selection unit 203 compares the value of the counter N with the number of face detections previously written in the memory 106. If the value of the counter N has not reached the face detection number, the process returns to step S603, the face detection result of another face is selected, and the same processing is repeated. When the value of the counter N reaches the number of face detections, the related subject selection process has been executed for all faces whose faces have been detected, so the related subject selection process ends.

  As described above, in the related subject selection algorithm shown in FIG. 6A, the related subject selection unit 203 is close in size to the face of the main subject among the faces other than the main subject detected by the face detection unit 201, and A face located near the center of the object scene is selected as the related subject.

  Next, another example of the related subject selection algorithm will be described with reference to FIG. Since the related subject selection algorithm shown in FIG. 6B is obtained by replacing the process in step S607 in FIG. 6A with the processes in steps S621 and S622, this process will be mainly described.

  In step S606 in FIG. 6B, when the correlation value R between the face detection information of the main subject and the face detection information of the referenced face is smaller than the threshold value TH, the referenced face is the main subject. It is determined that the size is close to the face, and the process proceeds to step S621.

  In step S621, the related subject selection unit 203 sets a second threshold value according to the face size of the main subject. In step S622, the related subject selection unit 203 sets the distance between the center position of the face determined to be close to the face of the main subject and the center position of the face of the main subject set in step S621. It is determined whether it is within 2 thresholds. If this distance is within the second threshold, the process proceeds to step S608. In step S608, the related subject selection unit 203 selects the referenced face as a related subject. As described above, in the related subject selection algorithm shown in FIG. 6B, the related subject selection unit 203 has a size close to that of the main subject among the faces other than the main subject detected by the face detection unit 201, and is the main subject. A face close to the subject's face is selected as the related subject. That is, unlike FIG. 6A, even if the face being referred to is located at the end of the object scene, it may be selected as a related subject if the distance from the face of the main subject is short.

  The second threshold value set in step S621 is set so as to increase as the face of the main subject increases. This is because even if the distance between the two persons is the same, the distance between the two persons in the image data varies depending on how far the persons are from the imaging device. However, the larger the face is, the larger the second threshold may not be increased without limitation, and an upper limit may be provided for the second threshold. This is likely to be a shot intended for a portrait-like composition when the face becomes large to some extent. In such a case, if the subject is a related subject, the distance is expected to be rather close. Because.

  Here, in both the examples of FIGS. 6A and 6B, the face size is used as a criterion, but the present invention is not limited to this. The related subject may be selected only by the distance between the face of the main subject and the reference face. It is also possible to measure the time that each face can be detected continuously and select a face that has the same detection time as the face of the main subject as the related subject. A movement locus may be accumulated for each face, and a face whose detected movement locus is similar to the face of the main subject may be selected as the related subject. Alternatively, a subject highly relevant to the subject determined to be the main subject by the main subject selection unit 202 may be selected using personal authentication. For example, by storing personal priorities in the memory in advance, authentication processing is performed on the detected face, and only persons with priorities close to those of the person selected as the main subject are related. It is also possible to select it as a subject.

[Blink detection algorithm]
Next, the above-described eyebrows detection algorithm will be described in detail with reference to FIG.

  The eyelid detection unit 204 executes each process described below under the control of the system control unit 107. At this time, the face detection information stored in the memory 106 in the face detection process is used.

  In step S701, the eyelid detection unit 204 reads an image of an area estimated to be an eye from the face area obtained by the face detection process.

  In step S <b> 702, the eyelid detection unit 204 reads eye image template data from the memory 106.

  In step S703, the eyelid detection unit 204 compares the image of the eye area obtained by the face detection process with the template data to determine whether or not the degree of eye opening is sufficient. If it is determined that the degree of opening is sufficient, the process proceeds to step S705, and the display to the effect that the eye-opening is detected is not performed.

  If it is determined in step S704 that the degree of opening of the eyes is not sufficient, the process proceeds to step S706, and the display unit 110 is notified of a target for performing a blind eye warning. Upon receiving this notification, the display unit 110 superimposes and displays on the image data that the subject is blinded when performing a REC review display in step 312.

  In addition to the method of combining characters and icons on the screen and displaying them as a blink warning, it is also possible to blink or light up voices, LEDs, or the like.

  As described above, according to the present embodiment, a subject having a high correlation with the selected main subject is selected as a related subject, and blinding detection is performed only on the selected main subject and the related subject. By doing so, it is possible to improve the probability of performing eye-blink detection only on a subject that the user intends to shoot.

  In particular, in digital cameras, there is often a gap between the timing for selecting the main subject as the target of AF processing and the timing for actually detecting eye-blinks. A situation that appears in the scene is also conceivable. Suppose that the target for detection of eye-browsing is not an object having a high correlation with the main object selected in advance, but an object having a large weight obtained from the face size or position. In this case, if a person appears near the camera immediately before shooting, the probability that the face will be selected as a target for detection of a blindness increases.

  On the other hand, in the present embodiment, a subject having a high correlation is selected as a target for eye detection. Therefore, even if an unintended person appears in the scene between the first operation for selecting the main subject (half-pressing the release button) and the second operation for selecting the related subject (full-pressing the release button). Thus, it is possible to reduce the probability that the person is selected as a target for detection of eyebrows.

  Furthermore, when the user selects an arbitrary subject as the main subject with the direction button, the main subject is not necessarily larger than the other subjects, or is located at the center of the screen. Is not limited. Under such circumstances, if a subject with a large weight obtained simply from the face size, position, etc. is selected as the target for eye-blink detection, there is a possibility that the user can perform eye-gaze detection on the subject intended by the user. Is low. On the other hand, in the present embodiment, even if the user changes an arbitrary subject to the main subject using the direction instruction button 113, which is an extra subject based on the main subject newly selected by the user. Determine whether. For this reason, it is possible to improve the probability that the eyebrows detection can be performed only on the subject that the user intends to shoot.

  Further, in this embodiment, after selecting the related subject in step S310, the detection of the eyebrows is detected for the main subject and the related subject in step S311. However, the order of this processing may be reversed. Absent. In other words, eye detection is performed on all the faces detected in step S309, and faces corresponding to the main subject and related subjects are selected from the faces determined to be eyebrows. This is indicated in the REC review. By doing so, the processing time for eye-opening is not shortened, but warning display for eye-opening is not performed for an extra subject, so that it can be prevented that the display becomes troublesome for the user.

(Embodiment 2)
This embodiment will be described with respect to an embodiment in which a main subject and a related subject are selected, personal authentication is performed on the selected face, and personal authentication is not performed on other faces. In addition to the first embodiment, the configuration of the image pickup apparatus according to the present embodiment includes the memory information including calendar information and date. It is assumed that the image processing unit 104 of the digital camera according to the present embodiment has a personal authentication unit instead of the eyelid detection unit 204.

[Overall operation of digital camera]
FIG. 8 is a flowchart showing the overall operation of the digital camera according to the second embodiment. In addition, the step which performs the same process as the process demonstrated in Embodiment 1 attaches | subjects the same code | symbol as FIG.

  Steps S301 to S310 and steps S312 to S316 are the same as the processing of each step shown in FIG.

  In step S801, the image processing unit 104 performs personal authentication by the method described above for the main subject and the face selected as the related subject, and does not perform personal authentication for other faces. Proceed to

  In step S <b> 802, the image processing unit 104 records a shooting history in the memory 106 for a person who has been personally authenticated. The image processing unit 104 stores the personal authentication result in the image data captured on the same day in the memory 106 according to a calendar installed in the digital camera. That is, on the same day, every time an individual is authenticated from the captured image data, a counter indicating the number of times the individual is captured is increased. Then, in the next step S312, the system control unit 107 causes the display unit 110 to display a number indicating which person was photographed and how many times on the day together with the photographed image. By comprising in this way, it becomes possible to help a user image | photograph a friend equally. Of course, the counting period can be set arbitrarily by the user, and the counting can be continued only for the period specified by the user, or the counting period can be set with the power on and off as a trigger. It is.

  With the above configuration, the related subject is selected based on the correlation with the main subject, and personal authentication is performed only for the main subject and the related subject and the selected subject, thereby performing personal authentication without distinguishing the subject. Compared to the case, the time required for personal authentication can be shortened. According to the present embodiment, since personal authentication is performed only on the main subject and the subject selected as the related subject, a person who has been photographed with a composition that is not conspicuous is less likely to be counted. . For this reason, a person who happens to enter the scene is not counted even if the person can be personally authenticated, and the number of shots can be counted by focusing on an image having a composition suitable for that person. It becomes possible.

(Embodiment 3)
In this embodiment, a main subject and a related subject are selected, and red-eye correction is performed on the selected main subject and related subject. The configuration of the imaging apparatus according to the present embodiment is the same as that of the first embodiment. Note that the image processing unit 104 of the digital camera according to the present embodiment includes a red-eye detection unit and a red-eye correction unit in place of the eyebrows detection unit 204.

[Overall operation of digital camera]
FIG. 9 is a flowchart illustrating the operation of the entire imaging apparatus according to the third embodiment. Parts that perform the same processing as in FIG. 3 of the first embodiment are denoted by the same reference numerals.

  After selecting the main subject and the related subject in steps S304 to S310, in step S901, the system control unit 107 performs red-eye detection to determine whether the main subject and the related subject have red eyes. The red-eye detection method first checks the hue of the eye area detected by the face detection process, detects the average hue of the eye area, and determines that red-eye has occurred when the red component is equal to or greater than a set threshold.

  If red eye is detected, the process proceeds to step S902, and red eye correction is performed. Red-eye correction is correction that lowers the saturation of a region where red-eye is detected.

  With such a configuration, red-eye correction is not performed except for a subject that is estimated to be taken by the user, so that it is possible to provide an imaging device that has a shorter processing time and is easier to use. It was.

  As in the first embodiment, the order of the related subject selection process and the red-eye detection process may be reversed. That is, red-eye detection is performed on all the faces detected in step S309, and faces corresponding to the main subject and related subjects are selected from the faces determined to be red-eye, and the REC review in step S312 is performed. A message to that effect is displayed. By doing so, since the display of the red-eye detection result is not performed for an extra subject, it can be prevented that the display becomes annoying for the user.

  Note that the image analysis performed on the main subject and related subjects by the present invention is not limited to eyebrows detection, red-eye correction, and personal authentication. Any device capable of performing detection processing and correction processing may be used.

  In the first, second, and third embodiments, a digital camera has been described as an example of the image processing apparatus. However, the present invention is not limited to this. It can also be applied to an image processing apparatus comprising a system in which image data is communicated between a camera and an information processing apparatus, and image data taken by the camera is analyzed on the information processing apparatus side. is there. In addition, if the purpose is to perform eyelid detection, red-eye correction, or personal authentication only for a required person, the information processing apparatus analyzes the image data received via the network. It is also possible to perform.

  That is, processing equivalent to any of the first, second, and third embodiments can be realized by a program. In this case, each of the components including FIG. 2 may function in a subroutine executed by the CPU in the information processing apparatus.

DESCRIPTION OF SYMBOLS 201 Face detection part 202 Main subject selection part 203 Related subject selection part 204 Eyelid detection part

Claims (9)

Face detection means for detecting a face from image data;
Main subject selection means for selecting any face as a main subject when a plurality of faces are detected from the face detection means;
Related subject selection means for selecting a face as a related subject based on the correlation with the face selected as the main subject from the faces not selected as the main subject by the main subject selection means;
Image processing means for performing predetermined image processing on the face detected by the face detection means ;
Having an operation part ,
The main subject selection means selects a face as the main subject in response to a first operation on the operation unit, and the related subject selection means selects a second in the operation unit after the first operation. Select the face as the related subject according to the operation of
The image processing means performs the predetermined image processing on the face selected by the main subject selection means and the related subject selection means, and performs the predetermined image processing on the other faces. There is no image processing apparatus. Face detection means for detecting a face from image data;
Main subject selection means for selecting any face as a main subject when a plurality of faces are detected from the face detection means;
Related subject selection means for selecting a face as a related subject based on the correlation with the face selected as the main subject from the faces not selected as the main subject by the main subject selection means;
Image processing means for performing predetermined image processing on the face detected by the face detection means ;
Having an operation part ,
The main subject selection means selects a face as the main subject in response to a first operation on the operation unit, and the related subject selection means selects a second in the operation unit after the first operation. Select the face as the related subject according to the operation of
The image processing means displays the result of the predetermined image processing for the face selected by the main subject selecting means and the related subject selecting means, and for the other faces, the predetermined image processing result is displayed. An image processing apparatus characterized by not displaying an image processing result. The related subject selection unit compares at least one of the position, size, detected time, and position movement trajectory of the face selected by the main subject selection unit and the unselected face, the image processing apparatus according to claim 1 or 2, characterized in that selecting the relevant object. Wherein the predetermined image processing eye head detection, red-eye correction, and image processing apparatus according to claims 1 to 3 any one, characterized in that among the personal identification is at least one. The said image processing apparatus has an image pick-up element which converts an optical image into an electric signal, and an optical system which adjusts the position of the focus of the said optical image, The any one of Claim 1 thru | or 4 characterized by the above-mentioned. Image processing apparatus. In response to the first operation in the operation unit, the main subject selection unit selects a face as the main subject, and the optical system adjusts a focus position to the face as the main subject, and the operation unit The image processing apparatus according to claim 5 , wherein the image data generated by the image sensor is recorded on a recording medium in accordance with the second operation. A face detection step of detecting a face from image data;
A main subject selection step of selecting one of the faces as a main subject when a plurality of faces are detected from the face detection step;
A related subject selection step of selecting a face as a related subject based on the correlation with the face selected as the main subject from the faces not selected as the main subject in the main subject selection step;
An image processing step for performing predetermined image processing on the face detected by the face detection step,
In the main subject selection step, a face as the main subject is selected according to a first operation in the operation unit,
In the related subject selection step, a face as the related subject is selected according to a second operation in the operation unit after the first operation,
The image processing step performs the predetermined image processing on the face selected by the main subject selection step and the related subject selection step, and performs the predetermined image processing on the other faces. There is no image processing method characterized by the above. A face detection step of detecting a face from image data;
A main subject selection step of selecting one of the faces as a main subject when a plurality of faces are detected from the face detection step;
A related subject selection step of selecting a face as a related subject based on the correlation with the face selected as the main subject from the faces not selected as the main subject in the main subject selection step;
An image processing step for performing predetermined image processing on the face detected by the face detection step,
In the main subject selection step, a face as the main subject is selected according to a first operation in the operation unit,
In the related subject selection step, a face as the related subject is selected according to a second operation in the operation unit after the first operation,
The image processing step displays the result of the predetermined image processing for the face selected by the main subject selection step and the related subject selection step, and the other predetermined face for the predetermined subject. An image processing method characterized by not displaying an image processing result. A program for causing an information processing apparatus to execute the method according to claim 7 or 8 .