JP2008181439A - Face detection device and method, and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To detect an edge face, that is, the state in which a part of the face runs off an input image. <P>SOLUTION: A face detection device is provided with an edge face detection processing part for detecting the edge face in which a part of the face is on the edge of the input image by adding a dummy image 201 around the input image 200 and performing face detection beyond the input image edge. The edge face detection processing part calculates similarity between the image and a reference face image from the image in a judgement area defined in a synthesized image and prescribed dictionary data, and judges whether or not there is an edge face in the judgement area by comparing the similarity with a judgement threshold. For each pixel value of the dummy image, a pixel value which does not influence the edge face detection judgement is set. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a face detection device and a face detection method for detecting a face in an image, and an imaging device using them.

  One of the failures at the time of shooting a still image or a moving image using an imaging device is subject cut. “Out of subject” means that the subject focused on by the photographer protrudes from the photographing region. In particular, when the subject is a person, the subject is recognized as a serious failure by the user.

  In many cases, a person's face is included as a subject at the time of shooting, and it is unlikely that the photographer intentionally protrudes the face from the shooting area when shooting a person. If it is possible to detect a face where the subject is cut off, it is possible to obtain a more desirable image by adjusting the composition using the detection result. For this reason as well, it is important to detect the face (edge face) that causes the subject to be cut.

  Conventionally, there are various face detection methods such as a method using face matching by pattern recognition, a method using color extraction based on skin color, and a method through learning processing that learns facial features from a large amount of data. A method has been proposed. However, these conventional face detection methods are based on the premise that the entire face is shown. That is, the conventional face detection method performs face detection processing on a complete face existing in an image, and thus cannot detect a face that partially protrudes from the image.

  In view of this, Patent Document 1 below discloses a method for detecting whether or not a subject's face protrudes from a frame (image). In this method, the detection of a subject cut of a face is performed through extraction of a skin color region. However, as is generally known, an algorithm using skin color extraction is greatly affected by a difference in skin color and a photographing condition (such as ambient light) according to race. Japanese Patent Application Laid-Open No. 2004-228561 also discloses that the detection of a face subject being cut off is performed through detection of feature points corresponding to eyes, nose, mouth, and the like. However, it is a technique on the premise that eyes, nose, and mouth, which are face parts, are detected independently, and as is generally known, face part detection has relatively many false detections. As described above, the method of Patent Document 1 has many problems.

  Patent Document 2 below discloses a method for detecting a subject cut of a face using a difference between frames. However, this is a technique for moving images and cannot be applied to still images. In addition, since the change from the state in which the face is reflected is detected using the inter-frame difference and the subject cut is detected by this, it is not possible to cope with an operation in which the face enters from outside the shooting area.

  In addition to these, various methods relating to face detection have been proposed (Patent Documents 3 to 5 below), but there is no method for detecting a subject cut of a face well.

JP 2005-65265 A JP 2003-323621 A JP 2006-101186 A JP 2005-217768 A JP 2005-117316 A

  As described above, the conventional face detection method is based on the premise that a complete face exists in the input image for the part where face detection is performed, and in this state, the face (end face) in which the subject is cut off is detected. I can't. In addition, when detecting a face where the subject has run out, if the detection conditions (conditions such as which range in the image is the detection range) are appropriately set, the processing time can be shortened. For this reason, it is also important how the above detection conditions are set.

  Accordingly, an object of the present invention is to provide a face detection device, a face detection method, and an imaging device that can perform good end face detection. Another object of the present invention is to provide a face detection device and an imaging device that contribute to shortening the processing time for edge detection.

  In order to achieve the above object, a first face detection apparatus according to the present invention is a face detection apparatus that performs face detection by adding a dummy image around an input image and performing face detection beyond the edge of the input image. The apparatus further comprises an end face detection means for detecting an end face where a part of the face is applied to the end of the input image.

  This makes it possible to realize good end face detection.

  Specifically, for example, the edge detection unit defines a determination area in a composite image of the input image and the dummy image, and calculates similarity between the image in the determination area and predetermined dictionary data. A degree calculation unit and a determination threshold setting unit for setting a determination threshold, and comparing the similarity with the determination threshold to determine whether the end face is present in the determination region To do.

  For example, the determination threshold value setting unit changes the determination threshold value according to a contribution component to the similarity of the overlapping area between the determination area and the dummy image.

  Thereby, it is possible to eliminate or suppress the influence of the pixel value of the image in the overlapping area on the edge detection.

  Instead of this, for example, the end face detection means sets a pixel value such that a contribution component to the similarity of the overlapping region between the determination region and the dummy image is zero, and the pixel of the image in the overlapping region. It may be set as a value.

  Further, for example, a face detection unit that detects a face present in the input image is further provided, and the end face detection unit changes the detection condition of the end face according to a face detection result of the face detection unit.

  Thereby, for example, it is possible to provide upper and lower limits to the end face detection size in accordance with the face size detected by the face detection means. If the upper and lower limits are provided, it is possible to expect a reduction in processing time for end face detection and an effect of suppressing end face misdetection or overdetection.

  Specifically, for example, the end face detection unit changes the end face detection condition based on at least one of the size, position, and orientation of the face detected by the face detection unit.

  Further, for example, the face detection device is mounted on an imaging device having an imaging unit, the input image is generated from a captured image of the imaging unit, and the end face detection unit is configured to capture the captured image at the time of capturing the captured image. The edge detection condition is changed according to the inclination of the apparatus.

  Thereby, for example, it is possible to limit the detection range of the face according to the inclination of the imaging device at the time of shooting. If the end face detection range is limited, it is possible to expect a reduction in processing time for end face detection and an effect of suppressing end face misdetection or overdetection.

  In order to achieve the above object, a second face detection device according to the present invention is a face detection device having a face detection means for detecting a face existing in an input image, wherein a part of the face is applied to an end of the input image. An end face detection unit for detecting an end face is further provided, and the end face detection unit changes the detection condition of the end face according to a face detection result of the face detection unit.

  Thereby, for example, it is possible to provide upper and lower limits to the end face detection size in accordance with the face size detected by the face detection means. If the upper and lower limits are provided, it is possible to expect a reduction in processing time for end face detection and an effect of suppressing end face misdetection or overdetection.

  In order to achieve the above object, a third face detection device according to the present invention is a face detection device mounted on an image pickup device having an image pickup means, receives an input image generated from a photographed image of the image pickup means, An end face detection unit that detects an end face partly applied to the end of the input image, and the end face detection unit detects the end face according to an inclination of the imaging device at the time of shooting the captured image. To change.

  Thereby, for example, it is possible to limit the detection range of the face according to the inclination of the imaging device at the time of shooting. If the end face detection range is limited, it is possible to expect a reduction in processing time for end face detection and an effect of suppressing end face misdetection or overdetection.

  In order to achieve the above object, an imaging apparatus according to the present invention is an imaging apparatus including an imaging unit and the face detection apparatus according to any one of the above, wherein the input image to the face detection apparatus is It is generated based on the captured image of the imaging means.

  In order to achieve the above object, a face detection method according to the present invention is a face detection method for performing face detection by adding a dummy image around the input image and performing face detection beyond the edge of the input image. It is characterized in that an end face partially covered with the input image edge is detected.

  According to the present invention, it is possible to provide a face detection device, a face detection method, and an imaging device that can perform good end face detection. Further, it is possible to provide a face detection device and an imaging device that contribute to shortening the processing time for edge detection.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle.

<< First Embodiment >>
First, a first embodiment of the present invention will be described. FIG. 1 is an overall block diagram of an imaging apparatus 1 according to the first embodiment of the present invention. The imaging apparatus 1 in FIG. 1 is a digital still camera that can capture and record a still image, or a digital video camera that can capture and record a still image and a moving image.

  The imaging device 1 includes an imaging unit 11, an AFE (Analog Front End) 12, a main control unit 13, an internal memory 14, a display unit 15, a recording medium 16, an operation unit 17, an inclination sensor 18, And a face detection unit 19. The operation unit 17 is provided with a shutter button 17a.

  The imaging unit 11 includes an optical system, an aperture, an imaging device such as a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and a driver (all not shown) for controlling the optical system and the aperture. And have. The driver controls the zoom magnification and focal length of the optical system and the aperture of the diaphragm based on the AF / AE control signal from the main control unit 13. The image sensor photoelectrically converts an optical image representing a subject incident through the optical system and the diaphragm, and outputs an electrical signal obtained by the photoelectric conversion to the AFE 12.

  The AFE 12 amplifies the analog signal output from the imaging unit 11 (imaging device), and converts the amplified analog signal into a digital signal. The AFE 12 sequentially outputs this digital signal to the main control unit 13.

  The main control unit 13 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and also functions as a video signal processing unit. Based on the output signal of the AFE 12, the main control unit 13 generates a video signal representing an image captured by the imaging unit 11 (hereinafter also referred to as “captured image”). The main control unit 13 also has a function as display control means for controlling the display content of the display unit 15, and performs control necessary for display on the display unit 15.

  The internal memory 14 is formed by SDRAM (Synchronous Dynamic Random Access Memory) or the like, and temporarily stores various data generated in the imaging device 1. The display unit 15 is a display device including a liquid crystal display panel and the like, and displays an image taken in the immediately previous frame, an image recorded on the recording medium 16, and the like under the control of the main control unit 13. The recording medium 16 is a non-volatile memory such as an SD (Secure Digital) memory card, and stores captured images and the like under the control of the main control unit 13.

  The operation unit 17 receives an operation from the outside. The content of the operation on the operation unit 17 is transmitted to the main control unit 13. The shutter button 17a is a button for instructing to capture and record a still image.

  The tilt sensor 18 detects the tilt of the imaging device 1 with respect to the vertical line. Now, a state in which the horizontal line on the photographed image when the horizontal line is photographed matches the horizontal direction (horizontal line) on the photographed image is defined as a horizontal photographing state. It is assumed that the imaging device 1 is not tilted in this horizontal shooting state (that is, the tilt is 0 degree), and the tilt sensor 18 detects the tilt of the imaging device 1 from this horizontal shooting state. When the image pickup apparatus 1 in the horizontal shooting state is tilted by 90 degrees, the horizontal line on the shot image when the horizontal line is shot matches the vertical direction (vertical line) on the shot image, and this state is called the vertical shooting state. . Tilt data indicating whether the current state of the imaging apparatus 1 is the horizontal photographing state or the vertical photographing state is transmitted to the main control unit 13.

  The operation mode of the imaging device 1 includes a shooting mode capable of shooting and recording a still image or a moving image, and a reproduction mode for reproducing and displaying the still image or moving image recorded on the recording medium 16 on the display unit 15. . Transition between the modes is performed according to the operation on the operation unit 17. In the shooting mode, the imaging unit 11 sequentially performs shooting at a predetermined frame period (for example, 1/60 seconds), and a captured image obtained in each frame is given to the face detection unit 19. When the face detection unit 19 detects a face, the captured image can be reduced. Therefore, in order to clearly distinguish the reduced image from the captured image itself, the captured image that has not been reduced is hereinafter referred to as an “original image”.

  FIG. 2 shows an internal block diagram of the face detection unit 19. 2 includes an image input unit 30, an all-face detection processing unit 31, an end face detection processing unit 32, a face detection result integration unit 33, and a face dictionary memory 34.

  The image input unit 30 gives an image based on the original image to the all face detection processing unit 31 and the end face detection processing unit 32 as an input image. Hereinafter, unless otherwise specified, the input image refers to an input image for the entire face detection processing unit 31 and the end face detection processing unit 32 (or 32a or 32b described later). This input image is an original image or a reduced image of the original image. The reduced image of the original image is generated by the image input unit 30 (or the main control unit 13 in FIG. 1).

  The all face detection processing unit 31 detects a face existing in the input image. With reference to FIG. 3, a method of face detection by the all-face detection processing unit 31 will be described.

  In FIG. 3A, reference numeral 100 represents an input image, and reference numeral 110 represents a determination area for face detection. When performing face detection, a state in which the determination area is arranged at the upper left corner of the input image is set as an initial state, and the determination area is scanned horizontally from left to right one pixel at a time in the input image. When the determination area reaches the right end of the input image, the determination area is shifted downward by one pixel, and scanning in the horizontal direction is performed again. In this manner, it is sequentially detected whether a face exists in the determination area while scanning the determination area in the horizontal direction and the vertical direction.

  The size of the input image is appropriately reduced so that faces of different sizes can be detected in one type of determination area. Specifically, the reduction ratio when generating an input image by reduction from the original image is changed in stages, and the original image and the reduced image generated at each reduction ratio are output from the image input unit 30 as input images. Is done. Reference numerals 101 and 102 in FIGS. 3B and 3C are input images obtained by reduction, and when the input image 100 in FIG. 3A is an original image, FIGS. Input images 101 and 102 shown in (b) correspond to reduced images of the original image.

  The size of the determination area 110 is the same for any input image (100, 101, and 102). Specifically, for example, the size of the determination area 110 is set to 24 × 24 pixels. However, for simplification of explanation and illustration, it is assumed that the size of the determination region 110 is 8 × 8 pixels.

  Although the determination area is scanned as described above, a method for determining whether or not a face exists in the determination area by paying attention to a state where the determination area exists at a specific position in a certain input image Will be described. FIG. 4 is a flowchart showing the operation procedure of this method. Each process of steps S1 to S5 in FIG.

  In step S1, four first edge-enhanced images (four-direction first edge-enhanced images) are generated by applying edge enhancement processing to each of the four types of differential filters on the image in the determination region. To do. As four types of differential filters, for example, four-direction Prewitt type differential filters (edges) corresponding to the horizontal direction, the vertical direction, the upper right diagonal direction, and the upper left diagonal direction as shown in FIGS. Detection operator). Each first edge-enhanced image obtained by applying these is referred to as a first edge-enhanced image in the horizontal direction, the vertical direction, the upper right oblique direction, and the upper left oblique direction. In addition, they are collectively referred to as a first edge enhanced image in four directions.

  FIG. 6 shows an example of each first edge enhanced image. Reference numerals 121, 122, 123, and 124 represent first edge emphasized images in the horizontal direction, the vertical direction, the right diagonally upward direction, and the diagonally left upward direction, respectively. In FIG. A part of the pixel value of each pixel to be formed is shown.

  Next, in step S2, a pixel having the maximum pixel value is specified for each corresponding pixel between the four edge-enhanced images in four directions, leaving only the maximum pixel value as it is, and setting the pixel values other than the maximum to zero. Thus, a second edge enhanced image in four directions is generated. In FIG. 6, reference numerals 131, 132, 133, and 134 represent second edge enhanced images in the horizontal direction, the vertical direction, the upper right diagonal direction, and the upper left diagonal direction, respectively. A part of pixel values of each pixel forming the edge enhanced image is shown.

  For example, when the pixel values of the first edge enhanced images 121, 122, 123, and 124 for a specific pixel position in the first edge enhanced image are 10, 4, 6, and 3, respectively, the second edge enhanced image 131 , 132, 133 and 134 at the specific pixel positions are 10, 0, 0 and 0, respectively (see FIG. 6).

  Next, in step S3, a smoothing process is performed on each of the four-direction second edge enhanced images, and the four-direction second edge enhanced image after the smoothing process is used as a four-direction feature image. FIGS. 7A to 7D show the generated feature images in the four directions, that is, feature images in the horizontal direction, the vertical direction, the right diagonally upward direction, and the diagonally left upward direction. Note that the four-direction second edge enhanced image itself may be used as the four-direction feature image without performing the smoothing process.

  In each of the four-direction feature images, the pixel position is specified by coordinates (x, y) in the feature image. x and y represent the horizontal coordinate and the vertical coordinate in each feature image. In the present example, x and y each take an integer of 0 or more and 7 or less. Further, the type of feature image is expressed by q. q is an integer of 0 or more and 3 or less, and the feature images of q = 0, 1, 2, and 3 mean feature images in the horizontal direction, the vertical direction, the right diagonally upward direction, and the left diagonally upward direction, respectively. . A pixel in each feature image is referred to as a feature pixel.

  The face dictionary memory 34 in FIG. 2 stores a weight table adapted to the size of the feature image (in other words, a weight table adapted to the size of the determination area). The weight table is created in advance based on a large amount of face images as teacher samples and stored in the face dictionary memory 34. FIG. 8 shows an example of the contents of the weight table when the size of the determination area is 8 × 8 pixels. The weight table stores the weight corresponding to the pixel value of the feature pixel for each feature pixel. The weight is a value representing the likelihood of a face, and in the present embodiment, this weight is referred to as a score.

  Further, variables q, x and y are considered as one variable n. From 4 × 8 × 8 = 256, the variable n is assumed to be an integer of 0 or more and 255 or less. The value of the variable n is uniquely determined if the variables q, x, and y are determined. Each feature pixel in the four-direction feature image (the total number of feature pixels is 256) is represented by F (n) using a variable n, and the pixel value in the feature pixel F (n) (corresponding to the horizontal direction in FIG. 8) Is represented by i (n). Since the score is also specified depending on the variable n, the score is represented by w (n). For example, assuming that q = x = y = 0 corresponds to n = 0, the pixel value at the pixel position (0, 0) of the horizontal feature image (corresponding to q = 0) is the feature pixel F (0 ) Matches the pixel value i (0). The score (0) is specified by the pixel value i (0) based on the weight table. When the pixel value i (0) is 3, the score (0) is 35 in the example of FIG.

  Such a weight table can be created using, for example, a known learning method called Adaboost (Yoav Freund, Robert E. Schapire, “A decision-theoretic generalization of on-line learning and an application to boosting” ", European Conference on Computational Learning Theory, September 20, 1995.).

  Adaboost is an adaptive boosting learning method. Based on a large number of teacher samples, Adaboost selects multiple weak classifiers that are effective for identification from among a plurality of weak classifier candidates. This is a learning method for realizing a highly accurate classifier by weighting and integrating. Here, a weak classifier refers to a classifier that has a higher discrimination ability than a coincidence but is not high enough to satisfy sufficient accuracy. When a weak classifier is selected, if there is a weak classifier already selected, learning is focused on a teacher sample that is erroneously recognized by the selected weak classifier. Thereby, the most effective weak classifier is selected from the remaining weak classifier candidates.

  Also, the weight table can be expressed by a function, and this function may be stored in the face dictionary memory 34 instead of the weight table. In this function, the score is determined depending on the pixel value of the feature pixel and the variable n. This function is expressed by a continuous function. In a three-dimensional space with the XYZ axes as coordinate axes, the X axis is taken as the pixel value of the feature pixel, the Y axis is taken as the variable n, and the Z axis is taken as the score. A simple curved surface. This function and weight table can be referred to as face dictionary data for deriving a similarity described later from an image in the determination area.

In step S4 following the process of step S3 in FIG. 4, the similarity S ₁ is calculated according to the following equation (1). Pixel value i (n) is the image of the determination area is a face image (i.e., image representing the face) can be referred to as feature amounts effective to identify whether the similarity S ₁ is focused This corresponds to the sum of the scores w (n) calculated from the pixel values i (n) based on the determination area. Since the weight table is obtained by converting the reference face image obtained through the learning process into a data, the sum of the scores w (n) is the sum of the reference face image represented by the weight table and the image in the determination area. It represents the degree of similarity.

Then, in step S5 in FIG. 4, by comparing the similarity S ₁ and predetermined face determination threshold value TH, when the following formula (2) is satisfied, the image of the determination area is a face image (in other words When the following expression (2) is not satisfied, the image in the determination area is not a face image (in other words, no face exists in the determination area) ).
S ₁ > TH (2)

The weights are set so that the similarity S ₁ takes a relatively large value when the image in the determination area is a face image and the similarity S ₁ takes a relatively small value when the image in the determination area is not a face image. The table is set through a learning process. For this reason, face detection is possible by the above-described processing.

  Next, the end face detection processing unit 32 in FIG. 2 will be described. The end face detection processing unit 32 detects a face by adding a dummy image around the input image to generate a composite image, and performing face detection in the composite image. An end face means a face in which a part of the face covers the end of the input image. That is, as shown in FIG. 9, the end face means a face that exists at the end of the input image 200 and a part of the face protrudes outside the input image 200.

  FIG. 10 shows the relationship between the input image, the dummy image, and the composite image. In FIG. 9, an area within a square frame denoted by reference numeral 200 represents an input image, and a donut-shaped broken line area denoted by reference numeral 201 represents a dummy image. The dummy image is provided so as to surround four sides of the input image, and the combination of the input image and the dummy image corresponds to the above-described composite image.

  In FIG. 10, a square with reference numeral 210 represents a determination area. Similar to that in the all-face detection processing unit 31, the determination area is scanned in the horizontal direction and the vertical direction in the composite image, and it is detected whether or not an end face exists in the determination area.

[Dummy image sizing method]
First, a method for determining the size of a dummy image will be described with reference to FIG. The width (ie, horizontal size) and height (ie, vertical size) of the input image are W ₁ and H _{1, respectively,} and the width and height of the composite image are W ₂ and H ₂ , respectively. Here, W ₁ ≦ W ₂ and H ₁ <H ₂ , or W ₁ <W ₂ and H ₁ ≦ H ₂ . FIG. 10 exemplifies the case where W ₁ <W ₂ and H ₁ <H ₂ as a typical example, and the following description will be given by taking this case as an example. Now, an image addition width when generating a composite image from an input image is represented by W_dum. Therefore, W_dum = W ₂ −W ₁ . Further, the width of the determination area (that is, the horizontal size) is represented by W_dic.

  For example, as shown in FIG. 10, the dummy image is set so that the center of the input image matches the center of the composite image. In this case, an image having a width of W_dum / 2 is provided as a part of the dummy image for each of the left end and the right end of the input image. In the example shown in FIG. 10, the image addition width W_dum is equally allocated to the left and right, but the width of the image added to the left and right of the input image may be set separately.

  The image additional width W_dum is set using any one of the following first to third additional width setting methods.

In the first additional width setting method, the image additional width W_dum is set based on the determination area width W_dic. More specifically, W_dum is determined based on the following formula (3a). Here, k ₁ is an arbitrary value that satisfies the inequality (3b). k ₁ may be a fixed value or a value determined according to the width W ₁ and / or the width W_dic.
W_dum = k ₁ · W_dic (3a)
0 <k ₁ <1 (3b)

  In the second additional width setting method, the image additional width W_dum is set based on the size of the face already detected by the all face detection processing unit 31. Here, “the size of the face already detected by the all face detection processing unit 31” is the size of the face on the same input image given to the all face detection processing unit 31 and the end face detection processing unit 32. The width (size in the width direction) of the face is represented by W_fd. The face width W_fd on the input image detected by the all-face detection processing unit 31 is equal to the width of the determination area used by the all-face detection processing unit 31, for example.

More specifically, W_dum is determined based on the following formula (4a). Here, k ₂ is an arbitrary value that satisfies the inequality (4b). k ₂ may be a fixed value or a value determined according to the width W ₁ and / or the width W_fd.
W_dum = k ₂ · W_fd (4a)
0 <k ₂ · W_fd <W_dic (4b)

In the third additional width setting method, the image additional width W_dum is set based on both the width W_dic and the width W_fd. More specifically, W_dum is determined based on the following formula (5a). Here, k _3A and k _3B are arbitrary values satisfying the inequality (5b). k _3A and k _3B may be fixed values or values determined according to the width W ₁ , the width W_dic, and / or the width W_fd.
W_dum = k _3A · W_dic + k _3B · W_fd (5a)
0 <k _3A · W_dic + k _3B · W_fd <W_dic (5b)

  Even if a face detection process is performed beyond a specific area determined by the width W_dic of the determination area, it is meaningless (because no face can exist in that area) or accurate face detection is performed. I can't. In consideration of this, the first, second, or third additional width setting method is adopted, and conditions such as equations (3b), (4b), and (5b) are provided. In addition, when there are a plurality of faces (including end faces) that the photographer wants to capture in the same input image, the sizes of the plurality of faces are generally the same. In consideration of this, the second or third additional width setting method may be adopted.

  Regardless of which additional width setting method is employed, useless or unnecessary detection processing is omitted, and the effect of reducing processing time is obtained and the effect of suppressing misdetection and overdetection of a face is obtained. The over-detection of the end face is an end face that is remarkably smaller or larger than the size of the face detected by the all-face detection processing unit 31 (this end face is assumed to be a face of a person not focused on by the photographer). Means detection.

Although the method for determining the size of the dummy image has been described focusing on the width, the image additional height H_dum (= H ₂ −H ₁ ) when generating the composite image from the input image is also set in the same manner as the image additional width W_dum. Is set. That is, the additional image height H_dum is set based on the height of the determination area and / or the height of the face already detected by the all-face detection processing unit 31.

[Operation after composite image generation]
Next, the operation of the end face detection processing unit 32 in FIG. 2 after generating the composite image will be described.

  When the end face detection processing unit 32 performs the end face detection process, the state in which the determination area is arranged at the upper left corner of the composite image is set as an initial state, and the determination area is horizontally shifted from left to right one pixel at a time in the composite image. To scan. When the determination area reaches the right end of the composite image, the determination area is shifted downward by one pixel, and scanning in the horizontal direction is performed again. In this way, it is sequentially detected whether an end face exists in the determination area while scanning the determination area in the horizontal direction and the vertical direction.

  However, the determination area defined in the end face detection processing unit 32 always overlaps a part of the dummy image as in the determination area 210 of FIG. The input image 200 and the dummy image 201 in FIG. 11 are the same as those shown in FIG. In FIG. 11, a shaded area 211 represents an area where the determination area 210 and the dummy image 201 overlap each other, and this area is hereinafter referred to as an overlapping area (or simply an overlapping area) between the determination area and the dummy image.

  Since the scanning is performed so that the determination region and the dummy image always have an overlapping region, as shown in FIG. 12A, when scanning the determination region in a certain composite image 202, the center coordinates of the determination region are It moves within the hatched area 203 near the outer periphery of the composite image 202.

  The determination area is scanned horizontally from left to right one pixel at a time, and when the determination area reaches the right end of the composite image, the determination area is shifted downward by one pixel and moved to the left end of the composite image to perform horizontal scanning again. Instead of scanning, it may be scanned in a spiral shape (spiral shape). In the former scanning method, when moving the determination region from the right end to the left end of the composite image, all pixel values of the image in the determination region read into a memory (not shown) for face detection are discarded from the memory, It is necessary to newly read all the pixel values of the image in the determination area after movement into the memory. Compared with this, the latter scanning method (spiral scanning method) has an advantage that the number of times of reading into the memory can be reduced.

  A conceptual diagram of the scanning direction of the spiral scan is shown in FIG. In the spiral scan, the determination area is scanned horizontally from left to right one pixel at a time, and when the determination area reaches the right end of the composite image, the scanning direction is changed to the vertical direction. When the determination area reaches the right end of the composite image, the determination area is scanned vertically from top to bottom without moving the determination area in the horizontal direction. By scanning in this way, when moving the determination area downward after the determination area reaches the right end of the composite image, a part of the pixel values read into the memory before the movement can be reused. is there. Therefore, the number of readings into the memory can be reduced, and high processing speed can be expected. When the determination area reaches the lower end of the composite image, the determination area is scanned in the horizontal direction from right to left without moving the determination area in the vertical direction. Thereafter, as shown in FIG. 12B, the determination region is scanned so that the movement locus of the determination region draws a spiral in the composite image.

  The input image for the end face detection processing unit 32 is the same as the input image for the all face detection processing unit 31. Accordingly, the input image to the edge detection processing unit 32 is also an original image or a reduced image of the original image, so that an edge having a different size can be detected in one type of determination area. Become.

  The size of the determination area 210 is the same as that of the determination area 110 in FIGS. Hereinafter, for simplification of description and illustration, it is assumed that the size of the determination area 210 is 8 × 8 pixels, similarly to the determination area 110.

  As described above, the determination area is scanned, but focusing on the state where the determination area exists at a specific position in a certain composite image, it is determined whether an end face exists within the determination area. The method will be described. FIG. 13 is a flowchart showing the operation procedure of this method.

  The end face detection processing unit 32 performs the same processing as steps S1 to S3 shown in FIG. That is, in step S1, a first edge-enhanced image in four directions is generated by performing edge enhancement processing on the image in the determination area defined in the composite image, and in the subsequent step S2, first images in four directions are generated. A second edge enhanced image in four directions is generated from the edge enhanced image. In step S3, a four-direction feature image is generated from the four-direction second edge enhanced image. The generation methods of the first edge enhanced image, the second edge enhanced image, and the feature image are the same as those in the all face detection processing unit 31. In the operation procedure in the end face detection processing unit 32, the process proceeds to step S14 after the process in step S3. The processes in steps S14 and S15 are executed by the end face detection processing unit 32.

In step S14, as described in the explanation of the operation of the all face detection processing unit 31, the variables q, x, and y are considered as the variable n, and the four-direction features generated by the end face detection processing unit 32 are considered. Based on the pixel value i (n) of each feature pixel F (n) of the image, the score w (n) is calculated from the above weight table. Then, the similarity S ₂ is calculated according to the following equation (6). The similarity S ₂ corresponds to the sum of the scores w (n) based on the four-direction feature images generated by the end face detection processing unit 32. The similarity calculation method in Equation (6) is the same as that in Equation (1) above, but the similarity (S ₁ ) calculated by the all face detection processing unit 31 and the end face detection processing unit 32 In order to distinguish and express the calculated similarity (S ₂ ), a symbol S ₂ different from S ₁ is introduced here.

After calculating the similarity S _2, the process proceeds to step S15. As described above, the all-face detection processing unit 31 determines whether or not the image in the determination region is a face image by determining whether S ₁ > TH is satisfied or not. However, although depending on the pixel value (hereinafter referred to as a dummy pixel value) of each pixel forming the dummy image, the similarity S ₂ includes a score component as an error in the overlapping region between the determination region and the dummy region. Can be. This is because a value not related to the face is set as the dummy pixel value.

Therefore, in step S15, a threshold correction term α is introduced to determine whether or not the following formula (7) is established. In step S15, when the following expression (7) is satisfied, it is determined that an end face exists in the determination area. When the following expression (7) is not satisfied, an end face exists in the determination area. Judge that it is not.
S ₂ > TH-α (7)

  The threshold correction term α is the sum of the scores w (n) calculated from the pixel values i (n) corresponding to the overlapping area. With reference to FIG. 14 and FIG. 15, a specific example for illustrating a method of calculating the threshold correction term α will be described. Now, as shown in FIG. 14, it is assumed that only one column at the left end of the determination area is an overlapping area. The first edge-enhanced images in four directions with respect to the image in the determination area are represented as reference numerals 221 to 224 in FIG. One row at the left end of each of the first edge-enhanced images 221 to 224, which is hatched, is obtained by performing edge enhancement processing on the pixels in the overlapping region. In FIG. 15, reference numerals 231 to 234 are four-direction feature images generated from the four-direction first edge enhanced images 221 to 224. The leftmost column of each of the feature images 231 to 234, which is shaded, is calculated based on the pixel values of the leftmost column of the first edge enhanced images 221 to 224. A total of 32 scores w (n) are calculated from the pixel values i (n) of a total of 32 pixels belonging to the leftmost column of each feature image 231 to 234. The total of 32 scores w (n) ) Is a threshold correction term α.

In the determination area, when an area different from the overlapping area is called a non-overlapping area, the similarity S ₂ is not overlapped with the scores corresponding to the images in the overlapping area (a total of 32 scores w (n) in the above example). The sum is a sum of the scores corresponding to the images in the area, the former corresponds to the contribution component of the overlapping area with respect to the similarity S ₂ and corresponds to the threshold correction term α, and the latter corresponds to the non-overlapping area with respect to the similarity S ₂ . It corresponds to the contribution component.

  As described above, by correcting the face determination threshold TH with the threshold correction term α, it is possible to perform end face detection independent of the dummy pixel value.

In consideration of the fact that the total number of feature pixels contributing to the similarity S ₂ decreases as the overlapping area increases, the establishment / non-establishment of the following expression (7a) may be determined in step S15. . In this case, when the following formula (7a) is satisfied, it is determined that an end face is present in the determination area. On the other hand, when the following expression (7a) is not satisfied, an end face is present in the determination area. Judge that there is no. Here, k ₄ is determined according to the total number of pixels belonging to the overlapping region (eight in the example of FIG. 14). K ₄ may be set to a value less than 1 when it is desired to increase the ease of detection of an end face, and k ₄ may be set to a value greater than 1 when it is desired to suppress erroneous detection and overdetection.
S ₂ > k ₄ · TH-α (7a)

  The face detection result of the all face detection processing unit 31 and the face detection result of the end face detection processing unit 31 are sent to the face detection result integration unit 33. The face detection result of the all-face detection processing unit 31 specifies the size and position of the face detected by the all-face detection processing unit 31 on the original image. The face detection result of the end face detection processing unit 32 specifies the size and position of the end face detected by the end face detection processing unit 32 on the original image. The face detection result integration unit 33 outputs an integrated face detection result corresponding to an integration of both face detection results.

  Each part in the imaging device 1 refers to the integrated face detection result as necessary, and performs necessary processing. For example, when a correction lens and a vari-angle prism (both not shown) are provided in the optical system of the imaging unit 11, the entire face of a person who is recognized as an end face is an original image by driving and controlling them. The optical image formed on the image sensor of the imaging unit 11 is shifted so as to be included in the image. In addition, the original image is usually formed from pixel signals of an effective imaging area included in the entire imaging area of the imaging device. This effective imaging area can be moved within the entire imaging area. Therefore, when an end face is detected, the effective image pickup area may be moved within the entire image pickup area so that the entire face of the person recognized as the end face is included in the original image. Further, by driving and controlling a zoom lens (not shown) provided in the optical system to increase the angle of view of the optical system, the entire face of a person recognized as an end face is included in the original image. Good

<< Second Embodiment >>
Next, an imaging apparatus according to the second embodiment of the present invention will be described. The imaging apparatus according to the second embodiment is similar to the imaging apparatus 1 according to the first embodiment, and the overall block diagram and the like are the same between the two. However, the method of calculating the similarity by the end face detection processing unit 32 in FIG. 2 differs between the first and second embodiments. In other respects, the first and second embodiments are the same, and therefore, duplicate description of similar parts is omitted, and description will be made focusing on the differences between the two. The matters described in the first embodiment are applied to the second embodiment as long as there is no contradiction.

  In the first embodiment, the face detection threshold TH is corrected using the threshold correction term α to realize edge detection independent of the dummy pixel value. In the second embodiment, such correction is performed. Accurate edge detection is realized without performing the process.

  In the second embodiment, when generating a composite image, a pixel value that does not affect the end face detection determination is set as the dummy pixel value of the dummy image forming the composite image. For example, if the score that does not affect the edge detection determination is 0 as exemplified in the first embodiment, the pixel value is such that the sum of the scores corresponding to the overlapping area between the determination area and the dummy image is 0. May be a dummy pixel value. That is, when generating a composite image, a pixel value from which a score having a value of 0 is derived (hereinafter referred to as a forced zero pixel value) may be set as a dummy pixel value in the overlapping region.

  This forced zero pixel value can be set to an arbitrary value in the design stage. If the possible value of each pixel value of the composite image is an integer between 0 and 255, a forced zero pixel value may be selected from an integer between 0 and 255.

By the way, as described in the first embodiment, the score w (n) is calculated from the pixel value i (n) of the corresponding feature pixel F (n), and if the pixel value and the variable n are determined, the weight It is uniquely determined by the table or function. As described above, this function is represented by a continuous function, and forms a continuous curved surface in a three-dimensional space in which the X-axis, Y-axis, and Z-axis are the pixel values of the feature pixels, the variable n, and the score. For this reason, even if the pixel value of the characteristic pixel is shifted by about “1”, there is a feature that the similarity S ₂ finally obtained does not greatly differ. This feature applies to the weight table created corresponding to this function as well. In addition, even if the pixel value of the feature pixel is shifted by about “1”, there is no significant difference in the similarity S ₂ finally obtained. Therefore, a shift of about “1” in the pixel value of the input image is also performed by the similarity S. _No significant impact on ₂ .

  Consider a case in which a pixel value of each pixel in the dummy image (overlapping region) is set to a forced zero pixel value and a mechanism for forcibly giving a score of 0 to a pixel for which the forced zero pixel value is set is considered. In this case, it is assumed that the pixels in the input image (non-overlapping region) have the same pixel value as the forced zero pixel value. FIG. 16 illustrates this situation, where pixel 300 in FIG. 16 is a pixel in the input image having a forced zero pixel value. In this case, if no treatment is made, the score corresponding to the pixel in the input image (pixel 300 in FIG. 16) is different from the score that should be originally. In order to avoid this, in the second embodiment, the above feature is used, and the pixel value of the pixel in the input image having the same pixel value as the forced zero pixel value is slightly increased or decreased.

  FIG. 17 shows an internal block diagram of the end face detection processing unit 32a according to the second embodiment. The end face detection processing unit 32a includes a composite image generation unit 41, a score calculation preprocessing unit 42, a score calculation unit 43, a similarity calculation unit 44, and an end face determination unit 45. FIG. It can be used as the end face detection processing unit 32.

  The composite image generation unit 41 generates a composite image by adding a dummy image around the input image to the face detection processing unit 32a. At this time, a forced zero pixel value is set as a dummy pixel value (pixel value of each pixel of the dummy image). Furthermore, when a pixel having a forced zero pixel value (corresponding to the pixel 300 in FIG. 16) is included in the input image, the pixel value is increased or decreased by slightly increasing or decreasing the pixel value of the pixel. Different from. Specifically, the pixel value is increased by 1 or decreased by 1. Whether to increase or decrease the pixel value may be determined according to the forced zero pixel value.

  The score calculation preprocessing unit 42 refers to the composite image generated by the composite image generation unit 41 and forcibly gives a score of 0 to pixels having a forced zero pixel value in the composite image. The area having a pixel value different from the forced zero pixel value, that is, the area of the input image is subjected to the same processing as that of the first embodiment by the score calculation unit 43, and each score is calculated from the weight table.

  That is, the composite image generation unit 41, the score calculation preprocessing unit 42, and the score calculation unit 43 all set the scores corresponding to the pixels in the dummy image to 0, and the scores corresponding to the pixels in the input image It comes to be calculated from. Each obtained score is given to the similarity calculation unit 44.

The similarity calculation unit 44 calculates the similarity S ₂ according to the above equation (6) based on the given scores. Similar to the first embodiment, the similarity S ₂ is calculated for each determination region. For a certain determination area, the total of 256 scores w (n) calculated from the images in the determination area is calculated as the similarity S ₂ to the determination area. Of the total 256 scores w (n), at least the scores corresponding to the overlapping regions are all zero. That is, the contribution component of the overlapping area to the similarity S ₂ is zero.

The end face determination unit 45 compares the similarity S ₂ calculated by the similarity calculation unit 44 with the face determination threshold TH, and when the former is larger than the latter, an end face exists in the focused determination region. On the other hand, if not, it is determined that there is no end face in the determination area. Alternatively, TH that should be compared with the similarity S ₂ may be multiplied by the coefficient k ₄ as described in the first embodiment by changing the above expression (7) into the expression (7a). That is, the edge determination unit 45 compares the similarity S ₂ calculated by the similarity calculation unit 44 with k ₄ · TH, and if the former is larger than the latter, the edge is present in the focused determination region. On the other hand, if not, it may be determined that there is no end face in the determination area. The determination result by the end face determination unit 45 is output as a face detection result. This face detection result is equivalent to that by the end face detection processing unit 32 of FIG.

  If configured as in the second embodiment, it is not necessary to sequentially calculate the threshold correction term α. Moreover, it is possible to calculate the similarity related to edge detection using the same similarity calculation method as that of the all-face detection processing unit 31 in FIG. 1 (this also applies to the first embodiment). That is, it is not necessary to employ a dedicated similarity calculation method or to prepare a dedicated weight table in order to perform end face detection. For this reason, it is possible to share the part for calculating the similarity between the all face detection processing unit and the end face detection processing unit.

In the above description, “the pixel value of each pixel of the dummy image is set to a forced zero pixel value, and a pixel having a forced zero pixel value is forcibly given a score of 0”. It can also be deformed. That is, the pixel value of each pixel of the dummy image is set to the forced zero pixel value, and the pixel having the forced zero pixel value is set so that the sum of the scores calculated from the pixel values of the respective pixels in the overlapping region becomes the value C _SCORE. Forcibly give some value. Then, the similarity S ₂ includes a component of the value C _SCORE . For example, the value C _SCORE is a predetermined value that is determined in advance. The value C _SCORE may be determined according to the total number of pixels in the overlap region. The value C _SCORE may be set to a positive value if it is desired to increase the ease of detection of the end face, and the value C _SCORE may be set to a negative value if it is desired to suppress erroneous detection and overdetection.

<< Third Embodiment >>
Next, a third embodiment of the present invention will be described. The overall block diagram of the imaging apparatus according to the third embodiment is the same as that in the first embodiment (FIG. 1), and the matters described in the first embodiment (or the second embodiment) are as long as there is no contradiction. This also applies to the third embodiment. Since the third embodiment has a feature inside the face detection unit 19 in FIG. 1, the description will be made by paying attention to the feature portion.

  FIG. 18 is an internal block diagram of a face detection unit that can be used as the face detection unit 19 of FIG. The face detection unit in FIG. 18 is similar to the face detection unit 19 in FIG. However, the end face detection processing unit 32 in the face detection unit 19 of FIG. 2 is replaced with an end face detection processing unit 32b. Except for this replacement, the face detection unit in FIG. 18 and the face detection unit 19 in FIG. 2 have the same configuration and function. The end face detection processing unit 32b has the same function as the end face detection processing unit 32, and also the face detection result of the all face detection processing unit 31 and / or the tilt output from the tilt sensor 18 of FIG. It has a unique function using data.

  Hereinafter, this unique function will be described. This peculiar function is a function that limits the detection conditions when detecting the end face. Since there are seven specific modes for adding restrictions to detection conditions, each will be described individually as first to seventh restriction examples. Each restriction example can be used in any combination as long as there is no contradiction, and the matters described in a certain restriction example can be applied to other restriction examples as long as there is no contradiction. In the following description, the face detected by the all-face detection processing unit 31 is also referred to as an all-face in order to clearly distinguish it from the end face detected by the end-face detection processing unit 32b.

[First restriction example]
First, a first restriction example will be described. Focus on one original image. In the first restriction example, an upper limit and a lower limit are set in the end face detection size range as a detection condition based on the size of all faces detected in advance by the all face detection processing unit 31. This will be specifically described.

  As described above, by using the reduced image of the original image as the input image and changing the reduction ratio in stages, the entire face detection processing unit 31 and the end face detection processing unit 32b can detect all the faces or edges having various sizes. The face can be detected. The all face detection processing unit 31 performs a process for detecting all faces on each input image based on the focused original image, and uses the detected all face size information for specifying the size of all detected faces as a face detection result. This is given to the face detection processing unit 32b (and the image input unit 30 as necessary).

Let _FW be the size of all faces on the original image specified by the detected total face size information. When a plurality of all faces are detected from the focused original image, any one of the average size, the maximum size, and the minimum size may be set as _FW .

In this case, the upper limit of the end face detection size range is (F _W + ΔA), and the lower limit is (F _W −ΔB). In other words, an end face is detected so that an end face having a size of (F _W + ΔA) or less and (F _W −ΔB) or more on the original image is detected, and an end face of any other size is not detected. The detection processing unit 32b operates (specifically, the reduction rate for the original image is limited). Here, ΔA and ΔB are positive values, and matching / mismatching of both is not questioned. Note that the size of the entire face or end face on the original image is represented, for example, by the total number of pixels of the face area (area where it is determined that the entire face or end face exists) on the original image.

  In commemorative photography or the like, the size of the face of the person to whom the photographer pays attention is often almost the same. Considering this, as described above, the upper and lower limits of the end face detection size range are set in accordance with the sizes of all faces detected in advance. Thereby, the effect of shortening the processing time is obtained and the effect of suppressing misdetection and overdetection of the face is obtained.

[Second restriction example]
Next, a second restriction example will be described. First, items necessary for explaining the second restriction example will be described. Now, as shown in FIG. 19, paying attention to the input image 400, the region 401 including the left end of the input image 400 as the center, the region 402 including the right end of the input image 400 as the center, and including the upper end of the input image 400 as the center. An area 403 and an area 404 including the lower end of the input image 400 as the center are defined. Further, the upper left corner of the input image 400 is set as the origin 410 of the input image 400. In FIG. 19, the vertical direction corresponds to the vertical direction of the image, and the horizontal direction corresponds to the horizontal direction of the image.

  On the left side of the region 403, the region 403 and the region 401 have a common region, and on the right side of the region 403, the region 403 and the region 402 have a common region. On the left side of the region 404, the region 404 and the region 401 have a common region, and on the right side of the region 404, the region 404 and the region 402 have a common region. The area 401 and the area 402 do not have a common area. The determination area for end face detection is scanned so as to be included in any of the areas 401 to 404. When the second restriction example is not applied, for example, the determination area for end face detection is scanned so as to be located in all of the areas 401 to 404. That is, the end face detection range is the entire area 401 to area 404.

  In the second restriction example, the end face detection range as a detection condition is restricted based on the positions of all faces detected in advance by the all face detection processing unit 31. For example, as shown in FIG. 19, when the entire face 420 is detected from the left side of the input image 400, the region 401 is included in the end face detection range, while the region 402 is excluded from the end face detection range. As a result, the determination area for end face detection is scanned in the area 401, but not scanned in the area 402, and end face detection is not performed on the area 402. In this case, whether or not to include the regions 403 and 404 in the end face detection range is arbitrary. Further, what kind of restriction is added to the end face detection range in accordance with the positions of all detected faces is set in advance.

  When a plurality of faces are included in the shooting area, such as when taking a commemorative photo, the faces are often concentrated on one side in the shooting area. Considering this, as described above, the end face detection range is set according to the positions of all faces detected in advance (limitation is added). Thereby, the effect of shortening the processing time is obtained and the effect of suppressing misdetection and overdetection of the face is obtained.

[Third restriction example]
Next, a third restriction example will be described. When the third and fourth restriction examples described later are used, the entire face detection processing unit 31 is configured to detect the orientation of all faces. In this case, the all-face detection processing unit 31 determines whether the face detected from the input image is a front face (a face seen from the front) or a side face (a face seen from the right side or the left side). Can be detected separately. Various methods have been proposed as methods for detecting the orientation of the face, and the all-face detection processing unit 31 can employ any method.

  For example, as in the technique described in Japanese Patent Laid-Open No. 10-307923, face parts such as eyes, nose and mouth are found in order from the input image, and the position of the face in the image is detected. The direction of the face is detected based on the projection data of the part. Alternatively, for example, a method described in JP 2006-72770 A may be used. This technique will be described later.

  In the third restriction example, the end face detection range as a detection condition is restricted based on the orientation of all faces detected in advance by the all face detection processing unit 31. For example, as shown in FIG. 20, when the orientation of all faces 430 detected from the input image 400 is rightward, the area 402 is included in the end face detection range, while the area 401 is excluded from the end face detection range. As a result, the determination area for end face detection is scanned in the area 402, but is not scanned in the area 401, and end face detection is not performed on the area 401. In this case, whether or not to include the regions 403 and 404 in the end face detection range is arbitrary. In addition, what kind of restriction is added to the end face detection range in accordance with the orientations of all detected faces is set in advance.

  When all the faces included in the input image are facing right, it is estimated that there is a high possibility that the end face that may exist is located on the right side in the composite image. Considering this, as described above, the end face detection range is set according to the orientations of all faces detected in advance (limitation is added). Thereby, the effect of shortening the processing time is obtained and the effect of suppressing misdetection and overdetection of the face is obtained.

[Fourth restriction example]
Next, a fourth restriction example will be described. When the fourth restriction example is used, the end face detection processing unit 32 is configured to detect the direction of the end face. In this case, the end face detection processing unit 32 determines whether the end face detected from the composite image is a front face (end face seen from the front) or a side face (end face seen from the right side or the left side). It is possible to detect whether it is As a method for detecting the direction of the end face, a method similar to that in the all-face detection processing unit 31 can be used.

  In the fourth restriction example, the orientation of the detected end face is restricted based on the orientation of all the faces detected in advance by the all face detection processing unit 31. In the fourth restriction example, the restriction on the direction of the end face corresponds to the detection condition when detecting the end face.

  For example, as shown in FIG. 21, when the orientation of all faces 440 detected from the input image 400 is rightward, the right-facing endface 441 is detected while the left-facing endface 442 is not detected (endface). 442 is not recognized as an end face). At this time, whether or not to detect an end face facing the front is arbitrary. In addition, it is set in advance what kind of restriction is imposed on the orientation of the face to be detected according to the orientation of all the detected faces.

  When all the faces included in the input image are facing right, it is highly likely that the left-facing end face that can be included in the composite image is a face of a person not focused on by the photographer. Considering this, as described above, the direction of the end face to be detected is set in accordance with the direction of all faces detected in advance (addition of a restriction). Thereby, the effect of shortening the processing time is obtained and the effect of suppressing misdetection and overdetection of the face is obtained.

  It is possible to combine the fourth restriction example with the third restriction example described above. For example, as shown in FIG. 21, when the orientation of all faces 440 detected from the input image 400 is rightward, the left-facing end face 442 located in the area 401 is not detected, while the area 402 A left-facing end face (not shown) is made to be detected.

[Fifth restriction example]
Next, a fifth restriction example will be described. In the fifth restriction example, inclination data from the inclination sensor 18 of FIG. 1 is used. The inclination data specifies whether the current state of the imaging apparatus 1 is the horizontal shooting state or the vertical shooting state.

  The end face detection processing unit 32b limits the end face detection range based on the inclination data. An input image 400 shown in FIGS. 19 to 21 is an original image taken in the horizontal shooting state or a reduced image thereof. Accordingly, the end face detection processing unit 32b includes the areas 401 and 402 in the end face detection range based on the tilt data (inclination data indicating the horizontal shooting state) at the time of shooting the original image, while the area from the end face detection range to the area. 403 and 404 are excluded. This is because a situation in which the face is cut off in the vertical direction side of the image in the horizontal shooting state is relatively unlikely. As a result, the determination area for edge detection is scanned in the areas 401 and 402, but is not scanned in the areas 403 and 403, and the edge detection is not performed on the areas 403 and 404.

  On the other hand, an input image 400 shown in FIG. 22 is an original image taken in a vertical shooting state or a reduced image thereof. In this case, the end face detection processing unit 32b includes the areas 403 and 404 in the end face detection range based on the tilt data at the time of shooting the original image (inclination data indicating the vertical shooting state), while starting from the end face detection range. The areas 401 and 402 are excluded. This is because a situation in which the face is cut off in the horizontal direction of the image in the vertical shooting state is relatively unlikely. As a result, the determination area for edge detection is scanned in the areas 403 and 404, but is not scanned in the areas 401 and 402, and edge detection is not performed on the areas 401 and 402.

  According to the fifth restriction example, the effect of shortening the processing time can be obtained, and the effect of suppressing misdetection and overdetection of the face can be obtained.

[Sixth restriction example]
Next, a sixth restriction example will be described. Prior to specific description of the sixth restriction example, a method for detecting a tilted whole face or end face will be described. Although common to all the embodiments, the whole face detection processing unit (31) and the end face detection processing unit (32, 32a, or 32b) each have a tilted full face and a tilted end face as shown in FIG. Can also be detected. In order to detect all the tilted faces, the input image is rotated by a necessary amount, and the above-described all face detection process (process for detecting all faces) is performed on the rotated input image. . Similarly, when detecting an inclined face, the above-described face detection process (process for detecting a face) may be performed on the rotated input image. Hereinafter, the sixth restriction example will be described focusing on the edge face, but the process is similarly performed for all faces.

  In order to detect an end face having various tilt angles, for example, the input image is rotated by a predetermined angle, and the above-described all face detection process and end face detection process are performed on each rotated input image. However, in the sixth restriction example, the angle range for rotating the input image is restricted based on the tilt data described above. That is, an end face having an inclination angle within a predetermined inclination angle range can be detected based on the inclination data, while an end face having an inclination angle outside the inclination angle range is excluded from detection targets. Depending on the inclination data, what kind of restriction should be added to the angle range (or inclination angle range) as a detection condition may be determined in advance.

  A specific example is given. When the tilt data at the time of shooting the original image indicates the horizontal shooting state, an end face 451 having a chin on the region 403 side and an eye on the region 404 side as shown in FIG. , Unlikely to exist. Therefore, when the tilt data represents the horizontal shooting state, the above-described angle range is limited so that an end face such as the end face 451 is not detected.

  Similarly, when the tilt data at the time of shooting the original image indicates the vertical shooting state, as shown in FIG. 24B, the end where the jaw is on the region 402 side and the eye is on the region 401 side of the composite image. The face 452 is unlikely to exist. Therefore, when the tilt data represents the vertical shooting state, the above-described angle range is limited so that an end face such as the end face 452 is not detected.

  According to the sixth restriction example, the effect of shortening the processing time can be obtained and the effect of suppressing misdetection and overdetection of the face can be obtained.

[Seventh restriction example]
Next, a seventh restriction example will be described. In the seventh embodiment, end face detection is performed by paying attention only to the direction in which a human voice can be heard. For this purpose, a plurality of directional microphones are mounted on the imaging apparatus 1 of FIG. The plurality of directional microphones are installed in the imaging apparatus 1 so that directions with high sensitivity are different from each other.

  A specific example will be given with reference to FIG. FIG. 25 is a diagram of the imaging apparatus 1 provided with the directional microphones 51 and 52 as viewed from above (empty side). In FIG. 25, reference numeral 501 represents an area where the directivity of the directional microphone 51 is high, and reference numeral 502 represents an area where the directivity of the directional microphone 52 is high. The directional microphone 51 collects sound that enters from the left side of the imaging apparatus 1, and the directional microphone 52 collects sound that enters from the right side of the imaging apparatus 1. That is, on the two-dimensional plane when the imaging device 1 is viewed from above, when the center of the imaging device 1 is considered as a reference, the directional microphone 51 is relatively high in sensitivity to sound entering from the left side. The sensitivity of the directional microphone 52 is relatively low with respect to sound coming from the right side, while the sensitivity of the directional microphone 52 is relatively high with respect to sound coming from the left side. Low.

  The main control unit 13 in FIG. 1 also has a function as an audio signal processing unit. Each audio signal representing each sound input to the directional microphones 51 and 52 is given to the main control unit 13. Based on each audio signal, the main control unit 13 determines whether or not a human voice is included in the sound input to the directional microphone 51 using a known speech recognition process and is input to the directional microphone 52. To determine whether the sound contains human voice. The result of this determination is called a voice determination result.

  FIG. 25 is also a diagram of the imaging device 1 in the horizontal shooting state as viewed from above (empty side). Therefore, the voices uttered by the person located on the side of the composite image area 401 (see FIG. 19 and the like) are collected by the directional microphone 51, and the voices uttered by the person located on the side of the composite image area 402 are collected. 52 is collected. In the seventh restriction example, this characteristic is used to restrict the end face detection range based on the voice determination result.

  For example, if it is determined that the sound input to the directional microphone 51 includes a human voice and the sound input to the directional microphone 52 does not include a human voice, the left side of the imaging device 1 And the region 401 is included in the end face detection range, while the region 402 is excluded from the end face detection range. Conversely, when it is determined that the sound input to the directional microphone 51 does not include a human voice and the sound input to the directional microphone 52 includes a human voice, the imaging apparatus 1 It is assumed that there is a person on the right side of the area, and the area 402 is included in the end face detection range, while the area 401 is excluded from the end face detection range. In these cases, whether or not to include the regions 403 and 404 in the end face detection range is arbitrary. Further, what kind of restriction is added to the end face detection range according to the sound determination result is set in advance. Further, although the restriction process in the horizontal shooting state has been described, it is possible to perform the same process in the vertical shooting state by applying a naturally anticipated modification.

  According to the seventh restriction example, the effect of shortening the processing time can be obtained and the effect of suppressing misdetection and overdetection of the face can be obtained.

<< Fourth Embodiment >>
The technique described in JP-A-2006-72770 can be applied to the all face detection processing unit (31) and the end face detection processing unit (32, 32a, or 32b) in the first to third embodiments. . This will be described as a fourth embodiment.

  In this method, one front face is divided into a left half (hereinafter referred to as a left face) and a right half (hereinafter referred to as a right face), and the parameters for the left face and the right face are determined through a learning process. Is generated. At the time of face detection, the left determination area and the right determination area are defined by dividing the determination area into right and left, the similarity based on the left determination area and the left face parameters, and the right determination area and right face parameters And the similarity based on. Then, when one or both of the similarities are equal to or greater than the threshold, it is determined that the determination area is a face area. Furthermore, the face orientation is detected from the magnitude relationship between the two similarities.

  When this method is applied to the first to third embodiments, a left face parameter and a right face parameter are generated as a weight table. That is, as a weight table stored in the face dictionary memory 34 of FIG. 1, a left face weight table representing a left face parameter and a right face weight table representing a right face parameter are prepared. .

Then, the feature pixels F (n) are classified into feature pixels for the left face and feature pixels for the right face in correspondence with dividing the decision area into a left decision area and a right decision area, and each feature for the left face The sum of the score w (n) calculated based on the pixel value i (n) of the pixel F (n) and the weight table for the left face is calculated as the similarity S _L for the left face, calculating the sum of the feature pixel F pixel value i of the (n) (n) and the score is calculated based on the weight table for right facial w (n) as the similarity S _R for the right face.

The all face detection processing unit determines whether or not the inequality “S _L > TH _B ” and the inequality “S _R > TH _B ” are satisfied. When it is determined that both are present and neither of them is established, it is determined that the entire face does not exist in the focused determination area. Here, TH _B is a predetermined threshold value.

For example, the end face detection processing unit determines whether or not the inequality “S _L > TH _B −α _L ” and the inequality “S _R > TH _B −α _R ” are satisfied. If it is determined that an end face exists in the focused determination area and neither of them is established, it is determined that no end face exists in the focused determination area. The threshold correction term α _L is the sum of the scores w (n) calculated from the pixel values i (n) corresponding to the overlapping area belonging to the left determination area. The threshold correction term α _R is the sum of the scores w (n) calculated from the pixel values i (n) corresponding to the overlapping area belonging to the right determination area.

In addition, the method described in Japanese Patent Application Laid-Open No. 2006-72770 may be applied, and the detected orientation of all faces or end faces may be detected based on the magnitude relationship between S _L and S _R.

<< Deformation, etc. >>
As modifications or annotations of the above-described embodiment, notes 1 to 4 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
In each of the above-described embodiments, the same input image is given to the whole face detection processing unit (31) and the end face detection processing unit (32, 32a, or 32b), and the face detection result of the whole face detection processing unit for the input image ( All face detection results) are reflected in the edge detection processing for the input image by the edge detection processing unit. That is, all face detection processing is performed on a captured image of a certain frame, and the result of all face detection is reflected in the end face detection processing for the captured image of the same frame.

  However, the entire face detection process may be performed on the captured image of the first frame, and the entire face detection result may be reflected in the edge detection process for the captured image of the second frame. The second frame is a frame after the first frame. Typically, for example, the second frame is a frame subsequent to the first frame.

  In this way, the following processing is also possible. That is, based on the result of the entire face detection process for the captured image of the first frame, a condition for the edge detection process for the captured image of the second frame is set, and a composite image based on the captured image of the second frame is generated. Here, the “end face detection processing conditions” refer to all the conditions to be set for performing end face detection. The image additional width W_dum (see FIG. 10) described in the first embodiment, the third embodiment. The detection conditions at the time of edge detection described in the form are included. Then, the determination area for performing all face detection processing and end face detection processing is scanned at a time on the composite image corresponding to the second frame. That is, both detection scans are performed on the composite image corresponding to the second frame without separately performing the determination region scan for performing all face detection and the determination region scan for performing end face detection. Do it at once. As a result, it is not necessary to perform scanning exclusively for edge detection.

[Note 2]
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values.

[Note 3]
The imaging device in each embodiment can be realized by hardware or a combination of hardware and software. In particular, the function of each part shown in FIG. 2, FIG. 17, or FIG. 18 can be realized by hardware, software, or a combination of hardware and software (except for memory), and each of these functions. Can also be realized outside the imaging device.

  When the imaging apparatus is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part. Also, all or part of the functions of the respective parts shown in FIG. 2, FIG. 17, or FIG. 18 are described as a program, and the program is executed on a program execution device (for example, a computer), whereby all of the functions are described. Or you may make it implement | achieve a part.

[Note 4]
In each of the above-described embodiments, the end face detection processing unit (32, 32a or 32b) shown in FIG. 2 or the like functions as an end face detection means, and the end face detection means has the similarity (S ₂₎ described above. Etc.) and a determination threshold value setting means for setting a determination threshold value with which the similarity should be compared. This determination threshold corresponds to (TH-α) described above.

1 is an overall block diagram of an imaging apparatus according to a first embodiment of the present invention. It is an internal block diagram of the face detection part of FIG. It is a figure showing a mode that the determination area | region is scanned within the input image with respect to the whole face detection process part of FIG. It is a flowchart showing the operation | movement procedure of the face detection process (all face detection process) by the all face detection process part of FIG. It is a figure showing the differential filter used when producing | generating the 1st edge emphasis image of 4 directions from the image in a determination area | region. It is a figure showing the 1st edge emphasis image of 4 directions and the 2nd edge emphasis image of 4 directions produced | generated from the image in a determination area | region. It is a figure showing the characteristic image of 4 directions produced | generated from the image in a determination area | region. It is a figure showing the example of the content of the weight table stored in the face dictionary memory of FIG. It is a figure showing the end face which may exist in an input image. It is a figure showing the synthesized image produced | generated by adding a dummy image to an input image. It is a figure which shows that an overlap area | region exists between a determination area | region and a dummy image. It is a figure which shows the locus | trajectory of the center coordinate of the determination area | region scanned within a composite image. 3 is a flowchart showing an operation procedure of face detection processing (face detection processing) by an edge detection processing unit in FIG. 2. It is a figure for demonstrating the calculation method of the threshold value correction | amendment term by the end face detection process part of FIG. It is a figure for demonstrating the calculation method of the threshold value correction | amendment term by the end face detection process part of FIG. It is a figure for demonstrating operation | movement of the end face detection process part which concerns on 2nd Embodiment of this invention. It is an internal block diagram of the end face detection processing part which concerns on 2nd Embodiment of this invention. It is an internal block diagram of the face detection part which concerns on 3rd Embodiment of this invention. It is a figure for demonstrating the relationship between an input image and an end face detection range concerning 3rd Embodiment of this invention. It is a figure for demonstrating the relationship between an input image and an end face detection range concerning 3rd Embodiment of this invention. It is a figure for demonstrating the relationship between the direction of all the detected faces and the direction of the end face used as a detection target concerning 3rd Embodiment of this invention. It is a figure for demonstrating the relationship between the inclination of an imaging device, and an end face detection range concerning 3rd Embodiment of this invention. It is a figure which concerns on 3rd Embodiment of this invention and shows the whole face and end face which inclined. It is a figure which concerns on 3rd Embodiment of this invention and shows the inclined end face excluded from a detection target. It is the figure which looked at the imaging device provided with the directional microphone concerning 3rd Embodiment of this invention from upper direction (sky side).

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 19 Face detection part 30 Image input part 31 Whole face detection process part 32, 32a, 32b End face detection process part 33 Face detection result integration part 34 Face dictionary memory

Claims (9)

In a face detection device that performs face detection,
A face detection unit is provided for detecting a face part of the face that has been applied to the edge of the input image by adding a dummy image around the input image and performing face detection beyond the edge of the input image. A face detection device characterized by the above. The edge detection means includes:
A determination area is defined in a composite image of the input image and the dummy image;
Similarity calculating means for calculating the similarity between the image in the determination area and predetermined dictionary data;
A determination threshold value setting means for setting the determination threshold value,
The face detection apparatus according to claim 1, wherein it is determined whether or not the end face is present in the determination region by comparing the similarity and the determination threshold. A face detecting means for detecting a face present in the input image;
The face detection apparatus according to claim 1, wherein the end face detection unit changes the detection condition of the end face according to a face detection result of the face detection unit. 4. The end face detection unit changes the end face detection condition based on at least one of a size, a position, and an orientation of the face detected by the face detection unit. Face detection device. The face detection device is mounted on an imaging device having imaging means,
The input image is generated from a captured image of the imaging means,
The face detection apparatus according to claim 1, wherein the end face detection unit changes the detection condition of the end face according to an inclination of the image pickup apparatus at the time of shooting the captured image. In a face detection device comprising a face detection means for detecting a face present in an input image,
Further provided is an end face detection means for detecting an end face where a part of the face is applied to the end of the input image,
The face detection device, wherein the end face detection unit changes the detection condition of the end face according to a face detection result of the face detection unit. In the face detection device mounted on the imaging device having the imaging means,
Receiving an input image generated from a captured image of the imaging means, and comprising an end face detecting means for detecting an end face where a part of the face is applied to the end of the input image,
The face detection device is characterized in that the face detection device changes a detection condition of the face according to an inclination of the image pickup device at the time of shooting the captured image. Imaging means;
An imaging device comprising: the face detection device according to claim 1,
The imaging apparatus according to claim 1, wherein the input image to the face detection apparatus is generated based on a captured image of the imaging means. In the face detection method for performing face detection,
A face detection method, wherein a dummy image is added around an input image and face detection is performed beyond the edge of the input image, thereby detecting a face part of the face that is applied to the edge of the input image. .

Images