JP2009017194A - Object detecting device and method therefor, and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To suppress breaks and fluctuations in facial detection. <P>SOLUTION: A face detection device includes a first detection part for provisionally detecting a facial position from each frame image and a second detection part for detecting the final facial position. When the facial position of a current frame image is detected, a center view area (220) and a peripheral view area (230) around the area (220) are set with the facial position of a preceding frame image as a reference. The center view area and the peripheral view area are defined for the preceding frame image and the current frame image, and image changes between both the frame images in the center view area and the peripheral view area are detected by the calculation of motion vectors, or the like. The second detection part ultimately detects the facial position of the current frame, on the basis of detection results of the first detection part for the current frame, and the preceding frame and the image changes. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an object detection apparatus and an object detection method for detecting an object such as a human face from an image. The present invention also relates to an imaging apparatus using them.

  In an imaging apparatus, a face detection function for detecting a face from an image has been put into practical use, and various camera controls using face detection results have been proposed. For example, the face detection result can be used for image compression processing. In this type of image compression processing, the resolution of the face portion is kept relatively high by increasing the amount of code assigned to the face portion as compared to other portions (see, for example, Japanese Patent No. 3748717).

  When performing camera control using a face detection result, stable face detection is required. Stable face detection means that face detection is not interrupted or wobbled (or less). In the case where the above-described image compression processing is performed using the face detection result, if face detection is interrupted (actually, a face exists in the image but face detection fails), the face portion The resolution of the screen suddenly increases or decreases. This is more problematic than shooting with low resolution of the face. A similar problem occurs when auto focus control, automatic exposure control, or auto white balance control is executed using the face detection result. As can be seen from these circumstances, stable face detection is important.

  However, face detection does not succeed under all conditions, and in fact, face detection may not be performed stably. For example, the face of a person faces every direction, but there is a limit to the direction of the face that can be detected, and face detection may fail near the limit. In addition, for example, face detection may fail due to noise, image blur due to face movement, sudden light source change or occlusion. Further, for example, a face search area may be thinned out to reduce the amount of face detection calculation, and in this case, the detected face position may fluctuate. When the camera control is performed based on such an unstable face detection result, the above-described problem occurs.

  Patent Document 1 below discloses a technique for performing shooting operation control based on a face detection result obtained within a predetermined time when face detection becomes impossible during continuous shooting. However, with this method, when the face is stationary for a predetermined time or more, the face detection result necessary for the shooting operation control is lost.

JP 2006-157617 A Hiroyoshi Fujiyoshi, “Video Understanding Technology and its Applications”, [online], Chubu University Faculty of Engineering, Department of Information Engineering, [Search May 25, 2007], Internet <URL: http: //www.vision.cs.chubu .ac.jp / VU / pdf / VU.pdf>

  Although the conventional problem has been described focusing on the case where the object to be detected is a face, the same problem occurs when an object other than the face is detected.

  Accordingly, an object of the present invention is to provide an object detection device, an object detection method, and an imaging device that contribute to stable object detection.

  A first object detection device according to the present invention is an object detection device that detects a position of a specific type of object in a moving image based on the moving image, and also includes an image change around the object in the moving image. In consideration of this, the position of the object is detected.

  Based on the image change around the object, it is possible to estimate the object itself or the movement around the object, and it is possible to detect the position of the object well based on the estimation result. Therefore, when configured as described above, stable object detection is expected.

  A second object detection device according to the present invention is an object detection device that detects a position of a specific type of object in a moving image based on the moving image, and tentatively detects the position of the object in the moving image. Based on the detection result of the first detection means, the detection result of the first detection means, and the image change in the reference area including the object and in the peripheral area located around the reference area. And second detection means for finally detecting the position of the object.

  In some cases, the position of the detected object can be corrected to a more appropriate position based on the image change. Therefore, it is configured as described above. This also allows stable object detection.

  Specifically, for example, the moving image is formed from a plurality of input images arranged in time series, and the first detection means tentatively detects the position of the object in each input image, and the second detection Means for setting the reference area and the peripheral area for the current input image based on the position of the object with respect to the past input image detected by the first detection means; The reference area and the peripheral area set for the current and past input images are defined, and image changes in the reference area and the peripheral area between the current and past input images are defined as the current input image. Image change detection means for detecting an image change for the first and second input images, and a detection result of the first detection means for the past and current input images, and the image change detection means. It detected Te, and the image changes for the current input image, and finally to detect the position of the object in the current input image based on.

  Further, for example, the peripheral area is set so as to surround the entire circumference of the reference area, and the image change detection unit excludes a part of the peripheral area and then performs the image change for the peripheral area. To detect.

  For example, when the object to be detected is a person's face, further stabilization of object detection is expected if the area corresponding to the body is excluded.

  Instead of this, for example, the peripheral area may be set so as to surround a part of the entire circumference of the reference area.

Further, for example, the peripheral area is hierarchized to include a first peripheral area that is relatively close to the reference area and a second peripheral area that is relatively far from the reference area,
The second detection means finally detects the position of the object based on a detection result of the first detection means and image changes in the reference area and the first and second peripheral areas. Also good.

  More specifically, for example, the specific type of object is a human face.

  An imaging apparatus according to the present invention includes an imaging unit that acquires a moving image according to a subject, and any one of the object detection devices that receives the moving image obtained by the imaging unit. It is characterized by.

  The object detection method according to the present invention is an object detection method for detecting a position of a specific type of object in a moving image based on the moving image, and also considers an image change around the object in the moving image. Then, the position of the object is detected.

  According to the present invention, it is possible to provide an object detection apparatus, an object detection method, and an imaging apparatus that contribute to stable object detection.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to fourth embodiments will be described later. First, matters that are common to each embodiment or items that are referred to in each embodiment will be described.

  FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging apparatus 1 in FIG. 1 is a digital still camera that can capture and record still images, or a digital video camera that can capture and record still images and moving images.

  The imaging apparatus 1 includes an imaging unit 11, an AFE (Analog Front End) 12, a main control unit 13, an internal memory 14, a display unit 15, a recording medium 16, an operation unit 17, and a face detection unit 18. It is equipped with.

  The imaging unit 11 includes an optical system, an aperture, an imaging device such as a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and a driver (all not shown) for controlling the optical system and the aperture. And have. The driver controls the zoom magnification and focal length of the optical system and the aperture of the diaphragm based on the control signal from the main control unit 13. The image sensor photoelectrically converts an optical image representing a subject incident through the optical system and the diaphragm, and outputs an electrical signal obtained by the photoelectric conversion to the AFE 12.

  The AFE 12 amplifies the analog signal output from the imaging unit 11 (imaging device), and converts the amplified analog signal into a digital signal. The AFE 12 sequentially outputs this digital signal to the main control unit 13.

  The main control unit 13 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and also functions as a video signal processing unit. Based on the output signal of the AFE 12, the main control unit 13 generates a video signal representing an image captured by the imaging unit 11 (hereinafter also referred to as “captured image” or “frame image”). The main control unit 13 also has a function as display control means for controlling the display content of the display unit 15, and performs control necessary for display on the display unit 15.

  The internal memory 14 is formed by SDRAM (Synchronous Dynamic Random Access Memory) or the like, and temporarily stores various data generated in the imaging device 1. The display unit 15 is a display device composed of a liquid crystal display panel or the like, and displays an image taken in the immediately previous frame or an image recorded on the recording medium 16 under the control of the main control unit 13. The recording medium 16 is a non-volatile memory such as an SD (Secure Digital) memory card, and stores captured images and the like under the control of the main control unit 13. Of course, the recording medium 16 may be formed using a recording medium other than the SD memory card (for example, HDD (Hard Disk Drive) or DVD (Digital Versatile Disc)). The operation unit 17 receives an operation from the outside. The content of the operation on the operation unit 17 is transmitted to the main control unit 13.

  The face detection unit 18 detects the face of a person from each captured image, and transmits information representing the face detection result to the main control unit 13. The main control unit 13 performs various processes using the face detection result of the face detection unit 18. For example, autofocus control is performed so that the face detected by the face detection unit 18 is in focus. Alternatively, for example, exposure control (control on the opening degree of the aperture and control on the exposure time of the image sensor) is performed so that the exposure amount for the face detected by the face detection unit 18 is optimized. Alternatively, for example, auto white balance control is performed on the face image detected by the face detection unit 18. Further, at the time of shooting and recording a moving image, each frame image forming the moving image is compressed by a predetermined compression method and then recorded on the recording medium 16. In this image compression process, the face of the face detection unit 18 is recorded. The detection result can also be used. For example, in the image compression process, it is possible to increase the amount of code assigned to the face part more than other parts.

  The operation modes of the imaging apparatus 1 include a shooting mode capable of shooting and recording still images or moving images, and a playback mode for playing back and displaying still images or moving images recorded on the recording medium 16 on the display unit 15. included. Transition between the modes is performed according to the operation on the operation unit 17. In the shooting mode, the imaging unit 11 sequentially performs shooting at a predetermined frame period (for example, 1/60 seconds). The following description is an explanation of the operation in the shooting mode unless otherwise specified.

  Now, every time the frame period elapses, the first, second, third,..., (N-2) th, (n-1) th and nth frames come in this order (n is (Integer greater than or equal to 3), first, second, third,..., (N-2), (n-1) and nth frames are taken as the first, The second, third,..., (N-2) th, (n-1) th and nth frame images are referred to. The first to nth frame images arranged in time series form a moving image. Each frame image is given to the face detection unit 18 as an input image to the face detection unit 18. The first to nth frame images forming the moving image can include a still image obtained in response to a still image shooting instruction to the operation unit 17 and recorded on the recording medium 16, and also includes a through display image. Can be. The through display image is an image acquired from the imaging unit 11 in order to update the display content of the display unit 15 even when a still image shooting instruction is not given.

  FIG. 2 shows an internal block diagram of the face detection unit 18. As shown in FIG. 2, the face detection unit 18 includes a face detection processing unit 21 and a face detection stabilization processing unit 22. An input image (each frame image) to the face detection unit 18 is given to the face detection processing unit 21 and the face detection stabilization processing unit 22. In the following description, an input image simply refers to an input image for the face detection unit 18. Further, in the following description, it is assumed that each input image includes a face unless otherwise specified.

  For each input image, the face detection processing unit 21 detects a human face from the input image based on the image data of the input image, and extracts a face region including the detected face. Various methods are known as a method for detecting a face included in an image, and the face detection processing unit 21 can employ any method. For example, a face (face region) may be detected by extracting a skin color region from an input image as in the method described in Japanese Patent Application Laid-Open No. 2000-105819, or Japanese Patent Application Laid-Open No. 2006-21111 or Japanese Patent Application Laid-Open No. 2006. A face (face area) may be detected using the method described in Japanese Patent No. -72770.

  Typically, for example, the image of the region of interest set in the input image is compared with a reference face image having a predetermined image size to determine the similarity between both images, and the region of interest is determined based on the similarity. It is detected whether or not a face is included (whether or not the region of interest is a face region). The similarity determination is performed by extracting a feature amount effective for identifying whether the face is a face. The feature amount is a horizontal edge, a vertical edge, a right diagonal edge, a left diagonal edge, or the like.

  In the input image, the region of interest is shifted in the horizontal direction or the vertical direction pixel by pixel. Then, the image of the region of interest after the shift is compared with the reference face image, the similarity between both images is determined again, and the same detection is performed. In this way, the region of interest is updated and set, for example, while being shifted pixel by pixel from the upper left to the lower right of the input image. In addition, the input image is reduced at a certain rate, and the same face detection process is performed on the reduced image. By repeating such processing, a face of any size can be detected from the input image.

  Further, the face detection processing unit 21 is configured to be able to detect the inclination of the face in each input image. For example, by rotating the input image and performing face detection processing similar to the above on the rotated image, it is possible to detect a tilted face and to detect the tilt (tilt angle) of the face. Here, the face inclination means the inclination of the face with respect to the vertical direction of the input image, for example, the inclination of the straight line connecting the center of the mouth and the eyebrows with respect to the straight line parallel to the vertical direction of the input image.

  The face detection processing unit 21 further detects the orientation of the face in each input image. That is, for example, the face detection processing unit 21 detects whether the face detected in each input image is a front face (face viewed from the front) or a side face (face viewed from the side). Is possible. Various methods have been proposed as methods for detecting the orientation of a face, and the face detection processing unit 21 can employ any method.

  For example, as in the technique described in Japanese Patent Laid-Open No. 10-307923, face parts such as eyes, nose and mouth are found in order from the input image, and the position of the face in the image is detected. The direction of the face is detected based on the projection data of the part.

  Alternatively, for example, a method described in JP 2006-72770 A may be used. In this method, one front face is divided into a left half (hereinafter referred to as a left face) and a right half (hereinafter referred to as a right face), and parameters related to the left face and right face are preliminarily determined through a learning process. Generate in advance. At the time of face detection, the region of interest in the input image is divided into left and right, and the similarity between each divided region and the corresponding parameter of the two parameters is calculated. Then, when one or both of the similarities are equal to or greater than the threshold, it is determined that the region of interest is a face region. Furthermore, the orientation of the face is detected from the magnitude relationship of the similarity to each divided region.

  The face detection processing unit 21 outputs face detection information representing a face detection result by itself. When a face is detected from an input image by the face detection processing unit 21, face detection information for the input image is “face position, face size, face tilt, and face orientation” on the input image. Is identified. Actually, for example, the face detection processing unit 21 extracts a rectangular area including the face as a face area, and expresses the position and size of the face by the position and image size of the face area on the input image. The face position indicates, for example, the center position of the face area for the face. The face detection information for each input image is given to the face detection stabilization processing unit 22. If no face is detected by the face detection processing unit 21, face detection information is not generated and output. Instead, information indicating that fact is transmitted to the face detection stabilization processing unit 22.

  Incidentally, face detection by the face detection processing unit 21 does not succeed under all conditions, and face detection may not be performed stably. For example, there is a limit to the direction of the face that can be detected, and face detection may fail near the limit. In addition, for example, face detection may fail due to noise, facial expression change that the face detection processing unit 21 does not assume, image blur due to face movement, sudden light source change or occlusion. Further, for example, a face search area may be thinned out to reduce the amount of face detection calculation, and in this case, the detected face position may fluctuate.

  As described above, face detection may not be performed stably only by the face detection processing unit 21, but in the face detection unit 18, the face detection stabilization processing unit 22 ensures the stability of face detection. The face detection stabilization processing unit 22 stabilizes face detection based on the input images for a plurality of frames and the face detection information from the face detection processing unit 21, thereby generating and outputting the stabilized face information. Stabilization of face detection means suppressing face detection interruptions and wobbling. The stabilized face information represents a face detection result after the face detection is stabilized. The stabilized face information includes at least information for specifying a face position and a face size on the focused input image, and further includes information for identifying a face inclination and a face orientation on the focused input image. sell.

  The stabilized face information for the focused input image is transmitted to the main control unit 13 in FIG. 1 as information representing the face detection result of the face detection unit 18 for the input image. That is, the main control unit 13 performs various processes (image compression processing, autofocus control, etc.) as described above based on the face position and size specified by the stabilized face information.

  FIG. 3 is an internal block diagram of the face detection stabilization processing unit 22. As shown in FIG. 3, the face detection stabilization processing unit 22 includes an area setting unit 31, an image change determination unit 32, and a stabilized face information generation unit 33.

  Now, for the sake of concrete explanation, it is assumed that the input image for which face detection is to be stabilized is the nth frame image, and that the stabilized face information for the nth frame image is generated (this assumption is This applies to all of the following descriptions unless otherwise stated). An input image (in the present example, the nth frame image) on which face detection is to be stabilized is also referred to as a “stabilization target image”.

  Schematically, the face detection stabilization processing unit 22 monitors the face area and image changes around the face area, and determines the position of the face to be finally detected in consideration of the image change.

  For example, in a certain section, it can be determined that the face itself and the face position have not changed if there is almost no image change in the face area and the periphery of the face area. Also, for example, if there is no image change around the face area, it can be determined that the face itself may have moved but the face position has not changed. When there is an image change in both the face area and the periphery of the face area, it can be determined that the movement of the face position has occurred.

  For example, if it can be determined that the face position has not changed, the face detection result for the frame image before the stabilization target image can be used as the face detection result for the stabilization target image, as shown in FIG. it can. Thereby, even if face detection of the face detection processing unit 21 with respect to the stabilization target image fails, occurrence of a face detection interruption or the like can be suppressed.

  The function of each unit in the face detection stabilization processing unit 22 of FIG. 3 will be schematically described (detailed description will be provided in each example described later).

  The area setting unit 31 is configured to monitor a central vision area and a peripheral vision area (for example, a central vision area 220 and a peripheral vision area in FIG. 5A to be described later) based on face detection information about the area setting image. 230) is set. When the stabilization target image is the nth frame image, the area setting image is the (n−1) th frame image.

  The image change determination unit 32 detects each image change in the central view area and the peripheral view area for the stabilization target image. The image changes in the central vision area and the peripheral vision area for the stabilization target image are image changes in the central vision area and the peripheral vision area between the two image change detection images including the stabilization target image. . When the stabilization target image is the nth frame image, the two image change detection images are the (n−1) th and nth frame images. The processing performed by the image change determination unit 32 is collectively referred to as “image change detection processing”.

  The stabilization face information generation unit 33 is based on the result of the image change detection process by the image change determination unit 32 and the face detection information about the reference past image and the stabilization target image, and the stabilization face for the stabilization target image. Generate and output information. When the stabilization target image is the nth frame image, the reference past image is the (n−1) th frame image.

  Below, the 1st-4th Example is described as a detailed Example related to the face detection function of the imaging device 1 of FIG.

<< First Example >>
First, the first embodiment will be described. In the first embodiment, an area setting method by the area setting unit 31 will be described. As area setting methods that can be used by the area setting unit 31, first to fourth area setting methods are individually illustrated. In the present specification, two words, an area and a region, are used as words representing a certain part in the image, but both are synonymous.

[First area setting method]
The first area setting method will be described. Reference is made to FIG. In FIG. 5A, reference numeral 200 denotes an (n−1) th frame image as an area setting image. A broken-line rectangular area denoted by reference numeral 210 is a face area extracted from the (n−1) th frame image by the face detection processing unit 21 in FIG. 2. The position and size of the face area 210 are specified by face detection information of the (n−1) th frame image.

  In FIG. 5A, a solid line rectangular area denoted by reference numeral 220 is a central viewing area (reference area) including the face area 210. The central vision area 220 is set based on the position, size, and orientation of the face with respect to the face area 210. Typically, for example, the center of the central viewing area 220 and the center of the face area 210 are made to coincide. Further, as shown in FIG. 5B, both centers may be shifted according to the orientation of the face. The size (image size) of the central viewing area 220 is set to a size proportional to the size (image size) of the face area 210 or a certain size according to the size of the face area 210.

  In FIG. 5A, a square-shaped hatched area denoted by reference numeral 230 is a peripheral vision area (peripheral area) arranged around the central vision area 220. The peripheral vision area 230 is an area that is adjacent to the central vision area 220 and is disposed so as to surround the entire circumference of the central vision area 220, and the central vision area 220 and the peripheral vision area 230 do not overlap each other. Therefore, the face area 210 does not exist in the peripheral vision area 230. The peripheral vision area 230 is typically a portion obtained by omitting the central vision area 220 from a rectangular region having a size larger than that of the central vision area 220 and the center of the central vision area 220 being coincident with the center.

  When the first area setting method is adopted, the area setting unit 31 in FIG. 3 sets the central vision area 220 and the peripheral vision area 230 based on the face detection information of the area setting image, and the central vision area on the input image. The position and size of 220 and the peripheral vision area 230 are transmitted to the image change determination unit 32. The image change determination unit 32 sets a central viewing area 220 and a peripheral viewing area 230 for two image change detection images (in the present example, the (n−1) th and nth frame images), and 2 Each image change in the central vision area 220 and the peripheral vision area 230 is detected between two image change detection images. Thereby, it is possible to grasp the movements in the central vision area 220 and the peripheral vision area 230 (whether the face has moved).

  FIG. 5A shows an example when the face is not tilted, but when the face is tilted, the central vision area 220 and the peripheral vision area 230 are tilted according to the tilt of the face ( The same applies to the second to fourth area setting methods).

[Second area setting method]
Next, the second area setting method will be described. Reference is made to FIGS. 6 (a) and 6 (b). In FIG. 6A, reference numeral 200 denotes an (n−1) th frame image as an area setting image, and the face area 210 and the central viewing area 220 are the same as those in FIG.

  In the second area setting method, the peripheral vision area is divided into four, and the same peripheral vision area 230 as shown in FIG. 5A is formed by the upper area 231, the lower area 232, the left area 233, and the right area 234. It is formed. FIG. 6B is a diagram in which only the peripheral vision area 230 is extracted. For simplification of description, the upper area, the lower area, the left area, and the right area are also simply referred to as areas.

  The forehead side of the face that matches the upper side of the page of FIG. 6A is defined as the upper side, and the chin side of the face that matches the lower side of the page of FIG. The upper area 231 is an upper partial rectangular area in the peripheral vision area 230, the lower area 232 is a lower partial rectangular area in the peripheral vision area 230, and the left area 233 is a left side in the peripheral vision area 230. The right area 234 is a right-side partial rectangular area in the peripheral vision area 230. The area obtained by combining the areas 231 to 234 is the peripheral vision area 230, and the areas 231 to 234 do not overlap each other.

  When the second area setting method is adopted, the area setting unit 31 in FIG. 3 sets the central viewing area 220 and the areas 231 to 234 based on the face detection information of the area setting image, and the central viewing area on the input image. 220 and the positions and sizes of the areas 231 to 234 are transmitted to the image change determination unit 32. The image change determination unit 32 sets the central vision area 220 and the areas 231 to 234 for two image change detection images (in this example, the (n−1) th and nth frame images), and 2 Each image change in the central vision area 220 and the areas 231 to 234 is detected between two image change detection images.

  In addition, since the human body exists below the face area, the human body is drawn in the lower area 232. And even if the position of the face has not changed, the body can move. In this embodiment, it is determined whether or not the position of the face has changed by monitoring changes in the vicinity of the face area, and thereby trying to stabilize face detection, so that the position of the face is not changed. Such body movement becomes a noise component for the stabilization of face detection. Considering the movement of the body, it is possible to take a method of setting a larger threshold for determining whether or not a change has occurred around the face area, but this can be dealt with by dividing the peripheral vision area It is possible.

  That is, in the image change detection process of the image change determination unit 32, a part of the areas 231 to 234 may be ignored. Which of the areas 231 to 234 is to be ignored can be determined according to the face detection information of the area setting image. For example, in the example of FIG. 6A, the lower area 232 that is an area on the jaw side of the face (that is, an area including a part of the body) is ignored in the image change detection process. The face detection processing unit 21 (FIG. 2) capable of detecting a face can naturally recognize the forehead side and the chin side of the face, and the recognition content is also included in the face detection information.

  In addition, an inclination sensor (not shown) capable of detecting the direction of gravity can be provided in the imaging apparatus 1, and in this case, any of the areas 231 to 234 is based on the detection result of the inclination sensor. You may decide to ignore. In other words, from the direction of gravity detected by the tilt sensor, it is specified by specifying which of the areas 231 to 234 is the area on the chin side of the face (that is, an area including a part of the body). Are ignored in the image change detection process.

  In the image change detection process, any one of the areas 231 to 234 is not completely ignored, but the image change detection process is performed after weighting each of the areas 231 to 234. May be. For example, by making the weight of the lower area 232, which is the area on the jaw side of the face, smaller than the weight of the other area (231, 233 or 234), the degree of contribution that the former image change gives to the result of the image change detection process Is smaller than that of the latter. For a certain area, setting the weight to zero is equivalent to ignoring that area in the image change detection process.

[Third area setting method]
Further, instead of dividing the peripheral vision area as in the second area setting method, it is possible to cope with body movement by making the shape of the peripheral vision area U-shaped as shown in FIG. This is the third area setting method.

  In FIG. 7, reference numeral 200 denotes an (n−1) th frame image as an area setting image, and the face area 210 and the central viewing area 220 are the same as those in FIG. A U-shaped oblique line area denoted by reference numeral 230a is a peripheral vision area according to the third area setting method, and the peripheral vision area 230a is an area obtained by combining the areas 231, 233, and 234 of FIG. It corresponds to.

  When the third area setting method is adopted, the area setting unit 31 in FIG. 3 sets the central vision area 220 and the peripheral vision area 230a based on the face detection information of the area setting image, and the central vision area on the input image. The position and size of 220 and the peripheral vision area 230 a are transmitted to the image change determination unit 32. The image change determination unit 32 sets the central view area 220 and the peripheral view area 230a for two image change detection images (in the present example, the (n−1) th and nth frame images), and 2 Each image change in the central vision area 220 and the peripheral vision area 230a is detected between two image change detection images.

[Fourth area setting method]
Next, the fourth area setting method will be described. Please refer to FIG. In FIG. 8, reference numeral 200 denotes an (n-1) th frame image as an area setting image, and the face area 210 and the central viewing area 220 are the same as those in FIG.

  In FIG. 8, reference numeral 240 denotes a first peripheral vision area arranged around the central vision area 220. The first peripheral vision area 240 is a square-shaped region and is the same as the peripheral vision area 230 in FIG. 5A (however, the size may be different). That is, the first peripheral vision area 240 is an area that is adjacent to the central vision area 220 and is disposed so as to surround the entire circumference of the central vision area 220. The central vision area 220 and the first peripheral vision area 240 overlap each other. Do not fit. The first peripheral vision area 240 is typically a portion obtained by omitting the central vision area 220 from a rectangular area having a size larger than that of the central vision area 220 and the center of the central vision area 220 being coincident with the center. .

  In FIG. 8, a square-shaped hatched area denoted by reference numeral 250 is a second peripheral vision area arranged around the first peripheral vision area 240. The second peripheral vision area 250 is an area that is adjacent to the first peripheral vision area 240 and is disposed so as to surround the entire circumference of the first peripheral vision area 240. The first peripheral vision area 240 and the second peripheral vision area 240 250 do not overlap each other.

  When the fourth area setting method is adopted, the area setting unit 31 in FIG. 3 sets the central vision area 220, the first peripheral vision area 240, and the second peripheral vision area 250 based on the face detection information of the area setting image. Then, the positions and sizes of the central vision area 220, the first peripheral vision area 240, and the second peripheral vision area 250 on the input image are transmitted to the image change determination unit 32. The image change determination unit 32 performs the central view area 220, the first peripheral view area 240, and the second view for two image change detection images (in the present example, the (n−1) th and nth frame images). The peripheral vision area 250 is set, and each image change in the central vision area 220, the first peripheral vision area 240, and the second peripheral vision area 250 is detected between the two image change detection images.

  The peripheral vision area in the fourth area setting method is an area obtained by combining the first peripheral vision area 240 and the second peripheral vision area 250, and the peripheral vision area is hierarchized. By hierarchizing the peripheral vision area, a more advanced image change state can be detected. For example, it is possible to accurately distinguish the case where the position of the face has moved from the case where the face is hidden by another object. This is because when the face moves, the movement propagates in the order of the central vision area 220, the first peripheral vision area 240, and the second peripheral vision area 250, whereas the face is hidden by another object. This is because the movement propagates in the order of the second peripheral vision area 250, the first peripheral vision area 240, and the central vision area 220, and it is possible to distinguish between the two by monitoring the image change in each area. is there.

  Note that each of the first peripheral vision area 240 and the second peripheral vision area 250 may be divided into a plurality of areas as if the first area setting method was modified to the second area setting method. Then, the area on the jaw side of the face (that is, the area including a part of the body) may be ignored in the image change detection process. Further, as in the third area setting method, each of the first peripheral vision area 240 and the second peripheral vision area 250 may be a U-shaped region that does not include the area on the jaw side of the face.

  Further, for the sake of clarification and concrete explanation, it is assumed that the central vision area 220 and the peripheral vision area 230 or 230a do not overlap each other, but a part of both may overlap (FIG. 5 ( a) and FIG. 7). This is also true for other areas described as “not overlapping each other”. That is, for example, part of the area 231 and part of the central viewing area 220 in FIG. 6A, part of the area 231 and part of the area 233 in FIG. 6A, part of the central viewing area 220 in FIG. And a part of the first peripheral vision area 240 or a part of the first peripheral vision area 240 and a part of the second peripheral vision area 250 in FIG. 8 may overlap each other.

<< Second Example >>
Next, a second embodiment will be described. In the second embodiment, an image change detection method performed by the image change determination unit 32 in FIG. 3 will be described.

  The image change determination unit 32 treats each of the areas set by the area setting unit 31 as a detection target area, and detects an image change in the detection target area for each detection target area.

  For example, when the area setting unit 31 adopts the first area setting method described above, each of the central vision area 220 and the peripheral vision area 230 in FIG. 5A is handled as the detection target area. When the area setting unit 31 adopts the above-described second area setting method, each of the central vision area 220 and the areas 231 to 234 in FIG. 6A is handled as the detection target area. When the area setting unit 31 adopts the above-described third area setting method, each of the central vision area 220 and the peripheral vision area 230a in FIG. 7 is handled as a detection target area. When the area setting unit 31 adopts the above-described fourth area setting method, each of the central vision area 220, the first peripheral vision area 240, and the second peripheral vision area 250 in FIG. 8 is handled as the detection target area.

  However, an area that is ignored in the image change detection process is not a detection target area. For example, in the example of the second area setting method described above, it has been described that the lower area 232 in FIG. 6A can be ignored in the image change detection process, but in this case, the lower area 232 is not a detection target area. .

  The image change determination unit 32 detects an image change in each detection target area between two image change detection images (in the present example, the (n−1) th and nth frame images). An image change detection method will be described by focusing on one detection target area.

[Image change detection based on motion vectors]
For example, the image change determination unit 32 calculates a motion vector based on the video signal (image data) in the detection target area in each image change detection image, and the image change in the detection target area is calculated by calculating the motion vector. To detect. The motion vector calculated for the focused detection target area specifies the direction and magnitude of the motion of the image in the detection target area.

  Since the motion vector calculation method itself is known, a detailed description of the calculation method is omitted. For example, a motion vector can be calculated using a representative point matching method disclosed in Japanese Patent Application Laid-Open No. 61-201581. Alternatively, the motion vector may be calculated by detecting the optical flow in the detection target area. The optical flow can be detected based on the block matching method, the gradient method, or the LK (Lucas-Kanade) method described in Non-Patent Document 1 (particularly, pages 36 to 40).

  One motion vector may be calculated for one detection target area. However, in order to perform more advanced image change detection, the detection target area is divided into a plurality of blocks, and the motion vector for each block is calculated. Should be calculated. In this case, a plurality of motion vectors are calculated for one detection target area.

  When calculating a motion vector, an image change in the detection target area can be determined as follows. That is, when the size of the motion vector in the detection target area of interest is less than a predetermined determination threshold, it is determined that there is no image change in the detection target area, and the size of the motion vector is greater than or equal to the determination threshold In this case, it is determined that there is an image change in the detection target area. “No image change” is a concept that not only means that there is no image change, but also means that there is almost no image change. In addition, an image change in the detection target area can be rephrased as a state change in the detection target area.

  In addition, when calculating a plurality of motion vectors for one detection target area, an average value of the sizes of the plurality of motion vectors may be calculated and the average value may be compared with a predetermined determination threshold. When the average value is less than the determination threshold, it is determined that there is no image change in the target detection target area, and when the average value is equal to or greater than the determination threshold, the image change in the detection target area Judge that there is. Note that the presence or absence of image change may be detected using the maximum value of the magnitudes of the plurality of motion vectors instead of the average value of the magnitudes of the plurality of motion vectors. This is because the maximum value may be more suitable for detecting the presence / absence of an image change. Similarly, instead of the average value of the sizes of the plurality of motion vectors, the mode value, the minimum value, or the median value of the sizes of the plurality of motion vectors can be used.

[Image change detection based on camera control information]
It is also possible to detect image changes based on camera control information instead of motion vectors. This will be described.

  The camera control information is, for example, an AE evaluation value that can be used for automatic exposure control. The AE evaluation value has a value corresponding to the brightness of the image in the focused area, and is, for example, an average value of luminance signal values in the focused area. An image change (movement of the face position, etc.) can be detected by a change in the brightness of the image represented by the AE evaluation value. When using the AE evaluation value, the AE evaluation value in the detection target area in the two image change detection images is calculated, and the AE evaluation value in one image change detection image and the AE evaluation in the other image change detection image. Compare the value. If the difference value between the two is less than the predetermined difference threshold, it is determined that there is no image change in the focused detection target area. Otherwise, it is determined that there is an image change in the focused detection target area.

  The camera control information is, for example, an AF evaluation value that can be used for autofocus control. The AF evaluation value has a value corresponding to the contrast amount (edge amount) of the image in the focused area, and is calculated from, for example, an integrated value of a predetermined high-frequency component included in the image in the area. An image change (movement of the face position, etc.) can be detected by a change in the contrast amount of the image represented by the AE evaluation value. Even when the AF evaluation value is used, it is possible to detect the presence or absence of an image change in the detection target area in the same manner as when the AE evaluation value is used. That is, when the AF evaluation value is used, the AF evaluation value in the detection target area in the two image change detection images is calculated, and the AF evaluation value in one image change detection image and the other image change detection image are calculated. The AF evaluation value is compared. If the difference value between the two is less than the predetermined difference threshold, it is determined that there is no image change in the focused detection target area. Otherwise, it is determined that there is an image change in the focused detection target area.

  The camera control information is, for example, an AWB evaluation value that can be used for auto white balance control. In the auto white balance control, automatic control is performed so that the ratio of the RGB signals of the image in the focused area becomes a desired ratio, and the evaluation value that is the basis of the automatic control is the AWB evaluation value. For example, the AWB evaluation value has a value corresponding to the ratio of the RGB signals of the image in the focused area. When a motion occurs in an image in the detection target area (when a face position is moved or the like), a change occurs in the AWB evaluation value for the detection target area. Therefore, based on the AWB evaluation value, it is possible to detect the presence or absence of an image change in the detection target area in the same manner as when the AE evaluation value or the AF evaluation value is used.

  Each area such as the central vision area and the peripheral vision area set by the area setting unit 31 may be matched with the area for calculating the camera control information. For example, the area for calculating camera control information is each of a total of 256 divided areas obtained by dividing the frame image into 16 parts in the vertical direction and 16 parts in the horizontal direction. The camera control information is calculated for each, but each area (central viewing area and peripheral viewing area) set by the area setting unit 31 may be formed from one or a plurality of divided areas. That is, for example, based on the position and size of the face area extracted by the face detection processing unit 21 in FIG. 2, the central area includes one divided area including the face area or a combined area of a plurality of adjacent divided areas. Set as Eliya. In this way, it is not necessary to recalculate the camera control information for the central vision area and the peripheral vision area.

  AE evaluation value calculated outside the face detection unit 18 to execute automatic exposure control, AF evaluation value calculated outside the face detection unit 18 to execute autofocus control, or auto white balance control In order to detect the image change in the detection target area by diverting the AWB evaluation value calculated outside the face detection unit 18, an AE evaluation value suitable for detecting the image change can be detected. A part for calculating the AF evaluation value or the AWB evaluation value may be provided in the image change determination unit 32 of FIG.

  Regardless of whether the motion vector is used or the camera control information is used, the image change determination unit 32 eventually determines the image in the detection target area based on the video signal (image data) in the detection target area in each image change detection image. A change is detected.

<< Third Example >>
Next, a third embodiment will be described. In the third embodiment, a method for generating stabilized face information based on the image change detection process of the image change determination unit 32 in FIG. 3 will be described. The third embodiment is implemented in combination with the first and second embodiments described above. The central vision area and the peripheral vision area are set as in the first embodiment, and each image change in the central vision area and the peripheral vision area as the detection target area is detected as in the second embodiment.

  In the third embodiment, the peripheral vision area refers to the peripheral vision area 230 in FIG. 5A or 6B when the first or second area setting method is adopted, and the third area setting method is referred to as the third area setting method. When it is adopted, it refers to the peripheral vision area 230a of FIG. 7, and when the fourth area setting method is adopted, it refers to the composite area of the first peripheral vision area 240 and the second peripheral vision area 250 of FIG. The central vision area refers to the central vision area 220 regardless of which area setting method is used (see FIG. 5A and the like).

  The image change determination unit 32 classifies the image change state for the stabilization target image into one of four states based on the detected presence or absence of each image change in the central vision area and the peripheral vision area. This classification process is referred to as “schematic classification process”. The four states are referred to as first, second, third and fourth classification states. When classified into the fourth classification state, the image change determination unit 32 further performs “detailed classification processing”. FIG. 9 roughly shows the contents of the rough classification process and the detailed classification process.

  In the rough classification process, when it is determined that there is no image change in both the central vision area and the peripheral vision area, the classification is performed in the first classification state. In this case, there is no change from the previous frame, and it is considered that there is no change in the face position. For example, when the first area setting method corresponding to FIG. 5A is adopted, each image change in the central vision area 220 and the peripheral vision area 230 is changed between the (n−1) th and nth frame images. When it is detected as in the second embodiment and it is determined that there is no image change in both the central vision area 220 and the peripheral vision area 230, the image is classified into the first classification state.

  In the rough classification process, when it is determined that there is an image change in the central vision area but no image change in the peripheral vision area, the image is classified into the second classification state. In this case, although there is a change in the face area (and the vicinity of the face area) due to a swinging motion or the like, it is considered that the face position itself does not change.

  In the rough classification process, when it is determined that there is no image change in the central vision area but there is an image change in the peripheral vision area, the image is classified into the third classification state. It can be classified into the third classification state by the influence of noise, the shaking of the leaves around the face area, the shaking of the body, and the like. However, also in this case, it is considered that the face position itself does not change.

  In addition, when the peripheral vision area is subdivided by adopting the above-described second area setting method, when there is no image change in all the areas 231 to 234 in FIG. If it is determined that there is no image change and there is an image change in any of the areas 231 to 234, it is determined that there is an image change in the peripheral vision area.

  Further, as described in the second embodiment, since the area ignored in the image change detection process is not the image change detection target area, the image change in the ignored area is performed by the rough classification process and the detailed classification. Does not affect processing (ignored in rough classification processing and detailed classification processing). For example, in the above-described example of the second area setting method, it has been described that the lower area 232 in FIG. 6A can be ignored in the image change detection process. When 232 is ignored, a peripheral vision area including only the upper area 231, the left area 233, and the right area 234 is assumed. If there is no image change in all of the upper area 231, the left area 233, and the right area 234, it is determined that there is no image change in the peripheral vision area, and any one of the upper area 231, the left area 233, and the right area 234 is determined. If there is an image change, it is determined that there is an image change in the peripheral vision area.

  In the case of classification into the first to third classification states, it is considered that there is no change in the face position itself. For this reason, the face detection information for the previous frame image can be used as the stabilization face information for the stabilization target image, and thereby it is possible to suppress face detection interruptions and fluctuations. However, the handling may be different depending on whether or not there is face detection information of the nth frame image that is the stabilization target image. That is, for example, in the case of classification into the first to third classification states, when there is no face detection information of the nth frame image, the face detection information of the (n−1) th frame image as the reference past image is changed to the nth frame image. It is used as it is as stabilized face information for the frame image. On the other hand, when there is face detection information of the nth frame image, the face detection information itself of the nth frame image is used as stabilized face information for the nth frame image, or (n-1) th and nth frames Stabilized face information for the nth frame image is generated in consideration of both face detection information of the image.

  In the rough classification process, when it is determined that there is an image change in both the central vision area and the peripheral vision area, the stabilization target image is classified into the fourth classification state, and “detailed classification process” is further performed.

  The fourth classification state is subdivided into any of the first to third subdivision states by the detailed classification process. The first subdivided state corresponds to a state in which the face is actually moved in the real space due to a walking person walking. The second subdivided state corresponds to a state in which the imaging device 1 itself is moved by a so-called camera shake, panning or tilting operation. The third subdivided state corresponds to a state in which another object enters in front of the face and the face is hidden by the other object (occurrence of occlusion).

  The procedure of the detailed classification process will be described assuming that image change is detected based on the motion vector. In this case, first, it is determined whether the direction and magnitude of the motion vector in the central vision area are the same as the direction and magnitude of the motion vector in the peripheral vision area. Thus, it is determined that the imaging device 1 itself has moved, and the stabilization target image is classified into the second subdivided state. Note that “same” regarding the direction and size of a motion vector is a broad concept that means not only “same” but also “same” substantially. Moreover, when the gyro sensor (not shown) for detecting the angular velocity of the imaging device 1 is provided in the imaging device 1, the whole motion of the imaging device 1 obtained from the detection result of the gyro sensor is represented. It is also possible to determine whether or not the stabilization target image is classified into the second subdivided state by using the entire motion vector.

  When the direction and magnitude of the motion vector in the central vision area are not the same as the direction and magnitude of the motion vector in the peripheral vision area, the stabilization target image is classified into one of the first and third subdivided states. A distinction between the first and third subdivision states can also be made based on the motion vectors of the central vision area and the peripheral vision area.

  For example, as described in the second embodiment, one detection target area is divided into a plurality of blocks, and a plurality of motion vectors are detected for each of the central vision area and the peripheral vision area. In this case, when the position of the face moves to the right between the (n−1) th and nth frames, the direction of each motion vector in the central viewing area and the peripheral viewing area as shown in FIG. Is rightward (direction going out of the central vision area) and a rightward motion vector having a significant magnitude only on the right side in the peripheral vision area is detected. If such a motion vector is detected, the stabilization target image is classified into the first subdivided state. Also, when another object enters the front of the face from the right side of the face between the (n−1) th and nth frames, as shown in FIG. 10B, each of the central vision area and the peripheral vision area The direction of the motion vector is leftward (direction entering the central vision area), and a leftward motion vector having a significant magnitude only on the right side is detected in the peripheral vision area. If such a motion vector is detected, the stabilization target image is classified into the third subdivided state.

  When classified into the fourth classification state, the stabilized face information generation unit 33 in FIG. 3 performs central vision between the face detection information of the (n−1) th frame image and the nth and (n−1) th frame images. The position of the face in the nth frame image is estimated based on the motion vectors of the area and the peripheral vision area.

For example, when the face position moves rightward between the (n−1) th and nth frames and a motion vector as shown in FIG. 10A is detected, the nth frame image is divided into the first subdivision. And the position of the face in the nth frame image is estimated to be shifted to the right from the position of the face specified by the face detection information of the (n−1) th frame image. The direction and amount of deviation can be determined from the direction and magnitude of the motion vector).
Further, for example, when another object enters the front of the face from the right side of the face between the (n−1) th and nth frames and a motion vector as shown in FIG. The frame image is classified into the third subdivision state, and the face position in the nth frame image is estimated to be the same as the face position specified by the face detection information of the (n−1) th frame image.
Further, for example, when the nth frame image is classified into the second subdivision state, the first frame image is divided by an amount corresponding to the motion vector of the central vision area and the peripheral vision area between the nth and (n−1) th frame images. It is estimated that the position of the face in the n frame image is deviated from the position of the face specified by the face detection information of the (n−1) th frame image (the direction and amount of deviation are the direction and magnitude of the motion vector). Can be determined). The face position estimated as described above is referred to as an estimated face position.

  When classified into the fourth classification state, the method for generating stabilized face information differs depending on whether or not there is face detection information of the nth frame image that is the stabilization target image.

  When the nth frame image is classified into the fourth classification state and there is no face detection information of the nth frame image, stabilized face information of the nth frame image is generated from the estimated face position. That is, the estimated face position is set as the face position specified by the stabilized face information of the nth frame image. The size of the face specified by the stabilized face information of the nth frame image is the same as that of the (n−1) th frame image.

  When the nth frame image is classified into the fourth classification state and there is face detection information of the nth frame image, either of the estimated face position and the face position specified by the face detection information of the nth frame image Is the face position in the stabilized face information of the nth frame image. Basically, the face closer to the face position in the stabilized face information of the (n−1) th frame image is set as the face position in the stabilized face information of the nth frame image. Thereby, the fluctuation of the detected face position is suppressed. Note that the size of the face specified by the stabilized face information of the nth frame image may follow the face detection information of the nth frame image.

<< 4th Example >>
Next, a fourth embodiment will be described. In the fourth embodiment, as in the third embodiment, a method for generating stabilized face information based on the image change detection process of the image change determination unit 32 in FIG. 3 will be described. The fourth embodiment is also implemented in combination with the first and second embodiments described above. However, in the fourth embodiment, only the case where the above-described fourth area setting method is adopted in the area setting unit 31 of FIG. 3 is assumed.

  Even when the fourth area setting method is adopted, the rough classification process is the same as that in the third embodiment. When the stabilization target image is classified into the fourth classification state in the rough classification process, the detailed classification process is executed. The adoption of the fourth area setting method is particularly useful for detecting the occurrence of occlusion, which is involved in the distinction between the first and third subdivided states. The principle of detection of occurrence of occlusion will be described with reference to FIGS.

  It is assumed that the face position in the input image is stationary between the (n−2) th frame and the nth frame. In this case, the central viewing area 220, the first peripheral viewing area 240, and the second peripheral viewing area 250 are set at the same position for each of the (n−2) th to nth frame images. Then, it is assumed that the object 300 enters the face from the right side of the face between the (n−2) -th frame and the n-th frame. 11 (a), (b), and (c) respectively show a central vision area 220 set in the (n-2) th frame image, the (n-1) th frame image, and the nth frame image, respectively. The 1 peripheral vision area 240 and the 2nd peripheral vision area 250 are shown with the said object 300. FIG.

  The object 300 is located in the second peripheral vision area 250 in the (n-2) th frame image, but the object 300 moves to the left between the (n-2) th and (n-1) th frames. Thus, it is assumed that the object 300 is located in the first peripheral vision area 240 in the (n−1) th frame image. Then, it is assumed that the object 300 is positioned in the central vision area 220 in the nth frame image by further moving the object 300 leftward between the (n−1) th and nth frames.

  In this case, an image change appears in both the first peripheral vision area 240 and the second peripheral vision area 250 between the (n-2) th and (n-1) th frame images, and the (n-1) th And between the nth frame images, image changes appear in both the first peripheral vision area 240 and the central vision area 220. By detecting such an image change, it is possible to detect the occurrence of facial occlusion by the object 300.

  Specifically, the detailed classification process is executed as follows. When the object 300 moves as shown in FIGS. 11B and 11C, it is determined that there is an image change in both the central viewing area and the peripheral viewing area, and the stabilization target image is classified into the fourth classification. The state is classified and the detailed classification process is executed. In the detailed classification process, first, it is determined whether or not the stabilization target image is classified into the second subdivided state. This determination can be made in the same manner as in the third embodiment.

  When the stabilization target image is not classified into the second subdivision state, the first and third subdivision states are distinguished. In this distinction, the image change determination unit 32 in FIG. 3 not only changes between the (n−1) th and nth frame images, but also between the (n−2) th and (n−1) th frame images. See also image changes. Therefore, in the fourth embodiment, the image change detection image for the image change determination unit 32 in FIG. 3 includes three frame images ((n-2) to nth frame images).

  The image change determination unit 32 determines whether or not each image change in the central vision area 220, the first peripheral vision area 240, and the second peripheral vision area 250 between the (n-2) th and (n-1) th frame images. To detect. Since this detection result has already been obtained in the image change detection process when the (n−1) th frame image is set as the stabilization target image, it can be used. When it is determined that there is an image change only in the first peripheral vision area 240 and the second peripheral vision area 250 between the (n-2) th and (n-1) th frame images, the nth frame It is determined that facial occlusion has occurred in the image, and the n-th frame image (stabilization target image) is classified into the third subdivided state. Otherwise, the n-th frame image (stabilization target image) Are classified into the first subdivided state.

  The distinction between the first and third subdivided states can be performed either by detecting the presence / absence of an image change based on a motion vector or by detecting the presence / absence of an image change based on camera control information.

  The processing after the classification into any of the first to third subdivided states is almost the same as in the third embodiment. That is, the stabilized face information generation unit 33 in FIG. 3 performs motion vectors of the central vision area and the peripheral vision area between the face detection information of the (n−1) th frame image and the nth and (n−1) th frame images. Is used to estimate the position of the face in the nth frame image and determine the estimated face position. In particular, when the nth frame image as the stabilization target image is classified into the third subdivision state, the estimated face position is the same as the face position specified by the face detection information of the (n−1) th frame image. And it is sufficient. The estimation process when classified into the first or second subdivided state is the same as in the third embodiment.

  If the nth frame image is classified into the fourth classification state and there is no face detection information of the nth frame image, stabilized face information of the nth frame image is generated from the estimated face position. That is, the estimated face position is set as the face position specified by the stabilized face information of the nth frame image. The size of the face specified by the stabilized face information of the nth frame image is the same as that of the (n−1) th frame image.

  When the nth frame image is classified into the fourth classification state and there is face detection information of the nth frame image, either of the estimated face position and the face position specified by the face detection information of the nth frame image Is the face position in the stabilized face information of the nth frame image. Basically, the face closer to the face position in the stabilized face information of the (n−1) th frame image is set as the face position in the stabilized face information of the nth frame image. Thereby, the fluctuation of the detected face position is suppressed. Note that the size of the face specified by the stabilized face information of the nth frame image may follow the face detection information of the nth frame image.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 to 6 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
In the fourth area setting method corresponding to FIG. 8, the peripheral vision area is hierarchized in two stages, but the peripheral vision area may be hierarchized in three or more stages. For example, if the peripheral vision area is hierarchized in three stages, it is possible to detect the occurrence of occlusion with higher accuracy based on four frame images.

[Note 2]
The area setting unit 31 in FIG. 3 sets a central vision area and a peripheral vision area where image changes should be monitored based on face detection information about the area setting image. In this case, in the above-described example, when the stabilization target image is the nth frame image, the area setting image is the (n−1) th frame image.

  Since we want to monitor the face area and image changes around the face area based on the nth frame image, it is desirable to set the above area using face detection information of the past frame image as close as possible (distant past frames The face position in the image is likely to be far from that in the nth frame image). Therefore, it is desirable to handle the (n−1) th frame image as the area setting image, but a past frame image (for example, the (n−2) th frame image) is used as the area setting image. It is also possible.

  If the area setting image is the (n-2) th frame image, the central viewing area and the peripheral viewing area for the nth frame image are set based on the face detection information of the (n-2) th frame image. The

[Note 3]
The stabilization face information generation unit 33 in FIG. 3 is based on the result of the image change detection process by the image change determination unit 32 and the face detection information on the reference past image and the stabilization target image. Generate stabilized face information. In this case, in the above-described example, when the stabilization target image is the nth frame image, the reference past image is the (n−1) th frame image.

  Since the reference past image is used to estimate the position of the face in the stabilization target image, it should be a past frame image as close as possible. Therefore, it is desirable to treat the (n-1) th frame image as the reference past image, but a past frame image (for example, the (n-2) th frame image) may be used as the reference past image. It is possible.

  For example, even when the reference past image is the (n-2) th frame image, the nth frame image as the stabilization target image is classified into the first to third classification states, and the face detection information of the nth frame image is When there is not, it is possible to perform processing such that the face detection information itself of the (n−2) th frame image is used as the stabilized face information of the nth frame image, thereby effectively preventing the face detection from being interrupted.

  As can be seen from the annotation 2 and the main annotation (annotation 3), the area setting image and the reference past image for the same stabilization target image can be different from each other.

[Note 4]
In the above-described embodiment, the face position is finally detected in consideration of the image change in both the central vision area and the peripheral vision area, but the face detection is performed only by considering only the image change in the peripheral vision area. Contributes to the stabilization of For example, when there is face detection information of the (n-1) th frame image and no face detection information of the nth frame image as the stabilization target image, the periphery between the (n-1) th and nth frame images When it is determined that there is no image change in the viewing area (such as the peripheral vision area 230 in FIG. 5A), it is estimated that there is no change in the face position. It is possible to perform processing such that the information itself is the stabilized face information of the nth frame image. This can effectively prevent face detection interruptions and the like.

[Note 5]
In the above-described embodiment, the face detection unit 18 as an object detection device is provided in the imaging device 1, and a target (a specific type of object) to be detected from the input image is a human face. It is not limited. That is, it is also possible to handle objects other than the face as objects to be detected from the input image. For example, if the object to be detected from the input image is a vehicle, the present invention can be applied to a parking lot management system or the like. Detection of an object other than a face can also be realized by using a known method (pattern matching or the like).

[Note 6]
The imaging apparatus 1 in FIG. 1 can be realized by hardware or a combination of hardware and software. In particular, the function of each part shown in FIGS. 2 and 3 can be realized by hardware, software, or a combination of hardware and software. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part.

  Further, all or part of the functions realized by the face detection unit 18 are described as a program, and the program is executed on a program execution device (for example, a computer) to realize all or part of the functions. You may make it do.

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is an internal block diagram of the face detection part of FIG. FIG. 3 is an internal block diagram of a face detection stabilization processing unit in FIG. 2. It is a figure for demonstrating the schematic function of the face detection stabilization process part of FIG. FIGS. 5A and 5B show examples of the central viewing area and the peripheral viewing area (first area setting method) set by the area setting unit of FIG. 3 according to the first embodiment of the present invention. FIG. 7A shows an example of the central vision area and the peripheral vision area (second area setting method) set by the area setting unit in FIG. 3 according to the first embodiment of the present invention, and only the peripheral vision area is extracted. It is the figure (b) shown. It is a figure which concerns on 1st Example of this invention and shows the example (3rd area setting method) of the central vision area and peripheral vision area which are set by the area setting part of FIG. It is a figure which shows the example (4th area setting method) of the central vision area and peripheral vision area which are related to 1st Example of this invention and are set by the area setting part of FIG. It is a figure which shows the content of the rough classification | category process and detailed classification | category process which are related to 3rd Example of this invention, and are implemented in the image change determination part of FIG. The figure which shows the motion vector which appears when a face moves to the right direction according to 3rd Example of this invention, and the figure which shows the motion vector which appears when another object approachs toward the face from the right side ( b). FIG. 6 is a diagram (a), (b), and (c) for explaining the principle of occlusion detection that can be performed when the peripheral vision area is hierarchized according to the fourth embodiment of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 18 Face detection part 21 Face detection process part 22 Face detection stabilization process part 31 Area setting part 32 Image change determination part 33 Stabilization face information generation part

Claims (9)

In an object detection apparatus for detecting the position of a specific type of object in a moving image based on the moving image,
An object detection apparatus for detecting a position of the object in consideration of an image change around the object in the moving image. In an object detection apparatus for detecting the position of a specific type of object in a moving image based on the moving image,
First detection means for tentatively detecting the position of the object in the moving image;
The position of the object in the moving image is finally determined based on the detection result of the first detection means and the image change in the reference area including the object and in the peripheral area located around the reference area. An object detection apparatus comprising: a second detection means for detecting. The moving image is formed from a plurality of input images arranged in time series,
The first detection means tentatively detects the position of the object in each input image,
The second detection means includes
Area setting means for setting the reference area and the peripheral area for the current input image based on the position of the object with respect to the past input image detected by the first detection means;
The reference area and the peripheral area set for the current input image are defined for the current and past input images, and image changes in the reference area and the peripheral area between the current and past input images are defined. Image change detecting means for detecting as an image change for the current input image,
Based on the detection result of the first detection means for the past and current input images and the image change for the current input image detected by the image change detection means, the object in the current input image The object detection apparatus according to claim 2, wherein the position is finally detected. The peripheral area is set so as to surround the entire circumference of the reference area,
The object detection apparatus according to claim 2, wherein the image change detection unit detects the image change for the peripheral area after excluding a part of the peripheral area. The object detection apparatus according to claim 2, wherein the peripheral area is set so as to surround a part of the entire circumference of the reference area. The peripheral area is hierarchized to include a first peripheral area that is relatively close to the reference area and a second peripheral area that is relatively far from the reference area,
The second detection means finally detects the position of the object based on a detection result of the first detection means and image changes in the reference area and the first and second peripheral areas. The object detection device according to claim 2. The object detection apparatus according to claim 1, wherein the specific type of object is a human face. Imaging means for acquiring a moving image according to the subject;
An imaging apparatus comprising: the object detection apparatus according to claim 1 that receives the moving image obtained by the imaging means. In an object detection method for detecting a position of a specific type of object in a moving image based on the moving image,
An object detection method for detecting a position of the object in consideration of an image change around the object in the moving image.

Images