JP2007280088A - Specific object detection device, specific object detection method, and specific object detection program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To more accurately and more efficiently detect a position and a size of a specific object in an image, by utilizing not only the relation between a distance and the size of an area for detection but a wider range of distance information to detect the specific object in an image and its size. <P>SOLUTION: A specific object in an image and its size are detected, by setting a suitable detection area according to distance, and analyzing not only the relation with size of an area for detection but also the spatial distribution of distance information in the detection area or extending the utilization of distance information to a periphery of the detection area. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a specific object detection device, a specific object detection method, and a specific object detection program for detecting the presence and position of a specific object as a subject in a captured image.

  Conventionally, a technique for detecting a specific object as a subject from a photographed image or the like has been developed. Particularly in recent years, in the field of security and the like, there has been an increasing demand for detection of a person using an image, particularly authentication of a specific individual.

  In the growing demand for so-called biometrics authentication, face recognition algorithm proposals with excellent usability are increasing.For example, even when face authentication is performed, It is necessary to specify the size.

  That is, it can be said that a technique for detecting a specific object (typically a human face) and specifying a position and a range in an image is an essential technique.

  However, since the size of the face area that exists as a subject in the captured image varies depending on the distance between the face as the subject and the photographing device, the template size is changed in various ways in the conventional template matching. Therefore, it is necessary to perform collation, which is time-consuming and time-consuming.

  Also, in terms of detection accuracy, use of only image information as in template matching has resulted in many false detections, and sufficient accuracy has not yet been obtained.

  To solve this problem, use a stereo camera as an image input device to obtain distance information about the subject in the captured image, and use the relationship between the distance and the size of the specific object to be detected. Techniques have been developed to increase accuracy and improve detection efficiency (see, for example, Patent Documents 1, 2, and 3).

  In the technique described in Patent Document 1, a stereo camera is used to detect a face and its size from an image, and then determine whether the detected face size is appropriate by comparing it with distance information. Yes. Thereby, it is possible to improve the accuracy with respect to the normal face detection.

  In the technique described in Patent Document 2, a stereo camera is used, the size of an appropriate face area is limited from the obtained distance information, and the face area is detected based on the face area size. Thereby, since the size of the face is limited before detection, the detection speed and efficiency can be improved.

The technique described in Patent Document 3 also uses a stereo camera, limits the size of the face area from the obtained distance information, and determines the size of the subject corresponding to the distance in the set detection area. Whether it is a face or not is determined depending on how much it is. As a result, improvements are made in accuracy and efficiency.
JP 2000-222576 A JP 2002-216129 A JP 2005-301833 A

  The techniques described in Patent Documents 1, 2, and 3 described above all calculate distance information of a subject in an image from a plurality of input images with different viewpoints using a stereo camera or the like, and use the distance information. Thus, the detection speed and accuracy are intended to be improved mainly from the relationship with the size of a specific object that is a human face.

  However, although these methods use distance information, they are supplementary support methods for conventional detection methods based only on the relationship between distance and size, and they are sufficiently accurate and efficient. It is not satisfactory.

  The object of the present invention is to solve the above-mentioned problems, and in using distance information to detect the position and size of a specific object in an image, not only the relationship with the size of the area to be detected, but also a wider range. To provide a specific object detection device, a specific object detection method, and a specific object detection program capable of detecting a specific object and its size in an image more accurately and more efficiently by using distance information .

  In order to solve the above problems, the present invention has the following features.

  1. Based on input image acquisition means for acquiring a plurality of input images from different viewpoints, distance image generation means for generating a distance image from the plurality of input images acquired by the input image acquisition means, and a predetermined specific object to be detected Based on the correspondence set by the correspondence setting means and the correspondence setting means for setting the correspondence between the distance value in the distance image and the size of the detection area for detecting the specific object, the distance image A detection area setting means for setting a detection area of a size corresponding to the distance value, and for the detection area set by the detection area setting means, based on a size of an area satisfying the distance value corresponding to the size An auxiliary area is set around the detection area set by the first feature quantity calculating means for calculating the first feature quantity and the detection area setting means, and the auxiliary A second feature amount calculating means for calculating a second feature amount from a relationship between a distance value between the area and the detection region, and the detection region based on the first feature amount and the second feature amount. A specific object detection device for determining whether or not the predetermined specific object is detected.

  2. In the detection area, the first feature amount calculation means uses, as the first feature amount, a ratio of the number of pixels having a distance value equivalent to the distance value corresponding to the detection region within a predetermined range to the total number of pixels. The second feature amount calculating means sets the auxiliary area above the detection area, and has the largest number of pixels having the average value, the median value, or the same distance value of the distance values in the auxiliary area. 2. The specific object detection device according to 1, wherein a distance value is compared with a distance value corresponding to the detection region, and an absolute value of the difference is calculated as a second feature amount.

  3. The specific object detection determination unit detects the specific object when the first feature value is larger than a predetermined first threshold value and the second feature value is larger than a predetermined second threshold value. 3. The specific object detection device according to 1 or 2, wherein the determination is performed.

  4). Based on input image acquisition means for acquiring a plurality of input images from different viewpoints, distance image generation means for generating a distance image from the plurality of input images acquired by the input image acquisition means, and a predetermined specific object to be detected Based on the correspondence set by the correspondence setting means and the correspondence setting means for setting the correspondence between the distance value in the distance image and the size of the detection area for detecting the specific object, the distance image A detection area setting means for setting a detection area of a size corresponding to the distance value, and for the detection area set by the detection area setting means, based on a size of an area satisfying the distance value corresponding to the size A distance corresponding to the size of the first feature value calculating means for calculating the first feature value and the detection area set by the detection area setting means. A third feature amount calculating unit configured to calculate a third feature amount based on a spatial distribution of a region satisfying the value; and the predetermined feature in the detection region based on the first feature amount and the third feature amount. And a specific object detection judging means for judging whether or not a specific object is detected.

  5. In the detection area, the first feature amount calculation means uses, as the first feature amount, a ratio of the number of pixels having a distance value equivalent to the distance value corresponding to the detection region within a predetermined range to the total number of pixels. And the third feature amount calculating means calculates a sum of distance values in each region obtained by dividing the detection region into two in the horizontal direction, and takes a horizontal difference value obtained by taking an absolute value of the difference of each sum, Alternatively, the sum of the distance values is calculated in each region obtained by dividing the detection region into two in the vertical direction, and the vertical direction difference value obtained by taking the absolute value of the difference of each sum is calculated as the third feature amount. 5. The specific object detection device according to 4,

  6). The specific object detection determination unit detects the specific object when the first feature value is larger than a predetermined first threshold value and the third feature value is smaller than a predetermined third threshold value. 4. The specific object detection device according to 4 or 5, wherein the determination is performed.

  7). The first image corresponding to the position of the detection area set in the distance image by the detection area setting means with respect to a first image that is one of a plurality of input images used for generating the distance image. A second detection area setting means for setting a second detection area at a position in the input image, and an input for detecting whether or not the specific object is detected in the second detection area. An image determination unit, and the specific object detection determination unit determines whether the predetermined specific object is detected in the detection region based on a determination result by the input image determination unit. The specific object detection device according to any one of 1 to 6.

  8). The specific object detection apparatus according to any one of 1 to 7, wherein the specific object detected and determined by the specific object detection determination unit is a face.

  9. Based on an input image acquisition step of acquiring a plurality of input images with different viewpoints, a distance image generation step of generating a distance image from the plurality of input images acquired in the input image acquisition step, and a predetermined specific object to be detected Based on the correspondence set in the correspondence setting step, the correspondence setting step for setting the correspondence between the distance value in the distance image and the size of the detection area for detecting the specific object, the distance image A detection region setting step for setting a detection region having a size corresponding to the distance value in the detection region, and for the detection region set in the detection region setting step, based on a size of a region satisfying the distance value corresponding to the size An auxiliary region is set around the detection region set in the first feature amount calculation step for calculating the first feature amount and the detection region setting step. Based on the second feature value calculating step for calculating the second feature value from the relationship between the distance value between the auxiliary region and the detection region, and the detection based on the first feature value and the second feature value. And a specific object detection determining step for determining whether or not the predetermined specific object is detected in a region.

  10. In the first feature amount calculating step, in the detection region, a ratio of the number of pixels having a distance value equivalent to the distance value corresponding to the detection region within a predetermined range to the total number of pixels is used as the first feature amount. In the second feature amount calculation step, the auxiliary region is set above the detection region, and the average value, median value, or the same distance value of the distance values in the auxiliary region is the largest. 10. The specific object detection method according to 9, wherein a distance value is compared with a distance value corresponding to the detection area, and an absolute value of the difference is calculated as a second feature amount.

  11. In the specific object detection determination step, the specific object is detected when the first feature amount is larger than a predetermined first threshold value and the second feature amount is larger than a predetermined second threshold value. 11. The specific object detection method according to 9 or 10, wherein the determination is performed.

  12 Based on an input image acquisition step of acquiring a plurality of input images with different viewpoints, a distance image generation step of generating a distance image from the plurality of input images acquired in the input image acquisition step, and a predetermined specific object to be detected Based on the correspondence set in the correspondence setting step, the correspondence setting step for setting the correspondence between the distance value in the distance image and the size of the detection area for detecting the specific object, the distance image A detection region setting step for setting a detection region having a size corresponding to the distance value in the detection region, and for the detection region set in the detection region setting step, based on a size of a region satisfying the distance value corresponding to the size The first feature value calculating step for calculating the first feature value and the size of the detection region set in the detection region setting step In the detection region based on the third feature amount calculating step for calculating the third feature amount based on the spatial distribution of the region satisfying the distance value to be performed, and on the first feature amount and the third feature amount. A specific object detection determination step for determining whether or not the predetermined specific object has been detected.

  13. In the first feature amount calculating step, in the detection region, a ratio of the number of pixels having a distance value equivalent to the distance value corresponding to the detection region within a predetermined range to the total number of pixels is used as the first feature amount. And in the third feature amount calculating step, a total sum of distance values is calculated in each region obtained by dividing the detection region into two in the horizontal direction, and a horizontal direction difference value obtained by taking an absolute value of each sum difference, Alternatively, the sum of the distance values is calculated in each region obtained by dividing the detection region into two in the vertical direction, and the vertical direction difference value obtained by taking the absolute value of the difference of each sum is calculated as the third feature amount. 13. The specific object detection method according to 12,

  14 In the specific object detection determining step, the specific object is detected when the first feature value is larger than a predetermined first threshold value and the third feature value is smaller than a predetermined third threshold value. 14. The specific object detection method according to 12 or 13, wherein the determination is performed.

  15. The first image corresponding to the position of the detection area set in the distance image in the detection area setting step with respect to a first image that is one of a plurality of input images used for generation of the distance image. A second detection region setting step for setting a second detection region at a position in the input image of the input, and an input for detecting whether or not the specific object is detected in the second detection region. An image determination step, wherein the specific object detection determination step determines whether or not the predetermined specific object is detected in the detection region based on a determination result in the input image determination step. The specific object detection method according to any one of 9 to 14.

  16. 16. The specific object detection method according to claim 9, wherein the specific object detected and determined in the specific object detection determination step is a face.

  17. A computer that controls the detection processing of the specific object; an input image acquisition unit that acquires a plurality of input images from different viewpoints; a distance image generation unit that generates a distance image from the plurality of input images acquired by the input image acquisition unit; Based on a predetermined specific object to be detected, the correspondence setting means for setting the correspondence between the distance value in the distance image and the size of the detection area for detecting the specific object, the correspondence setting means is set. Based on the correspondence, a detection area setting means for setting a detection area having a size corresponding to the distance value in the distance image, and a distance value corresponding to the size for the detection area set by the detection area setting means. A first feature amount calculating means for calculating a first feature amount based on a size of a region to be filled, and the detection region set by the detection region setting means. A second feature amount calculating means for calculating a second feature amount from a relationship between a distance value between the assist region and the detection region, and the first feature amount and the second feature. A specific object detection program that functions as specific object detection determination means for determining whether or not the predetermined specific object is detected in the detection region based on the amount.

  18. In the detection area, the first feature amount calculation means uses, as the first feature amount, a ratio of the number of pixels having a distance value equivalent to the distance value corresponding to the detection region within a predetermined range to the total number of pixels. The second feature amount calculating means sets the auxiliary area above the detection area, and has the largest number of pixels having the average value, the median value, or the same distance value of the distance values in the auxiliary area. 18. The specific object detection program according to 17, wherein a distance value is compared with a distance value corresponding to the detection area, and an absolute value of the difference is calculated as a second feature amount.

  19. The specific object detection determination unit detects the specific object when the first feature value is larger than a predetermined first threshold value and the second feature value is larger than a predetermined second threshold value. The specific object detection program according to 17 or 18, wherein the determination is performed.

  20. A computer that controls the detection processing of the specific object; an input image acquisition unit that acquires a plurality of input images from different viewpoints; a distance image generation unit that generates a distance image from the plurality of input images acquired by the input image acquisition unit; Based on a predetermined specific object to be detected, the correspondence setting means for setting the correspondence between the distance value in the distance image and the size of the detection area for detecting the specific object, the correspondence setting means is set. Based on the correspondence, a detection area setting means for setting a detection area having a size corresponding to the distance value in the distance image, and a distance value corresponding to the size for the detection area set by the detection area setting means. A first feature amount calculating means for calculating a first feature amount based on a size of a region to be filled, and the detection region set by the detection region setting means. A third feature amount calculating means for calculating a third feature amount based on a spatial distribution of a region satisfying a distance value corresponding to the size, the first feature amount and the third feature amount; Based on this, a specific object detection program that functions as specific object detection determination means for determining whether or not the predetermined specific object has been detected in the detection region.

  21. In the detection area, the first feature amount calculation means uses, as the first feature amount, a ratio of the number of pixels having a distance value equivalent to the distance value corresponding to the detection region within a predetermined range to the total number of pixels. And the third feature amount calculating means calculates a sum of distance values in each region obtained by dividing the detection region into two in the horizontal direction, and takes a horizontal difference value obtained by taking an absolute value of the difference of each sum, Alternatively, the sum of the distance values is calculated in each region obtained by dividing the detection region into two in the vertical direction, and the vertical direction difference value obtained by taking the absolute value of the difference of each sum is calculated as the third feature amount. 21. The specific object detection program according to 20,

  22. The specific object detection determination unit detects the specific object when the first feature value is larger than a predetermined first threshold value and the third feature value is smaller than a predetermined third threshold value. The specific object detection program according to 20 or 21, wherein the determination is performed.

  23. The computer corresponds to the position of the detection area set in the distance image by the detection area setting means with respect to a first image that is one of a plurality of input images used to generate the distance image. Second detection area setting means for setting a second detection area at a position in the first input image, detecting the specific object with respect to the second detection area, and determining whether or not it has been detected. Functioning as an input image determination unit for determining, wherein the specific object detection determination unit determines whether the predetermined specific object has been detected in the detection region based on a determination result by the input image determination unit. The specific object detection program according to any one of 17 to 22, characterized by:

  24. The specific object detection program according to any one of claims 17 to 23, wherein the specific object detected and determined by the specific object detection determination means is a face.

  According to the present invention, in using distance information to detect the position and size of a specific object in an image, an appropriate detection area corresponding to the distance is set and only the relationship with the size of the area to be detected is set. Rather than analyzing the spatial distribution of distance information in the detection area or extending the use of distance information to the periphery of the detection area, the specific object and its size in the image can be more accurately and more efficiently A specific object detection apparatus, a specific object detection method, and a specific object detection program that can be detected can be provided.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Functional configuration of specific object detection device)
FIG. 1 is a block diagram showing a functional configuration of a specific object detection device according to an embodiment of the present invention. An embodiment of the specific object detection apparatus will be described with reference to FIG.

  The specific object detection apparatus according to the present invention is generally configured mainly with an image information processing function by a computer or the like.

  In FIG. 1, reference numeral 10 denotes a specific object detection processing unit, which has various image information processing functions for detecting a specific object. Specifically, the function is executed by the computer executing each processing program. Is realized. An input unit 101, an operation unit 102, a storage unit 103, a display unit 104, and an output unit 105 are auxiliary devices or interface devices for operating the functions of the specific object detection processing unit 10. .

  The input unit 101 inputs an image that is a target for detecting a specific object to the specific object detection processing unit 10. An imaging device may be built in or connected, or image data may be received from the outside.

  The operation unit 102 is a device for inputting necessary instructions and parameters when the specific object detection processing unit 10 executes the function. In general, a keyboard or the like is used.

  In addition to the processing program and the image data itself, the storage unit 103 stores various types of data that the specific object detection processing unit 10 should temporarily store while executing the processing. For example, the detection determination result in the middle stage before the final detection result is obtained is temporarily stored.

  The display unit 104 performs a display for informing the situation during execution of the specific object detection processing unit 10 and a result necessary for issuing an instruction. Specifically, a display device such as a liquid crystal panel is generally used.

  The output unit 105 is a device that outputs the processing result of the specific object detection processing unit 10, specifically, a printer, a transmission device to an external device, or arranged for reuse such as authentication. It may be a device that sends it to the storage unit in a state that has been done.

  The specific object detection processing unit 10 includes the following functional elements.

  Reference numeral 11 denotes an input image acquisition unit that acquires an input image from the input unit 101. That is, it functions as an input image acquisition unit.

  In the present embodiment, the input image is composed of a plurality of input images having different viewpoints. Generally, it is a plurality of images for generating a stereo image. That is, the present invention intends to perform more efficient specific object detection by using distance information in an image.

  Various objects can be assumed as the specific object, but what has been actively performed in recent years is to use an image, particularly a face image, in order to specify a person. Development is being promoted as a simple personal authentication method that places little burden on individuals from security and other viewpoints. To this end, it is first required to easily and accurately identify the presence and position of a face in an image. . In the present embodiment, the following description will be given by taking a face as an example of the specific object.

  A distance image generation unit 12 generates a distance image in which each pixel has distance information using a plurality of input images received from the input image acquisition unit 11. That is, it functions as a distance image generation means. Details of the distance image generation will be described later in detail with reference to FIG.

  Reference numeral 13 denotes a correspondence setting unit, which determines the correspondence between the distance value in the distance image and the size of the detection area for detecting the specific object based on a predetermined specific object to be detected (a face in the present embodiment). Set. That is, it functions as correspondence setting means.

  The size of the specific object, that is, the face, changes in the image according to the distance. However, for efficient detection, it is desirable to set the detection area size according to the face size. If the relationship between the distance and the face size in the image can be set in advance, it is possible to set the size of the detection region that matches the face size using the distance information.

  The relationship between the distance and the general face size, that is, the detection area size is obtained from the imaging conditions. Since the set correspondence is frequently used during the specific object detection process, it is desirable to store it in the storage unit 103 in the form of a data table or the like.

  Reference numeral 20 denotes a detection region processing unit, which sets a detection region based on the correspondence relationship of the correspondence relationship setting unit 13 and performs a specific object detection process for each detection region. Each functional element constituting the detection area processing unit 20 will be described later.

  A specific object detection processing control unit 14 controls the entire specific object detection process performed by the specific object detection processing unit 10. The details of the operation flow will be described later with reference to FIG. 2, but the processing performed by the detection region processing unit 20 described above, that is, the detection processing performed by setting the detection region is performed while scanning the detection region over the entire image. It repeats and controls the flow of finally submitting the entire result while repeating the detection determination.

  Reference numeral 15 denotes a specific object detection processing output unit. The detection region processing repeatedly performed by the detection region processing unit 20, that is, the detection results for each detection region are collected, and the final specific object detection processing performed by the specific object detection processing unit 10 is finalized. As a detection result, it has a function of sending it to the output unit 105.

  The functional elements constituting the specific object detection processing unit 10 are as described above, but the detection area processing unit 20 includes the following functional elements.

  Reference numeral 21 denotes a detection area setting unit. Based on the correspondence relationship of the correspondence relationship setting unit 13, according to the position in the distance image generated by the distance image generation unit 12, the size corresponding to the distance value at that position is detected. Set the area. That is, it functions as detection area setting means. Details will be described later with reference to FIG.

  Reference numeral 23 denotes a first feature amount calculation unit. For the detection region set by the detection region setting unit 21, the ratio of the pixel region having a distance value equivalent to the distance value corresponding to the size to the entire region is calculated. Calculated as the first feature amount. That is, it functions as a first feature amount calculation unit.

  The first feature amount is a reference for determining whether the size of the detection area is appropriate if it is a face with respect to the size of the detection region. Details will be described later with reference to FIG.

  Reference numeral 24 denotes a second feature amount calculation unit, which sets an auxiliary region above the detection region set by the detection region setting unit 21, and calculates an average value, a median value, or a distance value in the auxiliary region. The distance value having the largest number of pixels having the same distance value is compared with the distance value corresponding to the detection area size, and the absolute value of the difference is calculated as the second feature amount. That is, it functions as a second feature amount calculation means.

  The second feature amount indicates whether or not there is an object of the same distance above the position of the detection area, and serves as a reference for determining whether or not the subject at the position of the detection area is a face. Details will be described later with reference to FIG.

  Reference numeral 25 denotes a third feature amount calculation unit, which calculates the sum of distance values in each region divided into two in the horizontal direction or the vertical direction with respect to the detection region set by the detection region setting unit 21, and calculates each sum. A horizontal direction difference value or a vertical direction difference value obtained by taking the absolute value of the difference is calculated as the third feature amount. That is, it functions as a third feature amount calculation unit.

  Of course, the third feature amount may be both a horizontal direction difference value and a vertical direction difference value. They serve as a reference for determining whether or not the subject at the position of the detection area is an object having a substantially equal size in the vertical and horizontal directions, for example, a face. Details will be described later with reference to FIG.

  A specific object detection determination unit 27 includes a first feature amount by the first feature amount calculation unit 23, a second feature amount by the second feature amount calculation unit 24, and a third feature amount calculation unit 25. Based on the third feature amount or the like, it is determined whether or not a specific object, that is, a face is detected in the target detection area. That is, it functions as specific object detection determination means.

  Only the first feature value and the second feature value may be used for the determination, or only the first feature value and the third feature value may be used. Further, in addition to these determinations, a determination result of an input image determination unit 26 described later may be added to make the determination.

  The criterion for detection determination is that the first feature amount is greater than a predetermined first threshold, and the criterion for detection determination is that the second feature amount is greater than a predetermined second threshold. The third feature The criterion for detection determination is that the amount is smaller than a predetermined third threshold. Of course, these criteria vary depending on the definition of the feature amount. When all the detection determination criteria are satisfied and the determination result of the input image determination unit 26 is taken into account, all the detection determinations are made, and in the event that all detection determinations are made, a specific object, that is, a face is finally detected in the detection region It is determined that Details of the above detection determination will be described later with reference to FIG.

  Reference numeral 22 denotes a second detection area setting unit, which is obtained by the input image acquisition unit 11 and is used for the first image that is one of the plurality of input images used for the distance image generation by the distance image generation unit 12. Then, the second detection area is set at a position in the first input image corresponding to the position of the detection area set in the distance image by the detection area setting unit 21. That is, it functions as second detection area setting means.

  The purpose of setting the second detection area is not to judge the face based on the relationship between the size of the face and the distance and the distribution of the distance information, but to the normal input image. It is to improve the accuracy of determination by combining with face determination. These can be arbitrarily combined in consideration of the speed and accuracy of face determination.

  Reference numeral 26 denotes an input image determination unit, which detects whether a specific object, that is, a face is detected in the second detection region set in the first input image by the second detection region setting unit 22 and is detected. Judgment is made. That is, it functions as an input image determination unit.

  The detection determination of the specific object may be performed using a known authentication technique such as template matching. The detection result is sent to the specific object detection determination unit 27, and the final specific object detection determination is performed together with the determination using the distance information.

  The operation of the specific object detection process using the specific object detection apparatus will be described below.

(Operation flow of specific object detection processing)
FIG. 2 is a flowchart showing an overall operation flow of the specific object detection process by the specific object detection apparatus of the present embodiment. A specific object detection processing method using the specific object detection apparatus will be described with reference to FIG.

  The specific object detection process is generally realized by image information processing by a computer or the like, and each process described below is performed by the specific object detection processing unit 10, specifically, the computer executes a processing program corresponding to each processing means. By executing, the function is realized.

  First, in step S11, which is an input image acquisition process, an input image is acquired, and thus the specific object detection process starts. The input image is provided from the input unit 101 and acquired by the input image acquisition unit 11.

  As described above, the input image includes a plurality of input images having different viewpoints and is used to generate a distance image. In the specific object detection processing of this embodiment, the detection accuracy of the face is improved by using the information of the distance image, instead of detecting the specific object, that is, the face from only the normal image. Increase efficiency as well.

  Next, in step S12 which is a distance image generation step, the distance image generation unit 12 generates a distance image in which each pixel has distance information using a plurality of input images received from the input image acquisition unit 11.

  Usually, a stereo image is generated from first and second images having different parallaxes from viewpoints corresponding to the right eye and left eye of the observer. At that stage, distance information can be calculated for each pixel of the first image, for example, and a distance image in which the distance value is expressed by a luminance value from 0 to 255 can be created. A detailed flow of the distance image generation process will be described later.

  In step S13, which functions as the next correspondence setting process, initialization for the scan process of the detection area is performed. In particular, the correspondence between the distance value in the distance image and the size of the detection area for detecting the specific object is set.

  In the following specific object detection processing, the specific object detection processing control unit 14 scans the detection region over the entire distance image and repeats the detection region setting and detection determination in the detection region processing unit 20 to repeat the detection in the distance image. A specific object detection process is performed. As an initial setting for this, the correspondence setting unit 13 sets the correspondence and stores it in the storage unit 103. In addition, the specific object detection processing control unit 14 sets an initial scan position of the detection area in the distance image.

  Next, in step S14, the specific object detection processing control unit 14 that controls scanning determines whether scanning of the detection area is completed, that is, whether the entire distance image has been scanned. As the normal scan direction, the upper left is the initial position, the scan is performed to the right in the horizontal direction, and when the horizontal scan is completed, the scan in the horizontal direction is repeated while sequentially shifting downward in the vertical direction, and the final position at the lower right is finished And the scan is over.

  If the scan is finished in step S14 (step S14: YES), the detection determination by the detection area processing unit 20 for the specific object detection process is finished, and step S33 is executed.

  In step S33, the specific object detection processing output unit 15 reads out the detection results temporarily stored in the storage unit 103, collects the detection results as specific object detection results, and outputs them to the output unit 105 or receives an instruction and outputs them. Wait in a possible state. This completes the specific object detection process.

  On the other hand, if the scan is not completed in step S14 (step S14: NO), the next step S15 is executed in order to repeat the process in the detection area processing unit 20.

  In step S15, which is a detection area setting step, the detection area setting unit 21 sets the detection area size at the scan position according to the correspondence between the distance value set in step S13 and the detection area size. Specifically, the detection area size corresponding to the distance value of the pixel at the scan position is read from the storage unit 103, and the detection area of the size is set around the scan position pixel of the distance image. A detailed flow of the detection area setting step will be described later.

  Steps from here are divided into processing for the distance image in steps S16, S17, and S18 and processing for the input image in steps S21 and S22. In the flow of FIG. 2, the procedure is such that both can be processed independently until they are integrated and determined in step S25. However, if the processing efficiency is taken into consideration, a flow in which any one is preferentially processed may be used. When priority is given to the input image, refer to FIG. 15, and when priority is given to the distance image, refer to FIG.

  In step S16, which is the first feature amount calculation step, the first feature amount calculation unit 23 calculates a pixel area having a distance value equivalent to the distance value corresponding to the size of the set detection area. The ratio to the entire area is calculated as the first feature amount. A detailed flow of the first feature amount calculation step will be described later.

  In step S17, which is the second feature amount calculation step, the second feature amount calculation unit 24 sets an auxiliary region above the set detection region, and averages the distance values in the auxiliary region. The median value or the distance value with the largest number of pixels having the same distance value is compared with the distance value corresponding to the detection area size, and the absolute value of the difference is calculated as the second feature amount. A detailed flow of the second feature amount calculation step will be described later.

  In step S18, which is the third feature amount calculation step, the third feature amount calculation unit 25 calculates the sum of the distance values in each region divided into two in the horizontal direction or the vertical direction with respect to the set detection region. The horizontal direction difference value or the vertical direction difference value obtained by calculating and taking the absolute value of the difference of each sum is calculated as the third feature amount. A detailed flow of the third feature amount calculation step will be described later.

  On the other hand, in step S21 which is the second detection area setting step, the second detection area setting unit 22 performs the second operation on the first image which is one of the plurality of input images used for the distance image generation. Set the detection area. The setting position of the second detection area is a position in the first input image corresponding to the position of the detection area set in the distance image by the detection area setting unit 21, and the setting size is the same as the detection area size. is there.

  Next, in step S22, which is an input image determination step, the input image determination unit 26 detects a specific object, that is, a face, in the second detection region set in the input image, and determines whether or not it has been detected. I do. As described above, the detection determination may be performed using a known authentication technique such as template matching.

  In step S25, which is a specific object detection determination step, the specific object detection determination unit 27 performs determination based on the first, second, and third feature amounts calculated for the detection region of the distance image, and the input image In combination with the detection determination made for the second detection area, the final specific object, that is, the face detection determination is performed.

  When all the feature quantities satisfy the detection determination criteria for the detection area of the distance image, and all the determination results including the determination result for the second detection area of the input image are finally detected, the detection area As described above, it is determined that the specific object, that is, the face is detected. A detailed flow of this specific object detection determination step will be described later.

  When the determination result in step S25 is “detected” (step S25: YES), step S31 is executed. If the determination result is “not detected” (step S25: NO), step S31 is omitted and step S32 is executed.

  In step S <b> 31, the detection result, that is, the fact that it has been detected and the position and size of the detection area are temporarily stored in the storage unit 103. After the scanning of the detection area is completed later, the specific object detection processing output unit 15 summarizes the results of the entire image.

  In step S <b> 32, the scan position is updated by the specific object detection processing control unit 14. The update order of scan positions is as already described. After updating the scan position, the process returns to step S14, and the processing for the detection area after step S14 is repeated while scanning the detection area.

  The repetition is continued under the control of the specific object detection processing control unit 14 until the scan is completed, the end determination is made in step S14, and the process proceeds to step S33.

  As described above, in step S33, the specific object detection processing output unit 15 reads all detection results temporarily stored in the storage unit 103, collects them as specific object detection results, and outputs them to the output unit 105. In this state, the specific object detection processing in the specific object detection processing unit 10 ends.

<Distance image generation>
The detailed flow of step S12 in FIG. 2, that is, the distance image generation process will be described with reference to FIG. FIG. 3 is a flowchart showing the distance image generation process.

  The distance image generation in FIG. 3 is executed by the distance image generation unit 12 using a plurality of input images received from the input image acquisition unit 11.

  It is assumed that the input image includes first and second images with different parallaxes from viewpoints corresponding to the right eye and the left eye of the observer. FIG. 4 shows an example of the input image. 31a is a first input image (right eye equivalent viewpoint), and 31b is a second input image (left eye equivalent viewpoint). Three persons A (front), B (middle), and C (back) standing at different distances are taken from different left and right viewpoints, and parallax is generated between these two images.

  Reference numeral 30 denotes a parallax image generated from the first and second input images, that is, a distance image. The parallax is obtained for each pixel of the first input image and replaced with a distance, and is expressed by a luminance value from 0 to 255. The person A in the foreground is expressed with high brightness, and the person C in the back is expressed with low brightness.

  A flow for obtaining the distance image 30 from the input images 31a and 31b will be described with reference to FIG.

  First, in step S41, initialization for distance detection scanning is performed. The distance detection scan (or distance scan) is an operation of scanning the first input image and obtaining distance information for all pixels of the first input image in order to convert the first input image into a distance image. The order of updating the scan position may be the same as in the case of scanning the detection area. Therefore, here, the upper left pixel is set as the initial scan position.

  Next, in step S42, it is determined whether or not the distance detection scan is completed. Judgment is made based on whether or not all pixels of the first input image have been passed as distance detection scan positions.

  If the distance detection scan has ended (step S42: YES), the generation of the distance image in which all the pixels of the first input image have distance information has been completed. The generation process is complete.

  On the other hand, when the distance detection scan is not completed (step S42: NO), step S43 is executed.

  In step S43, a distance detection window for obtaining the parallax (distance) at the scan position is set. The distance detection window is an area for obtaining parallax, and may be set as appropriate, but here it is assumed to have a size of w × w. Accordingly, a w × w region is set around the pixel at the scan position.

  In step S44, the parallax setting is initialized for parallax scanning. The parallax scan is a search for the most appropriate parallax while changing the parallax as described below, and sets an initial value of the parallax change.

  In order to obtain the parallax in the distance detection window, the parallax is sequentially changed (parallax scan), and the distance detection windows corresponding to the first input image and the second input image are shifted by the parallax and overlapped. Then, the parallax that best matches both is set as the parallax between the two in the distance detection window. Since the parallax is changed from 0 to max_disp (upper limit of parallax that can be generated by shooting the input image), the initial value is set to 0 here.

  In addition, the storage of the SAD minimum value (determination criteria for whether or not they match), which will be described later, is cleared.

  Next, a parallax scanning loop is entered from step S45. In step S45, it is determined whether the parallax scan is completed. If the set parallax exceeds max_disp which is the upper limit value, the process ends. If not, the process does not end.

  If the parallax scan has been completed (step S45: YES), the process exits the parallax scan loop, moves to step S51, and executes it. If not completed (step S45: NO), the next step S46 is executed.

  In step S46, SAD (sum of absolute values of differences: Sum_of_Absolute_Difference) between the distance detection windows shifted by the set parallax between the first input image and the second input image is calculated. This SAD is obtained by the following equation.

SAD = ΣΣ | Img1 (i, j) −Img2 (i, j + p) |
However, (i, j) is a pixel position in the distance detection window (w × w), and ΣΣ is a sum total of all the pixel positions in all (i, j), that is, the distance detection window (w × w). Represents. Img1 and Img2 are the pixel values of the first and second input images, respectively, and p is the set parallax.

  The parallax p that minimizes the SAD is obtained and considered as the most appropriate parallax in the distance detection window.

  In step S47, the SAD calculated in step S46 is compared with the SAD minimum value temporarily stored in the storage unit 103 to determine which is smaller. If the calculated SAD is smaller, or if there is no temporarily stored SAD minimum value (step S47: YES), step S48 is executed.

  In step S48, the calculated SAD is updated as the SAD minimum value, updated and stored together with the corresponding parallax p, and the process proceeds to step S49.

  In step S47, if the temporarily stored SAD minimum value is smaller (step S47: NO), the SAD minimum value and the parallax p at that time are not updated and stored. Step S48 is omitted, and step S49 is executed.

  In step S49, the set parallax of the parallax scan is updated, and the process returns to step S45. The set parallax starts from the initial value 0, increases by one pixel until the upper limit max_disp is exceeded, and repeats the parallax scan after step S45.

  If the upper limit max_disp is exceeded, this corresponds to the case where the parallax scan ends in step S45, and the process proceeds to step S51.

  In step S51, a parallax corresponding to the SAD minimum value, that is, an appropriate parallax for the distance detection window at that time is determined. Since the magnitude of the parallax corresponds to the distance of the object, the distance value can be obtained from the parallax. This correspondence may also be stored in advance in the storage unit 103 as a data table.

  In step S52, the distance image data is sequentially written and stored. The position of the distance detection window, that is, the pixel position in the distance image, and the pixel value obtained by converting the calculated distance value into a luminance value from 0 to 255 are stored in the storage unit 103 as distance information. When this is finished for all the pixels, the distance image data as shown at 30 in FIG. 4 is completed.

  In step S53, the distance detection scan position is updated and advanced to the next pixel position. Thereafter, the process returns to step S42, and the distance detection scan after step S42 is repeated until all the pixels of the first input image have been passed as scan positions.

  As described above, in the determination in step S42, when all the pixels of the first input image have been scanned, the distance detection scan loop is exited, and the generation of the distance image ends.

<Detection area setting>
The detailed flow of step S15 in FIG. 2, that is, the detection region setting step will be described with reference to FIG. FIG. 5 is a flowchart showing the detection area setting step.

  The detection area setting in FIG. 5 is executed by the detection area setting unit 21 for the distance image received from the distance image generation unit 12.

  The detection area is an area for detecting a face that is a specific object, and the face in the image and its position are detected by scanning the detection area for the entire distance image. Therefore, for the detection process in the detection region, it is desirable to determine the size of the detection region in accordance with the size of the face that is the specific object. However, the size of the face in the image varies depending on the distance. For this reason, scanning is usually repeated while changing the size of the detection area.

  The intention of using the distance image in this embodiment is to set the size of the detection region efficiently by using the distance information. For this purpose, in step S13 of FIG. 2, the correspondence setting unit 13 sets the correspondence between the distance value and the face size, that is, the detection area size suitable for face detection, and stores it in the form of a data table or the like. It is.

  A flow for setting a detection area at a scan position based on this correspondence will be described with reference to FIG.

  First, in step S61, a distance value at the scan position is acquired. The distance image is scanned, but the scan position has already been set in step S13 or step S32 in FIG. The pixel value is acquired for the pixel position in the distance image. As described in the distance image generation flow, the pixel value is expressed by a luminance value from 0 to 255, which corresponds to the distance value of the subject at the pixel position.

  In step S62, a detection area size corresponding to the distance value acquired in step S61 is set. This is to obtain the detection area size corresponding to the distance value at the scan position by referring to the correspondence relationship between the distance value and the face size, that is, the detection area size suitable for face detection. Therefore, if the scan position changes due to scanning, the distance value changes, and the corresponding face size, that is, the detection area size suitable for face detection also changes.

  FIG. 6 shows the correspondence between the parallax, that is, the distance value and the detection area size. It is assumed that the parallax has an upper limit from 0 to max_disp and is represented by a luminance value in the range of 0 to 255 here. The magnitude of this parallax represents a distance value from the subject, and the distance becomes closer as the parallax increases. That is, 0 is infinity, and 255, the parallax corresponding to the upper limit, is the closest shooting distance.

  As shown in FIG. 6, when the parallax is small (distance is long), the face size, that is, the detection area size is small, and when the parallax is large (distance is short), the detection area size is large. A lower part 40 in FIG. 6 indicates that the detection area size changes depending on the parallax (distance value). The specific numerical values are stored as correspondences and are referred to.

  In step S63, a detection area of that size is set at the scan position in the distance image according to the detection area size obtained by referring to the correspondence. What is necessary is just to set centering on the pixel of a scanning position.

  This completes the setting of the detection area for the scan position.

  As described above, as the detection area scan progresses, the specific detection area size changes. FIG. 7A shows how the size of the detection region 40 changes as the distance image 30 is scanned (from the upper left to the upper right in the figure). As described above, in the present embodiment, an appropriate detection area can be set according to each position in the distance image, so that it is not necessary to repeat scanning while changing the detection area size, and detection with good accuracy is performed efficiently. be able to.

  FIG. 7B shows a method of repeating scanning while changing the size of the detection area. First, as shown at 30a, an image is scanned with a detection area 40a having a certain size. For example, in the scan shown in 30b, the image is scanned with the detection areas 40b having different sizes. Thus, since the scan is repeated many times, the detection efficiency is deteriorated.

  However, in this case as well, the correspondence relationship is referred to in the same manner as in the present embodiment, and if the distance value at the scan position does not correspond to the detection area size, the efficiency can be improved by omitting the subsequent detection processing. Can be raised.

<First feature amount calculation>
A detailed flow of step S16 of FIG. 2, that is, the first feature amount calculation step will be described with reference to FIG. FIG. 8 is a flowchart showing the first feature amount calculation step.

  The first feature amount calculation in FIG. 8 is executed by the first feature amount calculation unit 23 for the detection region set by the detection region setting unit 21.

  The first feature amount is a feature amount indicating the size of the subject at the distance in the detection region for determining whether or not the subject in the detection region is a face having a size corresponding to the distance.

  First, in step S71, a distance value corresponding to the detection area size is acquired. This is obtained by back-calculating the distance value at the original scan position from the correspondence between the distance value and the detection area size. Later, pixels having a distance value equivalent to this distance value are counted, and the size of the subject is obtained.

  Next, in step S72, initial settings for scanning all the pixels in the detection area are performed. Here, the upper left pixel of the detection area is set as the pixel position for starting scanning. The scan direction is the same as the scan described above. Also, the total number of pixels C1 and the corresponding number of pixels C2 are set to an initial value of 0.

  In step S73, it is determined whether or not all the pixels in the detection area have been scanned. If the scan position passes the lower right pixel of the detection area, it may be determined that the scan is finished. If the scan has been completed (step S73: YES), the process proceeds to step S78. If the scan has not ended (step S73: NO), the next step S74 is executed.

  In step S74, one pixel is added to the total number of pixels C1. In order to obtain the sum of all scanned pixels, 1 is added for each scan.

  In step S75, the pixel value at the scan position, that is, the distance value is acquired, and it is determined whether or not it is equivalent to the corresponding distance value acquired in step S71. Whether or not they are equivalent may be determined in advance by determining a determination range and determining that the difference between the two is within the determination range.

  If it is equal to the corresponding distance value in step S75, the next step S76 is executed. If it is not equal to the corresponding distance value, step S76 is omitted and the process proceeds to step S77.

  In the next step S76, one pixel is added to the corresponding pixel number C2. In order to obtain the sum of the pixels having the same distance value as the corresponding distance value among all the scanned pixels, 1 is added every time it is determined that the scanned pixel is the corresponding pixel.

  In step S77, the scan position in the detection area is updated. Move to the next pixel position. Thereafter, the process returns to step S73, and the process for each pixel in the detection area in step S73 and subsequent steps is repeated until it is determined that the scan is completed.

  As described above, when the scan end determination is made in step S73, the process loop of each pixel in the detection area is exited, and step S78 is executed. In step S78, a first feature amount C2 / C1 is calculated.

  At this time, C1 is the total number of pixels in the detection area, and C2 is the total number of pixels having a distance value equivalent to the corresponding distance in the detection area. Therefore, the first feature amount C2 / C1 is a face at the corresponding distance in the detection region depending on whether or not the size of the subject in the detection region is commensurate with the size of the detection region. It can be used as a criterion for judging whether or not it can be considered.

  FIG. 9 shows an example of the determination. FIG. 9A is an example in which the subject area 50 composed of pixels having a distance value equivalent to the corresponding distance value occupies most of the detection area 40 in the detection area 40 in the distance image 30. If the subject in the detection area is a face, the first feature value is a large value close to 1.

  FIG. 9B is an example in which the subject area 50 composed of pixels having a distance value equivalent to the corresponding distance value occupies a small portion of the detection area 40 in the detection area 40 in the distance image 30. In this example, the subject area 50 is actually a face, but since the positions of the detection area 40 and the subject area 50 that is a face are shifted, it is not appropriate to detect and determine the detection area as a face. Therefore, if the first feature amount is considerably smaller than 1, it should not be determined that the face is a face.

  Although the determination method will be described later, as the determination based on the first feature amount, it may be determined whether it is larger than a predetermined first threshold value.

  Thus, the first feature amount calculation process ends.

<Second feature amount calculation>
A detailed flow of step S17 in FIG. 2, that is, the second feature amount calculation step will be described with reference to FIG. FIG. 10 is a flowchart showing the second feature amount calculation step.

  The second feature amount calculation in FIG. 10 is executed by the second feature amount calculation unit 24 for the detection region set by the detection region setting unit 21.

  The second feature amount is a region around the detection region for determining whether or not the subject in the detection region is a face corresponding to the distance, particularly in the auxiliary region set at the top, This is a feature amount indicating whether or not there is a subject at a distance equivalent to the distance of the viewable subject.

  First, in step S81, a distance value corresponding to the detection area size is acquired. This is obtained by back-calculating the distance value at the original scan position from the correspondence between the distance value and the detection area size. A pixel having a distance value equivalent to this distance value is examined in an auxiliary area set separately later, and the presence or absence of a subject in the auxiliary area is referred to.

  Next, in step S82, an auxiliary area is set. Specifically, for example, when the scan position of the detection area is (i, j) and the size of the detection area centered on it is m × m, the center position of the auxiliary area is (i−1.5 m, j ) And the size of the auxiliary area may be m × m. FIG. 11 shows an example in which the auxiliary area 41 is set for the detection area 40.

  Next, in step S83, initial settings for scanning all the pixels in the auxiliary area are performed. Here, the upper left pixel of the auxiliary area is set as the scan start pixel position. The scan direction is the same as the scan in the detection area described above. Further, an initial value for calculating an average value, median value, or most frequent value of distance values later is set.

  In step S84, it is determined whether or not all the pixels in the auxiliary area have been scanned. If the scan position passes through the lower right pixel of the auxiliary area, it may be determined that the scan has ended. If the scan has been completed (step S84: YES), the process proceeds to step S88. If the scan has not ended (step S84: NO), the next step S85 is executed.

  In step S85, the distance value is acquired from the pixel value at the scan position in the auxiliary region.

  Next, in step S86, the distance value at each pixel acquired in step S85 is stored and saved in preparation for later calculations such as an average value, median value, or most frequent value. Alternatively, if possible, the arithmetic processing may be performed sequentially.

  In step S87, the scan position in the auxiliary area is updated. Move to the next pixel position. Thereafter, the process returns to step S84, and the process for each pixel in the auxiliary area after step S84 is repeated until it is determined that the scan is completed. As described above, when it is determined in step S84 that scanning has ended, the processing loop of each pixel in the auxiliary region is exited, and step S88 is executed.

  In step S88, the average value, median value, or most frequent value of the distance values in the auxiliary region based on the distance value stored and saved in step S86 during the scan processing in the auxiliary region or the data that has already been calculated. Is calculated as a representative distance value.

  In step S89, the absolute value of the difference between the representative distance value of the auxiliary area calculated in step S88 and the detected area corresponding distance value acquired in step S81 is calculated as the second feature amount. This can be expressed as:

Second feature amount = | representative distance value−corresponding distance value |
At this time, the subject in the detection area is a target to be determined as a face, and the corresponding distance value represents the distance of the subject. If there is a face at that distance, the upper part, that is, the subject in the auxiliary area is usually far away. On the other hand, if the subject in the detection area is other than a face (for example, a torso), it is possible that the subject in the upper part is approximately the same distance.

  The representative distance value is a representative distance value with respect to the subject in the auxiliary region, and the distance relationship with the corresponding distance value of the subject in the detection region can be used as a criterion for determination according to the above-described concept. That is, when the second feature amount is large, the possibility of being a face is high, and when the second feature amount is small, the possibility of being a face is small.

  FIG. 11 shows an actual example of the determination. FIG. 11A shows a case where the subject in the detection area 40 in the distance image 30 is a human face 50. In the auxiliary area 41 corresponding to the upper part of the detection area 40, no subject exists at the same distance as the face 50, and the photographed subject becomes a background object far away. In such a case, the subject in the auxiliary area 41 usually does not have a distance value equivalent to the corresponding distance value in the detection area 40. Assuming that the subject in the detection area 40 is the face 50, the second feature amount is a relatively large value.

  FIG. 11B shows a case where the subject in the detection area 40 in the distance image 30 is a human torso 51 instead of the face 50. In this case, in the auxiliary region 41 corresponding to the upper portion of the detection region 40, a subject such as the face 50 exists at a distance equivalent to the body 51. Therefore, when the second feature amount is a relatively small value, there is a high possibility that the subject in the detection area 40 is not the face 50.

  Although the determination method will be described later, the determination based on the second feature amount may be performed based on whether or not it is larger than a predetermined second threshold.

  Thus, the second feature amount calculation process ends.

<Third feature amount calculation>
A detailed flow of step S18 of FIG. 2, that is, the third feature amount calculation step will be described with reference to FIG. FIG. 12 is a flowchart showing the third feature amount calculation step.

  The third feature amount calculation in FIG. 12 is executed by the third feature amount calculation unit 25 for the detection region set by the detection region setting unit 21.

  The third feature amount is a feature amount indicating a distribution of the distance of the subject in the detection region for determining whether or not the subject in the detection region is a face having a size corresponding to the distance. If the subject is a face, the distance distribution uses a small deviation in the horizontal direction or the vertical direction.

  First, in step S91, the detection area is divided into two equal parts in the horizontal direction or the vertical direction, and is divided into areas S1 and S2. It is desirable to perform the division, the subsequent scan, and the feature amount calculation processing for both the horizontal direction and the vertical direction.

  In step S92, S1 is first set as a scan area. Each of the divided areas S1 and S2 is scanned. In this order, scanning is first started from the area S1.

  Next, in step S93, initial setting is performed when all the pixels in the divided region to be scanned (initially S1) are scanned. Here, the upper left pixel of the divided area is set as the pixel position at which scanning is started. The scan direction is the same as the scan in the detection area described above. In addition, an initial value for calculating the sum of distance values later is set.

  In step S94, it is determined whether or not all the pixels in the divided area have been scanned. If the scan position passes through the lower right pixel of the divided area, it may be determined that the scan is finished. If the scan is complete (step S94: YES), the process proceeds to step S97. If the scan has not ended (step S94: NO), the next step S95 is executed.

  In step S95, the distance value is acquired from the pixel value at the scan position in the divided area, and an addition process for obtaining the sum of the distance values is performed.

  In step S96, the scan position in the divided area is updated. Move to the next pixel position. Thereafter, the process returns to step S94, and the process for each pixel in the divided area in step S94 and subsequent steps is repeated until it is determined that the scan is completed. As described above, when the scan end determination is made in step S94, the process loop of each pixel in the divided area is exited, and step S97 is executed.

  In step S97, it is determined whether or not the divided area is initially set S1. If the divided area is S1 (step S97: YES), step S98 is executed. If the divided area is not S1, that is, if it is S2 (step S97: NO), the process proceeds to step S99.

  In step S98, since the scan of the divided area S1 is completed, next, S2 is set as the scan area. Thereafter, the process returns to step S93, and the process similar to that in the area S1 is performed on the divided area S2 until the scan end is determined in step S94.

  In step S94, the scan end is determined, and the process proceeds to step S97. Further, since it is S2, the process proceeds to step S99. In step S99, the total sum corresponding to each divided area calculated in step S95 during the scan process in each divided area. Based on the distance values s1 and s2, the absolute value of the difference is calculated.

When divided in the horizontal direction, the horizontal difference value = | s1−s2 |
When divided in the vertical direction, the vertical difference value = | s1−s2 |
According to each case, the horizontal difference value or the vertical difference value is set as the third feature amount. Of course, both may be calculated and both may be used as the third feature amount. In that case, all the steps from step S91 to step S99 may be repeated twice in the horizontal and vertical directions.

  The horizontal difference value and the vertical difference value, which are the third feature amounts, indicate whether or not the distance of the subject in the detection area is evenly distributed in the horizontal direction and the vertical direction, respectively. If the subject in the detection area is a face, the distance is almost evenly distributed vertically and horizontally, and if the difference is large, the possibility of not being a face is high. That is, when the third feature amount is small, it can be determined that the subject in the detection region is a face, and when it is large, the possibility that the subject is not a face is high.

  FIG. 13 shows an example of the determination. FIG. 13A shows a case where the subject in the detection area 40 is a human face 50, and FIG. 13B shows a case where the face 50 is out of the detection area 40. In the detection area 40 of (a), the face 50 is substantially uniform in the horizontal direction, and the horizontal difference value becomes considerably small and can be determined as face detection. However, in (b), the horizontal difference value is growing. Therefore, it should not be determined that a face has been detected.

  The same applies to FIGS. 13C and 13D. A case (c) where the subject in the detection area 40 is a human face 50 and a case (d) where the face 50 is out of the detection area 40 are shown. In the detection area 40 of (c), the face 50 is substantially uniform in the vertical direction, and the vertical difference value is considerably small and can be determined as face detection. However, in (d), the vertical difference value is growing.

  Although the determination method will be described later, the determination based on the third feature amount may be determined based on whether or not it is smaller than a predetermined third threshold. If the third feature amount is both the horizontal difference value and the vertical difference value, the predetermined third threshold value may be determined in both the horizontal direction and the vertical direction.

  Thus, the third feature amount calculation process ends.

<Detection judgment>
A detailed flow of step S25 of FIG. 2, that is, the specific object detection determination step will be described with reference to FIG. FIG. 14 is a flowchart showing the specific object detection determination step.

  The specific object detection determination in FIG. 14 is executed by the specific object detection determination unit 27 for the detection region set by the detection region setting unit 21.

  What is used for the determination is the first, second, and third feature amounts calculated in the preceding step, and the input image determination result. In this specific object detection determination step, when all the determinations indicate detection, it is determined that they have been detected.

  First, in step S101, first, second, and third feature amounts for the detection region are acquired. The first, second, and third feature amounts are calculated in step S16, step S17, and step S18 in FIG. 2, respectively.

  Next, in step S102, detection determination based on the first feature amount is performed. The determination is made using the first threshold value TH1 determined in advance, and it is determined that the detection is made when the following expression is satisfied. When not satisfying, it determines with not detecting.

First feature amount> TH1
If it is determined that it has been detected (step S102: YES), the next step S103 is executed. If it is determined not to be detected (step S102: NO), the process proceeds to step S108 and is executed. In step S108, the detection result without detecting the specific object is immediately submitted, and the specific object detection determination is terminated.

  In step S103, detection determination based on the second feature amount is performed. The determination is made using the second threshold value TH2 determined in advance, and it is determined that the detection is made when the following expression is satisfied. When not satisfying, it determines with not detecting.

Second feature amount> TH2
If it is determined that it has been detected (step S103: YES), the next step S104 is executed. If it is determined not to be detected (step S103: NO), the process proceeds to step S108 and is executed. In step S108, the detection result without detecting the specific object is immediately submitted, and the specific object detection determination is terminated.

  In step S104, detection determination based on the third feature amount is performed. The determination is made using the third threshold value TH3 (either horizontal or vertical or both) determined in advance, and it is determined that the detection is made when the following expression is satisfied. When not satisfying, it determines with not detecting.

Third feature value (horizontal difference value) <TH3 (horizontal)
Third feature amount (vertical difference value) <TH3 (vertical)
If it is determined that it has been detected (step S104: YES), the next step S105 is executed. If it is determined not to be detected (step S104: NO), the process proceeds to step S108 and is executed. In step S108, the detection result without detecting the specific object is immediately submitted, and the specific object detection determination is terminated.

  In step S105, a detection determination result for the second detection region is acquired. The second detection area is set in step S21 in FIG. 2, and the input image determination has already been performed in step S22. Therefore, the detection determination result for the second detection region is output.

  Next, in step S106, determination based on the input image determination result is performed. If it is determined that the input image determination result has been detected (step S106: YES), the next step S107 is executed. If it is determined not to be detected (step S106: NO), the process proceeds to step S108 and is executed. In step S108, the detection result without detecting the specific object is immediately submitted, and the specific object detection determination is terminated.

  In step S <b> 107, a detection result indicating that the specific object has been detected is submitted, and the result is stored and saved in the storage unit 103.

  This completes the specific object detection determination. About each determination performed in the middle of a process, a part can be abbreviate | omitted and it can also add. It is also possible to appropriately change the order in which each determination is performed. However, only when all the determinations are detected, it is determined that the specific object is finally detected.

(Operation flow of specific object detection processing: another example)
In the description of the entire flow of the specific object detection process in FIG. 2, it has been described that the process after step S <b> 16 is divided into a process for a distance image and a process for an input image. In the flow of FIG. 2, the procedure is such that both may be processed independently until the determination processing is integrated in step S25. A flow that performs priority processing may be used. The flow will be described with reference to FIG. 15 when the input image is prioritized (another example 1) and with reference to FIG. 16 when the distance image is prioritized (another example 2).

<Example 1>
A specific example 1 of the specific object detection process in FIG. 2 will be described with reference to FIG. FIG. 15 is a flowchart illustrating another example 1 of the operation flow of the specific object detection process. In this flow, the determination process for the input image is preferentially executed, and the determination process for the distance image is performed only when it is detected there.

  Steps S11 to S15 are the same as those in FIG.

  Subsequently, determination processing for the input image in step S21 and step S22 is performed. The processing contents are as described above. In step S27, detection determination by input image determination is performed. The detection determination here corresponds to each step shown in (S27) in FIG.

  When it determines with not detecting here (step S27: NO), it progresses to step S32. That is, it is determined that the detection is not performed, and the scanning of the detection area proceeds. Only when it is determined to be detected (step S27: YES), the process proceeds to the next step S16.

  Next, a feature amount calculation process is performed on the distance image in steps S16 to S18. The processing contents are as described above. Thereafter, in step S26, detection determination based on the first to third feature amounts is performed. The detection determination here corresponds to each step shown in (S26) in FIG.

  When it determines with not detecting here (step S26: NO), it progresses to step S32. That is, it is determined that the detection is not performed, and the scanning of the detection area proceeds. Only when it is determined to be detected (step S26: YES), the process proceeds to the next step S31.

  In step S <b> 31, the specific object detection determination result is stored and saved in the storage unit 103.

  In the next step S32, the scan position of the detection area is updated, the process returns to step S14, and the scan continues, as described with reference to FIG.

  As described above, the specific object detection process by flow example 1 is completed. However, the determination process for the input image is performed with priority, and the determination process for the distance image is performed only when the input image is detected. This is a flow suitable for the case where the determination process is more efficient or more accurate.

<Another example 2>
A specific example 2 of the specific object detection process in FIG. 2 will be described with reference to FIG. FIG. 16 is a flowchart illustrating another example 2 of the operation flow of the specific object detection process. In this flow, the determination process for the distance image is preferentially executed, and the determination process for the input image is performed only when it is detected there.

  Steps S11 to S15 are the same as those in FIG.

  Subsequently, a feature amount calculation process for the distance image in steps S16 to S18 is performed. The processing contents are as described above. Thereafter, in step S26, detection determination based on the first to third feature amounts is performed. The detection determination here corresponds to each step shown in (S26) in FIG.

  When it determines with not detecting here (step S26: NO), it progresses to step S32. That is, it is determined that the detection is not performed, and the scanning of the detection area proceeds. Only when it is determined to be detected (step S26: YES), the process proceeds to the next step S21.

  Next, determination processing for the input image in step S21 and step S22 is performed. The processing contents are as described above. In step S27, detection determination by input image determination is performed. The detection determination here corresponds to each step shown in (S27) in FIG.

  When it determines with not detecting here (step S27: NO), it progresses to step S32. That is, it is determined that the detection is not performed, and the scanning of the detection area proceeds. Only when it is determined to be detected (step S27: YES), the process proceeds to the next step S31.

  In step S <b> 31, the specific object detection determination result is stored and saved in the storage unit 103.

  In the next step S32, the scan position of the detection area is updated, the process returns to step S14, and the scan continues, as described with reference to FIG.

  As described above, the specific object detection process by flow example 2 ends. However, the determination process for the distance image is performed with priority, and the determination process for the input image is performed only when it is detected there. This is a flow suitable for the case where the determination process is more efficient or more accurate.

  As described above, according to this embodiment, when using distance information to detect the position and size of a specific object in an image, an appropriate detection area is set according to the distance, and the size of the area to be detected is set. Analysis of the spatial distribution of the distance information in the detection area, or by extending the use of the distance information to the periphery of the detection area, the specific object in the image can be more accurately and more efficiently It is possible to provide a specific object detection apparatus, a specific object detection method, and a specific object detection program capable of detecting the size.

  Further, the scope of the present invention is not limited to the above embodiment. Unless it deviates from the meaning of this invention, those changed forms are also included in the range.

It is a block diagram which shows the function structure about one Embodiment of the specific object detection apparatus which concerns on this invention. It is a flowchart which shows the whole operation | movement flow of the specific object detection process by a specific object detection apparatus. It is a flowchart which shows a distance image generation process. It is a figure which shows the example of the distance image produced | generated based on the input image and it. It is a flowchart which shows a detection area setting process. It is a figure which shows the correspondence of parallax, ie, a distance value, and a detection area size. It is a figure which shows the change (a) of the detection area size by the scan of a detection area, and the scanning repetition (b) by changing a detection area size. It is a flowchart which shows a 1st feature-value calculation process. It is a figure which shows the specific example of the determination by a 1st feature-value. It is a flowchart which shows a 2nd feature-value calculation process. It is a figure which shows the specific example of the determination by a 2nd feature-value. It is a flowchart which shows a 3rd feature-value calculation process. It is a figure which shows the specific example of the determination by a 3rd feature-value. It is a flowchart which shows a specific object detection determination process. It is a flowchart which shows the example 1 according to the whole flow of a specific object detection process. It is a flowchart which shows the example 2 according to the whole flow of a specific object detection process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Specific object detection process part 11 Input image acquisition part 12 Distance image generation part 13 Correspondence relationship setting part 14 Specific object detection process control part 15 Specific object detection process output part 20 Detection area process part 21 Detection area setting part 22 2nd detection Area setting unit 23 First feature value calculation unit 24 Second feature value calculation unit 25 Third feature value calculation unit 26 Input image determination unit 27 Specific object detection determination unit 30, 30a, 30b Distance image 31a, 31b Input image 40, 40a, 40b Detection area 41 Auxiliary area 50 Subject (face)
51 Subject (torso)
101 Input unit 102 Operation unit 103 Storage unit 104 Display unit 105 Output unit

Claims (24)

Input image acquisition means for acquiring a plurality of input images from different viewpoints;
Distance image generating means for generating a distance image from a plurality of input images acquired by the input image acquiring means;
Correspondence setting means for setting a correspondence between the distance value in the distance image and the size of the detection region for detecting the specific object based on a predetermined specific object to be detected;
Detection area setting means for setting a detection area having a size corresponding to a distance value in the distance image based on the correspondence set by the correspondence setting means;
A first feature amount calculating means for calculating a first feature amount based on a size of a region satisfying a distance value corresponding to the size of the detection region set by the detection region setting means;
A second feature amount calculating unit that sets an auxiliary region around the detection region set by the detection region setting unit, and calculates a second feature amount from a relationship between a distance value between the auxiliary region and the detection region; ,
Specific object detection determination means for determining whether the predetermined specific object is detected in the detection region based on the first feature amount and the second feature amount;
A specific object detection device comprising: The first feature amount calculating means includes:
In the detection area, the ratio of the number of pixels having a distance value equivalent to the distance value corresponding to the detection area within a predetermined range to the total number of pixels is calculated as a first feature amount,
The second feature amount calculating means includes:
The auxiliary region is set above the detection region, and an average value, a median value, or a distance value with the largest number of pixels having the same distance value in the auxiliary region, and a distance value corresponding to the detection region Comparing, and calculating the absolute value of the difference as the second feature amount,
The specific object detection device according to claim 1. The specific object detection determination means includes
Determining that the specific object has been detected when the first feature value is greater than a predetermined first threshold value and the second feature value is greater than a predetermined second threshold value;
The specific object detection device according to claim 1, wherein Input image acquisition means for acquiring a plurality of input images from different viewpoints;
Distance image generating means for generating a distance image from a plurality of input images acquired by the input image acquiring means;
Correspondence setting means for setting a correspondence between the distance value in the distance image and the size of the detection region for detecting the specific object based on a predetermined specific object to be detected;
Detection area setting means for setting a detection area having a size corresponding to a distance value in the distance image based on the correspondence set by the correspondence setting means;
A first feature amount calculating means for calculating a first feature amount based on a size of a region satisfying a distance value corresponding to the size of the detection region set by the detection region setting means;
Third feature quantity calculating means for calculating a third feature quantity based on a spatial distribution of an area satisfying a distance value corresponding to the size of the detection area set by the detection area setting means;
Specific object detection determination means for determining whether the predetermined specific object is detected in the detection region based on the first feature amount and the third feature amount;
A specific object detection device comprising: The first feature amount calculating means includes:
In the detection area, the ratio of the number of pixels having a distance value equivalent to the distance value corresponding to the detection area within a predetermined range to the total number of pixels is calculated as a first feature amount,
The third feature amount calculating means includes:
A horizontal direction difference value obtained by calculating a sum of distance values in each region obtained by dividing the detection region into two in the horizontal direction, and taking an absolute value of a difference of each sum,
Alternatively, the sum of distance values is calculated in each region obtained by dividing the detection region into two in the vertical direction, and the vertical direction difference value obtained by taking the absolute value of the difference of each sum,
Is calculated as the third feature amount,
The specific object detection apparatus according to claim 4. The specific object detection determination means includes
Determining that the specific object has been detected when the first feature value is greater than a predetermined first threshold value and the third feature value is less than a predetermined third threshold value;
The specific object detection device according to claim 4 or 5, wherein For a first image that is one of a plurality of input images used to generate the distance image,
Second detection area setting means for setting a second detection area at a position in the first input image corresponding to the position of the detection area set in the distance image by the detection area setting means;
An input image determination unit that detects the specific object with respect to the second detection region and determines whether the specific object is detected;
The specific object detection determination means includes
Based on the determination result by the input image determination means, it is determined whether or not the predetermined specific object is detected in the detection region;
The specific object detection apparatus according to claim 1, wherein the specific object detection apparatus is a single object detection apparatus. The specific object detected and determined by the specific object detection determination means is a face;
The specific object detection apparatus according to claim 1, wherein the specific object detection apparatus is a single object detection apparatus. An input image acquisition step of acquiring a plurality of input images from different viewpoints;
A distance image generation step of generating a distance image from a plurality of input images acquired in the input image acquisition step;
A correspondence setting step for setting a correspondence between a distance value in the distance image and a size of a detection region for detecting the specific object based on a predetermined specific object to be detected;
A detection region setting step of setting a detection region of a size corresponding to a distance value in the distance image based on the correspondence set in the correspondence relationship setting step;
A first feature amount calculating step of calculating a first feature amount based on a width of a region satisfying a distance value corresponding to the size of the detection region set in the detection region setting step;
A second feature amount calculating step of setting an auxiliary region around the detection region set in the detection region setting step and calculating a second feature amount from a relationship between a distance value between the auxiliary region and the detection region; ,
A specific object detection determination step of determining whether or not the predetermined specific object is detected in the detection region based on the first feature value and the second feature value;
A specific object detection method characterized by comprising: In the first feature amount calculating step,
In the detection area, the ratio of the number of pixels having a distance value equivalent to the distance value corresponding to the detection area within a predetermined range to the total number of pixels is calculated as a first feature amount,
In the second feature amount calculating step,
The auxiliary region is set above the detection region, and an average value, a median value, or a distance value with the largest number of pixels having the same distance value in the auxiliary region, and a distance value corresponding to the detection region Comparing, and calculating the absolute value of the difference as the second feature amount,
The specific object detection method according to claim 9. In the specific object detection determination step,
Determining that the specific object has been detected when the first feature value is greater than a predetermined first threshold value and the second feature value is greater than a predetermined second threshold value;
The specific object detection method according to claim 9 or 10, wherein: An input image acquisition step of acquiring a plurality of input images from different viewpoints;
A distance image generation step of generating a distance image from a plurality of input images acquired in the input image acquisition step;
A correspondence setting step for setting a correspondence between a distance value in the distance image and a size of a detection region for detecting the specific object based on a predetermined specific object to be detected;
A detection region setting step of setting a detection region of a size corresponding to a distance value in the distance image based on the correspondence set in the correspondence relationship setting step;
A first feature amount calculating step of calculating a first feature amount based on a width of a region satisfying a distance value corresponding to the size of the detection region set in the detection region setting step;
A third feature amount calculating step of calculating a third feature amount based on a spatial distribution of a region satisfying a distance value corresponding to the size of the detection region set in the detection region setting step;
A specific object detection determination step of determining whether the predetermined specific object is detected in the detection region based on the first feature amount and the third feature amount;
A specific object detection method characterized by comprising: In the first feature amount calculating step,
In the detection area, the ratio of the number of pixels having a distance value equivalent to the distance value corresponding to the detection area within a predetermined range to the total number of pixels is calculated as a first feature amount,
In the third feature amount calculating step,
A horizontal direction difference value obtained by calculating a sum of distance values in each region obtained by dividing the detection region into two in the horizontal direction, and taking an absolute value of a difference of each sum,
Alternatively, the sum of distance values is calculated in each region obtained by dividing the detection region into two in the vertical direction, and the vertical direction difference value obtained by taking the absolute value of the difference of each sum,
Is calculated as the third feature amount,
The specific object detection method according to claim 12. In the specific object detection determination step,
Determining that the specific object has been detected when the first feature value is greater than a predetermined first threshold value and the third feature value is less than a predetermined third threshold value;
The specific object detection method according to claim 12 or 13, characterized in that: For a first image that is one of a plurality of input images used to generate the distance image,
A second detection region setting step of setting a second detection region at a position in the first input image corresponding to the position of the detection region set in the distance image in the detection region setting step;
An input image determination step for detecting the specific object with respect to the second detection region and determining whether the specific object has been detected;
In the specific object detection determination step,
Based on the determination result in the input image determination step, it is determined whether the predetermined specific object is detected in the detection region,
The method for detecting a specific object according to claim 9, wherein the specific object is detected. The specific object detected and determined in the specific object detection determination step is a face;
The specific object detection method according to any one of claims 9 to 15, wherein A computer that controls the detection process of a specific object,
Input image acquisition means for acquiring a plurality of input images from different viewpoints;
Distance image generation means for generating a distance image from a plurality of input images acquired by the input image acquisition means;
Correspondence setting means for setting the correspondence between the distance value in the distance image and the size of the detection area for detecting the specific object based on a predetermined specific object to be detected;
Detection area setting means for setting a detection area of a size corresponding to a distance value in the distance image based on the correspondence set by the correspondence setting means;
A first feature amount calculating means for calculating a first feature amount based on a size of a region satisfying a distance value corresponding to the size of the detection region set by the detection region setting means;
A second feature quantity calculating means for setting an auxiliary area around the detection area set by the detection area setting means and calculating a second feature quantity from a relationship between a distance value between the auxiliary area and the detection area;
Specific object detection determination means for determining whether or not the predetermined specific object is detected in the detection region based on the first feature value and the second feature value;
A specific object detection program characterized by causing it to function as The first feature amount calculating means includes:
In the detection area, the ratio of the number of pixels having a distance value equivalent to the distance value corresponding to the detection area within a predetermined range to the total number of pixels is calculated as a first feature amount,
The second feature amount calculating means includes:
The auxiliary region is set above the detection region, and an average value, a median value, or a distance value with the largest number of pixels having the same distance value in the auxiliary region, and a distance value corresponding to the detection region Comparing, and calculating the absolute value of the difference as the second feature amount,
The specific object detection program according to claim 17. The specific object detection determination means includes
Determining that the specific object has been detected when the first feature value is greater than a predetermined first threshold value and the second feature value is greater than a predetermined second threshold value;
The specific object detection program according to claim 17 or 18, characterized in that: A computer that controls the detection process of a specific object,
Input image acquisition means for acquiring a plurality of input images from different viewpoints;
Distance image generation means for generating a distance image from a plurality of input images acquired by the input image acquisition means;
Correspondence setting means for setting the correspondence between the distance value in the distance image and the size of the detection area for detecting the specific object based on a predetermined specific object to be detected;
Detection area setting means for setting a detection area of a size corresponding to a distance value in the distance image based on the correspondence set by the correspondence setting means;
A first feature amount calculating means for calculating a first feature amount based on a size of a region satisfying a distance value corresponding to the size of the detection region set by the detection region setting means;
A third feature amount calculating unit that calculates a third feature amount based on a spatial distribution of a region that satisfies a distance value corresponding to the size of the detection region set by the detection region setting unit;
Specific object detection determination means for determining whether or not the predetermined specific object is detected in the detection region based on the first feature value and the third feature value;
A specific object detection program characterized by causing it to function as The first feature amount calculating means includes:
In the detection area, the ratio of the number of pixels having a distance value equivalent to the distance value corresponding to the detection area within a predetermined range to the total number of pixels is calculated as a first feature amount,
The third feature amount calculating means includes:
A horizontal direction difference value obtained by calculating a sum of distance values in each region obtained by dividing the detection region into two in the horizontal direction, and taking an absolute value of a difference of each sum,
Alternatively, the sum of distance values is calculated in each region obtained by dividing the detection region into two in the vertical direction, and the vertical direction difference value obtained by taking the absolute value of the difference of each sum,
Is calculated as the third feature amount,
The specific object detection program according to claim 20. The specific object detection determination means includes
Determining that the specific object has been detected when the first feature value is greater than a predetermined first threshold value and the third feature value is less than a predetermined third threshold value;
The specific object detection program according to claim 20 or 21, wherein: The computer,
For a first image that is one of a plurality of input images used to generate the distance image,
Second detection area setting means for setting a second detection area at a position in the first input image corresponding to the position of the detection area set in the distance image by the detection area setting means;
An input image determination unit that detects the specific object with respect to the second detection region and determines whether or not the specific object is detected.
Function as
The specific object detection determination means includes
Based on the determination result by the input image determination means, it is determined whether or not the predetermined specific object is detected in the detection region;
The specific object detection program according to any one of claims 17 to 22, wherein The specific object detected and determined by the specific object detection determination means is a face;
The specific object detection program according to any one of claims 17 to 23, wherein:

Images