JP6580201B2 - Subject detection apparatus, subject detection method, and program - Google Patents

Description

  The present invention particularly relates to a subject detection apparatus, a subject detection method, and a program suitable for use in detecting a main subject.

  Conventionally, as a method for detecting a main subject from an input image, for example, a method described in Patent Document 1 has been proposed. In the method described in Patent Document 1, first, an input image is divided into a plurality of partial regions using an automatic division algorithm. Further, the degree of saliency in the partial area is calculated based on the weighted sum of the differences with respect to the feature amounts of one of the obtained partial areas and the other partial areas. Further, the main subject in the image is detected based on the obtained saliency.

  As another method for detecting the main subject from the input image, for example, a method as described in Non-Patent Document 1 has been proposed. In the method described in Non-Patent Document 1, first, a plurality of types of feature amounts are extracted from an input image, and a multi-resolution image for the feature amounts is generated. In addition, two partial areas having different sizes are set for each type of the obtained multi-resolution image, and the statistical distribution difference with respect to the feature amount extracted from the two partial areas (Calbach-Librer divergence). The saliency is calculated based on the above. Further, the saliency obtained for each type of multi-resolution image is integrated to generate a saliency image, and finally a main subject in the image is detected based on the obtained saliency image.

  Furthermore, as another method for detecting the main subject (or its partial region) from the input image, for example, a method described in Non-Patent Document 2 has been proposed. In the method described in Non-Patent Document 2, first, a plurality of types of feature quantities are extracted from an input image, a multi-resolution image for the feature quantities is generated, and two partial regions having different sizes for each type of multi-resolution image Set. Then, a statistical distribution difference (distance between scale weighted probability distributions) with respect to the feature amount extracted from the two partial regions, and an information amount (information entropy) with respect to the feature amount extracted from one of the two partial regions, The saliency is calculated based on the product of. Furthermore, a saliency image is generated by integrating the saliency obtained for each type of multi-resolution image, and finally, the main subject (or its partial area) in the image based on the obtained saliency image Is detected.

JP 2012-243313 A

A. Dominik et al., Center-surround Divergence of Feature Statistics for Salient Object Detection, ICCV2011 T. Kadir et al., An affine invariant salient region detector, ECCV2004. Noriyuki Otsu: Automatic threshold selection method based on discrimination and least squares method, IEICE Transactions, J63-D-4 (1980-4), 349.356. Lowe, D.G .: Object recognition from local scale invariant features, Proc. Of IEEE International Conference on Computer Vision, pp. 1150.1157 (1999). T. Lindeberg (1994). Scale-Space Theory in Computer Vision. Springer. ISBN 0-7923-9418-6. E. Sharon, A. Brandt, and R. Basri, Fast multiscale image segmentation Proc.IEEE Computer Vision and Pattern Recognition, pp. 70-77, 2000.

  As described above, in the methods described in Patent Document 1 and Non-Patent Document 1, the saliency is calculated based on the difference in the statistical distribution with respect to the feature amount in the input image, and the saliency is calculated based on the obtained saliency. The main subject is detected. However, when the main subject in the image is not visually noticeable, there is a problem that the detection accuracy of the main subject is lowered.

  In the method described in Non-Patent Document 2, the amount of information (information entropy) contained in the main subject in the input image is calculated, and the image is calculated based on the obtained amount of information (information entropy). The main subject inside is detected. However, there is a problem that the detection accuracy of the main subject is lowered because it is easily affected by noise due to environmental or observational factors.

  An object of the present invention is to make it possible to more robustly detect a main subject in an image in view of the above-described problems.

The subject detection apparatus according to the present invention sets a first area, a second area including the periphery of the first area, and a third area including the first area on the input image. Setting means; first extraction means for extracting the same type of feature quantity based on luminance or color from each of the first and second areas; and the first and second areas extracted from each of the first and second areas. Based on the derivation means for deriving the saliency based on the difference between the same type of feature quantity, the second extraction means for extracting the edge feature quantity from the third region, and the saliency and the edge feature quantity Detecting means for detecting a main subject in the input image.

  According to the present invention, the main subject in the image can be detected more robustly even when the main subject is not visually noticeable or when noise due to environmental or observational factors is generated.

It is a block diagram which shows the function structural example of the main subject detection apparatus which concerns on embodiment. It is a flowchart which shows an example of the process sequence which identifies a main to-be-photographed object. It is a figure for demonstrating the relationship between the 1st partial area | region and 2nd partial area | region in 1st and 2nd embodiment. It is a figure for demonstrating the square 3rd partial area | region in 1st Embodiment. It is a figure for demonstrating the rectangular 3rd partial area | region in 1st Embodiment. It is a figure for demonstrating the circular 3rd partial area | region in 1st Embodiment. It is a figure for demonstrating the elliptical 3rd partial area | region in 1st Embodiment. It is a figure for demonstrating the procedure which calculates saliency based on the difference in the statistical distribution of the feature-value in a partial area. It is a figure for demonstrating the procedure which detects a main to-be-photographed from a score map. FIG. 10 is a diagram for describing a procedure for obtaining a third partial region in the second embodiment. It is a figure for demonstrating the 3rd partial area | region in 2nd Embodiment. It is a figure for demonstrating the relationship between the 1st partial area | region and 2nd partial area | region in 3rd Embodiment.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is a block diagram illustrating a functional configuration example of a main subject detection device 100 according to the present embodiment. The functions of the main subject detection apparatus 100 according to the present embodiment are realized by a program executed by a semiconductor integrated circuit (LSI) (not shown) or a CPU.

  As shown in FIG. 1, the main subject detection apparatus 100 includes a first partial region setting unit 101, a second partial region setting unit 102, a third partial region setting unit 103, a saliency calculating unit 104, an information amount calculating unit 105, A score calculation unit 106 and an identification unit 107 are included. These components correspond to the functions performed by the main subject detection apparatus 100, respectively.

FIG. 2 is a flowchart illustrating an example of a processing procedure for identifying the main subject by the main subject detection apparatus 100 according to the present embodiment. Hereinafter, processing by the main subject detection apparatus 100 according to the present embodiment will be described with reference to FIGS. 1 and 2.
As shown in FIG. 2, the main subject detection apparatus 100 first sets a first partial area and a second partial area of a predetermined size for a certain point (pixel) on the input image. To do. In addition, the first information amount is calculated in the second partial area, and the third partial area is set based on the size. Further, the score is calculated based on the saliency based on the difference in the statistical distribution of the feature amount between the first partial region and the second partial region, and the information amount of the feature amount in the third partial region. Finally, the score at each point on the image is calculated, and the main subject is detected by applying predetermined statistical processing. The details will be described below.

  First, processing is started when an input image is input to the main subject detection apparatus 100. In step S201, it is determined whether or not processing in steps S202 to S208 described later has been completed at all points on the input image. . As a result of this determination, if all the points on the input image have been processed, the process proceeds to step S209, and if not, the process proceeds to step S202.

  Next, in step S202, the first partial region setting unit 101 sets the first partial region 301 on the image space of the input image input from the outside of the main subject detection apparatus 100 as shown in FIG. . Specifically, as shown in FIG. 3A, the first partial area 301 may be set as a square area, and as shown in FIG. 3B, the first partial area 301 as a rectangular area. May be set. Further, as shown in FIG. 3C, the first partial region 301 may be set as a circular region, and as shown in FIG. 3D, the first partial region 301 is set as an elliptical region. May be.

  Next, in step S203, the second partial region setting unit 102 sets the second partial region 302 on the image space of the input image as shown in FIG. Specifically, as shown in FIG. 3A, the second partial region 302 may be set as a square region including the first partial region, and as shown in FIG. The second partial region 302 may be set as a rectangular region including the first partial region. Further, as shown in FIG. 3C, the second partial region 302 may be set as a circular region including the circular first partial region, and as shown in FIG. The second partial region 302 may be set as an elliptical region including the first partial region.

  Next, in step S204, the second partial region setting unit 102 calculates an information amount (hereinafter referred to as a first information amount) included in a feature amount (luminance value, color component, edge strength, etc.) in the second partial region. To do. Here, the magnitude of the first information amount is calculated as entropy H by the following equation (1), for example.

In Expression (1), P _i represents the occurrence probability of the i-th gradation of the feature quantity in the second partial region. When the number of gradations of a certain feature quantity is 256 (= 2 ⁸ ), the entropy H The maximum value is given by 8, and the minimum value is given by 0.

  Next, in step S205, the third partial region setting unit 103 sets a third partial region based on the magnitude of the first information amount in the second partial region on the input image. Specifically, for example, when the first information amount is large, as shown in FIG. 4A, the first partial area 301 is included and the relative value included in the second partial area 302 is included. The third partial area 403 is set as a small square area. When the first information amount is small, as shown in FIG. 4B, a relatively large square area that includes the first partial area 301 and is included in the second partial area 302 is used. The third partial area 403 may be set.

When the first partial region and the second partial region are set as square regions, the length L ₃ of one side of the third partial region is, for example, the length L ₁ of one side of the first partial region. Using the length L ₂ of one side of the second partial region, the following formula (2) is used for calculation.

According to Equation (2), the length L ₃ of one side of the third partial region in FIG. 4 is given as a real value from the minimum value L ₁ to the maximum value L ₂ according to the size of the entropy H. .

When the first partial region and the second partial region are set as shown in FIG. 3B, the following settings are made. When the first information amount is large, as shown in FIG. 5A, the third partial area 301 is included as a relatively small rectangular area including the first partial area 301 and included in the second partial area 302. The partial area 403 is set. When the first information amount is small, as shown in FIG. 5B, a relatively large rectangular area that includes the first partial area 301 and is included in the second partial area 302 is used. A third partial area 403 is set. The third partial region length L ₃ of the long side of the in Figure 5, the length L ₁ of the long side of the first partial region, the long side of the second partial region and the length L ₂ And can be calculated by the equation (2), and the short side can be calculated in the same manner.

In addition, when the first partial region and the second partial region are set as shown in FIG. 3C, the following settings are made. When the first information amount is large, as shown in FIG. 6A, the third partial region 301 is included as a relatively small circular region including the first partial region 301 and included in the second partial region 302. The partial area 403 is set. In addition, when the first information amount is small, as shown in FIG. 6B, a relatively large circular area including the first partial area 301 and included in the second partial area 302 is obtained. A third partial area 403 is set. The length L ₃ of the diameter of the third partial region in FIG. 6, using the length L ₁ of the diameter of the first portion region, the diameter of the second portion region and the length L ₂ Formula It can be calculated by (2).

Further, when the first partial region and the second partial region are set as shown in FIG. 3D, the following settings are made. When the first information amount is large, as shown in FIG. 7A, the third partial area 301 is included, and the third partial area 301 is included as a relatively small elliptical area included in the second partial area 302. The partial area 403 is set. When the first information amount is small, as shown in FIG. 7B, a relatively large elliptical region that includes the first partial region 301 and is included in the second partial region 302 is used. A third partial area 403 is set. The length L ₃ of the major axis of the third partial area in FIG. 7, the length L ₁ of the long axis of the first partial region, the long axis of the second partial region and the length L ₂ And can be calculated by the equation (2), and the short axis can be calculated similarly.

  Next, in step S <b> 206, the saliency calculating unit 104 serves as a saliency deriving unit, and the first partial region obtained by the first partial region setting unit 101 and the second partial region obtained by the second partial region setting unit 102. The degree of saliency is calculated using the partial area. Specifically, as shown in FIG. 8, the visual saliency is calculated based on the difference in the statistical distribution of the feature amount in each partial region. Here, for example, the saliency is calculated using the histogram intersection HI by the following equation (3).

Alternatively, the prominence may be calculated using a Pearson-divergence D _PR of Equation (4).

The saliency may be calculated using the relative Pearson divergence D _{RP according} to the following equation (5). Here, β is an arbitrary real value between 0 and 1.

Moreover, remarkable degree, Kullback by the following equation (6) - may be calculated using Leibler divergence D _KL.

Further, saliency can be calculated using the Batacharia distance D _BT by the following equation (7).

  Further, the saliency may be calculated using the distance scale D by the following equation (8).

The saliency may be calculated using D _{abs according} to the following equation (9).

  Here, in Expressions (3) to (9), P (i) represents the probability of the i-th gradation of the probability density P with respect to the feature amount extracted from the first partial region, and Q (i) is , Represents the probability of the i-th gradation of the probability density Q with respect to the feature quantity extracted from the second partial region.

  Next, in step S207, the information amount calculation unit 105 includes an information amount (hereinafter referred to as a feature amount (luminance value, color component, edge strength, etc.) in the third partial region obtained by the third partial region setting unit 103. , Second information amount). Specifically, the second information amount may be given by, for example, calculating the gradient strength of the feature amount in the third partial region at each point of the third partial region and summing them. Here, the gradient strength may be calculated using a known image processing filter (Sobel filter, Canny filter, Laplacian filter, Gabor filter, etc.).

  Next, in step S208, the score calculation unit 106 performs processing on the input image based on the saliency obtained by the saliency calculation unit 104 and the second information amount obtained by the information amount calculation unit 105. A score at the target point (a scale indicating whether or not there is a main subject) is calculated. Here, the score at each point on the input image may be given by, for example, the product of the saliency obtained by the saliency calculator 104 and the second information amount obtained by the information amount calculator 105. Note that the sum of the saliency obtained by the saliency calculating unit 104 and the second information amount obtained by the information amount calculating unit 105 may be given as a score. Alternatively, a combination of the product and sum of the saliency obtained by the saliency calculator 104 and the information amount obtained by the information amount calculator 105 may be given as a score.

  In step S209, the identification unit 107 detects the main subject in the input image based on the score calculated by the score calculation unit 106. Specifically, first, for example, a score map in which scores at points on the input image calculated for the input image as shown in FIG. 9A are arranged on the input image (see FIG. 9B). Is generated. The obtained score map is adaptively learned by the binarization processing described in Non-Patent Document 3, specifically by minimizing the intra-class variance and maximizing the inter-class variance, and thresholding. Is applied to set a candidate area for the main subject as shown in FIG. Furthermore, by setting a rectangular area circumscribing the obtained candidate area for the main subject, the main subject in the input image is detected as shown in FIG. 9D, and the detection result is output.

  The main subject detection result obtained in this way is used in an apparatus that uses the main subject detection apparatus 100. For example, under the assumption that the area detected as the main subject in the digital still camera is focused and the area is improved in image quality, the CPU and program in the digital still camera that controls the main subject detection apparatus 100 Is transmitted to etc.

  As described above, according to the present embodiment, a score corresponding to the visual saliency and the information amount of the feature amount (luminance value, color component, edge strength, etc.) in the third partial region is calculated. Detect the main subject. This makes it possible to robustly detect the main subject in the image even when the main subject is not visually noticeable or noise occurs due to environmental or observational factors.

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described with reference to FIGS. 10 and 11. Note that the configuration of the main subject detection apparatus and the basic processing flow are the same as those in the first embodiment, and in this embodiment, only differences from the first embodiment will be described. In the first embodiment, the third partial region is set so that the center of gravity coincides with the first partial region and the second partial region. In contrast, in the present embodiment, the third partial region is included as a plurality of partial regions that are included in the second partial region and have different sizes regardless of the centroids of the first partial region and the second partial region. The setting points are different.

  In step S204 of FIG. 2, the second partial region setting unit 102 uses the information amount (hereinafter referred to as the first information amount) included in the feature amount (luminance value, color component, edge strength, etc.) in the second partial region. calculate. Here, the first information amount is calculated using, for example, Difference of Gaussian (DoG) described in Non-Patent Document 4, which is a type of bandpass filter. Specifically, the smoothed image L (x, y) obtained by applying the Gaussian function G (x, y, kσ) to the input image I (x, y) by the following equation (10). , Kσ) and a smoothed image L (x, y, σ) obtained by applying the Gaussian function G (x, y, σ). Then, a difference image D (x, y, σ) of these smoothed images is calculated. Here, (x, y) represents horizontal and vertical coordinates on the input image. Further, k represents an increase rate of the Gaussian parameter σ and is uniquely determined according to a calculation time allowed for the main subject detection apparatus 100 in the assumed application.

  The position and size of the third partial region set in step S205 of FIG. 2 are given as follows, for example. As shown in FIG. 10, the pixel of interest in the difference image D (x, y, σ) is compared with neighboring pixels. Then, based on the extreme value (maximum value and minimum value) obtained by this comparison or the coordinate (x, y) of the pixel value equal to or greater than the threshold value in the difference image D (x, y, σ), the Gaussian parameter σ (or A circular area whose diameter is a predetermined coefficient multiple) is obtained. As a result, the third partial region is given as a circular region having a plurality of positions and sizes in the second partial region, as shown in FIG.

  Alternatively, the position and size of the third partial region may be calculated using Laplacian of Gaussian (LoG) described in Non-Patent Document 5, which is a type of bandpass filter. In this case, the Gaussian parameter σ is based on the extreme value (maximum value and minimum value) obtained by comparing the pixel of interest in the LoG image with the neighboring pixel or the coordinates (x, y) of the pixel value equal to or greater than the threshold value in the LoG image. It is given by a circular area whose diameter is (or a predetermined coefficient multiple).

  Further, the position and the size of the third partial region may be calculated using a Gabor filter of a known image processing filter, which is a kind of band pass filter. In this case, based on the coordinates (x, y) of the extreme value (maximum value and minimum value) obtained by comparing the target pixel and the surrounding pixels in the filter output value, the Gaussian parameter σ (or a predetermined coefficient multiple thereof) ) Is a circular area with a diameter.

  When the third partial region is set as described above, the main subject can be detected by the same procedure as in the first embodiment.

(Third embodiment)
Hereinafter, a third embodiment of the present invention will be described with reference to FIG. Note that the configuration of the main subject detection apparatus and the basic processing flow are the same as those in the first embodiment, and in this embodiment, only differences from the first embodiment will be described. In the first embodiment, there is an inclusion relationship between the first partial region and the second partial region, whereas in the present embodiment, the first partial region and the second partial region The difference is that there is not necessarily an inclusive relationship between them.

  In step S <b> 202 of FIG. 2, the first partial region setting unit 101 sets the first partial region on the input image input from the outside of the main subject detection apparatus 100. Specifically, as shown in FIG. 12A, circular regions (or elliptical regions) of an arbitrary size are arranged at equal intervals on the image space, and one of them (for example, FIG. 12A) The circular area A) is set as the first partial area.

  As shown in FIG. 12B, circular regions (or ellipse regions) of an arbitrary size are partially overlapped and arranged at equal intervals on the input image, and one of them (for example, FIG. 12). The circular area A) in (b) may be set as the first partial area. Alternatively, as shown in FIG. 12C, a circular area (or an elliptical area) of an arbitrary size is randomly arranged on the input image, and one of them (for example, the circular area A in FIG. 12C). ) May be set as the first partial region.

  Alternatively, as shown in FIG. 12D, the input image is divided into a plurality of local regions, and one of them (for example, the local region A in FIG. 12D) is set as the first partial region. Also good. In this case, the input image is divided into a plurality of local regions based on a statistical distribution of feature values such as luminance values, color components, edge strength, and texture. Here, the statistical distribution is, for example, whether the histogram for the feature amount in the region is unimodal or multimodal, or whether the information amount for the feature amount in the region is greater than or equal to a threshold value. Content. The input image may be divided into a plurality of local regions using the method described in Non-Patent Document 6.

  Next, in step S203 of FIG. 2, the second partial region setting unit 102 sets a second partial region on the input image. Specifically, as shown in FIG. 12, any one of the partial areas adjacent to the first partial area (local area A in FIG. 12) set by the first partial area setting unit 101 (for example, , The local region B) in FIG. 12 is set as the second partial region.

  Next, in step S204 of FIG. 2, the first partial region setting unit 101 calculates the information amount of the feature amount (luminance value, color component, edge strength, etc.) in the first partial region on the input image. . Similarly, the second partial region setting unit 102 calculates the information amount of the feature amount (luminance value, color component, edge strength, etc.) in the second partial region on the input image.

  Next, in step S205 in FIG. 2, the third partial region setting unit 103 determines the amount of information included in the feature amount in the first partial region and the amount of information included in the feature amount in the second partial region. Based on the above, the third partial region is set. Specifically, for example, when the amount of information in the first partial region is compared with the amount of information in the second partial region, the information amount in the first partial region is larger. One partial area is set as the third partial area. On the other hand, when the amount of information in the second partial area is larger, the second partial area is set as the third partial area.

  Here, the magnitude of the information amount in the first partial region and the second partial region may be given by, for example, entropy H shown in the above-described equation (1). Alternatively, the magnitude of the information amount in each partial region may be given by calculating the gradient strength of the feature amount in the partial region at each point of the partial region and summing them. The gradient strength may be calculated using, for example, a known image processing filter (Sobel filter, Canny filter, Laplacian filter, Gabor filter, etc.).

  In step S209, the identification unit 107 detects the main subject in the input image based on the score calculated by the score calculation unit 106. Specifically, first, with respect to the input image, the score calculation unit 106 uses the first partial region obtained by the first partial region setting unit 101 and the second partial region obtained by the second partial region setting unit 102. Scores are calculated for all combinations with the partial region.

  In the first and second embodiments, the processes of steps S201 to S208 are repeated for all points on the input image. However, in this embodiment, all of the first partial region and the second partial region are all processed. The processes in steps S201 to S208 are repeated for the combination. The score calculation method is calculated using the saliency and the second information amount, as in the first embodiment.

  Then, the identifying unit 107 generates a score map in which scores for all combinations are arranged on the image space. Further, the candidate area of the main subject is set by applying the binarization processing described in Non-Patent Document 3 to the obtained score map. Furthermore, the main subject in the input image is detected by setting a rectangular region circumscribing the obtained main subject candidate region.

  As described above, according to the present embodiment, the main subject in the image is robustly detected even when the main subject is not visually noticeable or noise is generated due to environmental or observational factors. be able to.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

101 first partial region setting unit 102 second partial region setting unit 103 third partial region setting unit 104 saliency calculating unit 105 information amount calculating unit 106 score calculating unit 107 identifying unit

Claims (9)

Setting means for setting a first area, a second area including the periphery of the first area, and a third area including the first area on the input image;
First extraction means for extracting the same type of feature quantity based on luminance or color from each of the first and second regions;
Derivation means for deriving saliency based on the difference between the same type of feature values extracted from each of the first and second regions;
Second extraction means for extracting edge feature values from the third region;
Detecting means for detecting a main subject in the input image based on the saliency and the edge feature amount;
A subject detection apparatus comprising:   The subject detection apparatus according to claim 1, wherein the third region is a region inside the second region.   3. The subject detection apparatus according to claim 1, wherein the first area, the second area, and the third area have the same center of gravity.   The subject according to any one of claims 1 to 3, wherein the setting unit sets the third area to a size based on a feature value extracted from the second area. Detection device.   5. The subject detection apparatus according to claim 1, wherein the same type of feature amount is a statistical distribution of luminance values of a region. Score derivation means for deriving a score based on the saliency and the edge feature amount;
The subject detection apparatus according to claim 5, wherein the detection unit detects a main subject in the input image based on the derived score.   The detecting means generates a score map in which the scores derived at a plurality of points on the input image by the score deriving means are arranged, and detects a main subject in the input image based on a threshold for the score map. The subject detection apparatus according to claim 6, wherein Setting a first area, a second area including the periphery of the first area, and a third area including the first area on the input image;
Extracting the same type of feature quantity based on luminance or color from each of the first and second regions;
Deriving a saliency based on the difference between the same type of feature values extracted from each of the first and second regions;
Extracting an edge feature amount from the third region;
Detecting a main subject in the input image based on the saliency and the edge feature amount;
A method for detecting a subject characterized by comprising: The program for functioning a computer as each means of the to-be-photographed object detection apparatus of any one of Claim 1 to 7.

Images