JP6517652B2 - Object saliency map calculation device, method and program - Google Patents

Description

  The present invention relates to an object saliency map calculation apparatus, method, and program, and more particularly, to an object saliency map calculation apparatus, method, and program for calculating an object saliency map from an image.

  The object saliency map quantitatively indicates in which region in the image the main object is likely to be reflected, and for each pixel constituting the image, the likelihood that the pixel is included in the main object ( The saliency score is defined. Object saliency maps are very commonly used as components of object recognition (object segmentation and object detection) and image editing (resize and filtering), and their industrial utility value is extremely high.

  One of the representative methods of calculating the object saliency map is based on the detection of local contrast changes in the image. For example, in Non-Patent Document 1, an object is remarkable because of the Uniqueness of pixels (the higher the contrast with pixels close in space is higher) and the Distribution (where the pixels with similar appearance are concentrated locally in the image). A method of calculating gender is disclosed.

  As a method of calculating an object saliency map, a method based on extraction of an object-like region (object candidate region) in an image is also representative. For example, Non-Patent Document 2 discloses a method of extracting a plurality of object candidate regions from an image and calculating object saliency of each pixel as a sum of object likeness scores of all object candidate regions including the pixel.

F. Perazzi et al., Saliency Filters: Contrast Based Filtering for Salient Region Detection, in CVPR, 2012. J. C. Rubio et al., Unsupervised Co-segmentation through Region Matching, in CVPR, 2012.

  In the method based on local contrast change detection as in Non-Patent Document 1, basically, small areas with high contrast with the periphery are calculated with higher saliency, so if the appearance of the main object is complicated, There is a problem that only the saliency of a part of the main object is detected extremely high, and the entire object of the main object can not be accurately captured from the obtained object saliency map. For example, when an image as shown in FIG. 5A is used as the input, the object saliency map as shown in FIG. 5B can be obtained by using the method disclosed in Non-Patent Document 1. In Fig. 5 (B), only the saliency of a part (vinyl boat and the character above it) of the main object (plate on which the illustration is written) is detected extremely high, and a clear view of the main object is obtained. It has not been done.

  However, since the object candidate area based on the object saliency map calculation method as described in Non-Patent Document 2 is specified from a very basic image feature such as color or edge, each object candidate area to be extracted is an image There is a problem that not only the main objects in the map are correctly captured, and therefore the calculated object saliency map does not necessarily capture the entire surface of the main objects. FIG. 5C is an object saliency map calculated by the method disclosed in Non-Patent Document 2. As apparent from FIG. 5, although the saliency of the wider area of the main object is highly evaluated compared to FIG. 5 (B), it is a part of the main object, and the main object total view It can not be said that it is caught in (C).

  As described above, in the known technique of calculating the object saliency map, there is a problem that the main object can not be correctly captured by the calculated object saliency map.

  The present invention has been made to solve the above problems, and provides an object saliency map calculation apparatus, method, and program that can calculate an object saliency map that more accurately captures the main object in an image. The purpose is

  In order to achieve the above object, an object saliency map calculation device according to a first aspect of the present invention calculates an object saliency map calculation method in an object saliency map calculation device that calculates an object saliency map of the image using an image as an input. The object candidate area extraction unit extracts a plurality of object candidate areas in which an object-like part is captured from the input image; and the spatial overlap degree calculation unit calculates the object candidate area for each of the extracted object candidate areas. Calculating the degree of spatial redundancy for each of the pair of the candidate area and each of the other object candidate areas, and the object candidate area ranking unit for each of the pairs calculated for each of the object candidate areas Ranking the plurality of object candidate regions on the basis of the degree of overlap, and the salient score calculating unit further comprising: rankining obtained by the object candidate region ranking unit The object candidate region of the upper with one or more, and executes include a step of calculating a remarkable score of each pixel of the input image.

  In the object saliency map calculation method according to the first invention, the step of ranking the object candidate area ranking unit is performed based on the degree of duplication for each of the pairs calculated for each of the object candidate areas. The plurality of object candidate areas may be ranked such that the object candidate areas overlapping in a large area with another object candidate area having high likelihood of a main object are ranked higher.

  Further, in the object saliency map calculation method according to the first invention, the step of ranking the object candidate area ranking unit uses the duplication degree for each of the pairs calculated for each of the object candidate areas as an element Based on the matrix A and the order matrix D in which the sum of the overlapping degrees with each of the other object candidate areas for each of the plurality of object candidate areas is a diagonal component, until a predetermined condition is satisfied And updating the ranking score of each of the plurality of object candidate areas repeatedly so that the previous ranking becomes higher as the object candidate area overlaps in a broader area with the upper object candidate area. The object candidate areas may be ranked.

  Further, in the object saliency map calculation method according to the first invention, in the step of ranking by the object candidate area ranking unit, the object candidate not overlapping with all other object candidate areas in each of the object candidate areas Even if the ranking score for the region is zero and the ranking score of each of the plurality of object candidate regions is repeatedly updated until the predetermined condition is satisfied, the plurality of object candidate regions are ranked. Good.

  An object saliency map calculation device according to a second aspect of the present invention is an object saliency map calculation device that calculates an object saliency map of an image using an image as an input, and a plurality of object saliency map calculation devices that capture an object-like part from an input image. Calculate the degree of spatial redundancy for each of the pair of the object candidate area and each of the other object candidate areas, for each of the extracted object candidate areas, and an object candidate area extraction unit for extracting an object candidate area A spatial overlap degree calculation unit, an object candidate area ranking unit for ranking the plurality of object candidate areas based on the overlap degree for each of the pairs calculated for each of the object candidate areas, and the object candidate area And a saliency score calculation unit for calculating a saliency score of each pixel of the input image using one or more ranking candidate object candidates obtained by the ranking unit. It is.

  Further, in the object saliency map calculation device according to the second invention, the object candidate area ranking unit is a main object likeness based on the overlapping degree for each of the pairs calculated for each of the object candidate areas. The plurality of object candidate areas may be ranked such that the higher the object candidate area overlapping with the other high object candidate area in the larger area, the higher the priority.

  Further, in the object saliency map calculation device according to the second aspect, the object candidate area ranking unit includes a matrix A having the degree of duplication for each of the pairs calculated for each of the object candidate areas as an element; Based on the degree matrix D in which the sum of the degree of overlap with each of the plurality of object candidate regions with each of the other object candidate regions is a diagonal component, the previous ranking is performed until a predetermined condition is satisfied. Repeating the updating of the ranking scores of each of the plurality of object candidate areas so that the object candidate area overlapping with the upper object candidate area in a wider area becomes higher, the plurality of object candidate areas It may be ranked.

  The program concerning the 3rd invention is a program for making a computer perform each step of the object saliency map calculation method concerning the 1st invention.

  According to the object saliency map calculation apparatus, method, and program of the present invention, a plurality of object candidate regions in which a portion likely to be an object is captured is extracted from an input image, and an object candidate region is extracted for each extracted object candidate region. A degree of spatial redundancy is calculated for each of the pairs with each of the other object candidate regions, and a plurality of object candidate regions are ranked based on the degree of redundancy for each of the pairs calculated for each of the object candidate regions. By calculating the saliency score of each pixel of the input image by using one or more object candidates in the ranking ranking, an object saliency map in which the main object in the image is captured more correctly can be calculated. Be

It is a block diagram which shows the structure of the object saliency map calculation apparatus based on embodiment of this invention. It is a figure which shows an example of extraction of an object candidate area | region. It is a figure which shows an example of calculation of space duplication degree. It is a flowchart which shows the object saliency map calculation processing routine in the object saliency map calculation apparatus based on embodiment of this invention. It is an example of the object saliency map by the technique of a prior art and this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Overview of Embodiment of the Present Invention>

  First, an outline of the embodiment of the present invention will be described.

  The method according to the embodiment of the present invention has been made in view of the above problems. Among the known techniques, especially in the method based on the extraction of object candidate regions, the obtained object saliency map also necessarily correctly captures the main objects, because the extracted candidate regions do not necessarily capture the main objects. It was not possible. In order to solve this problem, the object candidate areas are ranked based on the degree of spatial overlap between one or more object candidate areas obtained by arbitrary object candidate area extraction processing, and the object candidate area of the ranking upper rank is used for the object By calculating the saliency map, an object saliency map that captures the main objects in the image more correctly is calculated. The ranking based on the degree of spatial overlap between object candidate areas is an object candidate area that correctly captured the main object, assuming that the object candidate area is ideally ranked with the degree of correctly capturing the main object. Is based on the consideration that it may share a larger area with another object candidate area having a high degree of correct capture of the main object. By ranking object candidate areas so that object candidate areas overlapping with many other object candidate areas come higher, it becomes possible to select an object candidate area that has captured the main object more correctly, By using the object candidate area, as shown in FIG. 5D, it is possible to calculate an object saliency map in which the main object in the image is captured more correctly.

<Configuration of Object Saliency Map Calculation Device According to Embodiment of the Present Invention>

  Next, the configuration of the object saliency map calculation device according to the embodiment of the present invention will be described. As shown in FIG. 1, the object saliency map calculation device 100 according to the embodiment of the present invention stores a CPU, a RAM, a program for executing an object saliency map calculation processing routine described later, and various data. And a ROM containing the This object saliency map calculation device 100 functionally includes an input unit 10, a calculation unit 20, and an output unit 50 as shown in FIG.

  The input unit 10 receives an input image.

  The calculation unit 20 is configured to include an object candidate area extraction unit 30, a spatial overlap degree calculation unit 32, an object candidate area ranking unit 34, and a salient score calculation unit 36.

  The object candidate area extraction unit 30 extracts, from the input image, a plurality of object candidate areas in which a portion that seems to be an object is captured, as described below.

More specifically, the object candidate area extraction unit 30 selects N ₁ object candidate area sets from the input image.

Extract N ₁ is a parameter which takes a positive integer value, and may be 300, for example. As a method of extracting the object candidate area set ^ R, for example, the method of evaluating the object likeness of any rectangular partial area in the image as disclosed in, for example, the reference 1 and the reference 2 It is sufficient to select the top N ₁ region with high likelihood. In addition, when representing a vector, a matrix, or a set, it is represented by attaching "^" to the front of a symbol.

[Reference 1] B. Alexe et al., Measuring the Objectness of Image Windows, TPAMI, 2012.

[Reference 2] M. M. Cheng et al., BING: Binarized Normed Gradients for Objectness Estimation at 300 fps, in CVPR, 2014.

Alternatively, as disclosed in Reference 3 and Reference 4, a plurality of partial regions of arbitrary shape having a consistent appearance are extracted, and the object likeness is evaluated, and the object likeness is particularly high. The top N ₁ regions may be selected as the object candidate region set ^ R. The object rectangle candidate region set B can be uniquely defined as a set of minimum rectangular regions surrounding each object candidate region r _i ∈R.

[Reference 3] J. Carreira et al., CPMC: Automatic Object Segmentation Using Constrained Parametric Min-Cuts, TPAMI, 2012.

[Reference 4] P. Arbelaez et al., Multiscale Combinatorial Grouping, in CVPR, 2014.

Furthermore, it is possible to use a method of extracting a plurality of partial regions of an arbitrary form that are consistent in appearance, as disclosed in Reference 5 and Reference 6. In this case, since the object-likeness of each partial region is not given, the N ₁ region may be selected at random from the partial regions, or all of the obtained partial regions may be selected as object candidate regions (in this case, The number of extracted partial areas is set to N ₁ ).

[Reference 5] J. R. R. Uijlings et al., Selective Search for Object Recognition, IJCV, 2013.

[Reference 6] P. Krahenbuhl et al., Geodesic Object Proposal, in ECCV, 2014.

  In addition, although the object candidate area | region directly obtained by the method currently disclosed by the reference document 3, the reference document 4, the reference document 5, and the reference document 6 can take arbitrary shapes, as shown in FIG. It may be converted into a rectangular object candidate area as obtained in Reference 1 and Reference 2 and used in subsequent processing. An object candidate area of an arbitrary shape can be converted into a rectangular object candidate area, for example, by defining a minimum rectangular area surrounding the object candidate area.

  The space overlap degree calculation unit 32 sets, for each of the object candidate areas extracted by the object candidate area extraction unit 30, a pair of the object candidate area and each of the other object candidate areas as described below. The degree of spatial redundancy is calculated for each of the object candidate area pairs).

In the space overlap degree calculation unit 32, the space overlap degree of the object candidate area pair is calculated based on the binary mask b corresponding to the object candidate area r. Here, the binary mask b is the same size as the input image, and is defined as data in which 1 is assigned to pixels included in the object candidate area r and 0 is assigned to pixels not included. Here, it is considered to calculate the degree of spatial overlap a _ij of a pair of the object candidate area r _i and the object candidate area r _j (1 ≦ i, j ≦ N ₁ , i) j). The degree of spatial overlap a _ij can be calculated using, for example, Jaccard similarity defined by the following equation (1).

  The Jaccard similarity is calculated intuitively as shown in FIG. 3 by dividing the area of the common area of the object candidate area pair by the area of the union region of the object candidate area pair. Generally, when an object rectangular area takes an arbitrary shape, it is necessary to compare high-dimensional features to calculate Jaccard similarity, but when an object candidate area obtained by the object candidate area extraction unit 30 is a rectangle, FIG. As shown in the lower part of the figure, taking advantage of the fact that the area of the common area of the object candidate area pair is also rectangular, it is preferable because the Jaccard similarity can be calculated at high speed by several operations (such as area calculation of rectangles). It is.

  The object candidate area ranking unit 34 is, as described below, based on the overlap degree for each of the object candidate area pairs calculated for each object candidate area by the space overlap degree calculation unit 32 and has high main object likeliness. A plurality of object candidate areas are ranked such that the object candidate areas overlapping in a wide area with the object candidate area are ranked higher.

  More specifically, a matrix AA having the degree of overlap as an element for each of the object candidate area pairs calculated for each of the object candidate areas by the space degree of overlap calculation unit 32, and for each of a plurality of object candidate areas Based on the order matrix ^ D having the sum of the degree of overlap with each of the other object candidate regions as a diagonal component, the previous ranking is performed in the upper object candidate region and the wide region until the predetermined convergence condition is satisfied. Updating the ranking score of each of the plurality of object candidate areas is repeated to rank the plurality of object candidate areas so that the overlapping object candidate areas become higher.

  Here, it is assumed that the object candidate area is ranked higher in the object candidate area that overlaps with the object candidate area higher in ranking. In the present embodiment, the convergence condition is a predetermined number of times.

  Specifically, the object candidate area ranking unit 34 uses, for example, the following method, to rank the object candidate areas in order of being higher as object candidate areas overlapping in a wider area with the object candidate area higher in ranking. .

First, a vector having the ranking score p _i > 0 of each object candidate region r _i as an element

And calculate the vector ^ p which satisfies the following equation (2).

  However,

I assume.

  Matrix here

The element has the spatial degree of overlap a _ij obtained by the spatial degree of overlap calculation unit 32. However, a _ii = 0. Also, the matrix

Is the order matrix of the matrix ^ A. Specifically, the matrix D D has only diagonal components, and the sum of the degrees of overlap with each of the other object candidate regions is used as the diagonal component. That is, the value of the diagonal component is

It is. The above equation (2) is

And if all the diagonal elements of the matrix ^ D are nonzero, the solution of the vector ^ p is uniquely given. (2) As a method of solving the equation, in the present embodiment, the initial value p ₀ of the vector ^ p is

Then, the vector ^ p is updated according to the following equation (3) until convergence.

Where t is an index representing the number of repetitions. Let _T be the number of times of convergence, and let ^ p = ^ p _T hereafter. By ranking the object candidate areas in descending order of the ranking score of the obtained vector pp, the object candidate areas may be ranked higher in order of object candidate areas overlapping with the object candidate area higher in ranking. it can.

  Further, the object candidate area ranking unit 34 uses the vector p p with the ranking score as zero for object candidate areas that do not overlap with all other object candidate areas in each of the object candidate areas, using the above (3) Repeat updating vector ^ p according to the equation.

This is because if there are object candidates that are completely spatially isolated from one another, that is, there is no spatial overlap with other candidate regions, among the extracted object candidate regions, N ₁ If the above equation (1) is to be calculated using all of the object candidate regions of, there is a problem that the solution vector ^ p can not be calculated because zero elements are mixed in the diagonal elements of the calculated matrix ^ D. is there. This can be solved, for example, as follows. First, a vector ^ p 'is calculated from the equation (3) for an N ₁ ' object candidate region obtained by excluding all spatially isolated object candidate regions. Next, the ranking score of the excluded isolated object candidate region may be set to zero and inserted into the corresponding position of the vector ^ p 'to calculate the solution vector ^ p. It is the principle in the embodiment of the present invention that the ranking score of the isolated object candidate area is zero, that is, the object candidate area that correctly captured the main object is another object candidate area having a high degree of correctly capturing the main object. Based on the viewpoint of sharing many regions, it is considered that an object candidate region in which there is no overlapping object candidate region is considered not to capture the main object.

  The saliency score calculation unit 36 calculates a saliency score of each pixel of the input image using one or more ranking candidate object regions obtained by the object candidate region ranking unit 34, and outputs the saliency score to the output unit 50.

Specifically, the saliency score calculation unit 36 uses the ranking upper N ₂ (≦ N ₁ ) object candidate region obtained by the object candidate region ranking means to set the saliency score s (x) of each pixel x in the image. Calculate N ₂ is a parameter. For example, when N ₁ = 300, it may be set as N ₂ = 30. Pronounced score s of the image in pixels x (x) is a binary mask b _i corresponding to the object candidate region r _i ∈ ^ R ', with the score vector ^ p, be calculated by the following equation (4) Can. Where ^ R 'is a set of object candidate region of ranking N _2.

<Operation of Object Saliency Map Calculation Device According to Embodiment of the Present Invention>

  Next, the operation of the object saliency map calculation device 100 according to the embodiment of the present invention will be described. When the input unit 10 receives an input image, the object saliency map calculation device 100 executes an object saliency map calculation processing routine shown in FIG. 4.

  First, in step S100, a plurality of object candidate regions in which a portion likely to be an object is captured are extracted from the input image received by the input unit 10.

  Next, in step S102, for each of the object candidate areas extracted in step S100, the degree of spatial redundancy for each of the pair of the object candidate area and each of the other object candidate areas according to the equation (1). Calculate

  In step S104, based on the matrix AA and the order matrix に 従 っ て D according to the equation (3), the previous ranking is higher in the object candidate area overlapping with the upper object candidate area in a wider area. Then, the vector ^ p consisting of the ranking score of each of the plurality of object candidate regions is updated.

  In step S106, it is determined whether the convergence condition is satisfied, and if the convergence condition is satisfied, the process proceeds to step S108. If the convergence condition is not satisfied, the process returns to step S104 to repeat the update.

  In step S108, a plurality of object candidate regions are ranked based on the vector p p updated in step S104.

In step S110, the salient score of each pixel of the input image is calculated according to the equation (4) using the object candidate region of ranking top N ₂ obtained in step S108, and output to the output unit 50 for processing. finish.

  As described above, according to the object saliency map calculation device according to the embodiment of the present invention, a plurality of object candidate regions capturing a portion likely to be an object are extracted from an input image, and for each extracted object candidate region Calculating a spatial degree of overlap for each of the pair of the object candidate area and each of the other object candidate areas, and based on the degree of overlap for each of the pairs calculated for each of the object candidate areas, By calculating the saliency score of each pixel of the input image by ranking object candidate regions and using one or more object candidates in the top ranking, an object saliency map that more accurately captures the main object in the image is calculated. it can.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the scope of the present invention.

DESCRIPTION OF REFERENCE NUMERALS 10 input unit 20 operation unit 30 object candidate area extraction unit 32 space overlap degree calculation unit 34 object candidate area ranking unit 36 salient score calculation unit 50 output unit 100 object saliency map calculation device

Claims (4)

An object saliency map calculation method in an object saliency map calculation device for calculating a saliency score of each pixel of the input image from an input image,
A step of object candidate region extraction unit extracts a plurality of object candidate area captured object seems portion from said input image,
Calculating a spatial degree of overlap for each of the pair of the object candidate area and each of the other object candidate areas, for each of the extracted object candidate areas;
An object candidate area ranking unit uses a matrix A having the degree of duplication for each of the pairs calculated for each of the object candidate areas as an element, and another object candidate area for each of the plurality of object candidate areas An object candidate area in which the previous ranking overlaps in a wide area with the upper object candidate area until a predetermined condition is satisfied based on the order matrix D in which the sum of the overlapping degrees with each other is a diagonal component Updating the ranking score of each of the plurality of object candidate regions so as to be approximately higher, and ranking the plurality of object candidate regions;
Calculating a saliency score of each pixel of the input image by using a one or more ranking candidate object regions obtained by the object candidate region ranking unit;
Object saliency map calculation method including. The step in which the object candidate area ranking unit performs ranking is performed by setting the ranking score of the object candidate areas that do not overlap all the other object candidate areas among the object candidate areas as zero, and satisfying the predetermined condition. up, repeatedly updating the ranking score of each of the plurality of object candidate region, the object saliency map calculation method according to claim 1 for ranking the plurality of object candidate region. An object saliency map calculation device that calculates a saliency score of each pixel of the input image from an input image,
An object candidate region extraction unit for extracting a plurality of object candidate area captured object seems portion from said input image,
A spatial overlap degree calculation unit that calculates a spatial overlap degree for each of the pair of the object candidate area and each of the other object candidate areas, for each of the extracted object candidate areas;
A matrix A having the degree of duplication for each of the pairs calculated for each of the object candidate areas as an element, and the degree of duplication with each of the other object candidate areas for each of the plurality of object candidate areas Based on the order matrix D having the sum as a diagonal component, the previous ranking becomes higher in the object candidate area overlapping with the upper object candidate area in a wider area until the predetermined condition is satisfied, An object candidate area ranking unit that ranks the plurality of object candidate areas by repeatedly updating the ranking score of each of the plurality of object candidate areas ;
A saliency score calculation unit that calculates a saliency score of each pixel of the input image using one or more ranking candidate object candidate regions obtained by the object candidate region ranking unit;
Object saliency map calculation device including. The program for making a computer perform each step of the object saliency map calculation method of Claim 1 or Claim 2 .