JP2015133101A - Method for constructing descriptor for image of scene - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide global descriptors for matching Manhattan scenes used for viewpoint-invariant object matching.SOLUTION: A descriptor is associated with a vanishing point 101 in an image by first quantizing an angular region around the vanishing point into a preset number of angular quantization bins, and a centroid of each angular quantization bin indicates a direction of the angular quantization bin. For each angular quantization bin, a sum of the magnitude of pixel gradients for pixels in the image at which a direction of the pixel gradient is aligned with the direction of the angular quantization bin is determined. These steps are performed in a processor 150.

Description

  The present invention relates generally to computer vision, and more particularly to global descriptors for matching Manhattan scenes that can be used for viewpoint invariant object matching.

  Viewpoint invariant object matching is difficult due to image distortion caused by factors such as rotation, translation, lighting, cropping, and occlusion. Understanding visual scenes is a well-known problem in computer vision. In particular, the identification of objects in a 3D scene based on projection onto a two-dimensional (2D) image plane presents an unmanageable challenge.

  It is known that the human visual cortex relies heavily on the presence of edges at physical object boundaries to identify individual objects within the field of view. Using clues from edges, textures, and colors, the brain can typically visualize and understand a three-dimensional (3D) scene regardless of viewpoint. In contrast, modern computers lack high-level processing architectures such as the visual cortex, so low-level viewpoint invariance must be explicitly incorporated into the scene descriptor.

  Methods for scene understanding include two broad categories. One class relies on local keypoints that can be accurately detected regardless of rotation, translation, and other viewpoint changes. In this case, descriptors for those keypoints are constructed to capture local structures of gradients, textures, colors, and other information that remain unchanged with respect to viewpoint changes. Scale-invariant feature transform (SIFT) and accelerated robust feature (SURF) are examples of two keypoint-based descriptors.

  Another class of methods involves capturing features in a global scope. Accuracy is obtained by local averaging and using other statistical properties of the color and gradient distribution. This global approach is used in gradient histograms (HOG) and GIST descriptors.

  Local and global approaches have complementary features. Local descriptors are accurate and discriminatory for the corresponding local keypoints, but lack global structural cues for larger objects, and between several local descriptors associated with those keypoints Can only be guessed after establishing the corresponding relationship. Global descriptors tend to capture aggregate statistical information about the image, but do not include specific geometric or structural cues that are often related to scene understanding.

  Many artificial scenes satisfy the Manhattan World Hypothesis. In this hypothesis, the line is oriented along three main orthogonal directions. A very important aspect of Manhattan geometry is that all parallel lines with dominant directions intersect at a vanishing point in the 2D image plane. In scenes where there may not be three orthogonal directions, a line can satisfy a single dominant direction, for example vertical or horizontal, or it can have multiple dominant non-orthogonal directions, for example, indoor furniture objects. Can be included.

  Embodiments of the present invention provide global descriptors for Manhattan scenes. Manhattan scenes typically have a dominant directional orientation in three orthogonal directions. Thus, all parallel edges in 3D that are in the dominant direction intersect without exception at the corresponding vanishing point (VP) in the 2D image plane. All scene edges retain their relative spatial location and intensity as viewed from the VP. The global descriptor is based on the spatial location and brightness of the image edges in the Manhattan scene around the vanishing point. The method uses 8 kilobits per descriptor and up to 3 descriptors per image (one for each VP), compared to local keypoint descriptors such as SIFT for efficient storage. And provide data transfer.

  The method constructs a global descriptor by maintaining the strict angular ordering of parallel lines across the image as they intersect at the vanishing point. The relative length and relative angle (orientation or direction) of these parallel lines that meet at the vanishing point are approximately the same.

  A compact global image descriptor for Manhattan scenes captures the relative location and intensity of edges along the disappearance direction. To construct this descriptor, an edge map is determined for each vanishing point. This edge map encodes the edge strength over the range of angles or directions measured for the vanishing point.

  For object matching, descriptors from two scenes are compared across multiple candidate scales and displacements. Matching performance is improved by comparing the edge shape at the maximum of the scale-displacement plot in the form of a histogram.

It is a figure which shows the image of the Manhattan scene containing two vanishing points with which the global descriptor by the embodiment of this invention is comprised. FIG. 6 is a schematic diagram illustrating various angles and angular quantization bins made with respect to a horizontal reference line at a vanishing point location according to an embodiment of the present invention. FIG. 4 is a schematic diagram of binned pixel luminance of an edge map according to an embodiment of the present invention. FIG. 6 shows a diagrammatic edge strength of angle bins of two different views of a building according to an embodiment of the present invention. 3 is a flowchart of a method for constructing a global descriptor according to an embodiment of the present invention; FIG. 4 is a schematic diagram of affine transformation of two images according to an embodiment of the present invention. 4 is a histogram of edge strength on a scale-displacement plot according to an embodiment of the present invention. 3 is a flowchart of a method for matching objects using a global descriptor according to an embodiment of the present invention; It is a figure explaining the metric for measuring the quality of the matching by embodiment of this invention.

  Embodiments of the present invention provide a global descriptor 250 for the Manhattan scene 100. A Manhattan scene usually has a dominant directional orientation in three orthogonal directions, and all parallel edges in 3D in one dominant direction intersect at a corresponding vanishing point (VP101) in the 2D image plane. Note that the Manhattan scene can be indoor or outdoor, and can include any number of objects.

  Descriptor 250 consists of image 120 acquired by camera 110 (500). These descriptors can then be used for object matching 800 or other related computer vision applications. These configurations and matching can be performed in a processor 150 connected to the memory and input / output interface by a bus as is known in the art.

Vanishing Point Based Image Descriptor The descriptor is based on the following perception of multiple images 120 (views) of the same object. First, parallel lines in an actual 3D scene maintain their angular order strictly across the 2D image (up to an inversion) as they intersect at the vanishing point. Second, the relative lengths and relative angles of the parallel lines that meet at the vanishing point are approximately the same. These recognitions suggest that the relative location and strength of edges oriented along the disappearance direction can be used to construct the descriptor. The steps involved in configuring (250) the descriptors 250 and using these descriptors for matching are described below.

Descriptor seeding at each vanishing point The vanishing point is defined as the intersection of projections of parallel lines 102 in the 3D scene where the 2D image 100 is available. A VP can be viewed as a 2D projection of a 3D point at infinity in the direction given by parallel lines in a 3D scene.

  In general, there are many vanishing points corresponding to a plurality of scene directions determined by parallel lines. However, many man-made structures, such as cityscapes, have a regular cubic geometry. Thus, typically three vanishing points are obtained from the image projection, two of which are shown in FIG.

  VP has been used in computer vision for image correction, camera calibration, and related problems. VP identification is simple when parallel lines in the underlying 3D scene are labeled, but becomes more difficult when labeling is not available. Methods for determining vanishing points include edge aggregation clustering, 1D Hough transform, multi-level random sample consensus (RANSAC) based method, and expectation maximization method (EM) for assigning edges to VPs It is.

によって示すことができる。ここで、通常、マンハッタンシーンの場合には、ｍ≦３である。さらに、ＶＰ

において水平基準ライン２０１に対してなす角度をθ_ｊ（ｘ，ｙ）とする。したがって、

である。 As shown in FIG. 2, the VP location 200 is

Can be indicated by Here, normally, in the case of a Manhattan scene, m ≦ 3. In addition, VP

Is the angle formed with respect to the horizontal reference line 201 at θ _j (x, y). Therefore,

It is.

Descriptor 250 is constructed by encoding the relative location and strength of edges that converge at each VP. Thus, the descriptor can be viewed as a function D: Θ → R ⁺ , whose domain includes the angular orientation of edges that converge to VP, whose range is a measure of the strength of these edges in the correct order. including. Descriptors are determined for each VP according to method 500 described below.

Edge Location Coding Line detection procedures often generate broken and cropped lines, fail to find important edges, and generate false lines. Therefore, as shown in FIG. 3, to obtain accuracy, we will deal directly with the brightness of the edge pixels, not the lines that match the image edges. The representation of the edge strength as a function of the angular location of the edge around the vanishing point is called the edge map 300. Specifically, when the gradient indicates that the pixels are oriented according to the vanishing points for constructing the descriptor, the luminance of the pixels in the angle bin 202 is stored and summed individually as shown in FIG. To do. To do this, a gradient g (x, y), which is a 2D vector, is first determined (510) for every pixel in the image, as shown in FIG.

The pixel gradient direction ψ _g (x, y) 511 at location (x, y) in the image refers to the direction along which a large luminance change exists. The gradient magnitude | g (x, y) | 512 refers to the luminance difference at the pixel along the gradient direction.

のピクセルセットＰ_ｊを求める（５２０）。

ここで、τは、勾配方向がＶＰの方向と一致していない量に基づいて選択された閾値である。このセットＰ_ｊが求められると、基礎となるエッジロケーションは、以下のように符号化される。 Next, the vanishing point VP is as follows:

The pixel set P _j is determined (520).

Here, τ is a threshold value selected based on an amount in which the gradient direction does not coincide with the VP direction. Once this set P _j is determined, the underlying edge location is encoded as follows:

であるような、画像にわたる角度範囲［θ_ｍｉｎ，θ_ｍａｘ］２０４内のφ_ｋ，１≦ｋ≦Ｋを中心とする（２０３）一様な角度ビン２０２の事前に設定された数（Ｋ）に量子化され、そのため、角度量子化ビンの重心は、角度量子化ビンの方向、すなわち、ピクセル角度を示す。 Pixel angle (direction) is

A pre-set number (K) of uniform angle bins 202 centered at φ _k , 1 ≦ k ≦ K in the angular range [θ _min , θ _max ] 204 over the image, such that Therefore, the centroid of the angle quantization bin indicates the direction of the angle quantization bin, ie the pixel angle.

Edge Strength Coding Research on the human visual system suggests that the relative prominence of edges plays a role in visualizing discriminatory object patterns. Image saliency is a function of edge length, thickness, and lateral change (luminance and falloff characteristics) in a direction perpendicular to the edge.

  There are several ways to construct the edge strength metric. For example, if an edge detector is used to construct a descriptor for a particular VP, the intensity can be a function of the length of the edge and the cumulative gradient in pixels along the edge. However, as explained above, using an edge detector is not always accurate. Therefore, methods based on pixel-by-pixel gradient clustering or quantization are preferred. This process is described in detail below.

When pixel set P _j is uniformly quantized into angle bins 202, one way to encode edge strength is to use the magnitude of gradient | g (x, y) | 512 in each angle quantization bin. It is to calculate the total. To do this, as shown in FIG. 2, any angular quantum with endpoints (r _{k, min} cos φ _k , r _{k, min} sin φ _k ) and (r _{k, max} cos φ _k , r _{k, max} sin φ _k ) Consider a line segment 203 that passes through the center of the bin.

ここで、φ_ｋ，１≦ｋ≦Ｋ_ｊは、ＶＰ

に対する量子化ビンに関連付けられた角度配向又は方向を表し、ｒは、半ピクセル（ｈａｌｆ−ｐｉｘｅｌ）解像度における範囲内で変化することができる。 In this case, the descriptor 250 is the following sum.

Here, φ _k , 1 ≦ k ≦ K _j is VP

Represents the angular orientation or direction associated with the quantization bin for, and r can vary within a range in half-pixel resolution.

  To obtain accuracy, bilinear interpolation is used to obtain the pixel gradient at the subpixel location. The configuration 500 of descriptor D (k) 250 is performed at sub-pixel resolution. An example descriptor obtained as above by determining the edge strength in each angle bin is shown in FIG. 4 for two different views of the same (building) object 401. The corresponding graph shows the normalized luminance sum as a function of bin index.

Configuration Method FIG. 5 summarizes the basic steps of the configuration method. For each pixel in the image 120, the gradient direction 511 and magnitude 512 will be determined. Next, a set of gradients 521 having a direction that coincides with the vanishing point is determined. There can be up to three vanishing points. The gradient magnitudes are then individually summed for each set and encoded as edge strength (530) to obtain a descriptor 250 for each vanishing point.

Projective Transformation Our motivation behind constructing the global descriptor 250 (500) is to perform a matching 800 of objects in images obtained from different viewpoints. Since each image is a 2D projection of the same real-world scene, there is usually a geometric relationship between corresponding keypoints or edges in the image pair. For example, there is a homography relationship between the flat front images of the composition. Our recognition suggests that there is an affine correspondence between descriptors D (k) 250 determined for images of the same object.

  In the following, it is explained that this recognition has theoretical validity. In particular, it shows that the transformation of the angle between the image lines (edges) used in the binning step between (500) constituting the descriptor is approximately affine.

ここで、λ∈Ｒである。 Consider two images (views) of the same scene consisting of “pencils” of lines passing through the vanishing point, as shown in FIG. It is assumed that the vanishing point of the first view is located at the origin. Using the same kind of representation, the x and y axes are given by e _x = (010) ^T and e _y = (100) ^T. Here, T is a transpose operator. Using these vectors, an arbitrary line l _λ is expressed as:

Here, λεR.

Without loss of generality, mutual angle being considered (inter-angle) is assumed to be the angle between the x axis and l _lambda. Note that θ _λ = tan ⁻¹ (−λ). Our goal is to show that the angle between the x-axis and l _λ undergoes an approximate affine transformation from one image to the other. To illustrate this, a 3 × 3 homography between two views will be shown using the matrix H. In general, under homography, the vanishing point is no longer at the origin of the second view, and He _x is no longer along the x axis. Here, as shown in FIG. 6, the transformation given by another 3 × 3 matrix T is selected which translates the vanishing point and returns it to the origin, and rotates He _x to return to the x axis.

ここで、

であり、（ａ_１，ａ_２，ｂ_１，ｂ_２）は、Ｔ及びＨの要素から導出された変換パラメーターである。消失点が画像から遠く離れており、そのため、θ_ｍａｘ−θ_ｍｉｎが小さいという仮定の下では、テーラー級数近似ｔａｎ^−１（α）≒αを用いることができる。ここで、αは、小さな角度（ラジアンで表される）である。したがって、以下の式となる。

The TH conversion of l _lambda indicated by l _gamma, will be indicated by the angle between the l _gamma and x-axis theta _gamma. In this case, the following equation is obtained.

here,

Where (a ₁ , a ₂ , b ₁ , b ₂ ) are transformation parameters derived from the elements of T and H. Under the assumption that the vanishing point is far from the image and θ _max −θ _min is small, the Taylor series approximation tan ⁻¹ (α) ≈α can be used. Here, α is a small angle (expressed in radians). Therefore, the following equation is obtained.

Using a small reciprocal angle assumption, the second order term θ _γ θ _λ is negligibly small. If this cross term is ignored, the conversion from θ _λ to θ _γ is approximately affine.

Descriptor Matching An object in a Manhattan scene can have up to 3 VPs and thus can have up to 3 descriptors. Therefore, matching objects visible from two viewpoints without prior orientation information involves up to nine matching operations. As explained above, angular edge locations undergo an approximate affine transformation with a change in viewpoint. We therefore propose to invert this transformation before comparing the relative shape of edge strengths in matched descriptor pairs. This inversion step is performed using several candidate scales and displacements, ie several candidate affine transformations. A dominant affine transformation (scale-displacement) pair can be selected from these candidate affine transformations. Method 800 is used to compare the descriptors as described below.

Corresponding mapping for edges To find an approximate affine transformation that translates descriptors between viewpoints, under the correct correspondence, a pair of coplanar edges is given by a scale-displacement pair (s, d) To generate an affine parameter equal to the approximated value. Therefore, the Hough transform type voting procedure in the (s, d) space of the edge pair results in a maximum value at the true scale s ^* and displacement d ^* .

  Multiple maxima occur when an object has multiple planes supported by the VP direction axis. In order to obtain accuracy and efficiency, prominent edges are identified based on their edge strength. Pixels on the edge that have an intensity greater than the specified percentile threshold are selected. In addition, to obtain accuracy for edge occlusion, only edges within a close range of angles are paired for voting, eg, each prominent edge is paired with C nearest edges. .

及びｄ＝ｍ_ｊ−ｓｋ_ｉを用いて（ｓ，ｄ）ヒストグラムの票が生成される。角度反転、すなわち、ＶＰの回りの上部／底部及び左／右の回転を可能にするために、上記２つのセットのうちの一方の中のピークの順序を逆にすることによって、追加の票が生成される。 Descriptor _{D 1 (k), 1 ≦} k ≦ K is, _{N 1} peaks pair _{(k i, k 'i)} , it is possible to generate a set of ₁ ≦ i ≦ _N 1. _{Similarly, D} 2 (m) is, _{N 2} peaks pair _{(m j, m 'j)} , generating a set of 1 ≦ j ≦ _{N 2.} These identified peak pairs are cross-mapped between the two sets,

And _d = m j -sk _i using (s, d) the vote of the histogram is generated. Additional votes can be obtained by reversing the order of the peaks in one of the two sets to allow angle reversal, ie top / bottom and left / right rotation around the VP. Generated.

As shown in FIG. 7, a coarse histogram 700 of (s, d) votes can be used here to locate the local maximum (s ^* , d ^* ). This histogram identifies the scale and displacement for which the two VP-based descriptors have the best match. The local maximum provides the relationship between the edges in the two views of the object. If the maximum value contains too few votes, a mismatch is declared for that (s ^* , d ^* ) pair. If none of the maxima contain enough votes, their descriptors do not represent the same object.

  Accordingly, each descriptor is changed so that the scale and displacement of the descriptors are the same. Next, the difference between the shape of the peak in the first descriptor and the shape of the corresponding peak in the second descriptor is determined, and when this difference is less than a threshold, it indicates a match between the two images. it can.

Matching Method FIG. 8 summarizes the basic steps of the matching method 800. For images 801 and 802, the respective descriptors 811 and 812 are configured as described above (500). Peaks 821 and 822 are identified (820) and a vote for histogram 700 is generated (830). These peaks identify the scale and displacement for which the two VP-based descriptors have the best match.

  It should also be noted that the descriptor can be used as a query to a database of images to retrieve images of similar scenes.

Shape Matching at Corresponding Edges Each local maxima (s ^* , d ^* ) improves the matching process by utilizing the edge strength plots in the two descriptors being compared, eg the local shape of the plot of FIG. can do. In essence, after compensating for the scaling factor s ^* and displacement d ^* , what remains is to compare the shape of the edge strength plots in the vicinity of the edge pair voted for (s ^* , d ^* ). It is. There are several ways to do this. One embodiment is described below.
a) As shown in FIG. 9, perform the following steps for each salient peak to construct a metric for measuring the quality of the match.
b) Consider a region near the peak angle of the first descriptor.
c) Find the accumulated edge strength vector in this neighborhood and normalize this vector so that the sum of all edge strengths is 1.
d) Repeat this process for each matching salient peak in the second descriptor.
e) Find the absolute distance between the normalized cumulative edge intensity vectors for each pair of matching peaks taken one by one from each descriptor.
f) The absolute distance obtained in step (d) is averaged over all matching peak pairs, possibly generated from multiple bins, and compared to a threshold value.
g) If the average distance between the normalized cumulative edge strength vectors is less than the threshold, a match between the two descriptors is declared.

Claims (16)

A method of constructing a descriptor for an image of a scene, said descriptor being associated with a vanishing point in the image,
The method
Quantizing an angular region around the vanishing point into a predetermined number of angular quantization bins, wherein a centroid of each angular quantization bin indicates a direction of the angular quantization bin; and ,
Determining, for each angular quantization bin, a sum of pixel gradient magnitudes of pixels in the image and a direction of the pixel gradient that matches the direction of the angular quantization bin;
Including
The steps are performed in a processor;
How to construct a scene image descriptor. The scene is a Manhattan scene with the Manhattan World Hypothesis,
The method of claim 1. The angular quantization bin is uniform;
The method of claim 1. The angular quantization bin is determined by clustering the direction of the pixel gradient;
The direction is measured with respect to the location of the vanishing point;
The method of claim 1. The pixel gradient is determined individually for each pixel,
The method of claim 1. The pixel gradient is used to determine edge strength by performing edge detection on the image, and to determine the pixel gradient of only the pixels having an edge strength greater than a specified percentile threshold as a peak. Is,
The method of claim 1. The gradient is determined at subpixel locations.
The method of claim 1. Comparing a first descriptor composed of two images acquired from different viewpoints of the scene with a second descriptor;
The method of claim 1, further comprising: Configuring a metric that measures the quality of the matching;
The method of claim 8, further comprising: Identifying, from the descriptor of each image, the pixel having an edge strength greater than a specified percentile threshold as a peak;
Scale-displacement such that the peak pair selected from the first descriptor, cross-mapped according to a given scale and displacement value, corresponds to the peak pair selected from the second descriptor. Generating a plot; and
Identifying one or more local maxima in the scale-displacement plot;
Comparing two descriptors using the scale and displacement values at each local maximum;
The method of claim 8, further comprising: The comparing step includes:
Changing each descriptor such that the scale and the displacement of the descriptor are the same;
Determining a difference between the peak in the first descriptor and the peak in the second descriptor;
Declaring a match between the two images when the difference is less than a threshold;
The method of claim 10, further comprising: The step of obtaining the difference includes
Calculating a cumulative edge strength near an angle of the peak for corresponding peaks in the first descriptor and the second descriptor;
Normalizing the cumulative edge strength so that the sum of the edge strengths near the angle of the peak is 1.
Calculating a distance between the normalized cumulative edge strength of the first descriptor and the normalized cumulative edge strength of the second descriptor;
The method of claim 11, further comprising: Retrieving a similar image from a database of images based on the descriptor;
The method of claim 1, further comprising: The vanishing point pixel set is

And
Where the direction of the gradient of the pixel at location (x, y) in the image is ψ _g (x, y);
θ _j (x, y) is an angle formed with respect to a horizontal reference line at the vanishing point,
τ is a threshold selected based on the amount that the direction does not match the direction of the vanishing point;
The method of claim 1.

Quantizing the direction into a predetermined number (K) of bins centered around φ _k , 1 ≦ k ≦ K within the angular range [θ _min , θ _max ], such that
The method of claim 1, further comprising: The descriptor is

And
Where φ _k , 1 ≦ k ≦ K _j represents the direction of the bin, and r varies in range with half-pixel resolution.
The method of claim 15.

Images