KR101057936B1 - 3D Object Retrieval and Pose Estimation Based on Object's Appearance at Arbitrary Angles - Google Patents

Abstract

The present invention relates to a three-dimensional object search and attitude estimation method based on the appearance of the object at any angle, the three-dimensional object search and attitude estimation method of the present invention (a) the appearance information of all three-dimensional objects stored in the database Calculate distance curve, combine images with mirror image generated by camera rotation into one feature information, and use distance curve of remaining images after removing overlapping pattern according to object symmetry. Constructing an index; (b) After calculating the distance curve of the query image, search similar objects based on the most similar appearance by applying a similarity comparison method to the distance curve of the query image through the three-dimensional object data index structure constructed in step (a). Making; (c) extracting a candidate pose of the object including an image having the same appearance as the most similar appearance found in step (b); And (d) estimating a posture determined to be the most similar from the posture of the object in the query image by applying a SIFT algorithm between the images extracted from the candidate posture of the object extracted in step (c) and the query image. Include.

Pose estimation, 3D object search, contour based search, distance curve

Description

SHAD-BASED 3D OBJECT DETECTION AND POSE ESTIMATION METHOD USING ARBITARY VIEW IMAGE}

The present invention relates to a three-dimensional object search and attitude estimation method, and more particularly to a three-dimensional object search and attitude estimation method based on the appearance of the object at any angle.

As 3D data is gradually generated and used in various areas, 3D object retrieval and pose prediction methods using a single image query are actively used in many areas such as security, CAD, virtual reality, medical image analysis, robot automation, and place recognition. Is being studied. In order to achieve this goal through a single image query, we must consider all possible images of the object according to the camera position. This requires an enormous amount of computation time and storage space. Therefore, an efficient index structure is needed.

On the other hand, one of the methods used in the three-dimensional object search using a single image query is to use the appearance information of the object. Similar to how a person recognizes an object, this method of using the object's appearance information can achieve satisfactory results in a relatively short time, but only the appearance information can be used because the same appearance can occur depending on the posture of the object. The disadvantage is that the exact position of the object is not known.

Another popular method for 3D object retrieval and posture prediction using a single image query is the Scale-Invariant Feature Transform (SIFT) algorithm. The SIFT algorithm has the advantage of being able to search accurately even if the rotation and size change of the image occurs, but it has a big disadvantage that it takes a lot of time in the preprocessing or similarity comparison process because a large amount of feature information is extracted and used. .

In order to speed up the speed of the SIFT algorithm, a Speed-Up Robust Feature (SURF) has been proposed. SURF can compare similarities in a much faster time than SIFT, but has a disadvantage in that accuracy is greatly reduced by using a small amount of feature information in order to speed up the similarity comparison.

Therefore, effective index structure is needed to complement the characteristics of SIFT. Various methods such as binary tree based on the number of points recognized as the index structure of the SIFT have been proposed, but due to the lack of the generalized index structure of the SIFT, many related studies have compared the similarity between the query image and the 3D data. It is done sequentially.

Therefore, in order to search for 3D objects in real time and to predict attitudes, there is an urgent need for a more effective index structure construction and a fast retrieval method.

The present invention efficiently reduces the amount of data for similar object search without affecting the accuracy of the three-dimensional object search, and achieves a fast search speed by using only the appearance information of the three-dimensional object. It is to provide.

Another object of the present invention is to provide a method for estimating a pose of a 3D object capable of predicting a pose most similar to a query image of a 3D object.

To achieve the above object, a three-dimensional object search and attitude estimation method based on the appearance of an object at any angle of the present invention comprises: (a) a distance curve as appearance information of all three-dimensional objects stored in a database; Calculating, combining images having the mirror image generated by the rotation of the camera into one feature information, and constructing an index using distance curves of the remaining images after removing the overlapping contour pattern according to the symmetry of the object; (b) After calculating the distance curve of the query image, search similar objects based on the most similar appearance by applying a similarity comparison method to the distance curve of the query image through the three-dimensional object data index structure constructed in step (a). Making; (c) extracting a candidate pose of the object including an image having the same appearance as the most similar appearance found in step (b); And (d) estimating a posture determined to be the most similar from the posture of the object in the query image by applying a SIFT algorithm between the images extracted from the candidate posture of the object extracted in step (c) and the query image. Include.

According to the three-dimensional object search and pose estimation method of the present invention through the above configuration, it is possible to quickly and accurately search the three-dimensional object and attitude estimation.

In order to search for 3D objects through a single camera image query, it is necessary to consider all possible images of the object. Therefore, an excessively large amount of data must be considered for similar object search. The present invention proposes a method that can dramatically reduce the amount of such data by using the symmetry of the object without affecting the accuracy of the search.

In addition, the present invention proposes a similar object search method based on the appearance information using only the appearance information of various camera images of the three-dimensional object to search for similar objects. Furthermore, when only the appearance information is used to search for an object, a plurality of objects having the same appearance may be generated. Accordingly, the present invention proposes a method capable of predicting a pose closest to the query image among possible poses of the object of the same appearance.

It is to be understood that both the foregoing general description and the following detailed description are exemplary, and that additional explanations of the claimed invention are provided. Reference numerals are shown in detail in preferred embodiments of the invention, examples of which are indicated in the reference figures. In any case, the same reference numerals are used in the description and the drawings to refer to the same or similar parts. DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings so that those skilled in the art may easily implement the technical idea of the present invention.

1 is a flowchart illustrating an example of a 3D object search and pose estimation method according to the present invention. Referring to FIG. 1, first, a distance curve is calculated as appearance information of all three-dimensional objects stored in a database, and the images having mirror images generated by the rotation of the camera are combined into one feature information. The index is constructed using the distance curves of the remaining images after removing the overlapping external pattern according to the symmetry (S101). Detailed procedures for the index construction method according to the present invention will be described in detail later with reference to FIG.

After calculating the distance curve of the query image in the next step, the similarity comparison method with the distance curve of the query image is applied through the three-dimensional object data index structure configured in step S101 to search for similar objects based on the most similar appearance (S103). ). In this case, dynamic indexing based on dynamic time warping (DTW) is configured to ensure rotational invariance.

Specifically, first, two functions, UB_Dist () and LB_Dist (), are used to remove the distance curves most similar to those of the query image. UB_Dist () always returns a distance greater than or equal to the distance of the search method used. Similarly, LB_Dist () always returns a distance less than or equal to the distance of the search method used. If this is expressed as an expression, it is as follows.

UB_Dist () ≥ Matching method () ≥ LB_Dist ()

The present invention uses Dynamic Time Warping (DTW) as a search method. Therefore, the LB_Keogh distance can be used as the LB_Dist function and the Euclidean distance can be used as the UB_Dist function. After finding the smallest distance among the UB_Dist values for all the distance curves, the number of candidate distance curves is reduced by removing all distance curves in which the LB_Dist value is larger than the smallest UB_Dist value. Through this process, after removing a plurality of candidates and dynamically indexing only a few candidates, a distance curve having a minimum distance from the distance curve of the query image is searched.

Subsequently, a candidate pose of an object including an image having the same appearance as the retrieved most similar appearance is extracted (S105). Specifically, the process follows.

FIG. 2 shows objects with different poses, in which (a) and (d) are different camera images, (b) and (e) are their outline contours, and (c) and (f) are their distances It is a curve. As shown in Fig. 2, although their postures (visual characteristics) are different, their outline contours (distance curves) are the same or opposite to each other. Since the duplicated appearance has been eliminated in the index structure construction step of step S101, such a candidate attitude can no longer be used.

Therefore, it is necessary to dynamically calculate candidate poses and retrieve corresponding images from the database, or generate candidate poses from three-dimensional objects in the database using software tools such as computer-aided design (CAD). In the latter case, the coordinates for the most similar pose and its angle from the candidate pose are needed. This is calculated as follows.

First, candidate postures of 180 / H _periods can be obtained according to the horizontal repetition period H _period . The horizontal angle between the most similar posture and the candidate posture can be easily calculated using Equation 1 below.

Where 1 ≦ i ≦ (180 / H _period ), j is the horizontal angle between the most similar pose and the candidate pose, and k is the angle of the most similar pose.

A SIFT algorithm between the images extracted from the candidate postures of the object extracted in step S105 and the query image is applied to finally estimate a posture determined to be the most similar from the posture of the object in the query image (S107). Since the SIFT algorithm is well known in the art to which the present invention pertains, a detailed description thereof will be omitted.

3 shows SIFT characteristics of two camera viewpoint change images for the same appearance. As shown in FIG. 3, since the camera viewpoint change image for the same appearance may have different SIFT features, SIFT may be used to effectively identify the most similar pose. In general, SIFT requires a significant amount of time to calculate a feature. However, in the present invention, since the SIFT feature for the small number of candidate poses is required through step S105, this is not a big problem.

4 is a flowchart illustrating a method of constructing an index structure for searching for similar objects according to the present invention. Referring to FIG. 4, first, a distance curve obtained by calculating the distance between the center point and the outline of all three-dimensional objects stored in the database as the appearance information of the object is calculated (S401). FIG. 5 exemplarily shows a detailed procedure of calculating a distance curve as the appearance information of an object. In FIG. 5, (a) is an image of a three-dimensional object, and (b) is a distance between a center point and an outline along an outline. (C) shows the distance curve of the image.

On the other hand, all three possible images of an object must be considered to search for a 3D object through a single image query, which requires a large amount of computation time and storage space. Most objects have left and right and / or symmetry, and they can be classified as follows according to the types of objects as shown in FIG. 6. (1) in the case of symmetrical objects, the mirror image has a mirror image appearance; (2) the rear image of any kind of object has the mirror image of the front image; (3) The object and the projected image of the object have the same appearance. Based on this feature, the present invention can minimize the number of images to be indexed without influencing the accuracy of the search by combining mirror images and eliminating duplicate appearance patterns. This will be described in detail as follows.

Regardless of the type of object, the front and back images have the same mirror image. In this case their distance curves are opposite to each other and thus this Discrete Fourier transform (DFT) returns the same value. Therefore, when the discrete Fourier transform value is used for indexing, the mirror image may be combined using one discrete Fourier transform feature value (S403). The index space can be reduced by half while maintaining the accuracy by combining the mirror image and reconstructing the mirror image when searching.

Since the entire viewpoint of the 3D object may be generated through a combination of horizontal and vertical camera movements, symmetry of the object in two horizontal and vertical planes may be considered. Depending on the symmetry of the object, four different classes can be defined for each plane. First, four different horizontal classes H1 to H4 are defined as follows according to the symmetry of the outer and front sides.

H1: This class contains the same object as all possible horizontal camera images, such as spheres.

H2 This class contains an object whose appearance pattern is repeated every 90 degrees, like a car.

H3: This class contains objects whose appearance patterns repeat more often than 90 degrees, such as dice. For example, in the case of cubes such as dice, the appearance pattern is repeated every 45 degrees.

H4: This class contains objects whose appearance patterns are not repeated from any horizontal camera image.

In the same way as the horizontal class, the vertical classes V1 to V4 are defined by the symmetry of the contours in the front and plane. The properties of the vertical class are the same as the horizontal class. As described above, the present invention configures 16 classes according to horizontal and vertical symmetry of objects, and classifies corresponding objects according to classes. Then, by analyzing the overlapping appearance patterns and removing the duplicated appearances (S405), index space and search time can be reduced without sacrificing the accuracy of the search.

The index structure is constructed using the distance curves of the remaining images after the mirror image combination in step S403 and the overlapping contour pattern removal process in step S405. An efficient index structure is essential for efficient three-dimensional object retrieval performance, because taking into account all possible images of an object according to the camera's position requires an enormous amount of database capacity. However, constructing an index structure for a complete distance curve is inefficient because of a problem known as "curse of dimensionality."

Therefore, the present invention reduces the dimension of the index by using the DFT used in the mirror image combination of step S403. The DFT can be used to construct a rotation invariant index structure because it guarantees rotation invariance.

Meanwhile, in the detailed description of the present invention, specific embodiments have been described, but various modifications may be made without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the above-described embodiments, but should be defined by the equivalents of the claims of the present invention as well as the following claims.

1 is a flowchart illustrating a three-dimensional object search and attitude estimation method according to the present invention.

2 is an exemplary view showing two different postures of the same appearance.

3 is an exemplary view showing SIFT characteristics of two camera viewpoint change images for the same appearance.

4 is a flowchart illustrating a method of constructing an index structure for searching for similar objects according to the present invention.

5 is an exemplary view showing a detailed procedure for calculating a distance curve as appearance information of an object.

6 is an exemplary view showing an appearance pattern according to symmetry of an object.

Claims (12)

In the three-dimensional object search and attitude estimation method based on the appearance of the object at any angle, (a) constructing an index using a distance curve as appearance information of all three-dimensional objects stored in a database; (b) After calculating the distance curve of the query image, search similar objects based on the most similar appearance by applying a similarity comparison method to the distance curve of the query image through the three-dimensional object data index structure constructed in step (a). Making; (c) extracting a candidate pose of the object including an image having the same appearance as the most similar appearance found in step (b); And (d) applying a SIFT algorithm between the images extracted from the candidate postures of the object extracted in step (c) and the query image to estimate the posture finally determined to be the most similar from the posture of the object in the query image; But In step (a), (a-1) calculating a distance curve obtained by calculating distances between the center point and the outline of all three-dimensional objects stored in the database as the appearance information of the object; (a-2) combining the images having a mirror image generated by the rotation of the camera into one feature information; (a-3) removing the overlapping shapes by analyzing the overlapping appearance patterns according to the symmetry of the object; And (a-4) 3D object search and pose estimation method comprising the step of constructing an index using the distance curve of the remaining images after removing the duplicated appearance through the step (a-3). delete The method of claim 1, wherein the step (a-2) combines the images having the mirror image into one feature information by using one discrete Fourier transform feature value. The method according to claim 1, wherein the step (a-3), Composing 16 classes according to the symmetry of objects and classifying the objects according to the classes; And 3. The method of claim 3, further comprising removing the overlapping shapes by analyzing the overlapping shapes of the classified objects. The three-dimensional object of claim 4, wherein the sixteen classes are a combination of four different horizontal classes defined according to the horizontal symmetry of the object and four different vertical classes defined according to the vertical symmetry of the object. Search and posture estimation method. The method of claim 1, wherein the step (a-4) reduces the dimension of the index using a Discrete Fourier Transform. The method according to claim 1, wherein step (b), (b-1) removing distance curves most similar to distance curves of the query image using two functions of UB_Dist () and LB_Dist (); (b-2) removing the plurality of candidates through step (b-1) and dynamically indexing only the remaining candidates; And (b-3) searching for a distance curve having a minimum distance from the distance curve of the query image; The method of claim 7, wherein the searching is performed by dynamic time warping. The method of claim 8, wherein the LB_Dist () is an LB_Keogh distance, and the UB_Dist () is an Euclidean distance. The method of claim 1, wherein the step (c) is performed by dynamically calculating the candidate pose and searching for a corresponding image in a database. The method of claim 1, wherein step (c) is performed by generating candidate poses from three-dimensional objects in a database using a computer-aided design (CAD). The three-dimensional object of claim 11, wherein the candidate pose is obtained by a number of 180 / H _periods according to a horizontal repetition period H _period , and a horizontal angle between the most similar pose and the candidate pose is calculated by the following equation. Search and posture estimation method.

Where 1 ≦ i ≦ (180 / H _period ), j is the horizontal angle between the most similar pose and the candidate pose, and k is the angle of the most similar pose.

Images