JP5554286B2 - Method for determining a 3D pose of a three-dimensional (3D) object in an environment - Google Patents

Description

  The present invention relates generally to estimating posture, and more particularly to estimating posture in an environment for the purpose of geolocation.

  Computer vision applications use image features such as points and lines to solve a wide variety of geometric problems. Conventional methods use all available features and solve the problem using a least-squares criterion across all those features. On the other hand, in many computer vision problems, it has been found that the minimum solution is less susceptible to noise than the non-minimum method.

  The minimum solution was very useful as a hypothesis generator in a hypothesis creation / inspection procedure such as random sample consensus (RANSAC) (see Patent Document 1).

  The minimum solution is known for some computer vision problems. These problems include automatic calibration of radial distortion, perspective three-point problem, five-point relative attitude problem, six-point focal length problem, six-point generalized camera problem, and nine points for estimating the parabolic catadioptric matrix. The problem, and the nine-point radial distortion problem. There are also attempts to unify all known solutions.

TECHNICAL FIELD The present invention particularly relates to posture estimation using points and lines. The posture is defined as having three degrees of freedom of translation and rotation. The goal is to determine the camera pose between the camera reference system and the world reference system, given the three correspondences between points and lines in the world reference system, and their projection on the image in the camera reference system. is there. Solutions for three-wire or three-point configurations are known. As defined herein, a “reference system” is a 3D coordinate system.

  In practice, both point and line features have complementary advantages. Considering our intended geolocalization application in urban environments, it is relatively easy to obtain correspondence between 3D models and image lines using image sequences and coarse 3D models.

  Point matching between two images is also known and is easier than line matching. Although the fusion of points and lines for tracking has been described, minimal solutions that are less sensitive to outliers, imperfect matches, and narrow fields of view have not been described in the prior art.

  In recent years, minimum attitude procedures for stereo cameras that use either points or lines but not both have been described.

Image-based geolocalization There is a growing interest in inferring geolocation from images. Geospatial localization can be obtained by matching the query image with the image stored in the database using the decreasing vertical direction.

  Several 3D reconstruction procedures have been described for the reconstruction of large urban environments. One localization method matches the lines from the 3D model with the lines in the image captured by the wide-angle camera from the controlled room environment.

  However, finding an attitude in an outdoor environment such as “Urban Canyon” is a more difficult task. Attitude estimation for geolocation purposes in such environments is often difficult with GPS based solutions when radio reception is poor.

US Patent No. 7,359,526

  It is therefore desirable to provide a camera-based solution for such geolocation applications.

  Embodiments of the present invention provide a method for solving geometric problems in computer vision using a minimum number of image features. The method uses a geometric collinearity constraint and a geometric coplanarity constraint to provide solutions for four minimal geometric problems.

  Simplify the equations involved in posture estimation using an intermediate reference system. As defined herein, a “reference system” is a 3D coordinate system. Conventional application of constraints results in a solution of a 64th order polynomial. Using our intermediate reference system, the problem is reduced to 4th and 8th order polynomials.

  For minimum pose estimation, points and lines are used to give the last two missing terms.

  Pose estimation refers to the problem of recovering the camera pose given the known features (points and lines) in the world and their projections on the image. There are four minimal configurations that use point and line features: three points, two point one line, one point two line, and three lines. Solutions for the first and last configurations that rely only on either points or lines are known.

  However, mixed configurations that are more common in practice have not been solved so far.

  We can actually develop a generic technique that can solve all four configurations using the same approach, and the solution is either a 4th order polynomial or an 8th order polynomial. Indicates that it involves seeking roots.

  The key to the solution is the selection of an appropriate intermediate reference system and the use of collinearity and coplanarity constraints in solving the pose estimation problem. It also provides a powerful application for geolocalization using a rough 3D model of image sequences and poor GPS radio urban environments with poor radio reception.

  In many real world applications, it is not always possible to seek a three-point or three-line correspondence. The present invention improves on this situation by giving the option of mixing these features, thereby enabling solutions in cases that were not possible before.

  Our problem formulation using collinearity and coplanarity constraints and the technique of selecting an intermediate reference system are very useful in many applications such as geolocation and object detection in urban environments.

FIG. 5 is a schematic diagram of collinearity constraints where points in space are on corresponding backprojected rays from the camera. FIG. 5 is a schematic diagram of coplanarity constraints where a line in space is on a corresponding interpretation plane formed by a corresponding backprojected ray from the camera. FIG. 6 is a schematic diagram of an intermediate conversion technique in which a camera reference system and a world reference system are converted to corresponding intermediate camera reference systems and intermediate world reference systems according to an embodiment of the present invention. It is the schematic of the intermediate conversion for the minimum attitude | position estimation procedure using 2 points | pieces 1 line by embodiment of this invention. It is the schematic of the intermediate conversion for the minimum attitude | position estimation procedure using 1 point | piece 2 line by embodiment of this invention. It is a flow block diagram of the minimum attitude | position estimation method by embodiment of this invention. 1 is a schematic diagram of a method for building a 3D model of an urban environment used for enhanced geolocation and object detection in an urban environment. FIG.

  FIG. 6 illustrates a method for determining a 3D pose 641 of an object in the environment 602 according to an embodiment of the present invention. The posture has 3 degrees of freedom in translation (T) and 3 degrees of freedom in rotation (R).

  Features are extracted from points and lines in the image 601 captured from the environment by the camera 603 (610). These features are matched to the 3D model 609 of the environment and a correspondence 611 is determined.

  The correspondence is used to convert the camera reference frame of the image and the world reference frame of the environment into the corresponding intermediate camera reference frame and intermediate world reference frame 621. As defined herein, a “reference system” is a 3D coordinate system. Geometric constraints are applied to the intermediate reference system (630). Next, a 3D pose is determined using the constrained reference system parameters (640).

  FIG. 7 illustrates the steps that can be performed to build a 3D model of the environment, eg, an outdoor city scene. For example, a rough 3D model with no texture is captured from Google 3D Walehouse. Alternatively, the coarse model is constructed from images captured through the environment by a camera mounted on vehicle navigating.

  Textures can be added using, for example, Google and SketchUp (702). People can be put into the textured model (703), and road signs can be added using projection mapping to generate a detailed 3D model (704). During construction, several points and lines 700 of the urban environment can be extracted from the image. Next, using the methods described herein, the poses of various objects are determined using these points and lines. This detailed 3D model can then be used for geolocation and enhanced vehicle navigation services that perform automatic detection of pedestrians, vehicles, traffic signals, and road signs.

  Alternatively, the method can be used in robot settings where a minimum number of point and line features are used during automated manufacturing during posture estimation for grasping industrial parts and robot navigation in indoor scenes. Can be used. For example, the environment is an indoor factory setting and the 3D posture is used to grab small industrial parts using a robot arm.

  All of the above method steps may be performed in a processor 600 connected to memory and input / output interfaces known in the art.

  The method will now be described in more detail.

Collinearity and coplanarity As shown in FIGS. 1 and 2, our method uses only two geometric constraints, namely collinearity and coplanarity, and all four minimal configurations. To solve. The collinearity constraint arises from the 2D-3D correspondence 611. In a typical imaging setup, every pixel in the image corresponds to a 3D projection ray and the corresponding 3D point is on this ray. Here, x, y, and z are 3D coordinates.

The minimal solution described herein uses a projected ray CD ₁ connected to a 2D image feature, and the associated 3D point P ₁ in the environment is collinear when represented in the reference system. .

In FIG. 2, the two projected rays CD ₁ and CD ₂ connected to the endpoints of the 2D line and the associated 3D line represented by the _two points L ₁ and L ₂ are all coplanar.

In FIG. 1, the projected ray CD ₁ and the point P ₁ are on the ray when represented in the same intermediate reference system. Our goal is to find the orientation (R, T) where point P ₁ is on ray CD ₁ under orientation (R, T), where R is rotation. , T is translation. Stack these points into the following matrix: This matrix is called a collinearity matrix.

  The first collinearity constraint forces the determinant of any submatrix with the above matrix size 3 × 3 to disappear. In other words, four constraints are obtained by removing one row at a time. Four equations are obtained from the above matrix, but only two are independent and are useful for this reason.

The second coplanar geometric constraint arises from the 2D-3D line correspondence. As shown in FIG. 2, points C, D ₁ , D ₂ , P ₁ , and P ₂ are on a single plane when represented in the same reference frame. In other words, in the case of the correct posture [R, T], two constraints are acquired from a single 2D-3D line correspondence. That is, the quadruples (C, D ₁ , D ₂ , [R, T] L ₁ ) and (C, D ₁ , D ₂ , [R, T] L ₂ ) are coplanar, respectively. The coplanarity constraint of the quaternion (C, D ₁ , D ₂ , [R, T] L ₁ ) forces the determinant of the following matrix to disappear.

Similarly, the other quadruple (C, D ₁ , D ₂ , [R, T] L ₂ ) also gives coplanarity constraints. Thus, every 2D-3D line correspondence yields two equations from two points on the line.

  Our goal is to determine 6 parameters (3 for R and 3 for T) where 3D features (both points and lines) satisfy collinearity and coplanarity constraints. Thus, it has four possible minimum configurations (3 points, 2 points 1 line, 1 point 2 lines, and 3 lines).

Reference System As shown in FIG. 3, the camera reference system and the world reference system in which points and lines exist are represented by C ₀ and W ₀ , respectively.

The goal of the present inventors is to find the transformation between the world reference system W ₀ and the camera reference system C _0. First, C ₀ and W ₀ are converted into an intermediate camera reference system C ₁ and an intermediate world reference system W ₁ , respectively. Next, determine the attitude between these intermediate world reference system W ₁ and the intermediate camera reference system C _1. This can be used to determine the attitude between the camera reference system and the world reference system. Solutions using these intermediate reference systems greatly reduce the order of the resulting polynomial. That is, the conventional 64 (2 ⁶ ) degree polynomial is reduced to the fourth order polynomial and the eighth order polynomial of the present invention.

The transformation (R _w2c , T _w2c ) is obtained. This represents 3D points and lines in the world reference system and the camera reference system. The direct application of the collinearity and coplanarity constraints results in six linear equations with 12 variables (9 R _ij , 3 T _i ). To solve these variables, further equations are needed. These equations can be six quadrature orthogonality constraints.

The solution of such a system will eventually result in a 64 (2 ⁶ ) degree polynomial with 64 solutions. Finding such a solution may not be feasible for some computer vision applications. Our main goal is to provide a way to overcome this difficulty. To do this, first, both the camera reference system C ₀ and the world reference system W ₀ are converted into an intermediate camera reference system C ₁ and an intermediate world reference system W ₁ , respectively.

After this transformation, our goal is to find the attitude (R, T) between the intermediate reference frames. The intermediate camera reference system C ₁ and the intermediate world reference system W ₁ are selected so that the resulting polynomial is the lowest possible order. A posture estimation procedure for two minimum configurations will be described. The other two configurations can be handled in a similar manner.

Minimum solution 2 point 1 line The posture estimation procedure from the correspondence of two 2D-3D points and one 2D-3D line is provided. From the 2D coordinates of the points, the corresponding projected rays can be determined using calibration. In a 2D line configuration, the corresponding projected ray at the end point of the line in the image can be determined. In the following, only 3D projection rays associated with point and line features on the image will be considered.

4A to 4D show the camera reference system C ₀ and the world reference system W ₀ , and the intermediate camera reference system C ₁ and the intermediate world reference system W ₁ in posture estimation from the two-point one-line configuration. Show. The camera reference systems before and after conversion are shown in FIGS. 4A and 4C, respectively. Similarly, the world reference system before and after conversion is shown in FIGS. 4B and 4D, respectively.

Intermediate camera reference system C ₁
4A and 4B show camera projection rays (associated with 2D points and lines) and 3D features (points and lines) in camera reference system C ₀ and world reference system W ₀ , respectively. ing. The center of the camera is C ⁰ and the projected rays corresponding to the two 2D points are given by their direction vectors d ₁ (→) and d ₂ (→), and the projected rays corresponding to the 2D line are Are given by the vectors d ₃ (→) and d ₄ (→) (where the notation (→) means that a symbol is added above the symbol before the parenthesis to represent a vector).

In the intermediate camera reference system C _1, always represents the projection ray camera using two points (a point on the center and light). The projected rays corresponding to the two 2D points are CD ₁ and CD ₂ , and the lines are CD ₃ and CD ₄ . The plane formed by the triplet (C, D ₃ , D ₄ ) is called the interpretation plane.

Selecting satisfies intermediate camera reference system C ₁ below.
a. The camera center is at coordinates C (0, 0, -1),
b. Is one CD ₃ of the projection rays corresponding to line _L 3 _{L 4,} it is on the Z-axis such that D 3 = (0,0,0), and c. The other projected ray CD ₄ corresponding to line L ₃ L ₄ is on the XZ plane such that D ₄ is on the X axis.

Using a constructive declination, any set of projected rays corresponding to a two-point-one line can be transformed. Let P ₀ and P represent the coordinates of any point in the reference systems C ₀ and C, respectively. According to this notation, points D ₃ and D ₄ are represented by C ₀ and C ₁ using algebraic differentiation.

The attitude (R _c1 , T _c1 ) between C ₀ and C ₁ is given by what transforms the triplet (C ₀ , D ₃ , D ₄ ) into (C, D ₃ , D ₄ ).

Intermediate World Reference System W ₁
The intermediate world reference system can be described as follows. Let the Euclidean distance between any two 3D points P and Q be represented by d (P, Q). Two 3D points and one 3D point on the 3D line in W ₁ are represented by the following equations.

Here, X ₃ and Y ₃ can be obtained using trigonometry.

The posture (R _w1 , T _w1 ) between the world reference system W ₀ and the intermediate world reference system W ₁ is a triple (P ⁰ ₁ , P ⁰ ₂ , L ⁰ ₃ ) (P ₁ , P ₂ , Is given by what converts to P ₃ ). For simplicity, the following notation is used in the posture estimation procedure.

Pose estimation first step between the intermediate camera reference system C ₁ and the intermediate world reference system W ₁ is to stack the matrices all available collinearity limitations and coplanarity constraints. In this case, we have two collinear matrices for triplets (C, D ₁ , P ₁ ) and (C, D ₂ , P ₂ ) corresponding to 3D points P ₁ and P ₂ respectively. As shown in equation (1), these two collinearity matrices give four equations.

In addition, it has two coplanarity equations from quadruples (C, D ₃ , D ₄ , L ₃ ) and (C, D ₃ , D ₄ , L ₄ ) corresponding to 3D line L ₃ L ₄ . After stacking the constraints from the (partial) matrix determinant, we get the linear system AX = B. Here, A, X, and B are the following expressions.

The matrix A has known variables and has rank 6. Since there are eight variables in the linear system, the solution can be obtained in a subspace in which two vectors are spread.
a. X = u + l ₁ v + l ₂ w
Where u, v, and w are known vectors of size 8 × 1.

Next, the unknown variables l ₁ and l ₂ are estimated using orthogonality constraints from the rotation matrix. Two orthogonality constraints with rotation variables can be written.

Substituting these rotational variables as a function of l ₁ and l ₂ and solving the above simultaneous quadratic equations, we obtained four solutions for (l ₁ , l ₂ ), and therefore (R ₁₁ , R ₂₁ , R ₃₁ , R ₂₂ , R ₂₃ ), obtain four solutions.

  The five elements in the rotation matrix uniquely determine other elements using orthogonality constraints. For this reason, the two-point one-line configuration provides a total of four solutions for posture (R, T).

One-point two-line This section describes a posture estimation procedure from one 2D-3D point and two 2D-3D line correspondences.

Intermediate camera reference system W ₁
FIGS. 5A and 5B show the camera projection rays (associated with 2D points and lines) and 3D features (points and lines) in the camera reference system C ₀ and the world reference system W ₀ , respectively. ing. In the camera reference system C ₀ , the camera center is at C ⁰ , and the projection rays corresponding to the 2D points are given by their direction vectors d ₁ (→), and the projection rays corresponding to the two 2D lines are Given by the pair of direction vectors (d ₂ (→), d ₃ (→)) and (d ₄ (→), d ₅ (→)). In the intermediate camera reference system C ₁ , the ray corresponding to the 2D point is CD ₁ , and the rays connected to the two lines are pairs (CD ₂ , CD ₃ ) and (CD ₃ , CD ₄ ), respectively. is there.

By satisfying the following conditions, to select the C _1.
a. The camera center is at coordinates C (0, 0, -1).
b. The projected ray CD ₃ from the intersection of the two interpretation planes is on the Z axis such that D ₃ = (0, 0, 0), and c. Ray CD ₂ is on the XZ plane, but D ₂ is on the X axis.

Similar to the above configuration, we prove that such a transformation is possible by construction. The unit normal vectors of the interpretation planes (C ₀ , d ₂ (→), d ₃ (→)) are n ₁ (→) = d ₂ (→) × d (→) and d ₂ (→) = d _4. (→) × d (→). The direction vector of the intersecting line of the two planes can be obtained as d ₁₂ (→) = d ₁ (→) × d ₂ (→). Direction vectors d ₂ (→), d ₁₂ (→), and d ₄ (→) at C ₀ correspond to projected rays CD ₂ , CD ₃ , and CD ₄ , respectively.

Using algebraic transformation, the points D ₂ and D ₃ before and after the transformation to the intermediate reference system are as follows:

The transformation between the camera reference system C ₀ and the intermediate camera reference system C ₁ is given by what projects the triplet (C ⁰ , D ⁰ ₂ , D ⁰ ₃ ) onto (C, D ₂ , D ₃ ) It is done.

Intermediate World Reference System W ₁
The intermediate world reference system W ₁ is selected such that a single 3D point P ₁ is at the origin (0, 0, 0). The transformation between the world reference frame W ₀ and the intermediate world reference frame W ₁ translates P ⁰ ₁ to P ₁ . For simplicity, the following notation is used for some points in C ₀ and W ₀ .

He described the attitude estimation using the pose estimation 1 point 2-wire correspondence between the intermediate camera reference system C ₁ and the intermediate world reference system W _1. From two collinearity equations from triplicates (C, D ₁ , P ₁ ) and from quadruples (C, D ₂ , D ₃ , L ₂ ) and (C, D ₂ , D ₃ , L ₃ ) Are stacked with the four coplanarity equations. Construct the following linear system: AX = B. Here, A, X, and B are represented by the following expressions.

In the linear system AX = B, the first and second rows are obtained for the triplet (C, D ₁ , P ₁ ) using the collinearity constraint of equation (1). The third row, the fourth row, the fifth row, and the sixth row are divided into four pairs (C, D ₂ , D ₃ , L ₂ ), (C, D ₂ , D ₃ , L ₃ ), (C , D ₃ , D ₄ , L ₄ ), and (C, D ₃ , D ₄ , L ₅ ), using the coplanarity constraint of equation (2).

The matrix M has known variables and has rank 6. Since there are nine variables in the linear system, a solution can be obtained as shown below in a subspace in which two vectors spread. That is, X = u + l ₁ v + l ₂ w + l ₃ y, where u, v, w, and y are known vectors of size 9 × 1, and l ₁ , l ₂ , and l ₃ are unknown Variable. The three orthogonality constraints include the rotation variables R ₁₁ , R ₁₂ , R ₁₃ , R ₂₁ , R ₂₂ , and R ₂₃ (individual elements in X are l ₁ , l ₂ , and l ₃ Represented as a function).

In order to obtain an efficient and stable solution of the above equation, eight different solutions of (l ₁ , l ₂ , l ₃ ) are obtained using an automatic solver constructed using the Gröbner basis method. As a result, this generated eight solutions of posture (R, T).

Degenerate configuration and other scenarios Between 3D features, if the 3D point is on the 3D line, the configuration is degenerate. With the same idea of coordinate transformation and the use of collinearity and coplanarity constraints, it is possible to solve 3 points and 3 lines.

Claims (12)

A method for determining a 3D pose of a three-dimensional (3D) object in an environment,
Extracting features from an image captured from the environment by a camera;
Matching the features to a 3D model of the environment, seeking correspondence, matching;
Converting the camera reference system of the image and the world reference system of the environment into a corresponding intermediate camera reference system and a corresponding intermediate world reference system using the correspondence;
Applying a geometric constraint to the intermediate camera reference system and the intermediate world reference system, obtaining a constrained intermediate world reference system and a constrained world reference system;
Determining the 3D pose from parameters of the constrained intermediate world reference system and the constrained world reference system, wherein these steps are performed in a processor.   The method of claim 1, wherein the environment is an outdoor city scene and the 3D pose is used for geolocation.   The method of claim 1, wherein the posture is used for automatic navigation.   The method of claim 1, wherein the environment is an indoor factory setting and the 3D attitude is used to grab a small industrial part using a robotic arm.   The method of claim 1, wherein the geometric constraints are collinearity constraints and coplanarity constraints.   The method of claim 1, wherein the feature includes two points and a line.   The method of claim 1, wherein the feature includes a point and two lines.   The method of claim 1, wherein the parameters include three rotational parameters and three translation parameters of the attitude.   The method of claim 1, wherein the attitude is obtained using a fourth-order polynomial.   The method according to claim 1, wherein the posture is obtained using an eighth order polynomial.   In the intermediate camera reference system, the center of the camera is at the coordinates C (0, 0, −1), the projected light of the first line corresponds to the line on the Z axis, and the projection of the second line The method of claim 1, wherein the rays correspond to lines on the XZ plane.   The method of claim 1, wherein the pose is estimated using a random sample consensus framework.

Images