CN102221358A - Monocular Vision Positioning Method Based on Inverse Perspective Projection Transformation - Google Patents

Abstract

The invention provides a monocular vision positioning method based on inverse perspective projection transformation. The adopted technical scheme is as follows: fix the attitude sensor and the camera together, install them on the wheeled vehicle, and perform the following processing on the images taken during driving : The first step is to perform inverse perspective projection transformation on the image sequence; the second step is to calculate the transformation matrix between adjacent images; the third step is to determine the trajectory curve of the wheeled vehicle. The beneficial effect of the present invention is that the real-time attitude information obtained by the attitude sensor is used to assist the positioning of the wheeled vehicle, and a high-precision positioning result is obtained; the image is subjected to inverse perspective projection transformation, which eliminates the perspective effect and further improves the positioning of the wheeled vehicle precision.

Description

Monocular Vision Positioning Method Based on Inverse Perspective Projection Transformation

technical field

The invention relates to the technical field of camera measurement and image processing, and further relates to a method for monocular vision positioning of a motion platform by using an inverse perspective projection transformation algorithm.

Background technique

Automobile autonomous driving technology is a hot research topic at home and abroad in recent years, and the key technology is how to realize real-time and accurate self-positioning of wheeled vehicles. At present, the commonly used positioning solution is the integrated navigation system using GPS (Global Position System, Global Positioning System) plus IMU (Inertial Measurement Unit, Inertial Measurement Unit), but the price is generally expensive, and when the environment makes the motion platform unacceptable When there is no GPS signal, this solution cannot complete the positioning function.

The existing visual localization methods are mainly divided into two categories: stereo vision localization and monocular vision localization. Stereo vision positioning method detects environmental feature points in three-dimensional space and estimates the motion of wheeled vehicles on this basis. It has the following disadvantages: first, the algorithm is complex and time-consuming, and it is difficult to meet the real-time requirements; second, when the environmental background has no obvious texture features, the number of extracted feature points is limited, which will lead to large measurement errors in the algorithm. Therefore, it will still take time for the stereo vision positioning method to reach the real engineering application level.

In contrast, the monocular vision positioning method assumes that the road surface is relatively flat, and realizes the calculation of the wheeled vehicle trajectory by solving the simple displacement relationship between image sequences. Its algorithm is simple, time-sensitive, and easy to install. However, the traditional monocular vision positioning method needs to meet the application conditions such as the road surface is flat and the camera attitude does not change with the movement of the wheeled vehicle. Lu Qiang et al. studied the monocular vision positioning technology in the article "The realization of monocular visual odometer based on SIFT feature extraction in navigation system" (Journal of Sensing Technology, Vol. 20, No. 5, 1148-1152 Page), it directly performs SIFT feature matching on the perspective images captured by the camera, and calculates the motion state of the wheeled vehicle from the matching points according to the derived theoretical model. Since the perspective image has the effect of distorting the real scene into a larger one and a smaller one in the far distance, the farther the point in the world coordinate system corresponding to the feature points extracted on the image is from the wheeled vehicle, the greater the error in the calculation of the image mapping relationship will be. big. If the environmental texture features are single and the feature points are difficult to extract near the wheeled vehicle, the algorithm will have large measurement errors and even fail. Since the theoretical model in this paper is approximated to a certain extent, at the same time, it does not take into account that when the camera moves with the wheeled vehicle, the problem of attitude changes will inevitably occur, and there are certain limitations in directly performing feature matching on perspective images. As a result, the final positioning accuracy is not ideal.

Contents of the invention

The technical problem solved by the present invention is: aiming at the deficiencies of the existing visual positioning technology, a monocular visual positioning method based on inverse perspective projection transformation is proposed. Compared with the stereo vision positioning method, the algorithm of the present invention is simple and easy to implement; and the existing Compared with the monocular vision positioning method, the present invention has higher positioning accuracy.

The concrete technical scheme that the present invention adopts is as follows:

The attitude sensor and the camera are fixed together, and the camera is positioned so that the camera can capture the road surface in any direction around the vehicle body. Set the camera's frame rate and resolution as desired. The setting of the camera frame rate needs to ensure that the field of view areas corresponding to two adjacent images captured by the camera overlap during the normal driving of the wheeled vehicle. Assuming that the starting moment of recording wheeled vehicle trajectory curve is t ₁ , the image captured by the camera at this moment is P ₁ ; at time t _i (i=1, 2...,n, where n is the total number of images), the camera The i-th image P _i is captured, and the camera attitude angle obtained by the attitude sensor is a _i . Carry out the following steps:

In the first step, an inverse perspective projection transformation is performed on the image sequence.

At time t _i , the camera extrinsic parameter matrix A _i is obtained from the camera attitude angle a _i , combined with the camera intrinsic parameter matrix to obtain the inverse perspective projection transformation matrix B _i of the image. According to the obtained inverse perspective projection transformation matrix B _i , the inverse perspective projection transformation is performed on the image P _i captured at time t _i to obtain the front view P′ _i of the road surface.

Assume that the front-down view P′ _i of the road surface at all times constitutes the front-down view sequence P′ ₁ , P′ ₂ , L, P′ _n .

In the second step, the transformation matrix between adjacent images is calculated.

For any adjacent images P′ _q and P′ _q+1 of the front view sequence P′ ₁ , P′ ₂ , L, P′ _n , q=1, 2, L, n-1, perform the following processing:

Step (1), extract feature points.

Use the SURF feature description operator to perform feature extraction on adjacent images P′ _q and P′ _q+1 . After feature extraction, the feature point set of image P′ _q is F _q , and the feature point set of image P′ _q+1 is F _q+1 . The number of feature points in each image feature point set is related to the image resolution, texture complexity and parameter settings of the SURF operator, and can be determined according to the needs of use.

In step (2), the parameters of the rigid body transformation model between images are obtained.

For the adjacent images P′ _q and P′ _q+1 , establish a rigid body transformation model to describe the transformation relationship between the two images. The rigid body transformation model is as follows:

$\left[\begin{array}{c}{x}_j\\ {\mathrm{the\; y}}_j\end{array}\right]=\left[\begin{array}{cc}{\cos \mathrm{\θ}}_q& -\sin {\mathrm{\θ}}_q\\ {\sin \mathrm{\θ}}_q& {\cos \mathrm{\θ}}_q\end{array}\right]\left[\begin{array}{c}{x}_k^{\′}\\ {\mathrm{the\; y}}_k^{\′}\end{array}\right]+\left[\begin{array}{c}{\mathrm{dx}}_q\\ {\mathrm{dy}}_q\end{array}\right]$ q=1, 2..., n-1

Among them, assuming that the feature point set F _q of the image P′ _q corresponds to the image feature pixel point coordinate set is {(x _j , y _j )}, j=1...m, the feature point set of the image P′ _q+1 The image feature pixel coordinate set corresponding to F _q+1 is {(x′ _k , y′ _k )}, k=1...m′, where m and m′ are feature point sets F _q and F _{q+ 1} contains the number of feature points, θ _q represents the rotation angle of image P′ _q+1 relative to P′ _q , and (dx _q , dy _q ) is the pixel coordinate translation of image P′ _q+1 relative to P′ _q .

Using the coordinate sets of image feature pixels corresponding to images P′ _q and P′ _q+1 , and using the RANSAC estimation algorithm to solve the rigid body transformation model, the parameters θ _q and (dx _q , dy _q ) of the rigid body transformation model can be calculated.

The third step is to determine the trajectory curve of the wheeled vehicle.

q＝1，2...，n-1，其中，M为正下视图的纵向分辨率，正下视图序列中每一幅图像的纵向分辨率相同；D为正下视图纵向分辨率为M时对应的纵向实际视场范围，M和D的值可根据需要进行确定。结合航向信息θ_q，即可推断出时刻t_q+1轮式车辆在世界坐标系下所处位置坐标Loc_q+1，其中Loc₁位于坐标原点，即 

其余Loc_q+1(q＝1，2，L，n-1)的横纵坐标 

可类推如下：According to the obtained rigid body transformation model parameters θ _q and (dx _q , dy _q ), calculate the moving distance dL _q of the wheeled vehicle between adjacent moments t _q and t _q+1 ,

q=1, 2...,n-1, where M is the vertical resolution of the front view, and the vertical resolution of each image in the front view sequence is the same; D is the vertical resolution of the front view. The corresponding vertical field of view range, the values of M and D can be determined according to the needs. Combined with heading information θ _q , it can be deduced that the location coordinate Loc _q+1 of the wheeled vehicle at time t q+ ₁ in the world coordinate system, where Loc ₁ is located at the origin of the coordinates, namely

The horizontal and vertical coordinates of the remaining Loc _q+1 (q=1, 2, L, n-1)

It can be deduced as follows:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{{\mathrm{<span\; class="notranslate">\; Loc</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{{\mathrm{<span\; class="notranslate">\; Loc</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; cos</span>}<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; q</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; )</span>\\ {\mathrm{<span\; class="notranslate">\; Y</span>}}_{{\mathrm{<span\; class="notranslate">\; Loc</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; Y</span>}}_{{\mathrm{<span\; class="notranslate">\; Loc</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; q</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; )</span>\end{array}\right.$

By sequentially connecting the coordinate points Loc _i (i=1, 2, L, n) corresponding to each time, the driving track curve of the wheeled vehicle can be obtained.

The beneficial effects of the present invention are:

1. On the basis of the traditional monocular vision positioning method, the real-time attitude information obtained by the attitude sensor is used to assist the positioning of the wheeled vehicle, and a higher precision positioning result is obtained.

2. The image is transformed by inverse perspective projection to eliminate the perspective effect. Based on this, feature extraction and model parameter calculation are performed, which avoids the problem of large error in the calculation of image mapping relationship caused by traditional methods, and improves the positioning accuracy of wheeled vehicles.

3. The present invention only uses one camera and one attitude sensor to calculate the driving track information of the wheeled vehicle. Therefore, the present invention has a relatively simple structure, easy installation, and no complicated operations such as calibration.

Description of drawings

Fig. 1 is the flowchart of concrete implementation of the present invention;

Fig. 2 is a schematic diagram of the calculation principle of the wheeled vehicle trajectory curve;

Figure 3 is a schematic diagram of the working principle of monocular vision positioning;

Figure 4 is the 34th road surface image taken by the camera in the experiment;

Fig. 5 is the front view of the road surface after inverse perspective transformation corresponding to the 34th image in Fig. 4;

Figure 6 is a curve diagram of the wheeled vehicle trajectory obtained in the experiment.

Detailed ways

Fig. 1 has provided the flow chart of the concrete implementation of the present invention. In the first step, an inverse perspective projection transformation is performed on the image sequence. In this step, for the method of solving the camera internal parameter matrix and the camera external parameter matrix, refer to pages 22 to 33 of the book "Principles and Applications of Photogrammetry" (published by Science Press, written by Yu Qifeng/Shang Yang). In the second step, the transformation matrix between adjacent images is calculated. Among them, step (1) is to extract feature points. In this step, the feature detection operator used is the SURF operator. For the properties and usage of the SURF feature operator, refer to the document "Distinctive image features from scale invariant key points" (Journal: International Journal of Computer Vision, 2004, 60( 2), page number 91-110, author David G.Lowe) and literature "SURF: Speeded up robust features" (Conference: Proceedings of the 9th European Conference on Computer Vision, 2006, 3951(1), page number 404-417, author Herbert Bay, Tinne Tuytelaars and Luc Van Gool). The second step is to obtain the parameters of the rigid body transformation model between images. In this step, the RANSAC estimation algorithm is used to solve the rigid body transformation model. The RANSAC algorithm is fast and the estimated parameters are highly accurate. It is currently a relatively commonly used estimation algorithm. For a detailed introduction to the principle of the RANSAC algorithm, see the document "Preemptive RANSAC for Live Structure and Motion Estimation" (Conference: Proceedings of the Ninth IEEE International Conference on Computer Vision, ICCV 2003, by David Nist'er, Sarnoff Corporation and Princeton). The third step is to determine the trajectory curve of the wheeled vehicle. Figure 2 shows the calculation principle diagram of the wheeled vehicle trajectory curve. The XY coordinate axes in the figure represent the world coordinate system, and the origin of the coordinate system corresponds to the position coordinate Loc ₁ of the wheeled vehicle in the world coordinate system when the camera captures the first image, which corresponds to the initial moment t ₁ of positioning, θ ₁ is the angular rotation between the first image and the second image, dL ₁ is the translation distance between the first image and the second image, Loc ₂ corresponds to the wheel type when the camera captures the second image The position coordinates of the vehicle in the world coordinate system. θ ₂ is the angular rotation between the second image and the third image, dL ₂ is the translation distance between the second image and the third image, Loc ₃ corresponds to the wheel type when the camera captures the third image The position coordinates of the vehicle in the world coordinate system. θ _n-1 is the angular rotation between the n-1th image and the nth image, where n is the total number of images captured by the camera, and dL _n-1 is the distance between the n-1th image and the nth image Loc _n-1 corresponds to the position coordinates of the wheeled vehicle in the world coordinate system when the camera captures the n-1th image, and Loc _n corresponds to the wheeled vehicle's position in the world coordinate system when the camera captures the nth image location coordinates. The curve obtained by connecting the position coordinates Loc ₁ , Loc ₂ , . . . Loc _n in the world coordinate system is the driving track curve of the wheeled vehicle.

Utilize the specific embodiment of the present invention to carry out the experiment, the experiment is chosen to carry out in the outdoor flat place, the camera is set up in front of the wheeled vehicle, also can be set up in any side or the rear of the wheeled vehicle in practical application, as long as the visual field captured by the camera is guaranteed The field only includes the road surface. In this experiment, the camera collects pictures at a speed of 3 frames per second, and a total of 93 pictures are collected, that is, n=93. The attitude sensor (model MTI) and the camera (model FLE2-14S3) collect data synchronously. The working principle of the experiment is shown in the figure 3. Assuming that the number 1 in Fig. 3 indicates the position of the wheeled vehicle at time t _q (q=1, 2, ... n-1, n is the total number of images taken), the field of view corresponding to the camera at this time is like a trapezoid As shown in area 3, the image captured by the camera at this time is P _q , and the number 2 indicates the position of the wheeled vehicle at time t _q+1 , and the corresponding field of view of the camera at this time is shown in trapezoidal area 4. The captured image is P _q+1 , the black arrow represents the travel distance of the wheeled vehicle between time t _q and t _q+1 , and the overlapping area of the field of view of images P _q and P _q+1 is shown in figure 5. The acquisition of the moving distance and direction of the wheeled vehicle is obtained through the relative positional relationship of the overlapping areas of the two fields of view in each field of view.

In the experiment, the car traveled a certain distance along a serpentine curve. Fig. 4 is the 34th image P ₃₄ in the image sequence captured by the camera in the experiment, and Fig. 5 is the front view P _{' 34} of the road after the inverse perspective transformation of the image P ₃₄ . Fig. 6 has provided the wheeled vehicle running track curve that utilizes the present invention to obtain in the experiment, and XY represents the world coordinate system in the figure, and the unit is meter, and each green square point in the figure represents the position of the wheeled vehicle at the corresponding moment of each pair of images Coordinate Loc _i , connect all the points to obtain the wheeled vehicle trajectory curve. It can be seen from Fig. 6 that the traveling distance obtained by the method of the present invention is 22.76 meters, while the actual traveling distance measured by a tape measure on the spot is 22.63 meters, with an error of about 6‰.

Claims (1)

1. the monocular vision localization method based on contrary perspective projection transformation is characterized in that, attitude sensor and camera are fixed together, and the placement location of camera makes camera can photograph the vehicle body road surface on any one direction on every side; The frame frequency and the resolution of camera are set as required; Being provided with of camera frame frequency need guarantee that in wheeled vehicle cruising process the field of view of adjacent two width of cloth image correspondences that camera is taken has overlapping part; The initial moment of hypothetical record wheeled vehicle driving trace curve is t ₁, the image that this moment camera is taken is P ₁At moment t _i(i=1,2.., n, wherein n is a total number of images), camera photographs i width of cloth image P _i, the camera attitude angle that attitude sensor obtains is a _iImplement following step:

The first step is carried out contrary perspective projection transformation to image sequence;

At moment t _i, by camera attitude angle a _iObtain Camera extrinsic matrix number A _i, combining camera intrinsic parameter matrix obtains the contrary perspective projection transformation matrix B of image _iAccording to the contrary perspective projection transformation matrix B that obtains _i, at moment t _iThe image P that takes _iCarry out contrary perspective projection transformation, obtain the road surface and just playing view P ' _i

Suppose that all road surfaces are constantly just playing view P ' _iForm and just descending view sequence P ' ₁, P ' ₂, L, P ' _n

In second step, calculate the transformation matrix between adjacent image;

Align down view sequence P ' ₁, P ' ₂, L, P ' _nAny adjacent image P ' _qAnd P ' _Q+1, q=1,2, L, n-1, carry out following processing:

(1) step, extract minutiae;

Use SURF feature description operator to adjacent image P ' _qAnd P ' _Q+1Carry out feature extraction, image P ' after the feature extraction _qFeature point set be F _q, image P ' _Q+1Feature point set be F _Q+1It is relevant that the unique point number of every width of cloth characteristics of image point set and the parameter of the resolution of image, texture complexity and SURF operator are provided with, can be according to using needs to determine;

(2) goes on foot, and obtains the parameter of rigid body translation model between image;

To adjacent image P ' _qAnd P ' _Q+1, set up the transformation relation between rigid body translation model description two width of cloth images, the rigid body translation model is as follows:

$\left[\begin{array}{c}{x}_j\\ y_j\end{array}\right]=\left[\begin{array}{cc}{\cos \mathrm{\&theta;}}_q& -\sin {\mathrm{\&theta;}}_q\\ {\sin \mathrm{\&theta;}}_q& {\cos \mathrm{\&theta;}}_q\end{array}\right]\left[\begin{array}{c}{x}_k^{\&prime;}\\ y_k^{\&prime;}\end{array}\right]+\left[\begin{array}{c}{\mathrm{dx}}_q\\ {\mathrm{dy}}_q\end{array}\right]$ q＝1，2...，n-1

Wherein, suppose image P ' _qFeature point set F _qCorresponding characteristics of image pixel coordinate set is { (x _j, y _j), j=1...m, image P ' _Q+1Feature point set F _Q+1Corresponding characteristics of image pixel coordinate set be (x ' _k, y ' _k), k=1...m ', wherein m, m ' are respectively feature point set F _qAnd F _Q+1The unique point quantity that comprises, θ _qPresentation video P ' _Q+1With respect to P ' _qThe angle of rotation, (dx _q, dy _q) be image P ' _Q+1With respect to P ' _qThe pixel coordinate translational movement;

Utilize image P ' _qAnd P ' _Q+1Corresponding characteristics of image pixel coordinate set adopts the RANSAC algorithm for estimating to find the solution the rigid body translation model, just can calculate the parameter θ of rigid body translation model _q(dx _q, dy _q);

In the 3rd step, determine wheeled vehicle driving trace curve;

According to the rigid body translation model parameter θ that obtains _q(dx _q, dy _q), calculate adjacent moment t _qAnd t _Q+1Between the displacement dL of wheeled vehicle _q,

Q=1,2..., n-1, wherein, M is a longitudinal frame of just playing view; D is vertical practical field of view scope of correspondence when just playing the view longitudinal frame to be M, and the value of M and D can be determined as required; Wheeled vehicle is present position coordinate Loc under world coordinate system _Q+1, Loc wherein ₁Be positioned at true origin, promptly All the other Loc _Q+1(q=1,2, L, horizontal ordinate n-1) and ordinate

For:

$\left\{\begin{array}{c}{X}_{{\mathrm{Loc}}_{q+1}}=X_{{\mathrm{Loc}}_q}+{\mathrm{dL}}_q\&times;\cos \left(\underset{l=1}{\overset{q}{\mathrm{\&Sigma;}}}{\mathrm{\&theta;}}_l\right)\\ Y_{{\mathrm{Loc}}_{q+1}}=Y_{{\mathrm{Loc}}_q}+{\mathrm{dL}}_q\&times;\sin \left(\underset{l=1}{\overset{q}{\mathrm{\&Sigma;}}}{\mathrm{\&theta;}}_l\right)\end{array}\right.$

With each moment correspondence position coordinate points Loc _i(i=1,2, L n) connects successively, just can obtain the driving trace curve of wheeled vehicle.

Images