CN117173214A - High-precision map real-time global positioning tracking method based on road side monocular camera - Google Patents

Abstract

The invention discloses a high-precision map real-time global positioning and tracking method based on a roadside monocular camera, which includes: self-calibration of the roadside camera, learning semantic clues from traffic scenes to realize camera self-calibration; target detection and tracking, and tracking of vehicles on the road. The image plane is detected and tracked to obtain the pixel position changes within a period of time; the angle between the target vehicle and the camera is estimated, and the angle between the vehicle and the camera is calculated based on the pixel position and the calibrated camera parameters; the distance between the target vehicle and the camera is estimated , combine the calibrated camera parameters to calculate the distance between the vehicle and the camera; for final positioning calculation, use the obtained distance and angle combined with the camera's longitude and latitude to achieve global positioning of the target; display the global positioning results obtained by tracking on a high-precision map in real time superior. Automatically realize monocular global positioning of uncalibrated cameras on any roadside without manual operation, and achieve positioning and tracking in images and high-precision planes.

Description

A real-time global positioning and tracking method for high-precision maps based on roadside monocular cameras

Technical field

The present invention relates to the field of computer vision, and in particular to a camera calibration method based on deep learning.

Background technique

With the rapid development of autonomous driving technology, autonomous driving safety is receiving more and more attention. As the first step in autonomous driving perception, precise positioning is the prerequisite to ensure the safety of autonomous driving.

Global positioning and tracking of vehicles using Global Satellite Navigation System (GNSS) or various commercial products with built-in GNSS technology has been widely used. However, GNSS-based positioning schemes require line of sight (LOS) of at least four navigation satellites, and in In many areas, especially in highly built-up areas where satellite signals are severely attenuated or even rejected, the accessibility of so many navigation satellites cannot be guaranteed due to the urban canyon effect. In addition, the root mean square error of pure GNSS positioning in cities ( RMSE) is usually greater than 5m, and even if enhanced with inertial sensors, differential correction or multi-constellation receivers, the required navigation performance cannot be guaranteed in terms of availability and accuracy. On the other hand, with the large-scale deployment of roadside surveillance cameras, in this case, we need to find a method to use large-scale traffic surveillance cameras to achieve global positioning of roadside monocular vehicles in the absence of GNSS. Although there are already a large number of intelligent roadside applications using roadside surveillance cameras, including speed estimation, traffic flow statistics and prediction, illegal parking identification and other tasks to improve road safety and efficiency, there are currently few systematic methods to specifically solve the problem. Roadside vehicle positioning problem.

Most current positioning methods are based on solving the homography matrix at four points by manually selecting at least four corresponding points in maps and images to achieve conversion from the camera coordinate system to the world coordinate system. Although this method is simple to implement, it is easy to introduce manual errors. More importantly, in the current large-scale deployment scenario of smart traffic cameras, this method has to manually establish the initial point correspondence and solve the coordinate system transformation matrix for each camera, which seriously limits its application scale and deployment. efficiency.

Most of the current monocular positioning is tested in experimental scenarios. The camera calibration has been completed by default, or it relies on manual methods to perform camera calibration in advance. For example, Zhang Zhengyou and others completed the calibration by inserting a chessboard. These methods also require manual visits to the scene, and traffic flow restrictions make this method unsuitable for actual traffic scenarios. More importantly, for some roadside scenes, the camera angle is often affected by physical factors. At the same time, due to the current popularity of PTZ cameras, the camera focal length will also change at any time, so manual camera calibration methods are not applicable.

There is currently a self-calibration method for roadside camera calibration, which relies on vehicle vanishing point detection. However, this method requires the road to be approximately straight, or the vehicle trajectory to be approximately straight, and cannot be used in some turning scenarios.

To sum up, there is currently no complete solution to solve the above roadside monocular positioning problem. It can achieve automatic camera calibration without requiring scene prior information or any manual operation. It is suitable for any unknown Monocular vehicle localization of scenes.

Contents of the invention

In order to solve the above practical problems, the present invention provides a high-precision map real-time global positioning and tracking method based on a roadside monocular camera. The technical solution is as follows:

The technical solution provided by the present invention can automatically realize calibration based on existing roadside surveillance cameras without manual participation and without any requirements on the scene. This algorithm can be applied to any intersection, directly global positioning of vehicles in the scene, and at the same time, the global positioning results obtained after positioning are tracked on a high-precision map.

A global positioning and tracking method based on roadside monocular cameras, which includes:

Step A, the camera is automatically calibrated to obtain the internal and external parameters of the camera;

Step B: The target detector detects the vehicle in the image and obtains the pixel position of the vehicle in the image when it is in a certain frame;

Step C: The target tracker establishes an association between the previous and next frames on the basis of detection to obtain the pixel position changes of the vehicle over time;

Step D, vehicle positioning accepts pixel position changes to calculate the angle and distance between the vehicle and the camera to obtain the global position of the vehicle;

Step E, high-precision map positioning tracking calculates the global positioning information and displays it on the high-precision map.

In step A, in the roadside scene, a deep learning-based model is used to directly learn semantic clues from the scene and infer the vertical field of view angle, roll angle, and pitch angle without relying on manual input and scene prior information.

In step A, the focal length is not directly estimated, but the focal length is converted into a vertical field of view angle for indirect reasoning; the continuous values of the pitch angle, roll angle and vertical field of view angle are discretized into 256 buckets for classification prediction. Then use three fully connected layers to predict these three quantities respectively, and finally use the expected value of the probability distribution of each fully connected head as their predicted value; for the vertical field of view, use the Softargmaxbiased-L2 loss, and use the standard for roll and pitch Softargmax-L2 loss.

In steps B and C, only the center point at the bottom of the detection frame needs to be detected and tracked as the input for positioning. The precise positioning depends on the global parameters of the camera, such as latitude and longitude, camera height, camera heading angle, and the camera's own parameters, including Camera internal and external parameters.

In step D, the angle estimation between the vehicle and the camera depends on the camera heading angle and horizontal field of view, and the longitudinal distance between the vehicle and the camera depends on the camera height. On the basis of obtaining the camera parameters, the vehicle and camera are simultaneously estimated. Calculate the distance and angle between them, and then use the distance and angle information combined with the camera's own heading angle and longitude and latitude to calculate the camera's longitude and latitude.

A global high-precision map positioning and tracking system based on roadside monocular cameras, including: camera self-calibration, vehicle detection, vehicle tracking, vehicle positioning and high-precision map positioning and tracking;

The camera self-calibration predicts camera focal length, roll angle, and pitch angle information through a deep learning network;

The vehicle detection and tracking part detects and tracks the vehicle in real time, and obtains the change of the coordinates of the center point at the bottom of the vehicle detection frame over time;

The angle estimate and distance estimate between the vehicle and the camera are then used to obtain the angle and distance between the vehicle and the camera, and then combined with the geographical location conversion formula to obtain the longitude and latitude coordinates of the target vehicle;

The high-precision vehicle positioning and tracking input pixel changes and camera parameters obtained by camera calibration are combined with the positioning algorithm to display the geolocation results on a high-precision map in real time, while performing real-time positioning and tracking on the image plane and the high-precision map plane.

The camera calibration algorithm only infers the vertical field of view angle, pitch angle and roll angle, does not require any manual input, and has no requirements for the scene.

According to the target detection and tracking algorithm, the pixel position of the center point at the bottom of the 2D target detection frame is used to calculate the distance and angle between the vehicle and the camera.

Angle estimation is achieved by combining the horizontal field of view angle obtained by camera calibration, the camera's own heading angle and the vehicle pixel position obtained by target tracking. The angle between the vehicle and the camera is calculated by the following formula:

ω _d =ω _h +ω _c

In the formula, ω _h represents the camera heading angle;

ω _c represents the angle between the vehicle and the camera heading angle;

D represents the distance between the vehicle and the camera;

ωd represents the clockwise angle between the distance D between the vehicle and the camera and the north direction.

Distance estimation is achieved by combining the field of view angle obtained by camera calibration and the vehicle pixel position obtained by target tracking. The angle between the vehicle and the camera is calculated by the following formula:

In the formula, (x, y) is the pixel coordinate of the center point at the bottom of the detection frame of the vehicle in the image;

H is the camera installation height;

Y represents the longitudinal distance between the vehicle and the camera;

w, h are the pixel width and pixel height of the image respectively;

θ represents the pitch angle of the camera;

f represents the camera focal length;

The final distance between the vehicle and the camera is determined by the longitudinal distance and the angle between the vehicle and the camera. The global positioning result is converted by combining the distance and angle between the camera and the vehicle and the camera's own longitude and latitude. The final global positioning longitude and latitude result is calculated by the following formula :

R＝6371.393×1000

In the formula, R is the radius of the earth in meters;

l and g respectively represent the latitude and longitude of the vehicle;

l _c and g _c represent the latitude and longitude of the camera respectively;

High-precision map tracking and positioning is performed by displaying the longitude and latitude calculated from the image on the map without human participation.

The proposed solution of the present invention can greatly reduce the deployment process and manual operations in roadside positioning scenarios, reduce human errors caused by manual operations, improve the adaptability of the roadside positioning algorithm to scenarios, and adapt to different camera models. adaptability. In addition, monocular positioning results can be used as a supplementary service for locations where GNSS signals are missing, providing real-time positioning results for all autonomous vehicles in traffic scenarios.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

Figure 1 is a schematic diagram of an angle estimation method for monocular positioning according to an embodiment of the present invention;

Figure 2 is a schematic diagram of the vehicle located on the right side of the camera field of view during angle estimation according to the embodiment of the present invention.

Figure 3 is a schematic diagram of the vehicle located on the left side of the camera field of view during angle estimation according to the embodiment of the present invention.

Figure 4 is a schematic diagram of a distance estimation method for monocular positioning according to an embodiment of the present invention.

Detailed ways

The purpose of this invention is to achieve single-purpose positioning on the roadside without manual calibration of the camera, and to be applicable to different roadside scenarios, to achieve global vehicle positioning based on calibration, and to perform real-time positioning and tracking in high-precision maps.

To realize the above solution, the problem of camera calibration must be solved, and the global positioning of the vehicle must be achieved without manual methods based on calibration.

In response to the above problems, the present invention provides a high-precision map real-time global positioning and tracking algorithm based on roadside monocular cameras, which does not rely on manual input, does not limit deployment scenarios, and does not limit camera types, installation positions and orientations.

In the present invention, for the geometric camera model, the relationship between the 3D point p _w in the scene and the image pixel position p _im can be expressed as:

p _im ＝[λu λv λ] ^T ＝K[R|t][p _w |1] ^T ,

Among them, K is the camera projection matrix (camera internal parameters), R and t are the rotation and translation of the camera in the world coordinate system (camera external parameters). This invention adopts the mainstream assumptions, that is, a pinhole camera model is used, the center of the image is the main point, the camera is not tilted, and the proportions in the horizontal and vertical directions are equal (aspect ratio is 1, square pixels), and the focal length is in both directions are equal, then f _x =f _y =f, and the projection matrix K can be expressed as K = diag ([ff1]), where f is the focal length (in pixels).

Since the range of focal length is infinite and changes when we resize the image, in this invention, the arc of the vertical field of view α is directly estimated and converted to focal length by, where h is the image pixel height :

The rotation matrix can be described by three angles, which are roll angle, pitch angle and yaw angle. Since there is no natural frame of reference for estimating yaw (left and right) from arbitrary images, we do not estimate the yaw angle, so our rotation matrix consists of two angles, the pitch angle θ and the roll angle ψ, expressed through the horizontal line Representing these two quantities, the camera's pitch angle θ is represented by the intersection position of the horizontal midpoint and the image abscissa center point, and the roll angle ψ is represented by the rotation angle of the estimated line relative to the horizontal line.

The angles of the pitch angle θ, roll angle ψ and vertical field of view α are discretized into 256 buckets in the spatial domain, and the regression problem is transformed into a B classification problem. The present invention uses ResNet-50 as the backbone network and uses a separate fully connected layer to predict the pitch angle θ, roll angle ψ and vertical field of view α. The center value of each bucket can be expressed as θ=[θ ₁ ,...θ _i ,...θ _B ], ψ = [ψ ₁ ,...ψ _i ,...ψ _B ], α = [ α ₁ ,...α _i ,...α _B ], let represents the probability mass of the fully connected layer of each head separately. The expected value of the probability mass of each fully connected joint can be expressed as:

Next, for the vertical field of view α, the Softargmaxbiased-L2 loss is used, and for the pitch angle θ and roll angle ψ, the standard Softargmax-L2 is used. The final loss function is expressed as:

The model can be trained on some currently public camera calibration data sets, such as the pano360 data set. After training, the camera field of view, pitch angle, and roll angle can be obtained.

While calibrating, we also perform target detection and tracking on the vehicles in the scene to obtain the pixel position corresponding to each vehicle. The target detection and tracking algorithm can be replaced with any algorithm. After calibration, we obtain the camera’s pitch angle θ, roll angle ψ, and vertical field of view α. In traffic scenes, relative to the pitch angle θ, the deviation of the roll angle ψ has less impact on the final positioning. Therefore, the present invention does not consider the influence of roll angle.

As shown in Figure 1, given the position of the camera, we only need to know the distance D and angle ω _d between the camera and the target vehicle to estimate the geographical location of the target vehicle. In the present invention, ω _d represents the clockwise angle between the distance D and the camera relative to the true north N of the map, and ω _c represents the angle between the vehicle and the camera orientation angle ω _h . Given the image width w and the camera horizontal field of view hfov, the coordinates of the bottom center of the vehicle detection frame are (x, y), and ω _c can be expressed as:

ω _h represents the camera heading angle;

ω _c represents the angle between the vehicle and the camera heading angle;

D represents the distance between the vehicle and the camera;

ω _d represents the clockwise angle between the distance D between the vehicle and the camera and the north direction.

The two different scenes in Figure 1 are shown in Figures 2 and 3, in which the driving direction of the vehicle is located on the right and left sides of the camera's field of view respectively. For both cases, ω _d can be expressed as:

ω _d =ω _h +ω _c

The distance estimation model is shown in Figure 4. Assume that a vehicle is detected in the road scene, and its position is in the ground coordinate system, (X _w , Y _w , Z _w ). Let θ _v be the rear of the detected vehicle or The projection ray (relative to the camera) of the intersection between the front and the road plane, H is the installation height of the camera. The present invention assumes a horizontal road. According to the angle relationship, it is easy to know that the longitudinal distance Y is expressed as follows:

Among them, β can be expressed as:

Then the longitudinal distance Y between the camera and the vehicle can be calculated in the following way:

(x, y) are the pixel coordinates of the center point at the bottom of the detection frame of the vehicle in the image;

H is the camera installation height;

Y represents the longitudinal distance between the vehicle and the camera;

w, h are the pixel width and pixel height of the image respectively;

θ represents the pitch angle of the camera;

f represents the camera focal length;

The final distance D is:

After obtaining the distance D and angle ω _d , assuming that the camera's own latitude and longitude are: l _c , g _c The vehicle latitude l and longitude g can be obtained by the following formula:

R=6371.393×1000,

where R is the radius of the earth in meters. l and g respectively represent the latitude and longitude of the vehicle;

l _c and g _c represent the latitude and longitude of the camera respectively;

After obtaining the global position of the vehicle, the positioning results can be displayed on a high-precision map in real time, ultimately achieving real-time positioning and tracking on the roadside monocular high-precision map. Through the above method, positioning can be achieved without relying on manual conditions. There is no need to manually go to the site for calibration, and it is plug and play. And in the future, the positioning accuracy can be improved by optimizing the camera calibration accuracy.

The above are only specific embodiments of the present invention. It should be pointed out that those of ordinary skill in the art can also make several improvements and modifications without departing from the principles of the present invention. These improvements and modifications can also be made. should be regarded as the protection scope of the present invention.

Claims (10)

1. A global positioning tracking method based on a road side monocular camera comprises the following steps:

step A, a camera performs automatic calibration to obtain internal and external parameters of the camera;

step B, detecting the vehicle in the image by a target detector to obtain the pixel position of the vehicle in the image when the vehicle is in a certain frame;

step C, the target tracker establishes association between the front frame and the rear frame on the basis of detection to obtain the pixel position change of the vehicle along with time;

step D, the vehicle positioning receives the pixel position change to calculate the angle and distance between the vehicle and the camera, and the global position of the vehicle is obtained;

and E, displaying the global positioning information obtained through calculation in the high-precision map positioning tracking.

2. The global positioning tracking method based on roadside monocular camera of claim 1, wherein,

in the step A, under a road side scene, a model based on deep learning is adopted to directly learn semantic clues from the scene, and the vertical view angle, the rolling angle and the pitch angle are inferred without depending on manual input and prior information of the scene.

3. The global positioning tracking method based on roadside monocular camera of claim 1, wherein,

in the step A, the focal length is not directly estimated, but is converted into a vertical field angle for indirect reasoning; the pitch angle, the roll angle and the vertical view angle continuous values are respectively discretized into 256 barrels for classification prediction, then three full-coupling layers are respectively used for predicting the three amounts, and finally expected values of probability distribution of each full-coupling head are used as predicted values of the full-coupling heads; for vertical field angle, softargmaxbiased-L2 loss was used, and for roll and pitch standard Softargmax-L2 loss was used.

4. The global positioning tracking method based on roadside monocular camera of claim 1, wherein,

in the steps B and C, only the center point at the bottom of the detection frame obtained by detection and tracking is needed to be used as an input of positioning, and the accurate positioning depends on global parameters of the camera, such as longitude and latitude, camera height, camera course angle and parameters of the camera, including parameters inside and outside the camera.

5. The global positioning tracking method based on roadside monocular camera of claim 1, wherein,

in the step D, the angle estimation between the vehicle and the camera depends on the course angle and the horizontal view angle of the camera, the longitudinal distance between the vehicle and the camera depends on the height of the camera, the distance and the angle between the vehicle and the camera are calculated simultaneously on the basis of obtaining the parameters of the camera, and then the longitude and the latitude of the camera are calculated by combining the course angle and the longitude and the latitude of the camera by using the distance and the angle information.

6. A global high-precision map positioning and tracking system based on a road side monocular camera is characterized by comprising the following components: camera self-calibration, vehicle detection, vehicle tracking, vehicle positioning and high-precision map positioning tracking;

the camera self-calibration predicts the focal length, rolling angle and pitch angle information of the camera through a deep learning network;

the vehicle detection and tracking part detects and tracks the vehicle in real time to obtain the change of the center point coordinate at the bottom of the vehicle detection frame along with time;

the longitude and latitude coordinates of the target vehicle are obtained by combining a geographic position conversion formula after the angle and distance between the vehicle and the camera are obtained through the angle estimation and the distance estimation;

the high-precision vehicle positioning and tracking input pixel change and camera parameter obtained by camera calibration are combined with a positioning algorithm to display the geographic positioning result on a high-precision map in real time, and meanwhile, real-time positioning and tracking are carried out on an image plane and a high-precision map plane.

7. The roadside monocular camera-based global high-precision map positioning and tracking system of claim 6, wherein,

the camera calibration algorithm only infers the vertical field angle, the pitch angle and the roll angle, does not need any manual input, and has no requirement on scenes.

8. The roadside monocular camera-based global high-precision map positioning and tracking system of claim 6, wherein,

the target detection and tracking algorithm, the 2D target detection frame bottom center pixel location is used to calculate the distance and angle between the vehicle and the camera.

9. The roadside monocular camera-based global high-precision map positioning and tracking system of claim 6, wherein,

the angle estimation is realized by combining a horizontal field angle obtained by camera calibration, a self course angle of a camera and a vehicle pixel position obtained by target tracking, and the angle between the vehicle and the camera is calculated by the following formula:

ω _d ＝ω _h +ω _c

wherein omega is _h Representing camera headingA corner;

ω _c representing an angle between a vehicle and a camera heading angle;

d represents the distance between the vehicle and the camera;

ωd represents the clockwise angle of the distance D between the vehicle and the camera from the north direction.

10. The roadside monocular camera-based global high-precision map positioning and tracking system of claim 6, wherein,

the distance estimation is realized by combining a field angle obtained by camera calibration and a vehicle pixel position obtained by target tracking, and the angle between the vehicle and the camera is calculated by the following formula:

wherein, (x, y) is the pixel coordinate of the bottom center point of the detection frame of the vehicle in the image;

h is the camera mounting height;

y represents the longitudinal distance between the vehicle and the camera;

w, h are the pixel width and pixel height of the image, respectively;

θ represents a pitch angle of the camera;

f represents a camera focal length;

the final distance between the vehicle and the camera is determined by the longitudinal distance and the angle between the vehicle and the camera, the global positioning result is calculated by combining the distance and the angle between the camera and the vehicle, the longitude and latitude of the camera are converted, and the final global positioning longitude and latitude result is calculated by the following formula:

R＝6371.393×1000

wherein R is the radius of the earth, and the unit is meter;

l, g represent latitude and longitude of the vehicle, respectively;

l _c 、g _c representing the latitude and longitude of the camera, respectively;

the high-precision map tracking and positioning is performed by displaying longitude and latitude obtained through image calculation on a map without human participation.