CN112308913B

CN112308913B - Vehicle positioning method and device based on vision and vehicle-mounted terminal

Info

Publication number: CN112308913B
Application number: CN201910687280.0A
Authority: CN
Inventors: 李天威; 徐抗; 刘一龙; 童哲航
Original assignee: Beijing Momenta Technology Co Ltd
Current assignee: Beijing Momenta Technology Co Ltd
Priority date: 2019-07-29
Filing date: 2019-07-29
Publication date: 2024-03-29
Anticipated expiration: 2039-07-29
Also published as: CN112308913A; WO2021017211A1

Abstract

The embodiment of the invention discloses a vehicle positioning and device based on vision and a vehicle-mounted terminal. The method comprises the following steps: acquiring a road image acquired by camera equipment; determining an initial positioning pose corresponding to the road image according to the data acquired by the motion detection equipment; determining a target map point corresponding to the road image from a preset map according to the initial positioning pose; extracting an edge feature map of the road image according to the preset edge intensity; determining a mapping difference between a target map point and a point in the edge feature map according to the initial positioning pose, and determining the vehicle positioning pose according to the mapping difference; the initial positioning pose is a pose in a world coordinate system where a preset map is located; each map point in the preset map is as follows: and carrying out three-dimensional reconstruction and selection on points in the edge feature map of the sample road image in advance. By applying the scheme provided by the embodiment of the invention, the effectiveness of positioning the vehicle based on vision can be improved.

Description

Vehicle positioning method and device based on vision and vehicle-mounted terminal

Technical Field

The invention relates to the technical field of intelligent driving, in particular to a vehicle positioning method and device based on vision and a vehicle-mounted terminal.

Background

In the technical field of intelligent driving, positioning a vehicle is an important link in intelligent driving. In general, the pose of a vehicle may be determined from a satellite positioning system while the vehicle is traveling. However, when the vehicle travels into a scene where satellite signals are weak or no signals, in order to accurately determine the positioning pose of the vehicle, positioning may be performed based on a visual positioning manner.

Vision-based localization is typically localization based on a match between semantic information of a road image acquired by a camera device and semantic information in a high-precision map. Semantic information in the high-precision map is obtained by modeling according to common markers on roads. The markers may generally include ground lane lines, ground sign lines, traffic signs, light poles, and the like.

When the number of the effective markers in the scene is enough, the vision-based positioning mode can effectively determine the vehicle positioning pose. But when there are few or even no markers in the scene, it is difficult for a high-precision map to give enough effective information for visual localization; alternatively, visual localization may not be possible when the markers cannot completely match the high-precision map due to occlusion or aging. These all result in less effective visual localization.

Disclosure of Invention

The invention provides a vehicle positioning method and device based on vision and a vehicle-mounted terminal, so as to improve the effectiveness of positioning a vehicle based on vision. The specific technical scheme is as follows.

In a first aspect, an embodiment of the present invention discloses a vision-based vehicle positioning method, including:

acquiring a road image acquired by camera equipment;

determining an initial positioning pose corresponding to the road image according to the data acquired by the motion detection equipment; the initial positioning pose is a pose in a world coordinate system where a preset map is located;

determining a target map point corresponding to the road image from the preset map according to the initial positioning pose; wherein, each map point in the preset map is: carrying out three-dimensional reconstruction and selection on points in an edge feature map of a sample road image in advance to obtain the road image;

extracting an edge feature map of the road image according to preset edge intensity;

and determining a mapping difference between the target map point and a point in the edge feature map according to the initial positioning pose, and determining the vehicle positioning pose according to the mapping difference.

Optionally, the step of determining a mapping difference between the target map point and a point in the edge feature map according to the initial positioning pose, and determining a vehicle positioning pose according to the mapping difference includes:

Taking the initial positioning pose as an initial value of an estimated pose, mapping the target map point and the point in the edge feature map into the same coordinate system according to the value of the estimated pose, and determining the mapping difference between the target map point mapped into the same coordinate system and the point in the edge feature map;

when the mapping difference is larger than a preset difference threshold, modifying the value of the estimated pose according to the mapping difference, and returning to execute the step of mapping the target map point and the point in the edge feature map to the same coordinate system according to the value of the estimated pose;

and when the mapping difference is smaller than a preset difference threshold, determining the vehicle positioning pose according to the current value of the estimated pose.

Optionally, the step of mapping the target map point and the point in the edge feature map to the same coordinate system according to the estimated pose value, and determining a mapping difference between the target map point mapped to the same coordinate system and the point in the edge feature map includes:

determining a conversion matrix between the world coordinate system and a camera coordinate system according to the value of the estimated pose, mapping the target map point into the image coordinate system according to the conversion matrix and the projection relation between the camera coordinate system and the image coordinate system to obtain a first mapping position of the target map point, and calculating the mapping difference between the first mapping position and the position of the point in the edge feature map in the image coordinate system; the camera coordinate system is a three-dimensional coordinate system in which the camera equipment is located, and the image coordinate system is a coordinate system in which the road image is located;

Or,

determining a conversion matrix between the world coordinate system and a camera coordinate system according to the estimated pose value; and according to the transformation matrix and the projection relation between the camera coordinate system and the image coordinate system, mapping the points in the edge feature map into the world coordinate system to obtain second mapping positions of the points in the edge feature map, and calculating the mapping difference between the second mapping positions and the positions of the target map points in the world coordinate system.

Optionally, the step of determining, according to the initial positioning pose, a target map point corresponding to the road image from the preset map includes:

taking the position of the initial positioning pose in the preset map as a center, and taking a map point contained in a sphere with a preset distance as a radius as a map point to be selected;

and screening map points in the acquisition range of the camera equipment from the map points to be selected to obtain target map points corresponding to the road image.

Optionally, the step of screening map points in the acquisition range of the camera device from the map points to be selected to obtain the target map point corresponding to the road image includes:

Determining a conversion matrix between the world coordinate system and a camera coordinate system according to the initial positioning pose; the camera coordinate system is a three-dimensional coordinate system where the camera equipment is located;

mapping each map point to be selected into the camera coordinate system according to the conversion matrix to obtain a third mapping position of each map point to be selected;

and screening target map points corresponding to the road image from all map points to be selected according to screening conditions that the third mapping position is in the acquisition range of the camera equipment in the vertical height direction.

Optionally, the determined map point to be selected includes a coordinate position and a normal vector of the map point to be selected; the step of screening map points in the acquisition range of the camera equipment from the map points to be selected to obtain target map points corresponding to the road image comprises the following steps:

determining a connecting line between the camera equipment and each map point to be selected according to the coordinate position of each map point to be selected;

calculating the included angle between each connecting line and the normal vector of the corresponding map point to be selected;

and screening the target map points corresponding to the road image from each map point to be selected according to the screening condition that the included angle is in the preset included angle range.

Optionally, each map point in the preset map is constructed in the following manner:

acquiring a sample road image, and extracting a sample edge feature map of the sample road image according to preset edge intensity;

according to the data acquired by the motion detection equipment, determining a sample positioning pose corresponding to the sample road image; wherein the sample positioning pose is a pose in the world coordinate system;

determining the position information of each point in the sample edge feature map in the world coordinate system based on a three-dimensional reconstruction algorithm and the sample positioning pose;

and selecting map points from all points of the sample edge feature map according to preset point density, and adding the position information of all map points in the world coordinate system to the preset map.

In a second aspect, an embodiment of the present invention discloses a vision-based vehicle positioning device, including:

the road image acquisition module is configured to acquire a road image acquired by the camera equipment;

an initial pose determining module configured to determine an initial positioning pose corresponding to the road image according to data acquired by the motion detection device; the initial positioning pose is a pose in a world coordinate system where a preset map is located;

A map point determining module configured to determine a target map point corresponding to the road image from the preset map according to the initial positioning pose; wherein, each map point in the preset map is: carrying out three-dimensional reconstruction and selection on points in an edge feature map of a sample road image in advance to obtain the road image;

the edge feature extraction module is configured to extract an edge feature map of the road image according to preset edge intensity;

and the vehicle pose determining module is configured to determine a mapping difference between the target map point and a point in the edge feature map according to the initial positioning pose, and determine the vehicle positioning pose according to the mapping difference.

Optionally, the initial pose determining module is specifically configured to:

when the mapping difference is larger than a preset difference threshold, modifying the value of the estimated pose according to the mapping difference, and returning to execute the operation of mapping the target map point and the point in the edge feature map to the same coordinate system according to the value of the estimated pose;

Optionally, the initial pose determining module maps the target map point and the point in the edge feature map to the same coordinate system according to the value of the estimated pose, and when determining the mapping difference between the target map point mapped to the same coordinate system and the point in the edge feature map, the initial pose determining module includes:

or,

Optionally, the map point determining module is specifically configured to:

Optionally, the map point determining module screens map points in the acquisition range of the camera device from the map points to be selected, and when obtaining the target map point corresponding to the road image, the map point determining module includes:

Optionally, the determined map point to be selected includes a coordinate position and a normal vector of the map point to be selected; the map point determining module screens map points in the acquisition range of the camera equipment from various map points to be selected, and when obtaining a target map point corresponding to the road image, the map point determining module comprises:

Optionally, the apparatus further includes: a map point construction module configured to construct each map point in the preset map using:

In a third aspect, an embodiment of the present invention discloses a vehicle-mounted terminal, including: a processor, a camera device, and a motion detection device; the processor includes:

the road image acquisition module is used for acquiring road images acquired by the camera equipment;

the initial pose determining module is used for determining an initial positioning pose corresponding to the road image according to the data acquired by the motion detecting equipment; the initial positioning pose is a pose in a world coordinate system where a preset map is located;

the map point determining module is used for determining a target map point corresponding to the road image from a preset map according to the initial positioning pose; wherein, each map point in the preset map is: carrying out three-dimensional reconstruction and selection on points in an edge feature map of a sample road image in advance to obtain the road image;

The edge feature extraction module is used for extracting an edge feature image of the road image according to preset edge intensity;

and the vehicle pose determining module is used for determining the mapping difference between the target map point and the point in the edge feature map according to the initial positioning pose and determining the vehicle positioning pose according to the mapping difference.

Optionally, the initial pose determining module is specifically configured to:

or,

Optionally, the map point determining module is specifically configured to:

taking the position of the initial positioning pose in a preset map as a center, and taking a map point contained in a ball with a preset distance as a radius as a map point to be selected;

Optionally, the processor further includes: the map point construction module is used for constructing each map point in the preset map by adopting the following operations:

As can be seen from the foregoing, the vision-based vehicle positioning method, apparatus and vehicle-mounted terminal provided by the embodiments of the present invention may determine an initial positioning pose corresponding to a road image according to data collected by a motion detection device, determine a target map point corresponding to the road image from a preset map according to the initial positioning pose, and determine a vehicle positioning pose according to the initial positioning pose and a mapping difference between the target map point and a point in an edge feature map of the road image, where the map point in the preset map is obtained by performing three-dimensional reconstruction on a point in the edge feature map of a sample road image in advance and selecting the point. Because the edge feature map of the road image can extract structural features in the image, the structural features are richer and noise-resistant, and even if the markers in the scene are sparse or blocked, the vehicle positioning pose can be determined through the matching between the target map points and the points in the edge feature map. Therefore, the embodiment of the invention can improve the effectiveness of positioning the vehicle based on vision. Of course, it is not necessary for any one product or method of practicing the invention to achieve all of the advantages set forth above at the same time.

The innovation points of the embodiment of the invention include:

1. the edge feature map is a structural feature in the road image, is rich and noise-resistant, is not easy to be influenced by the number of the lane lines, the traffic signs, the lamp poles and other markers in the scene, is not easy to be influenced by illumination in the early and late, weather and other changes, and has better robustness, so that the vehicle is positioned according to the matching of the points in the edge feature map and the map points in the preset map, and the effectiveness of the vehicle in positioning can be improved.

2. When the vehicle positioning pose is determined, the mapping difference between the target map point and the point in the edge feature map is obtained by continuously adjusting the value of the estimated pose, so that the value of the estimated pose gradually approaches to the true value, the vehicle positioning pose is obtained by iterative solution, and the solved vehicle positioning pose is more accurate.

3. Aiming at map points in a ball with the position of the initial positioning pose in a preset map as the center, map points in the acquisition range of camera equipment are screened as target map points, so that effective map points can be selected from the preset map, and the accuracy of the determined vehicle positioning pose is improved.

4. When the preset map is constructed, a structured sample edge feature map is extracted from the sample road image, point cloud data is extracted from the sample edge feature map for map construction, so that more dense map information can be obtained, and the effective information amount in the preset map is increased.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is apparent that the drawings in the following description are only some embodiments of the invention. Other figures may be derived from these figures without inventive effort for a person of ordinary skill in the art.

Fig. 1 is a schematic structural diagram of a vision-based vehicle positioning method according to an embodiment of the present invention;

fig. 2 is a schematic diagram of an included angle between a camera device and a normal vector of a map point according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a vehicle positioning method based on vision according to an embodiment of the present invention;

FIG. 4 is a schematic structural view of a vision-based vehicle positioning device according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram of a vehicle-mounted terminal according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without any inventive effort, are intended to be within the scope of the invention.

It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. A process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed but may alternatively include other steps or elements not listed or inherent to such process, method, article, or apparatus.

The embodiment of the invention discloses a vehicle positioning method based on vision, a processor and a vehicle-mounted terminal, which can improve the effectiveness of positioning a vehicle based on vision. The following describes embodiments of the present invention in detail.

Fig. 1 is a schematic flow chart of a vision-based vehicle positioning method according to an embodiment of the present invention. The method is applied to the electronic equipment. The electronic device may be a general computer, a server, an intelligent terminal device, or the like, or may be a vehicle-mounted terminal such as a vehicle-mounted computer or a vehicle-mounted industrial control computer (Industrial personal Computer, IPC). In this embodiment, the vehicle-mounted terminal may be mounted in a vehicle, which refers to an intelligent vehicle. A variety of sensors are provided in the vehicle, including sensors such as camera devices and motion detection devices. The camera device provided in the vehicle may be one or more. The motion detection device may include inertial measurement units (Inertial Measurement Unit, IMU) and/or wheel speed meter or like sensors. The method specifically comprises the following steps.

S110: and acquiring a road image acquired by the camera equipment.

The camera device may collect road images at a preset frequency. The road image may comprise image data of road markers or any other object within the image acquisition range of the camera device.

In this embodiment, when there are a plurality of camera devices, the road image may be acquired for one camera device disposed in front of the vehicle, or may be acquired by stitching images acquired for a plurality of cameras disposed in front of the vehicle. The place where the vehicle is located can be outdoors or a parking lot.

The road image may be an image of the surroundings of the vehicle captured by the camera device when the vehicle is traveling on various roads. The road may be any place where a vehicle can travel, such as an urban road, a rural road, a mountain road, a parking lot road, etc., and an image acquired during driving into a parking space may be included in the road image.

S120: and determining an initial positioning pose corresponding to the road image according to the data acquired by the motion detection equipment.

The initial positioning pose is a pose in a world coordinate system where a preset map is located. The preset map may be a high-precision map installed in the vehicle. The time of the data collected by the motion detection device and the time of the road image collected by the camera device may be related time, for example, the collecting time of the two may be the same time or the time difference is very short.

According to the data collected by the motion detection device, determining the initial positioning pose corresponding to the road image can specifically include: and acquiring the upper positioning pose, and determining the initial positioning pose corresponding to the road image according to the upper positioning pose and the data acquired by the motion detection equipment. The last positioning pose can be the determined positioning pose of the vehicle at the last moment.

In another embodiment, the step may further include: and determining the initial positioning pose of the vehicle corresponding to the road image according to the data acquired by the motion detection equipment and the matching result of the road characteristics in the road image and the road characteristics in the preset map. In this embodiment, the road features in the road image are matched with the road features in the preset map, corresponding to another vision-based positioning method.

In another embodiment, the motion detection device may also include a global positioning system (Global Positioning System, GPS). When the motion detection equipment comprises a GPS, the accumulated errors in the positioning process according to the IMU and/or the wheel speed meter and the like can be eliminated as much as possible, and the accuracy of the positioning pose is improved.

The initial positioning pose determined in the step is the initial positioning pose of the vehicle at the current moment of collecting the road image, and is used for determining the more accurate vehicle positioning pose at the current moment.

S130: and determining a target map point corresponding to the road image from the preset map according to the initial positioning pose.

The determination of the target map point corresponding to the road image from the preset map may be understood as the determination of the position information of the target map point corresponding to the road image from the preset map. The target map point is a map point that may be observed in the road image. In this step, according to the position of the initial positioning pose in the world coordinate system, a map point in a preset range around the position in the preset map may be used as the target map point.

Wherein, each map point in the preset map is: and carrying out three-dimensional reconstruction and selection on points in the edge feature map of the sample road image in advance. The corresponding relation between each map point and the position information is stored in a preset map, and the position information is the position information of the map point in a world coordinate system.

The location information of each map point in the world coordinate system includes: the coordinate position of each map point in the world coordinate system and the normal vector information of the map point. The normal vector information of the map point represents a normal vector of a plane in which the map point is located. In the world coordinate system, the coordinate position of the map point can be represented by three coordinates (a, B, C), and the normal vector information of the point can be represented by three parameters (A, B, C). The position information of each map point contains 6 dimensions of information, and this representation can be equivalent to representing each map point by using a plane a (x-a) +b (y-B) +c (z-C) =0.

S140: and extracting an edge characteristic diagram of the road image according to the preset edge intensity.

An edge in an image refers to a collection of pixel points where the gray level of surrounding pixels is changed stepwise. There is a significant difference in the gray values of the pixels on both sides of the edge point. Edge strength can be understood as the magnitude of the edge point gradient. An edge feature map may be understood as data containing edge features consistent with the dimensions of a road image. The edge feature map contains position information of edge lines in the road image. The edge line is composed of edge points.

The extracted edge feature map is position information represented in an image coordinate system in which the road image is located. The edge points in the edge feature map are positions in the image coordinate system.

The extracting of the edge feature map of the road image may be understood as extracting features of the road image to obtain the edge feature map. Specifically, when the edge feature map of the road image is extracted, the preset edge intensity can be used as a threshold value, and the edge with the edge intensity greater than the threshold value can be extracted from the road image to obtain the edge feature map. In the local area of the road image, an edge having a high edge strength may be extracted, for example, in a pillar having a ridge, the gradient of the ridge inside the pillar may not be large in the entire road image area, but the edge strength of the ridge is large in the pillar area, and thus the edge extraction is also an object. The two schemes can also be combined.

The edge feature map extracted according to the preset edge intensity can embody structural features in the road image. Specifically, a canny operator, a sobel operator and a LoG operator can be adopted to extract an edge feature map of the road image.

After the edge feature map is extracted, a Gaussian Blur (Gaussian blue) algorithm process can be further carried out on the values in the edge feature map, so that the processed edge feature map is smoother.

In another embodiment, the step may further include: and extracting an edge feature map of the road image based on the edge feature extraction model.

The edge feature extraction model is obtained through training according to a sample road image and an edge feature map marked according to preset edge intensity. The edge feature extraction model associates road images with corresponding edge feature maps.

Based on the edge feature extraction model, extracting an edge feature map of the road image may include: inputting the road image into an edge feature extraction model, and obtaining an edge feature image of the road image output by the edge feature extraction model.

In the training stage, a plurality of sample road images and marked edge features can be obtained, and the sample road images are input into an edge feature extraction model; the edge feature extraction model extracts feature vectors of the sample road image according to model parameters, and carries out regression on the feature vectors to obtain reference edge features of the sample road image; comparing the reference edge characteristics with the marked edge characteristics to obtain a difference; when the difference is larger than a preset difference threshold, returning to execute the step of inputting the sample road image into the edge feature extraction model; and when the difference is not greater than a preset difference threshold, determining that the training of the edge feature extraction model is completed.

The marked edge features show preset edge strength. That is, the edge features in the sample road image are labeled according to a preset edge intensity as a standard. The edge feature extraction model obtained by training by the machine learning method can extract more accurate edge features. The edge feature extraction model may also be referred to as an edge detector.

S150: and determining the mapping difference between the target map point and the point in the edge feature map according to the initial positioning pose, and determining the vehicle positioning pose according to the mapping difference.

The initial positioning pose can reflect the pose of the vehicle within a certain precision range. In order to more accurately determine the pose of the vehicle, the vehicle positioning pose is determined according to the above-described map difference in the present embodiment. When the determined vehicle positioning pose is close to a true value, the mapping difference between the target map point and the point in the edge feature map is smaller than a preset difference threshold.

As can be seen from the foregoing, the present embodiment may determine, according to data collected by the motion detection apparatus, an initial positioning pose corresponding to a road image, determine, according to the initial positioning pose, a target map point corresponding to the road image from a preset map, determine, according to the initial positioning pose and a mapping difference between the target map point and a point in an edge feature map of the road image, a vehicle positioning pose, where the map point in the preset map is obtained by three-dimensional reconstruction of a point in the edge feature map of a sample road image in advance and selection. Because the edge feature map of the road image can extract structural features in the image, the structural features are richer and noise-resistant, and even if the markers in the scene are sparse or blocked, the vehicle positioning pose can be determined through the matching between the target map points and the points in the edge feature map. Therefore, the present embodiment can improve the effectiveness in locating the vehicle based on the vision.

In summary, the difference between the embodiment and the conventional vision-based vehicle positioning method further includes that the conventional positioning method detects semantic features corresponding to the lane lines, the sidewalks, the traffic signs, the lamp posts and other artificial objects in the road image. If these semantic features are not present in the scene, vehicle localization may not be possible. The edge features extracted from the road image in the embodiment are more generalized, and the adaptability is stronger due to the fact that the edge features of the artificial objects and the natural objects in the scene are included, so that the vehicle positioning process is more effective.

On the other hand, since the embodiment extracts the structured features, the illumination effect is more robust for changes in the morning and evening, weather, etc. The traditional method for extracting the characteristic points from the image is not anti-noise for the condition of illumination change; traditional trained methods do not generalize for scenarios, requiring extensive data training to ensure the effectiveness of the scheme. The solution of the present embodiment can be used as a mapping and positioning solution across the morning, evening and/or season.

In another embodiment of the present invention, based on the embodiment shown in fig. 1, step S150, a mapping difference between a target map point and a point in an edge feature map is determined according to an initial positioning pose, and a step of determining a vehicle positioning pose according to the mapping difference includes the following steps 1a to 3a.

Step 1a: and taking the initial positioning pose as an initial value of the estimated pose, mapping the target map points and the points in the edge feature map into the same coordinate system according to the value of the estimated pose, and determining the mapping difference between the target map points mapped into the same coordinate system and the points in the edge feature map.

In this embodiment, the target map points are located in the world coordinate system, and the points in the edge feature map are located in the image coordinate system, and the position information of the target map points may be mapped to the image coordinate system, or the points in the edge feature map may be mapped to the world coordinate system. The same coordinate system may be an image coordinate system, a world coordinate system, or other coordinate systems.

Determining the mapping difference between the target map point mapped to the same coordinate system and the point in the edge feature map may be understood as determining the difference between the position information of the target map point mapped to the same coordinate system and the position information of the point in the edge feature map as the mapping difference. The above difference may also be referred to as a residual error.

For example, the above-described mapping difference may be calculated according to the formula e=i (u) - α. Where the pixel coordinate obtained after the mapping of the target map point into the image coordinate system is u, I (u) represents the response of the target map point to the pixel coordinate u in the edge feature map, and α may be set according to the characteristics of the edge detector, for example, set to 255. If the edge detector has statistically regular observations of the position of the individual map points and the viewing angle of the camera device, a function α=f (p _c ) Instead of a fixed value of a, p _c Is the position of the map point in the camera coordinate system.

Step 2a: when the mapping difference is greater than the preset difference threshold, modifying the value of the estimated pose according to the mapping difference, and returning to the step of executing the point mapping of the target map point and the point in the edge feature map to the same coordinate system according to the value of the estimated pose in the step 1 a.

When the mapping difference is larger than a preset difference threshold, the estimated pose is considered to be far away from the real pose of the vehicle, the estimated pose can be modified continuously, and the iterative process is repeated.

When modifying the estimated pose value according to the mapping difference, the estimated pose value can be modified according to the mapping difference based on a jacobian matrix, a hessian matrix, a gaussian newton iteration method or a Levenberg-Marquardt (LM) algorithm.

Step 3a: and when the mapping difference is smaller than a preset difference threshold, determining the vehicle positioning pose according to the current value of the estimated pose.

When the mapping difference is less than the preset difference threshold, the estimated pose is considered to be very close to the true pose of the vehicle. According to the current value of the estimated pose, the vehicle positioning pose is determined, specifically, the current value of the estimated pose can be directly determined as the vehicle positioning pose, or the current value of the estimated pose can be determined as the vehicle positioning pose after preset transformation.

When the mapping difference is equal to the preset difference threshold, the value of the estimated pose can be modified according to the mapping difference, the step 1a is executed in a return mode, and the vehicle positioning pose can be determined according to the current value of the estimated pose.

In summary, in this embodiment, when determining the vehicle positioning pose, the value of the estimated pose is continuously adjusted to obtain the mapping difference between the target map point and the point in the edge feature map, so that the value of the estimated pose gradually approaches the true value, and thus the vehicle positioning pose is obtained by iteratively solving, so that the solved vehicle positioning pose is more accurate.

In another embodiment of the present invention, based on the embodiment shown in fig. 1, the step of mapping the target map point and the point in the edge feature map to the same coordinate system in step 1a according to the estimated pose value, and determining the mapping difference between the target map point mapped to the same coordinate system and the point in the edge feature map may include the following embodiments.

In a first embodiment, a conversion matrix between a world coordinate system and a camera coordinate system is determined according to the estimated pose, a target map point is mapped to an image coordinate system according to the conversion matrix and a projection relationship between the camera coordinate system and the image coordinate system, a first mapping position of the target map point is obtained, and a mapping difference between the first mapping position and a position of a point in an edge feature map in the image coordinate system is calculated.

The camera coordinate system is a three-dimensional coordinate system where the camera equipment is located, and the image coordinate system is a coordinate system where the road image is located. The estimated pose is the pose of the vehicle in the world coordinate system, and the conversion matrix between the world coordinate system and the camera coordinate system can be determined according to the value of the estimated pose.

In this embodiment, when mapping the target map point into the image coordinate system, the target map point may be first converted into the camera coordinate system according to the following formula:

wherein p is _c For the position of the target map point in the camera coordinate system, p _w For the location of the target map point in the world coordinate system,for the conversion relation between the world coordinate system and the car body coordinate system,/->Is a transformation matrix between a car body coordinate system and a camera coordinate system,/for the car body coordinate system>Representing a transformation matrix between the world coordinate system and the camera coordinate system. And then converting the coordinates in the camera coordinate system into an image coordinate system according to the projection model of the camera equipment to obtain the pixel coordinates u=pi (p) of the target map point _c ). Where pi (-) represents the projection model of the camera device and u represents the target map point in the image coordinate system.

In the second embodiment, a conversion matrix between the world coordinate system and the camera coordinate system is determined according to the estimated pose value, the points in the edge feature map are mapped into the world coordinate system according to the conversion matrix and the projection relationship between the camera coordinate system and the image coordinate system, the second mapping positions of the points in the edge feature map are obtained, and the mapping difference between the second mapping positions and the positions of the target map points in the world coordinate system is calculated.

In summary, in this embodiment, according to the mutual conversion relationship among the world coordinate system, the camera coordinate system and the image coordinate system, the position information of the target map point may be mapped to the image coordinate system, or the point in the edge feature map may be mapped to the world coordinate system, which provides a specific implementation way for determining the mapping difference.

In another embodiment of the present invention, based on the embodiment shown in fig. 1, step S130, a step of determining a target map point corresponding to a road image from a preset map according to an initial positioning pose, includes steps 1b and 2b.

Step 1b: and taking the position of the initial positioning pose in a preset map as a center, and taking the preset distance as a map point contained in the sphere with the radius determined as a map point to be selected. The preset distance may be a distance value determined empirically. The determined map points to be selected may be plural.

Step 2b: and screening map points in the acquisition range of the camera equipment from the map points to be selected to obtain target map points corresponding to the road images.

The target map points obtained through the above screening process are map points that may be observable in the road image.

In summary, in this embodiment, for the map points in the sphere centered on the position of the initial positioning pose in the preset map, the map points in the acquisition range of the camera device are screened as target map points, so that an effective map point can be selected from the preset map, and the accuracy of determining the positioning pose of the vehicle is improved.

In another embodiment of the present invention, based on the embodiment shown in fig. 1, step 2b, a step of screening map points within an acquisition range of a camera device from among the map points to be selected to obtain a target map point corresponding to a road image, includes the following steps 2b-1 to 2b-3.

Step 2b-1: and determining a conversion matrix between the world coordinate system and the camera coordinate system according to the initial positioning pose. The camera coordinate system is a three-dimensional coordinate system where the camera equipment is located.

Step 2b-2: and mapping each map point to be selected into a camera coordinate system according to the conversion matrix to obtain a third mapping position of each map point to be selected.

Specifically, this step may map each map point to be selected into the camera coordinate system according to the following formula:

p _b ＝Tp _w ,p∈A

wherein p is _b For a third mapping position of each map point to be selected in the camera coordinate system, p _w And (3) for the position information of each map point to be selected in the world coordinate system, T is the conversion matrix, p is any map point to be selected, and A is a point set formed by the map points to be selected.

Since the camera apparatus is fixed in the vehicle, the camera coordinate system can be replaced with the vehicle body coordinate system for screening map points to be selected according to a known conversion relationship between the camera coordinate system and the vehicle body coordinate system.

Step 2b-3: and screening target map points corresponding to the road images from the map points to be selected according to screening conditions that the third mapping position is in the acquisition range of the camera equipment in the vertical height direction.

Wherein the acquisition range of the camera device in the vertical height direction may be expressed as the range of the z-axis [ z1, z2]. For example, [ z1, z2] may be [ -0.1m,4m ]. Specifically, the step may determine the map point to be selected whose z-axis value is within the acquisition range as the target map point. The screened target map points are still represented by coordinates of a world coordinate system.

The step may specifically be that the map points to be selected, whose third mapping position is in the acquisition range of the camera device in the vertical height direction, are screened as target map points corresponding to the road image.

In summary, the embodiment screens the map points to be selected according to the range of the camera device in the height direction to obtain the target map points, and map points outside the secondary height range can be filtered from the map points to be selected.

In another embodiment of the present invention, the determined map points to be selected include coordinate positions and normal vectors of the map points to be selected based on the embodiment shown in fig. 1. The position information of each map point to be selected includes a coordinate position and a normal vector. And 2b, screening map points in the acquisition range of the camera equipment from the map points to be selected to obtain target map points corresponding to the road image, wherein the step comprises the following steps 2b-4 to 2b-6.

Step 2b-4: and determining the connection line between the camera equipment and each map point to be selected according to the coordinate position of each map point to be selected.

The coordinate position of the map point to be selected is a position in the world coordinate system. The position of the camera device in the world coordinate system may be determined by an initial positioning pose.

Step 2b-5: and calculating the included angle between each connecting line and the normal vector of the corresponding map point to be selected.

For example, the four map points A, B, C, D to be selected are AO, BO, CO, DO connected with the camera device O, the included angles between the connecting line AO and the normal vector of the map point a to be selected are calculated, the included angles between the connecting line BO and the normal vector of the map point B to be selected are calculated, the included angles between the connecting line CO and the normal vector of the map point C to be selected are calculated, and the included angles between the connecting line DO and the normal vector of the map point D to be selected are calculated, so that four included angles for the four map points to be selected are obtained.

Step 2b-6: and screening the target map points corresponding to the road image from each map point to be selected according to the screening condition that the included angle is in the preset included angle range.

The step may specifically include screening map points to be selected whose included angles are within a preset included angle range as target map points corresponding to the road image. The preset included angle range may be determined empirically in advance.

Referring to fig. 2, fig. 2 is a schematic diagram of an included angle between a camera device and a normal vector of a map point according to the present embodiment. The incident light is emitted from the map point, projected to the optical center of the camera device, and imaged on the imaging plane to obtain the road image. The line on which the incident light is located may be a line on which a line between the camera device and the map point is located. The normal vector of the map point is perpendicular to the plane. When the relative positions of the camera device and the map points are different, the included angles between the incident light and the normal vector are different. As can be seen from fig. 2, the map point corresponding to the included angle 1 can be acquired by the camera device, and the map point corresponding to the included angle 2 cannot be acquired by the camera device. Therefore, the range of the preset included angle is reasonably set, and map points which can be acquired by the camera equipment can be screened out from a large number of map points to be selected, so that the target map points are obtained.

In summary, in this embodiment, the map points to be selected are screened according to the normal vector of the map points to be selected, so that the map points which cannot be observed by the camera are filtered, and the accuracy of the target map points is improved.

In another embodiment of the present invention, based on the embodiment shown in fig. 1, each map point in the preset map is constructed using the following steps 1c to 4 c.

Step 1c: and acquiring a sample road image, and extracting a sample edge feature map of the sample road image according to the preset edge intensity.

In this step, a sample edge feature map is extracted, and reference may be made to step S140.

Step 2c: and determining the sample positioning pose corresponding to the sample road image according to the data acquired by the motion detection equipment.

The sample positioning pose is a pose in a world coordinate system. The sample positioning pose can be regarded as the positioning pose of the vehicle at the current moment, and the position of a map point in a preset map can be constructed according to the sample positioning pose.

Step 3c: and determining the position information of each point in the sample edge feature map in a world coordinate system based on the three-dimensional reconstruction algorithm and the sample positioning pose.

In this step, determining the location information of each point in the sample edge feature map in the world coordinate system may include:

based on a three-dimensional reconstruction algorithm, determining a three-dimensional coordinate of each point in the sample edge feature map in a camera coordinate system, determining a conversion matrix between the camera coordinate system and a world coordinate system according to the sample positioning pose, and converting the three-dimensional coordinate according to the conversion matrix to obtain the position information of each point in the sample edge feature map in the world coordinate system.

Step 4c: and selecting map points from all points of the sample edge feature map according to preset point density, and adding the position information of all map points in a world coordinate system to a preset map.

The method specifically comprises the following steps: constructing octree cube grids in a preset map according to an octree algorithm with preset voxel sizes; for each octree cube mesh, one point is selected from the points of the sample edge feature map that are in the octree cube mesh as a map point corresponding to the octree cube mesh.

And forming point cloud data in a three-dimensional world coordinate system according to map points selected from various points of the sample edge feature map according to preset point densities.

In summary, in this embodiment, when a preset map is constructed, a structured sample edge feature map is extracted from a sample road image, and point cloud data is extracted from the sample edge feature map for map construction, so that more dense map information can be obtained, and the effective information amount in the preset map can be increased.

Fig. 3 is a schematic diagram of a frame of a vision-based vehicle positioning method according to an embodiment of the present invention.

The frame comprises a front end and a repositioning end. The front-end motion detection device may include an intelligent vehicle common sensor such as an IMU or wheel speed meter. The front end is used for estimating the initial positioning pose of the vehicle at the current moment according to the last positioning pose of the vehicle and various sensor data, and inputting the initial positioning pose into the map manager and the MVS positioning optimizer. The initial localization pose may be understood as a predicted global pose. The front end may also be based on the road image when estimating the initial positioning pose of the vehicle.

The repositioning end comprises a map manager, an edge detector and an MVS positioning optimizer. The map manager loads a preset map of the environment in which it is located and manages the map by the octree. The map manager may query a preset map for map points that may be observed by the camera according to the initial positioning pose. The map manager can also screen the queried map points according to the acquisition range of the camera equipment, and remove map points beyond the range. And the map manager inputs the target map points obtained after screening into an MVS positioning optimizer.

The edge detector takes the road image as input, outputs an edge response of the road image, namely an edge feature map, and inputs the edge feature map into the value MVS positioning optimizer.

After receiving the input initial positioning pose, the target map points and the edge feature map, the MVS positioning optimizer adopts an iterative optimization solving mode to determine the global pose of the vehicle according to the mapping difference between the target map points and the points in the edge feature map. The global pose of the vehicle is the vehicle positioning pose.

Fig. 4 is a schematic structural diagram of a vision-based vehicle positioning device according to an embodiment of the present invention. The embodiment of the device corresponds to the embodiment shown in fig. 1, and the device is applied to electronic equipment. The device comprises:

A road image acquisition module 410 configured to acquire a road image acquired by the camera device;

an initial pose determination module 420 configured to determine an initial positioning pose corresponding to the road image from data collected by the motion detection device; the initial positioning pose is a pose in a world coordinate system where a preset map is located;

a map point determining module 430 configured to determine a target map point corresponding to the road image from a preset map according to the initial positioning pose; wherein, each map point in the preset map is: carrying out three-dimensional reconstruction and selection on points in an edge feature map of a sample road image in advance to obtain the road image;

an edge feature extraction module 440 configured to extract an edge feature map of the road image according to a preset edge intensity;

the vehicle pose determination module 450 is configured to determine a mapping difference between the target map point and a point in the edge feature map according to the initial positioning pose, and determine the vehicle positioning pose according to the mapping difference.

In another embodiment of the present invention, based on the embodiment shown in fig. 4, the initial pose determination module 420 is specifically configured to:

taking the initial positioning pose as an initial value of the estimated pose, mapping the target map points and points in the edge feature map into the same coordinate system according to the value of the estimated pose, and determining the mapping difference between the target map points mapped into the same coordinate system and the points in the edge feature map;

When the mapping difference is larger than a preset difference threshold, modifying the value of the estimated pose according to the mapping difference, and returning to execute the operation of mapping the target map point and the point in the edge feature map into the same coordinate system according to the value of the estimated pose;

In another embodiment of the present invention, based on the embodiment shown in fig. 4, the initial pose determining module 420 maps the target map point and the point in the edge feature map to the same coordinate system according to the estimated pose value, and determines the mapping difference between the target map point mapped to the same coordinate system and the point in the edge feature map, where the mapping difference includes:

determining a conversion matrix between a world coordinate system and a camera coordinate system according to the value of the estimated pose, mapping the target map point into the image coordinate system according to the conversion matrix and the projection relation between the camera coordinate system and the image coordinate system to obtain a first mapping position of the target map point, and calculating the mapping difference between the first mapping position and the position of the point in the edge feature map in the image coordinate system; the camera coordinate system is a three-dimensional coordinate system where the camera equipment is located, and the image coordinate system is a coordinate system where the road image is located;

Or,

determining a conversion matrix between the world coordinate system and the camera coordinate system according to the estimated pose value; according to the transformation matrix and the projection relation between the camera coordinate system and the image coordinate system, mapping the points in the edge feature map to the world coordinate system to obtain second mapping positions of the points in the edge feature map, and calculating the mapping difference between the second mapping positions and the positions of the target map points in the world coordinate system.

In another embodiment of the present invention, based on the embodiment shown in fig. 4, the map point determining module 430 is specifically configured to:

taking the position of the initial positioning pose in a preset map as a center, and taking a map point contained in a ball with a preset distance as a radius to be determined as a map point to be selected;

and screening map points in the acquisition range of the camera equipment from the map points to be selected to obtain target map points corresponding to the road images.

In another embodiment of the present invention, based on the embodiment shown in fig. 4, the map point determining module 430, when selecting a map point within the acquisition range of the camera device from the map points to be selected, obtains a target map point corresponding to the road image, includes:

Determining a conversion matrix between a world coordinate system and a camera coordinate system according to the initial positioning pose; the camera coordinate system is a three-dimensional coordinate system where the camera equipment is located;

mapping each map point to be selected into a camera coordinate system according to the conversion matrix to obtain a third mapping position of each map point to be selected;

and screening target map points corresponding to the road images from the map points to be selected according to screening conditions that the third mapping position is in the acquisition range of the camera equipment in the vertical height direction.

In another embodiment of the present invention, the determined map points to be selected include coordinate positions and normal vectors of the map points to be selected based on the embodiment shown in fig. 4; the map point determining module 430, when selecting a map point within the acquisition range of the camera device from the map points to be selected, obtains a target map point corresponding to the road image, includes:

and screening the target map points corresponding to the road image from the map points to be selected according to screening conditions that the included angle is in the preset included angle range.

In another embodiment of the present invention, based on the embodiment shown in fig. 4, the apparatus further comprises: a map point construction module (not shown in the figure) configured to construct each map point in the preset map using:

according to the data acquired by the motion detection equipment, determining a sample positioning pose corresponding to the sample road image; the sample positioning pose is a pose in a world coordinate system;

determining the position information of each point in the sample edge feature map in a world coordinate system based on a three-dimensional reconstruction algorithm and a sample positioning pose;

and selecting map points from all points of the sample edge feature map according to preset point density, and adding the position information of all map points in a world coordinate system to a preset map.

The device embodiment corresponds to the method embodiment, and has the same technical effects as the method embodiment, and the specific description refers to the method embodiment. The apparatus embodiments are based on the method embodiments, and specific descriptions may be referred to in the method embodiment section, which is not repeated herein.

Fig. 5 is a schematic structural diagram of a vehicle-mounted terminal according to an embodiment of the present invention. The vehicle-mounted terminal comprises: a processor 510, a camera device 520, and a motion detection device 530; the processor 510 includes:

A road image acquisition module (not shown in the figure) for acquiring a road image acquired by the camera device 520;

an initial pose determining module (not shown in the figure) for determining an initial positioning pose corresponding to the road image according to the data acquired by the motion detection device 530; the initial positioning pose is a pose in a world coordinate system where a preset map is located;

a map point determining module (not shown in the figure) for determining a target map point corresponding to the road image from a preset map according to the initial positioning pose; wherein, each map point in the preset map is: carrying out three-dimensional reconstruction and selection on points in an edge feature map of a sample road image in advance to obtain the road image;

an edge feature extraction module (not shown in the figure) for extracting an edge feature map of the road image according to a preset edge intensity;

a vehicle pose determining module (not shown in the figure) for determining a mapping difference between the target map point and a point in the edge feature map according to the initial positioning pose, and determining the vehicle positioning pose according to the mapping difference.

In another embodiment of the present invention, based on the embodiment shown in fig. 5, the initial pose determining module is specifically configured to:

In another embodiment of the present invention, based on the embodiment shown in fig. 5, the initial pose determining module maps the target map point and the point in the edge feature map to the same coordinate system according to the value of the estimated pose, and determines the mapping difference between the target map point mapped to the same coordinate system and the point in the edge feature map, where the mapping difference comprises:

Or,

In another embodiment of the present invention, based on the embodiment shown in fig. 5, the map point determining module is specifically configured to:

map points within the acquisition range of the camera device 520 are screened from the map points to be selected, and target map points corresponding to the road image are obtained.

In another embodiment of the present invention, based on the embodiment shown in fig. 5, the map point determining module, when selecting a map point within the acquisition range of the camera device 520 from the map points to be selected, obtains a target map point corresponding to the road image, includes:

determining a conversion matrix between a world coordinate system and a camera coordinate system according to the initial positioning pose; wherein, the camera coordinate system is a three-dimensional coordinate system where the camera device 520 is located;

and screening target map points corresponding to the road image from the map points to be selected according to screening conditions that the third mapping position is in the acquisition range of the camera device 520 in the vertical height direction.

In another embodiment of the present invention, the determined map points to be selected include coordinate positions and normal vectors of the map points to be selected based on the embodiment shown in fig. 5; the map point determining module, when selecting map points within the acquisition range of the camera device 520 from the map points to be selected, obtains a target map point corresponding to the road image, includes:

determining a connection line between the camera device 520 and each map point to be selected according to the coordinate position of each map point to be selected;

In another embodiment of the present invention, based on the embodiment shown in fig. 5, the processor 510 further includes: the map point construction module is used for constructing each map point in the preset map by adopting the following operations:

The terminal embodiment and the method embodiment shown in fig. 1 are embodiments based on the same inventive concept, and the relevant points can be referred to each other. The terminal embodiment corresponds to the method embodiment, and has the same technical effects as the method embodiment, and the specific description refers to the method embodiment.

Those of ordinary skill in the art will appreciate that: the drawing is a schematic diagram of one embodiment and the modules or flows in the drawing are not necessarily required to practice the invention.

Those of ordinary skill in the art will appreciate that: the modules in the apparatus of the embodiments may be distributed in the apparatus of the embodiments according to the description of the embodiments, or may be located in one or more apparatuses different from the present embodiments with corresponding changes. The modules of the above embodiments may be combined into one module, or may be further split into a plurality of sub-modules.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vision-based vehicle positioning method, comprising:

acquiring a road image acquired by camera equipment;

2. The method according to claim 1, wherein the step of mapping the target map point and the point in the edge feature map into the same coordinate system according to the estimated pose value, and determining a mapping difference between the target map point mapped into the same coordinate system and the point in the edge feature map, includes:

Or,

3. The method of claim 1, wherein the step of determining a target map point corresponding to the road image from the preset map according to the initial localization pose comprises:

4. The method of claim 3, wherein the step of screening map points within the acquisition range of the camera device from among the respective map points to be selected to obtain a target map point corresponding to the road image comprises:

5. A method as claimed in claim 3, wherein the determined map points to be selected comprise coordinate positions and normal vectors of the map points to be selected; the step of screening map points in the acquisition range of the camera equipment from the map points to be selected to obtain target map points corresponding to the road image comprises the following steps:

6. The method of claim 1, wherein each map point in the preset map is constructed in the following manner:

7. A vision-based vehicle positioning device, comprising:

a vehicle pose determination module configured to

8. The apparatus of claim 7, wherein the initial pose determination module is specifically configured to:

9. A vehicle-mounted terminal, characterized by comprising: a processor, a camera device, and a motion detection device; the processor includes:

vehicle pose determining module for