CN114088083B

CN114088083B - Graph construction method based on top view semantic object

Info

Publication number: CN114088083B
Application number: CN202111319475.3A
Authority: CN
Inventors: 周宏涛; 王璀; 范圣印; 李雪
Original assignee: Beijing Yihang Yuanzhi Technology Co Ltd
Current assignee: Beijing Yihang Yuanzhi Technology Co Ltd
Priority date: 2021-11-09
Filing date: 2021-11-09
Publication date: 2023-10-31
Anticipated expiration: 2041-11-09
Also published as: CN114088083A

Abstract

The invention discloses a drawing construction method based on a top view semantic object, which comprises the steps of carrying out initialization drawing construction based on detected enhanced feature parking space information; performing incremental mapping based on the KM algorithm matching result; a step of optimizing the map based on the key frame observation information; according to the invention, the characteristics of the current frame and the examples of the map are matched by using a KM algorithm in the process of establishing the map, a key frame confirmation and rejection strategy is provided, the position constraint among the example parking spaces is used, the relative relation among the example parking spaces is adjusted according to the multiple observation conditions, and the scale factors are added into the optimization function, so that the accuracy of establishing the map is improved, and the calculation speed is increased. The invention solves the problems of difficult image correction, difficult matching algorithm, difficult key frame processing and difficult map optimization in the process of creating the map based on the looking-around vision in the prior art.

Description

Graph construction method based on top view semantic object

Technical Field

The invention relates to the technical field of autonomous parking in the automatic driving field and the technical field of computer vision, in particular to a graph building method based on a top view semantic object.

Background

In the application field of autonomous parking technology of automatic driving, various sensors are required to complete the functions of mapping and positioning of surrounding environment. The current map construction and self-positioning technology based on the slam algorithm can be divided into laser slam, binocular slam and monocular slam according to different sensors, and the three map construction and self-positioning technologies have the defects: the laser slam is limited by equipment cost and is difficult to deploy into mass production vehicle types; the calculation complexity of the binocular slam is high, and real-time processing is difficult; monocular slam suffers from the problem of scale drift, and it is difficult to obtain absolute scale.

In order to overcome the defects and shortcomings, a scheme of looking around vision fusion IMU (inertial measurement unit) and wheel speed meter based on slam algorithm is provided, and the scheme can influence the precision in the process of drawing and positioning due to the adoption of different distorted picture processing methods, different matching algorithms, different key frame processing strategies, the addition of different optimization factors and different semantic object constraints. How to select and design more reasonable image correction methods, matching algorithms, key frame processing strategies, semantic object constraints and map optimization become a critical issue.

The scheme of the circular vision fusion IMU (inertial measurement unit) and the wheel speed meter based on the slam algorithm has the following implementation difficulties: compared with the laser mapping, the method based on the vision of looking around has the advantages that the former is greatly affected by the environment, and even if many remedial measures are taken, the effect of laser mapping is difficult to achieve. The laser mapping data processing amount is large, and the computing platform is required to have high computing power for performing point cloud matching, but the mapping accuracy is high. In the circular vision scheme, because the images are distorted and blurred, the matching difficulty is high, the image quality is affected, and the image construction precision is difficult to improve. Therefore, the difficulty and complexity of laser mapping are much greater than those of the mapping based on the looking-around vision. The method comprises the following steps:

the first difficulty is: difficulties in image correction: in the process of building the map by entering the underground parking lot, the map building vehicle is required to pass through the uphill and downhill bump zones of the parking lot, and the position of the fisheye camera can move up and down along with the bump of the vehicle at the joint of the uphill and downhill bump zones and the parking lot bump zones, so that the overhead view shot by the fisheye camera can be spliced incorrectly, and the map building error of the parking lot up and down slopes and the parking lot near the bump zones based on the overhead view is further caused. The prior art patent number CN111862673A, namely a self-positioning and map construction method of a parking lot vehicle based on a top view, adopts a real-time correction method in the map construction process: the method obtains the relative pose change of the current frame and the previous frame by calculating the homography matrix, fully considers the situation that the vehicle bumps or goes up and down the slope and adopts real-time correction. However, the correction process of the image is complex, and the homography matrix and the splicing matrix of the fisheye camera are required to be calculated in real time, so that the calculation amount is far higher than that of a flat road surface when the vehicle jolts;

The second difficulty is: difficulties of the matching algorithm: it is difficult for any single matching algorithm to not have a matching error. When a map parking space is newly built, the seen parking space in the current top view and the already built parking space in the map are used for matching, if the matching is not completed, the new parking space is built into the map, and the matching under the single condition is difficult to achieve accurate matching, for example, in the prior art, for the parking space which is not matched, the distance between the midpoints of two corner points of the parking space is further calculated, if the distance between the midpoints of the two corner points meets the requirement, the new parking space is considered, however, if the positions of the two corner points are recognized to be misplaced in the recognition process, the distance between the two corner points meets the requirement, but 1 corner point falls on the middle of 2 corner points of the existing parking space, and the other corner point falls on the middle of 2 corner points of the new parking space, and although the parking space is newly built, the two parking spaces are not adjacent but overlap. If the recognition condition is added, the 2 corner points of the current parking space and the parking space number are matched, and there is still a risk of misrecognition, for example, the parking space numbers "1" and "7" are easy to recognize as the same type when being recognized, and when the map parking space of the "7" is established, if the positions of the 2 corner points of the parking space are recognized without mistake, but the parking space number is recognized as "1", the current parking space can not be still determined to be the "7" parking space.

Third difficulty: key frame processing difficulties. The important "frame" is generally called a key frame, and the "frame data" refers to two basic information, namely "parking space information" observed by the frame and "vehicle pose" holding the frame. The difficulty of key frame processing is that two basic information, namely 'parking space information' and 'vehicle pose information' holding the frame are dynamically changed, the 'parking space position' comprises 2 corner points of the parking space and the midpoint of a parking space number detection frame, the 'parking space information' is dynamically changed, for example, the same parking space position on a map, as the vehicle is photographed while driving, the positions of 3 points of the same parking space seen by top view key frames of different positions are different, how to finally obtain a value closest to an actual parking space according to the different positions of the same parking space recorded by a plurality of key frames is difficult, because the value is influenced by not only the distance between the current vehicle and the map parking space, but also the driving condition (the bumping condition of the vehicle) and the illumination of the vehicle, and the scale factor of the top view is also influenced.

Fourth difficulty: difficulty in map optimization. The map optimization is to optimize the position of each parking space in the map and the vehicle posture of the key frame again on the basis of the established map parking spaces. Even if the method of optimizing matching is found during incremental mapping, and accurate mapping is realized, the accurate mapping cannot be realized only by one-time optimization, and the method of multiple optimization is needed. For example, a map is already built, the actual parking space is an adjacent parking space, but two adjacent parking spaces on the map do not have a common corner point; three points of two adjacent parking spaces of the actual parking space are collinear, but three vertexes of the two adjacent parking spaces on the map form included angles, so that the reasons for the errors are related to image correction of a fisheye camera, a matching algorithm, key frame data processing and the pose of a key frame. The pose of the key frame corresponds to a group of observed values, the situation that the tire of the vehicle is fully inflated and not fully inflated is detected, and the high and low positions of the camera and the observed values of the key frame are influenced, so that the problem of deep optimization and multidimensional optimization based on visual image construction is solved.

Disclosure of Invention

Aiming at the defects of the prior art, the invention provides a graph building method based on a top view semantic object, and aims to solve the problems of difficult image correction, difficult matching algorithm, difficult key frame processing and difficult map optimization which are difficult to solve in the graph building process based on the look-around vision in the prior art.

The invention provides the following technical proposal for solving the technical problems

The mapping method based on the top view semantic object is characterized by comprising the following steps of:

step one, initializing and building a map based on detected enhanced feature parking space information;

step two, performing incremental mapping based on a KM algorithm matching result;

step three, map optimization is carried out based on key frame observation information;

the first step is based on the detected enhanced feature parking space information, and the initialization map building is carried out, and the specific process is as follows:

1) Initializing before completing the drawing construction;

the initialization before the map construction is to determine the initial position and the preset track of the current vehicle;

2) Generating a top view based on image data shot by a plurality of fisheye cameras acquired by a vehicle end, and completing real-time correction of the top view;

3) Based on the top view corrected in real time, obtaining enhanced feature parking space information;

4) According to the enhanced feature parking space information, and combining with the inertia measurement unit and the wheel speed meter information, completing initialization map building; the initialization map is called an instance map, and the instance map is used as a basis for the subsequent incremental map construction;

and step two, performing incremental mapping based on a KM algorithm matching result, wherein the specific process is as follows:

1) repeating the process 2) and the process 3) of the first step to obtain a new top view after real-time correction and enhanced characteristic parking space information in the top view;

2) The enhanced feature parking space of the example map is matched with the enhanced feature parking space of the new top view after real-time correction by using a KM algorithm;

3) New example parking space: converting the enhanced feature parking space information of the current top view from a top view coordinate system to a world coordinate system; if the new parking space image features of the new top view are not matched with the new parking space image features of the initialized example map in the matching process, the example parking space is newly built in the example map, and the enhanced feature parking space information of the current top view is converted from a top view coordinate system to a world coordinate system;

and thirdly, map optimization is carried out based on key frame observation information, and the specific process is as follows:

1) Finishing corner fusion between the newly added parking space and the adjacent parking spaces in the example map;

2) Marking the collinear relative relation of corner points between adjacent parking spaces;

3) Confirming the key frames with parking spaces and the key frames without parking spaces, and eliminating the key frames for later optimization of the map;

4) Optimizing a local map by utilizing a key frame with a parking space, and optimizing the position of each parking space of adjacent parking spaces in the map and three-point collineation and image scale factors of the adjacent parking spaces; each parking space position comprises two parking space corner points of each parking space and a parking space number detection frame midpoint;

the step 2) generates a top view based on image data captured by a plurality of fisheye cameras acquired by a vehicle end, and completes real-time correction of the top view, and the specific process is as follows:

1) Converting the fisheye camera coordinates to top view coordinates;

wherein the rightmost u and v represent coordinates on the fisheye camera by pi ^-1 The bracket transformation of (2) converts the fisheye camera coordinates into corrected image coordinates, and then [ R ] is passed through _p t _p ]The inverse transformation of (1) converts the corrected image coordinates into top view coordinates, i.e. leaves the region within the marked range of the corrected image, [ x ] _p y _p ]Representing coordinates in top view;

2) R in the different cases of the formula (1) according to the formula (2) _P 、t _P Solving to obtain a corresponding external parameter matrix;

p＝HP....(2)

the association relationship between the formula (1) and the formula (2) is as follows: r of the formula (1) is obtained by decomposing H in the matrix of the formula (2) _P 、t _P The method comprises the steps of carrying out a first treatment on the surface of the Thereby obtaining R under different conditions _p t _p ]R in different cases _p t _p ]Comprising the following steps:

A. a fisheye camera external parameter matrix under a flat pavement;

B. the fisheye camera extrinsic matrix is used for the vehicle under the working conditions of different pitch angles and roll angles;

3) Generating a corresponding geometric lookup table based on the calculation result;

4) Correcting the top view image information in real time when jolting;

5) And obtaining a top view of the spliced multiple fish-eye camera pictures after real-time correction at each moment.

The top view based on the real-time correction in the step 3) obtains the information of the enhanced feature parking space, and the specific process is as follows:

1) Obtaining a top view corrected in real time at each moment;

2) Based on the deep neural network model, parking space feature detection is performed on the top view: the method comprises the steps of parking space position information detection and parking space type classification; the parking space position information is two corner points of a parking space entrance line, and the parking space type is as follows: dividing the relative position relationship between the parking space and the road into a horizontal parking space, a vertical parking space or an inclined parking space, wherein the horizontal parking space, the vertical parking space or the inclined parking space are specifically expressed as slots (x 1, y1; x2, y2; type); wherein (x 1, y 1) and (x 2, y 2) are position information coordinates under a top view coordinate system of two corner points clockwise, and type is a parking space type;

3) Based on the deep neural network model, the parking space number feature detection is carried out on the top view: the method comprises the steps of detecting the parking space number and identifying the parking space number, so that the position information of a detection frame of the parking space number characteristic and a parking space number identification result are obtained; the position information of the detection frame comprises the midpoint and the length and the width of the detection frame, and is specifically expressed as number (x, y, w, h, alpha; NUM), wherein (x, y) is the midpoint of the detection frame for the parking space number, (w, h) is the length and the width in the detection of the parking space number, alpha is the clockwise rotation angle value of the detection frame relative to the vertical direction, and NUM represents the recognition result of the parking space number;

4) Integrating the parking space features and the parking space number features to obtain the information of the enhanced feature parking space on the top view: and according to the position information of the two corner coordinates of the parking space and the parking space number detection frame under the top view coordinate system, the parking space and the parking space number information are associated, so that the enhanced characteristic parking space with the parking space number information is obtained, and the enhanced characteristic parking space is specifically expressed as (x 1, y1; x2, y2; type; x, y, w, h, alpha; NUM).

The step one, the process 4) completes initialization map building according to the enhanced feature parking space information and combining the inertia measurement unit and the wheel speed meter information, and the method specifically comprises the following steps:

1) Initializing and building a diagram: based on the enhanced feature parking space information, combining the inertia measurement unit and the wheel speed meter information to complete initialization map building, wherein the initialization map building is to acquire a key frame of a first observed parking space and build an example map by using the key frame of the first observed parking space; the key frame is a frame for observing the parking space for the first time, and comprises observed parking space information and vehicle pose information for holding the frame.

2) The example map is specifically configured to project an enhanced feature parking space under a top view coordinate system in a key frame of a first observed parking space under a vehicle coordinate system, and then convert the enhanced feature parking space under the vehicle coordinate system under a world coordinate system.

The step two, the step 2), the KM algorithm is used for matching the example parking space of the example map with the new real-time corrected enhanced feature parking space of the top view, and the method specifically comprises the following steps:

1) The method comprises the steps of carrying out multidimensional information matching on the information of the characteristic parking space and the example parking space of the example map by using the current new top view enhanced characteristic parking space information after real-time correction, and specifically comprises the following steps: matching is performed using the following 5-point information: the new top view after real-time correction is simply called a new top view, and the new top view contains the information of the enhanced feature parking spaces;

a. matching by using the parking space position information to obtain f _d ,f _d Representing the superposition degree of the parking space position information detected based on the new top view and the parking space position information of the example map; f (f) _d Also known as position error cost;

b. matching by using parking space category information to obtain f _t ,f _t Representing whether the currently detected parking spot category based on the new top view is the same as the parking spot category of the example map, f _t Also known as a parking spot category cost;

c. Matching by using the similarity of the parking space numbers to obtain f _b ,f _b Representing whether the currently detected parking space number based on the new top view is similar to the parking space number of the example map; f (f) _b Also known as space number similarity cost;

d. matching by utilizing the overlapping degree of the parking space number detection frames to obtain f _n ，f _n Representing the overlapping degree of the parking space number detection frame detected based on the new top view and the parking space number detection frame of the example map; f (f) _n Also known as checkbox overlap cost;

e. matching by using the parking space relative position information to obtain f _r ，f _r Representing the similarity degree of the current parking space relative position information detected based on the top view and the parking space relative position information of the example map; f (f) _r Also known as relative position cost;

the adjacent parking spaces refer to the situation that a common corner point exists between two parking spaces in real world space, each parking space possibly has an upper adjacent parking space and a lower adjacent parking space according to the clockwise direction, and the specific judgment formula of the adjacent parking spaces is as follows:

||P _A -P _B ||＜ΔS....(4)

wherein P is _A Representing a certain angular point position of the A parking space, namely PA _＝ [x _A ，y _A ]，P _B Representing a certain angular point position of B parking space, namely P _B ＝[x _B ，y _B ]Δs represents the distance threshold between two adjacent corner points of the parking space;

2) Performing optimal matching calculation through a KM algorithm: and combining the 5 kinds of information to obtain a total association cost function for matching between the enhanced feature parking spaces of the example map and the enhanced feature parking spaces of the new top view, wherein the specific formula is as follows: (KM (Kuhn and Munkres) algorithm for optimal matching of bipartite graphs)

f＝ω _d f _d +ω _t f _t +ω _b f _b +ω _n f _n +ω _r f _r ....(5)

Wherein f is the total cost function, ω _d 、ω _t 、ω _b 、ω _n 、ω _r The weight coefficients of the five factors are respectively; according to the formula of the total cost function f, the space between the enhanced feature parking spaces associated with all potential matches in the current new top view and the enhanced feature parking spaces of the map instance can be calculatedConstructing a corresponding incidence matrix, and finally carrying out optimal matching calculation by carrying in a KM algorithm; the potential matching association is: the space distances are similar or the parking space numbers are similar, and the potential matching association is considered;

3) And projecting the enhanced feature parking space of the current new top view into a world coordinate system.

Said f _d 、f _t 、f _b 、f _n 、f _r The calculation formula of (2) is as follows:

wherein x is _a 、y _a Is the midpoint position information of two corner points of the example map parking space, x _b 、y _b Is the position information of the midpoint of two angular points of the parking space projected to the world coordinate system in the new top view;

wherein, type _a Type of parking space, which is an example map parking space _b Is a new type of parking space in the top view;

wherein a is the stall number character string of the example map stall, b is the stall number character string of the stall in the new top view,representing exclusive OR, wherein i is the index of the parking space number character string;

wherein A is the area of a parking space number detection frame in the example map, and B is the area of the new top view, projected to the world coordinate system, of the parking space number detection frame;

f _r ＝ω _nl f _nl +ω _nn f _nn +ω _at f _at ....(10)

Wherein omega _nl 、ω _nn 、ω _at Respectively corresponding weight coefficients, f _nl Is the similarity degree of the example map parking space of the adjacent parking spaces on the parking space and the parking space number of the observed characteristic parking space, f _nn The example map parking space of the adjacent parking spaces under the parking space is similar to the parking space number of the observed characteristic parking space; f (f) _at Indicating whether the space distribution type of the parking space in the sliding window is the same as the space distribution type of the example parking space in the map, and type _ar The type of the space distribution of the example map parking spaces is type _br Is the space distribution type of the parking spaces in the sliding window.

The corner fusion between the newly added parking space and the adjacent parking spaces in the example map is completed in the step III, the process is as follows:

fusing public corner points of adjacent parking spaces: and (3) adjusting the common corner points among the parking spaces of the example map by using the observation result of the relative relation between the adjacent parking spaces in the new top view: if the situation that two adjacent parking spaces share one corner point exists in multiple observations in the sliding window in the new top view, fusing the two corner points with errors of the corresponding two adjacent parking spaces in the example map, so that the two parking spaces all have the corner point, and optimizing the position information of only one corner point in the later optimization process;

The step 2) of marking the relative relationship of corner points collineation between adjacent parking spaces, and marking the relative relationship of corner points collineation between parking spaces in an example map according to the relative relationship between parking spaces in a new top view, wherein the specific process is as follows:

adjacent map parking spaces in the new top view, including the already matched and unmatched parking spaces in the example map, wherein the unmatched parking spaces are the new parking spaces to be established in the map; the relative relation refers to whether angular points between adjacent parking spaces are collinear or not; the adjacent parking spaces refer to the common angular points of the two parking spaces in the top view, each parking space possibly has an upper adjacent parking space and a lower adjacent parking space in the clockwise direction, and the step e in the 1) is specifically defined as in claim 5;

the specific judgment formula of the corner collineation of the adjacent parking spaces is as follows:

wherein x is _A1 、y _A1 And x _A2 、y _A2 Is the position information of two angular points of the parking space A, x _B1 、y _B1 And x _B2 、y _B2 The method is characterized in that the method comprises the steps that the method is that the position information of two corner points of a parking space B is obtained, delta omega represents the threshold value of an included angle of connecting lines of the two corner points of two adjacent parking spaces, if the absolute value of the included angle of the connecting lines of the two corner points of the adjacent parking spaces is smaller than the threshold value of the included angle, the corner points of the two parking spaces are considered to be collinear, and the corner points of the two parking spaces are marked as collinear.

And step three, the step 3) is to confirm the key frames with the parking spaces and the key frames without the parking spaces, reject the key frames and optimize the map in the later period, and comprises the following specific processes:

1) Validating key frames

a. Determining key frame according to whether new parking space exists

Judging whether a frame corresponding to the current new top view is a new parking space in the example map, if so, confirming that the frame corresponding to the current new top view is a key frame and storing the key frame;

b. determining key frames based on distance

If the frame corresponding to the current new top view does not observe a parking space or the observed parking space is not a newly-built parking space in the example map but a historical parking space, confirming a key frame when the distance or course angle difference value between the current frame and the last key frame is larger than a certain threshold value, and if the key frame does not observe the parking space information, the key frame information does not comprise image data and only comprises pose information of the current vehicle, wherein the pose of the current vehicle is the pose of the current vehicle when the image is shot;

the formula for inserting key frames according to distance is as follows:

‖P _k+1 -P _k ‖＞ΔP....(15)

wherein P is _k Representing the central position of the vehicle at time k, i.e. P _k ＝[x _k ，y _k ]，P _k+1 Representing the vehicle centre position at time k+1, i.e. P _k+1 ＝[x _k+1 ，y _k+1 ]Δp represents a set vehicle center distance threshold, and if the vehicle center distance between time K and time k+1 is greater than the distance threshold, a new key frame is confirmed;

The formula for inserting the key frame according to the heading angle difference is as follows:

‖θ _k+1 -θ _k ‖＞Δθ....(16)

wherein θ _k Representing the heading angle, θ, of the vehicle at time k _k+1 Representing the heading angle of the vehicle at the moment k+1, wherein delta theta represents a set heading angle threshold value, and if the absolute value of the difference between the heading angles at the moment K and the moment k+1 is larger than the heading angle threshold value, confirming a new key frame;

2) Removing key frames:

a. calculating the observation score of the frame corresponding to the current new top view to the same example parking space: each parking space of the example map records the observation results of a plurality of key frames on the same parking space: wherein the farthest observations and the nearest observations are recorded: the observation data of the nearest observation and the farthest observation are recorded according to the following formula, and the observation score of each key frame to the parking space of the example is calculated:

L _min ＝min(L _min ,L)....(17)

L _max ＝max(L _max ,L)....(18)

wherein L is _min The minimum distance from the midpoint of the parking space corner or the parking space number detection frame in the top view to the center of the top view is the nearest observation; l (L) _max For the furthest observation, namely the maximum distance from the point of the parking space corner or the parking space number detection frame in the top view to the center of the top view, L is the distance from the point of the parking space corner or the parking space number detection frame in the top view to the center of the top view of the key frame with the current calculated score, and g is the observation score in the current key frame observation weight;

b. According to the calculation result of the observation score, different observation weights are given;

c. if the observation weight of a certain key frame is lower than a set threshold value, the key frame is rejected; the calculation of the observation weight is based on the distance between the corner point of the characteristic parking space and the center of the image and the distance between the midpoint of the parking space number detection frame and the center of the image, and is also influenced by the running condition of the vehicle during observation and the shadow of the illumination condition during observation;

the calculation of the observation weight is mainly divided into two parts: observing the angular points of the top view parking spaces and the midpoints of the top view parking space number detection frames; the shortest distance from the center of the top view to the corner point of the image feature parking space is taken as the nearest observation of the corner point, the shortest distance from the center of the top view to the midpoint of the detection frame of the image feature parking space is taken as the nearest observation of the parking space, and the specific calculation formula of the observation weight of the key frame is as follows:

f＝w _b w _l (∑(w _c g _c )+∑(w _n g _n ))....(20)

wherein f represents the observation weight of the key frame, w _b 、w _l Weight coefficient representing the influence of vehicle jolt, illumination g _c 、g _n Representative anglePoint and parking space number observation score, w _c 、w _n Whether the representative corner and the parking space number are the nearest observations or not is 10 for the nearest observations and 1 for the non-nearest observations;

the key frames with parking spaces are utilized to perform local map optimization in the step 4), and each parking space position of adjacent parking spaces in a map and three-point collineation and image scale factors of the adjacent parking spaces are optimized; each parking space position of the adjacent parking spaces comprises a parking space corner point of each parking space and a parking space number detection frame midpoint;

The specific expression is as follows:

the above formula (21) is a difference value between the ith parking space coordinate information and the ith parking space coordinate information of the example map in the j-th key frame, and the information is converted into world coordinate system parking space coordinate information, wherein the difference value is a difference value under the world coordinate system, and specifically is: the 1 st item on the right of the equal sign in the formula (21) is world coordinate information of the i-th parking space in the example map, the initial value of the value is coordinate information under the world coordinate system, wherein the observation information with highest observation scores in the observation results of the same parking space of the reserved plurality of key frames in the process 3) of the step three is converted into the coordinate information under the world coordinate system; the data in the bracket on the right of the equal sign of the equation (21) and the item 2 are coordinate information obtained by converting the ith parking space observed by the jth key frame in the vehicle coordinate system into the world coordinate system, and the difference value between the ith parking space and the ith parking space of the example map can be obtained by subtracting the item 2 from the item 1 on the right of the equal sign of the equation (21).

The method comprises the following steps:

a. item 1 to the right of the equal signRepresenting coordinates of two corner points in the ith example map parking space or the midpoint of the parking space number detection frame in a world coordinate system, wherein the value range of l is { a, b, c }, and represents the two corner points of the parking space and the midpoint of the parking space number detection frame respectively, and w represents the generation of The table is in a world coordinate system;

b. s inside the right-hand item 2 of the equal sign bracket _f Representing the scale factor of the image, solving the formula (25) to obtain s _f A value;

c. the 1 st and 3 rd matrices within the brackets on the right of the equal sign represent the changes in rotation and translation of the vehicle, where 1 st is the change in rotation, 3 rd is the change in translation, matrix [ x ] _j y _j θ _j ]The position of a jth key frame vehicle in a world coordinate system in a sliding window is the combination of continuous several frames of key frame observation information and key frame position and posture information of the vehicle in a moving state;

d. matrix 2 in brackets on right of equal signRepresenting coordinates of two corner points in the ith example map parking space or coordinates of a vehicle coordinate system of a jth key frame of a parking space number detection frame in a sliding window, wherein the value range of l is { a, b, c }, and the two corner points of the parking space and the midpoint of the parking space number detection frame are represented respectively; />Coordinate information representing different parking spaces observed by the vehicle in different pose states;

the above formula (23) and formula (24) respectively represent vectors formed by 2 corner points of adjacent parking spaces, and formula (22) represents the cross multiplication between the two vectors of formula (23) and formula (24); if the vector cross multiplication result approaches zero, the three points tend to be collinear; wherein i and i-1 represent the ith and parking spaces and the ith-1 parking space;

Respectively representing two corner coordinates of the ith example parking space, and for the ith-1 parking space, also respectively representing two corner coordinates of the ith-1 example parking space;

the equation (25) above is the total optimization function, the first part on the right of the equal sign is to sum the squares of the position errors observed by the current parking space for the different key frames of the equation (21) and then multiply the squares by Λ _ijl ，Λ _ijl For assigning weights, Λ _ijl Is a diagonal matrix in which elements are present only at diagonal lines, the elements of the matrix being observation weights corresponding to the observation information used in formula (21); the second part on the right of the equal sign is to sum squares of three-point collinear errors observed by the current parking space by different key frames of the formula (22) and then multiply by lambda _k ，Λ _k Only diagonal elements exist, the diagonal elements are all 1, and weight distribution is carried out on each three-point collineation error square.

The third step further comprises a process 5): and (3) carrying out loop detection by utilizing the key frames with the parking space information, optimizing the pose of the key frames in the map by utilizing the key frames with the parking space information and the key frames without the parking space, and optimizing the parking space corner points and the midpoint of the parking space number detection frame in the map again: the loop detection comprises the following steps: searching for a parking space with a similar number or a similar distance to the parking space in the map, matching and associating the parking space in the sliding window with an example parking space in the map by using a KM algorithm, and if a certain amount of matching exists, considering that loop-back occurs, wherein the specific process is as follows:

1) Optimizing the pose of the key frame, and the specific formula for optimizing the pose of the key frame is as follows:

wherein T is _ij Representing the relative motion between the i and j th frame key frames, T _i And T _j Representing the pose of the key frames of the ith frame and the jth frame respectively;is T _i Is equal to the identity matrix;

wherein e _ij Representing an error formed by pose transformation between an ith frame and a jth frame key frame;is T _ij An inverse matrix of (a); ln represents a logarithm operation; the V symbol represents the transformation from the form of the Liqun to the form of the Lialgebra;

f is a total cost function, and represents pose transformation to form the square sum of errors; e, e _ij Representing the error formed by the pose transformation between the ith frame and the jth frame key frame,e is _ij A diagonal transformation matrix;

in the above formula, i and j each represent a key frame number, T _i And T _j Representing the pose of the key frames of the ith frame and the jth frame respectively; [ x ] _j y _j θ _j ]Is the pose of the jth key frame vehicle in the world coordinate system in the sliding window, and is constructed into T by form transformation _j Form (iv);

2) Returning to the process 4) of the third step, and carrying out the optimization of the example map parking space position information again, wherein the scale factors are unchanged.

Advantageous effects of the invention

1. The invention corrects the spliced top view by using a pre-calculated fisheye camera external parameter matrix under the condition that a large error occurs in the top view when the vehicle bumps. On the premise of ensuring the precision, the calculation speed is accelerated.

2. In the invention, a KM (Kuhn-Munkras) algorithm is used for completing the matching of the characteristics of the current frame and the examples of the map in the process of establishing the map. Compared to ICP (iterative nearest neighbor) algorithms, KM algorithms can be based on more dimensional information, such as: the parking space category, the overlapping degree of the parking space number detection frames, the similarity degree of the parking space numbers, the relative position information of the parking spaces and the like. In case of poor initial values, still more robust results can be obtained.

3. The invention provides a key frame confirmation and rejection strategy. When a frame of spliced top view is input, judging whether to newly establish a map instance parking space and whether to update the observed feature parking space, and calculating the weight of the feature parking space; if new construction or updating is needed, the weight of the current frame is increased, and the weight of the key frame of the feature parking space which is previously built or observed is reduced. And if the weight of the key frame is lower than the threshold value, performing key frame removal. Under the condition that the new map instance parking space is not matched with the characteristic parking space of the current frame in the local map instance parking space, the Euclidean distance between the characteristic parking space of the current frame and the matched map instance parking space is smaller than the previous observation distance. While improving the precision, the scale of the calculated amount is reasonably controlled.

4. The invention uses the position constraint among the example parking spaces, adjusts the relative relation among the example parking spaces according to the observation condition for a plurality of times, and improves the precision of the map construction.

5. When the map construction optimization function is constructed, the problem of scale change caused by the change of the camera height, the change of the tire pressure and the error of the wheel speed meter of the vehicle is taken into consideration, so that scale factors are added into the optimization function, and the map construction with higher precision is completed.

Drawings

FIG. 1 is a schematic block diagram of the method of the present invention;

FIG. 1-1 shows a specific process of step one of FIG. 1 according to the present invention;

FIGS. 1-2 illustrate a second embodiment of the process of FIG. 1 according to the present invention;

FIGS. 1-3 illustrate a third embodiment of the process of FIG. 1 according to the present invention;

FIG. 2-1 is a schematic diagram of a key frame for confirming a new parking space according to the present invention;

FIG. 2-2 is a schematic diagram of a second key frame for confirming a new parking space according to the present invention;

FIGS. 2-3 are schematic diagrams of key frame validation according to distance according to the present invention;

FIGS. 2-4 are schematic diagrams of key frame validation according to heading angle in accordance with the present invention;

FIGS. 2-5 are schematic diagrams of a parking space corresponding to a plurality of key frames according to the present invention;

FIG. 3-1 is a schematic diagram of overlapping degree matching of a parking space number detection frame according to the present invention;

FIG. 3-2 is a schematic diagram of matching of relative position information among parking spaces according to the present invention;

FIG. 4 is a schematic diagram of pose optimization for a keyframe according to the present invention;

FIG. 5-1 is a top view of the neural network of the present invention, not including parking space position information before learning;

fig. 5-2 is a top view of the neural network of the present invention including parking space position information after learning.

Detailed Description

The design principle of the invention is as follows:

the invention is further explained below with reference to the drawings:

1. dynamic key frame design principle

First, overview: the key frame is the most important frame. The purpose of dynamically designing the key frames is to find the most excellent key frame in a plurality of key frames corresponding to the same parking space, and the purpose of finding the most excellent key frame is to optimize the skew and twisted parking spaces which are not consistent with the actual parking space pose in the map by taking the parking space pose seen by the key frame as a reference, so that the most approximate degree to the actual parking space pose is achieved after the optimization.

And the second one parking space corresponds to the principle of a plurality of key frames. As shown in fig. 2-5, detecting the line of sight of a vehicle from finding a parking space to leaving the parking space is a far-to-near and then near-to-far process. This process will have multiple keyframes "seeing" the same spot, so one spot in the map retains multiple keyframes.

Third, design difficulties: the difficulty is that the same keyframe, when the vehicle is in different positions, its score varies with the position of the vehicle, and if only 1 score is calculated for each keyframe, the keyframe with the highest score is never obtained. The key frame score is determined according to the distance between the key frame score and the target point, and the maximum value and the minimum value, wherein the maximum value is the value with the largest distance between the current all key frames and the target point, and the minimum value is the value with the smallest distance between the current all key frames and the target point, and taking the number 2 key frames in fig. 2-5 as an example: when the vehicle is detected at the No. 2 key frame position, the "current all key frames" only include No. 1 and No. 2, the maximum value is the distance between the vehicle and the target point, and the minimum value is the distance between the No. 1 key frame and the target point, so that the No. 2 key frame score is high (0.8 score is assumed) relative to the No. 1 key frame, but when the vehicle reaches the No. 3 key frame position, the No. 3 key frame score should be higher than the No. 2 score because the distance between the vehicle and the target point is closer, but the No. 2 key frame score may be the same as the No. 3 key frame score, so that a conclusion that the No. 3 key frame score is higher than the No. 2 score cannot be obtained. Therefore, the score for key frame number 2 cannot be calculated only once. Since the score of the first time the key frame No. 2 is calculated relative to the key frame No. 1, but the minimum value is changed when the vehicle reaches the key frame No. 3, the score of the key frame No. 2 should be recalculated, and the minimum value should be the distance from the key frame No. 3 to the target point, so that the score is reduced when the score of the key frame No. 2 is recalculated. Similarly, when the vehicle reaches the key frame number 7 position, the score of key frame number 2 is recalculated, and the score is lower.

Fourth, the solution of the present invention: as shown in fig. 2-5, in addition to calculating the score for the current key frame, the key frames for which the score has been calculated are recalculated. A loop calculation section for recalculating key frames is set, and this loop calculation section may be a section of 10 frames of data or a section of 5 frames of data. For example, the key frame number 2 in fig. 2-5 has a loop calculation interval of 5 frames of data, which means that when the self scores of key frames number 3, 4, 5, 6 and 7 are calculated, the score of the key frame number 2 is also calculated again. When the vehicle reaches the No. 7 key frame position from the No. 2 key frame position, the score of the No. 2 key frame is lower and lower along with each calculation, and the same loop calculation method is adopted for the No. 3, 4, 5 and 6 key frames, so that the conclusion that the score of the No. 7 key frame is highest can be obtained at the No. 7 key frame position.

Fifthly, eliminating the key frames. The purpose of finding the most excellent key frame is to reject the key frame with the lowest score, but the so-called most excellent key frame cannot be used for positioning, and because only one frame of data cannot be used for positioning even very good data, a plurality of good and best key frame data are reserved after the worst key frame is rejected.

2. Map optimization design principle

1) Necessity of map optimization. Each parking space of the map reserves a plurality of key frames for 'seeing' the parking space, and the key frames are key frames which are good in quality and are not removed. Although many parking spaces are built in the map through incremental map building, the pose of the parking spaces is not ideal and can be distorted and twisted, and the actual parking spaces are quite different, for example, the actual adjacent parking spaces have common angular points and three points are collinear, but the adjacent parking spaces are separated in the map and have no common angular points, and angles exist between two lines of the three points. At this time, the physical pose of each parking space in the map needs to be optimized through a plurality of better key frames reserved in the parking space.

2) A method for optimizing a map. The map optimization is divided into three steps, the angular points of the first and adjacent parking spaces are fused, the two separated angular points are fused together to form a common angular point, but after fusion, the two adjacent parking spaces only have the common angular point, but the positions of the three angular points are not optimized yet, and the positions of the three angular points are optimized secondly: firstly, as shown in a formula (21), taking the pose of a parking space 'seen' by the key frame with the highest most excellent score as a basis (x, y and angle), and then respectively comparing the pose with the data of other key frames to respectively obtain a group of error values; second, as shown in the first part of the equal sign of equation (25), when a group of error values are obtained by equation (21), they are multiplied by the sum of squares _ijl ，(Λ _ijl For assigning weights, Λ _ijl The method is a diagonal matrix and a weight distribution method), an optimal value is obtained, and finally the optimal value is used as the physical position of the current parking space closest to the actual parking space after optimization. Third, the optimization of the collineation of the three points. Although the positions of three corner points of adjacent parking spaces are optimized to be accurate, whether the positions of the three points are accurate or not is finally judged, if the positions of the three points are not collinear and have angles, the positions of the three corner points are not accurate enough, but the angular point collineation is optimized by adopting another method, so that the collineation of the three points is realized, and the positions of the three corner points are further optimized after the optimization. The three-point collineation optimization is divided into two steps, firstly, the errors of the three-point collineation of a plurality of key frames are respectively calculated, as shown in formulas (22), (23) and (24), then the errors are passed through the right second part of the equal sign of formula (25), and then the errors are passed through the square sum and then multiplied by lambda _k ，(Λ _k Only the diagonal has elements, and the diagonal elements are all 1, and the weight distribution is carried out on the square of each three-point collineation error), and finally the optimization of the three-point collineation is realized.

3. Multidimensional matching design principle

First, design difficulties. The incremental map building is to match the currently seen parking space in the top view with an example map in the map, and the parking space is considered to be a new parking space if the map is not matched. The difficulty is that: it is difficult to determine whether a new parking space, an existing parking space, or a third pending parking space is not newly added nor existing by any single information alone: 1) The position of the new vehicle is often not outside or adjacent to the old vehicle, but one of the 2 corner points of the new vehicle falls on the existing vehicle space in the map, and the other falls outside the vehicle space and occupies 50 percent of the existing vehicle space, so that whether the new vehicle is the new vehicle cannot be judged only by the positions of the 2 corner points; 2) If the judgment can not be performed only by the parking space numbers on the parking spaces, the identification errors of the parking space numbers '1' and '7', '3' and '8' are often generated; 3) Even if 2 corner points of the new parking space in the top view meet the condition (adjacent corner points or public corner points) of the new parking space, the position of the new parking space in the top view is 90 degrees different from the position of the example map, and the true position of the new parking space cannot be determined to be 90 degrees different, or the situation is caused by misjudgment, and in this case, the incremental map building still encounters a problem.

Second, solution: as shown in the formula (6) -the formula (14), 1) adopts 5 kinds of information comprehensive judgment. Besides adopting three kinds of information of parking space position, parking space number and parking space posture (parking space type) to judge, a fourth kind of information is adopted: judging the adjacent parking space information: when the parking space numbers of 1 and 7, 3 and 8 are mixed, the judgment can be carried out according to the parking space numbers of the adjacent parking spaces of the current parking space up, down, left and right, the left and right of the 7-numbered parking space should be 6 and 8, and the adjacent parking space of the 1-numbered parking space should be 3, so that the problem of incorrect identification of 1 and 7, 3 and 8 is solved. Fifth information is also employed: the overlapping degree of the parking space number detection frame, if the current top view is a new parking space, the parking space number detection frame (a frame tightly attached to the design of the parking space number, which is the parking space number detection frame) seen by the parking space must not coincide with the existing parking space, and should be at the set position of the adjacent parking space, when one of the 2 corner points of the new parking space falls on the existing example map parking space, the overlapping degree can be further judged by the position of the parking space number detection frame of the new parking space, because the area of the detection frame is smaller and more accurate than the area of the 2 corner points, if the detection frame falls at the position of the new parking space but not at the position of the existing example map parking space, although one corner point position falls on the existing example map parking space, the parking space can be judged to be the new parking space, so that incremental building is performed. 2) The weight coefficients are allocated to the above 5 kinds, the weight is not fixed, and the 5 weight coefficients can be dynamically adjusted according to the situation, which is not described herein.

Based on the principle of the invention, the invention designs a graph building method based on a top view semantic object.

The mapping method based on the top view semantic object is shown in fig. 1, 1-2 and 1-3, and is characterized by comprising the following steps:

1) Initializing before completing the drawing construction;

supplementary explanation:

the top view of fig. 5-1 is different from fig. 5-2. The top view of fig. 5-1 is the original top view, which is formed by stitching top view images of multiple fish-eye cameras over a range, the effect of which is shown in fig. 5-1. The physical space range of the top view is 10 meters by 10 meters, the image resolution is 720 by 720, and the spatial Scale of each pixel is 13.88 millimeters, namely the Scale factor Scale is 13.88. At this time, the top view retains the picture information, but does not have the detected information such as the parking space corner coordinates, the detecting frame midpoint coordinates, the parking space number identification result and the like. The initial top view includes only the image center coordinates and the top view vehicle rear axle center coordinates.

1) Converting the fisheye camera coordinates to top view coordinates;

p＝HP....(2)

A. a fisheye camera external parameter matrix under a flat pavement;

4) Correcting the top view image information in real time when jolting;

1) Obtaining a top view corrected in real time at each moment;

supplementary explanation:

as shown in fig. 5-2, the top view is a top view after learning by the neural network, and compared with the top view in fig. 5-1, the top view has the parking space position information added, which are the parking space position information identified after learning by the deep neural network. Fig. 5-2 is a schematic diagram of a detection frame for detecting corner points and number of parking spaces in a top view and a recognition result, wherein round small points in the diagram represent coordinates of corner points of the parking spaces, adjacent parking spaces are relatively close, and the corner points may be blocked. The triangle dots in the figure represent the coordinates of the points in the detection frame. The diamond shaped dots in the figure represent the image center coordinates. The hexagonal dots in the figure represent the center coordinates of the rear axle of the vehicle in top view. The portion pointed to by the dotted line is the result of recognition of the content of the detection frame. After the detection of the deep neural network model, the information of the top view not only comprises the center coordinates of the image and the center coordinates of the rear axle of the top view vehicle, but also comprises the information such as the coordinates of the parking space corner points and the midpoint of the parking space number detection frame in the top view, the identification result of the parking space number and the like.

Likewise, the neural network can also identify top view parking space type information, and the parking space types mainly comprise three types of horizontal parking spaces, vertical parking spaces and inclined parking spaces, and fig. 5-2 only shows the parking space positions identified after the deep neural network is learned, but does not comprise the parking space types.

Supplementary explanation:

the patent multi-point content relates to the transformation from top view coordinates to world coordinates (i.e. world coordinates), comprising a first step, a second step, a third step and a fourth step. Specifically, in the top view coordinate system, with the top left corner of the top view as the origin, the x-axis is directed to the rear of the vehicle, the y-axis is directed to the right of the vehicle, and the z-axis is directed upward. The vehicle coordinate system takes the center of the rear axle of the vehicle as an origin, the x-axis points to the front of the vehicle, the y-axis points to the left of the vehicle, and the z-axis faces upwards. The transformation relationship between the two is as follows:

in the above formula, scale is a scale factor between the top view coordinate system and the vehicle coordinate system describing the ratio between the picture pixels and the real world length; x is x _p 、y _p Is the position, x, of a point in the top view coordinate system _b 、y _b Is the position of a point in the vehicle coordinate system, deltax is the absolute value of the difference in distance between the center point of the top view and the center of the rear axle of the vehicle in the top view. The first term on the right of the equation is capable of converting the coordinate information of the point in the top view coordinate system represented by the second term into the vehicle coordinate system with the vehicle center as the origin, and adding the third term representing the distance from the top view center point to the vehicle rear axle center multiplied by the scale factor, so that the coordinate in the vehicle coordinate system with the vehicle rear axle center as the origin can be obtained.

Further, the enhanced feature parking space under the vehicle coordinate system is converted into the world coordinate system, and the conversion relationship between the enhanced feature parking space and the world coordinate system is as follows:

in the above formula, x _b 、y _b Is the position, x, of a point in the vehicle coordinate system _w 、y _w Is the coordinate in the world coordinate system, x, y and theta are the world coordinate systemLower vehicle pose. The first item on the right of the equation equal sign represents a rotation matrix obtained by calculating the heading angle of the vehicle pose, the second item represents the position information of the point under the vehicle coordinate system, and the coordinates under the world coordinate system can be obtained finally through the rotation of the first item and the translation of the third item.

The two corner points and the middle points (x 1, y1; x2, y2; x, y) of the enhanced feature parking space in the top view can be converted into the world coordinate system by carrying out the coordinate transformation of the two steps, so that the first observed enhanced feature parking space is built into an example map, and the initial map building is completed. The length and width information (w, h) of the detection frame of the enhanced feature parking space is only subjected to scale transformation, namely multiplied by a scale factor in a formula, and then the length and width information in the global coordinate can be obtained through the top view pixel and the scale factor. And adding the angle information alpha of the detection frame of the enhanced feature parking space to the heading angle theta of the vehicle to obtain the angle information in the global coordinates. The parking space category of the enhanced feature parking space and the recognition result of the parking space number are used as non-space position information and are not transformed.

a. matching by using the parking space position information to obtain f _d ，f _d Representing the superposition degree of the parking space position information detected based on the new top view and the parking space position information of the example map; f (f) _d Also known as position error cost;

c. matching by using the similarity of the parking space numbers to obtain f _b ,f _b Representing the similarity of the currently detected parking space number based on the new top view and the parking space number of the example map; f (f) _b Also known as space number similarity cost;

||P _A -P _B ||＜ΔS....(4)

wherein P is _A Representing a certain angular point position of the A parking space, namely P _A ＝[x _A ，y _A ]，P _B Representing a certain angular point position of B parking space, namely P _B ＝[x _B ，y _B ]Δs represents the distance threshold between two adjacent corner points of the parking space;

f＝ω _d f _d +ω _t f _t +ω _b f _b +ω _n f _n +ω _r f _r ....(5)

Wherein f is the total cost function, ω _d 、ω _t 、ω _b 、ω _n 、ω _r The weight coefficients of the five factors are respectively; according to the formula of the total cost function f, the cost between all the potential matching associated enhanced feature parking spaces in the current new top view and the enhanced feature parking spaces of the map instance can be calculated, so that a corresponding association matrix is constructed, and finally, the corresponding association matrix is brought into a KM algorithm to perform optimal matching calculation; the potential matching association is: the space distances are similar or the parking space numbers are similar, and the potential matching association is considered;

supplementary explanation:

two corner coordinates of an example parking space such as parking space N123 are [2,2]、[2，4.5]，x _a ＝(2+2)/2＝2，y _a = (2+4.5)/2=3.25; the parking space N123 is converted into two corner coordinates [1.8,2.1 ] under the world coordinate system in the new top view]、[1.8，4.6]，x _b ＝(1.8+1.8)/2＝1.8，y _b ＝(2.1+4.6)/2＝3.35；f _d ＝0.22；

wherein a is the stall number character string of the example map stall, b is the stall number character string of the stall in the new top view, Representing exclusive OR, wherein i is the index of the parking space number character string;

supplementary explanation:

for example, the number character string of the example map parking space is N124, the recognition result in the top view is N123, where n=n, 1=1, 2=2, 3+.4, and if the above results are accumulated, f _n The result of (3). I.e., comparing, bit by bit, whether the letters or numbers of each bit of the identification result of the space number of the example map and the space number of the top view are the same.

supplementary explanation:

as shown in fig. 3-1, a is the area of the space number detection frame in the example map, B is the area of the space number detection frame projected under the world coordinate system in the new top view, and it is assumed that the two areas are equal. The left part of the figure is that A and B have no overlapping area at all, A and B mean that the area of the overlapping part of two detection frames is calculated, a & -B means that the sum of the areas of the two detection frames is calculated only once, and the repeated part is counted once, thus substituting formula (9), a ≡b=0, a ≡b=2a=2b, f _b Equal to 0; the middle of the figure is that A and B have half of the overlapping area, so A is equal to half of the area of the detection frame, the repeated part is counted only once, and thus the formula (9) is substituted, A is equal to B=A/2=B/2, A is equal to U.S. B=3A/2=3B/2, f _b Equal to 1/3; the right side of the figure is A and BThe two detection frames are completely overlapped, so that A U B is equal to the detection frame area, A U B means that the sum of the two detection frame areas is calculated, and the repeated part is counted only once, thereby substituting the formula (9), A U B=A=B, f _b Equal to 1.

f _r ＝ω _nl f _nl +ω _nn f _nn +ω _at f _at ....(10)

Supplementary explanation:

the formula (10) is composed of formulas (11), (12), and (13), and coefficients are assigned according to a certain weight. Wherein formulas (11) and (12) respectively represent the similarity degree of the parking space numbers of the example parking spaces of the upper adjacent parking space and the lower adjacent parking space of the parking space and the observed characteristic parking space, and the specific calculation refers to the supplementary explanation of formula (8). The space distribution type of the parking space in the formula (13) refers to whether the parking space is at an intersection, as shown in fig. 3-2, an example parking space in a sliding window is shown in a dotted line frame, an example parking space of a map is shown in a solid line frame, the space distribution type of the parking space mainly can be divided into an intersection parking space and an non-intersection parking space, wherein N124, N125, N128, N111, N112 and N115 are intersection parking spaces, and N123, N126, N127, N110, N113 and N114 are non-intersection parking spaces. The sliding window is composed of the position information of the observed parking space of the most recent consecutive key frames and the pose information of the key frames.

fusing public corner points of adjacent parking spaces: and (3) adjusting the common corner points among the parking spaces of the example map by using the observation result of the relative relation between the adjacent parking spaces in the new top view: if the situation that two adjacent parking spaces share one corner point exists in multiple observations in the sliding window in the new top view, fusing the two corner points with errors of the corresponding two adjacent parking spaces in the example map, so that the two parking spaces all have the corner point, and optimizing the position information of only one corner point in the later optimization process; marking the collinear relative relation of corner points between adjacent parking spaces in the process 2) of the third step,

according to the relative relation between the parking spaces in the new top view, the relative relation of corner collineation between the parking spaces in the example map is marked, and the specific process is as follows:

1) Validating key frames

a. Determining key frame according to whether new parking space exists

supplementary explanation:

as shown in fig. 2-1, assuming that the traveling direction of the vehicle is from left to right, when the vehicle "sees" two newly added spaces N123, N124 in fig. 1, the frame of the current top view is confirmed as a key frame because the newly added spaces are "seen"; as shown in fig. 2-2, when the vehicle continues to move forward from left to right, a new N125 parking space is seen in addition to the original N123 and N124 parking spaces, so that the current frame is confirmed to be a key frame.

b. Determining key frames based on distance

the formula for inserting key frames according to distance is as follows:

‖P _k+1 -P _k ‖＞ΔP....(15)

supplementary explanation:

as shown in fig. 2-3, when the vehicle continues to travel, no new "seen" space is in front of the N125 space for a certain distance, and the distance from the vehicle to continue to travel forward beyond the N125 space of fig. 3 exceeds a certain threshold, it is confirmed as a key frame by equation (15); however, the key frame at this time only has the pose information of the frame, but does not have the parking position information. The key frames without "spot location" information are confirmed because the following step three, process 5), is to optimize the location of the key frames first, rather than the spot. If the distance between two frames is too far, the pose effect of optimizing the key frames is poor, in order to maintain the optimizing effect, the key frames are confirmed according to the distance, and the key frames are confirmed as the key frames as long as the distance is up.

‖θ _k+1 -θ _k ‖＞Δθ....(16)

supplementary explanation:

as shown in fig. 2-4, the heading angle of the vehicle changes at different moments, and the change of the heading angle affects the observation of the parking space position, so the pose of the frame at the moment is important, and is marked as a key frame. The key frames whose heading angle difference exceeds the threshold are confirmed because the following procedure 5) of step three is to optimize the vehicle pose of the key frame first, not the parking spot. If the difference value of the course angle between the two frames exceeds the threshold value, the posture effect of the optimized key frame is poor, in order to keep the optimized effect, the key frame is confirmed according to the difference value of the course angle, and the current frame is confirmed as the key frame as long as the difference value of the course angle between the current frame and the last key frame exceeds the threshold value.

2) Removing key frames:

L _min ＝min(L _min ,L)....(17)

L _max ＝max(L _max ,L)....(18)

Wherein L is _min The minimum distance from the midpoint of the parking space corner or the parking space number detection frame in the top view to the center of the top view is the nearest observation; l (L) _max For the furthest observation, namely the maximum distance from the point of the parking space corner or the point of the parking space number detection frame in the top view to the center of the top view, L is the distance from the point of the parking space corner or the point of the parking space number detection frame in the top view to the center of the top view, and g is the observation score in the current key frame observation weight;

supplementary explanation:

1) The conditions for the formulas (17), (18), (19) are that the current frame is above the 2 nd frame, if only the 1 st frame is stillWithout frame 2, minimum value L _min Maximum value L _max The score g defaults to a maximum value (10, 10), respectively, because: when the first frame observes N123 slots, there is no comparison of the other frames, so it is considered the maximum. For example, the vehicle finds the parking spot from a position farther from the parking spot N123, and is at a maximum value by default because only this distance is present, although the distance is farther. However, as the vehicle moves forward, the distance from the N123 parking space becomes closer and the score becomes higher as the distance becomes closer, so the score of the first frame changes and becomes lower before the shortest distance is reached.

2) Minimum value L of formula (19) _min Maximum value L _max : minimum value L _min Maximum value L _max Minimum and maximum values of class a corner points A, A ' … …, minimum and maximum values of class B corner points B, B ' … …, minimum and maximum values of class C detection frame center points C, C ' … … for all keyframes of N123 parking spaces (provided that 2 keyframes of N123 parking spaces are seen); for example: assuming that when N123 is observed by the first frame, the observed values of the two corner points and the middle point of the detection frame are {2.8,4.9,4.4}, and when N123 is observed by the second frame, the observed values of the two corner points and the middle point of the detection frame are {3.2,2,3.25}, calculating the minimum value L _min When comparing the same kind of angular points A, A ', B, B' of the first frame and the second frame, comparing the same kind of detection frame midpoint C, C ', wherein the minimum value relative to A, A' is 3.2 to 2.8, the minimum value relative to B, B 'is 4.9 to 2, the minimum value is 2, and the minimum value relative to C, C' is 4.4 to 3.25; similarly, the maximum value for A, A ' is 4.9 compared to 2.8 and 4.9, the maximum value for B, B ' is 4.9 and 2 compared to 4.9, the minimum value for C, C ' is 4.4 and 3.25 compared to 4.4; if the key frame of the N123 parking space is 6, the minimum value of the class A corner points is the minimum value of 6 class A corner points, and is not the minimum value of the previous frame compared with the next frame.

3) The formulas (17), (18) and (19) are designed for each corner point and the center point of the detection frame, if only two frames of data exist, 3 minimum values, 3 maximum values and 3L values are obtained, and after the minimum values, the maximum values and the L values are respectively substituted into the formula (19), 3 score values are obtained;

4) Regarding L: l is the distance from the midpoint of the parking space corner or parking space number detection frame in the top view to the center of the top view of the key frame with the current calculated score, which means that: if the currently calculated key frame score is the score of the last frame, L is the distance between the point of the parking space corner or the point of the parking space number detection frame in the top view of the last frame and the center of the top view, and if the currently calculated key frame score is the score of the latest generated key frame, L is the distance between the point of the parking space corner or the point of the parking space number detection frame in the latest generated top view and the center of the top view.

5) The significance of the score of the last key frame is recalculated: the comparison of the latest key frame score and the last key frame score after recalculation is used, if the detected vehicle is stopped from the point A to the point C by N123 parking spaces, and A, B, C three points are passed, wherein the vehicle is closest to the point N123 parking spaces at the point B, but the vehicle does not stop when reaching the point B, but continues to travel forward until reaching the point C, when the vehicle reaches the point C, the point B is known to be the key frame closest to the point B only by comparing the key frame score of the point C with the key frame score of the point B, the key frame score of the last frame B needs to be recalculated after the key frame score of the point C is calculated, and although the key frame score of the point B is calculated, the minimum value and the maximum value of the formula (19) change when the vehicle reaches the point C, the minimum value, the maximum value and the L value in the formula (19) change when the key frame score of the point B is recalculated, and the first calculated value is different from the calculated key frame score of the point B, and if the key frame score of the point B is not calculated to be closest to the point B. The reason is that: when the point B key frame score is calculated for the first time, the vehicle is positioned at the point B, and the minimum value, the maximum value and the L value are determined by comparing the point B with the point A, so that the score of the point B can be lower than the score when the vehicle reaches the point C, the score of the point B is recalculated by using the new minimum value, the maximum value and the L value, and because the distance between the point C and the point B is shortest, the score of the point B is higher than the score of the point C by using the new minimum value, the new maximum value and the new L value.

f＝w _b w _l (∑(w _c g _c )+∑(w _n g _n ))....(20)

wherein f represents the observation weight of the key frame, w _b 、w _l Weight coefficient representing the influence of vehicle jolt, illumination g _c 、g _n Represents the observation fraction of the corner point and the parking space number, w _c 、w _n Whether the representative corner and the parking space number are the nearest observations or not is 10 for the nearest observations and 1 for the non-nearest observations;

after other key frames are confirmed, calculating the observation score of the key frames to the parking spaces according to the steps, updating the observation weight of the key frames, and eliminating the key frames when the observation weight is smaller than a threshold value.

the specific expression is as follows:

the above formula (21) is a difference value between the ith parking space coordinate information and the ith parking space coordinate information of the example map in the j-th key frame, and the information is converted into world coordinate system parking space coordinate information, wherein the difference value is a difference value under the world coordinate system, and specifically is: the 1 st item on the right of the equal sign in the formula (21) is world coordinate information of the i-th parking space in the example map, and the initial value of the value is coordinate information of the highest observation score in the observation result of the plurality of reserved key frames on the same parking space in the step eight, which is converted into the world coordinate system; the data in the bracket on the right 2 th item of the equal sign of the formula (21) is that the ith parking space observed by the jth key frame in the vehicle coordinate system is converted into coordinate information in the world coordinate system, and the difference value between the coordinate information of the ith parking space and the coordinate information of the ith parking space in the jth key frame observed by the example map can be obtained by subtracting the 2 nd item from the 1 st item on the right of the equal sign of the formula (21), and the specific steps are as follows:

a. Item 1 to the right of the equal signRepresenting coordinates of two corner points in the map parking space or the midpoint of the parking space number detection frame of the ith example in a world coordinate system, wherein the value range of l is { a, b, c }, respectively represents the two corner points of the parking space and the midpoint of the parking space number detection frame, and w represents the coordinates in the world coordinate system;

c. the 1 st and 3 rd matrices within the brackets on the right of the equal sign represent the changes in rotation and translation of the vehicle, where 1 st is the change in rotation, 3 rd is the change in translation, matrix [ x ] _j y _j θ _j ]Is the j-th key in the sliding windowThe pose of the frame vehicle in the world coordinate system, wherein the sliding window refers to the combination of the keyframe observation information and the keyframe pose information of a plurality of frames of continuous frames of vehicles in a moving state;

Supplementary explanation:

the 2 nd column of the bracket to the right of the equal sign of equation (21) is placed behind the 1 st column rotation matrix to be expressed: the 2 nd column matrix [ x ] _j y _j θ _j ] ^T The pose of the jth key frame vehicle in the world coordinate system is the "seen" pose after the 1 st column rotation change, and the 3 rd column translation matrix of the bracket is placed behind the 2 nd column to express: and translating the pose of the jth key frame vehicle in the world coordinate system in the sliding window after rotation.

supplementary explanation:

the application premise of step three, process 5) is that the vehicle travels through a certain area a and a certain area B of the parking lot, and the physical positions of the two areas are very similar, so that a closed loop can be formed. The physical positions of the region A and the region B are close, the physical positions of the observed parking spaces contained in the region A and the region B are close, the two key frames can be fused, the pose of the two fused key frames is the same, the error is zero, after the error is zero, the error of each key frame is reduced one by pushing back the key frame of the first parking space until the error of the last key frame is 0, so the design premise of the step three, the process 4) of the step three can be repeated after the pose optimization is that the pose optimization of the key frames can be carried out. If the physical position of the parking space of the underground parking garage cannot form a closed loop, but the two end points of a straight line, key frames at one end and one end cannot be fused, and the precondition of pose optimization is not provided, and at the moment, the repeated process 4) of the step three is meaningless.

in the above formula (26), T _ij Representing the relative motion between the i and j th frame key frames, T _i And T _j Representing the pose of the key frames of the ith frame and the jth frame respectively;is T _i Is multiplied by the inverse of the identity matrix;

in the above formula (27), e _ij Representing an error formed by pose transformation between an ith frame and a jth frame key frame;is T _ij An inverse matrix of (a); ln represents the logarithm of the result in brackets; the V symbol represents the transformation from the form of the Liqun to the form of the Lialgebra;

in the above formula (28), F is the total cost function, representing the pose transformation to form the square sum of errors; e, e _ij Representing the error formed by the pose transformation between the ith frame and the jth frame key frame,e is _ij A diagonal transformation matrix;

supplementary explanation:

as shown in fig. 4, the vehicle maintains the pose of 9 key frames, respectively, T1, T2..t9, during driving. Corresponding to equation (26), Then it canThrough T ₁ Inversion to obtain T _j ＝T ₂ By->Then T can be found _ij ＝T ₁₂ . During the running of the vehicle, T can be obtained by the formula (26) ₁₂ 、T ₂₃ ...T ₈₉ . At this time, T can be passed through ₉ Directly calculate the sum T ₁ The relative pose change between the two is caused by T ₁ Can be calculated and T ₂ Pose transformation condition, T ₂ Can be calculated and T ₃ The pose transformation conditions of (2) are synthesized to obtain T ₁ And T is ₃ The position and posture change condition of (2) can obtain T from the step by step ₁ To T ₉ The pose transformation condition between the two. As shown by the broken line in the figure, T ₉₁ Can be obtained by recursion from the relative relation among a plurality of poses, and at the same time, T ₉₁ May also pass through T ₁ And T is ₉ The same parking space N123 is observed to directly calculate the relative pose transformation, and consideration of which pose transformation is more believed to be more accurate is needed at the moment. In contrast, the former involves recursion among a plurality of poses, and accumulated errors are relatively large, so that the former needs to be adjusted by using the latter, and thus T is formed ₉₁ T of (2) ₁₂ 、T ₂₃ ...T ₈₉ Are adjusted to form equation (27). The error function composed of two poses and the relative pose is represented in equation (27) because +.>Is T _ij Is known from equation (26)>The product of the reciprocal matrix is the identity matrix, so +. >Therefore->ln should go to zero after taking the logarithm internally. And then the pose between every two poses between T1 and T2 ₁₂ 、T ₂₃ ...T ₈₉ 、T ₉₁ The sum of the squares of the error functions formed yields the total cost F in equation (28). Formulas (29) and (30) represent the pose of two neighboring keyframes.

The above description is not intended to limit the invention, and it should be noted that: it will be apparent to those skilled in the art that various changes, modifications, additions or substitutions can be made without departing from the spirit and scope of the invention and these modifications and variations are therefore considered to be within the scope of the invention.

Claims

1. The mapping method based on the top view semantic object is characterized by comprising the following steps of:

1) Initializing before completing the drawing construction;

b. matching by using parking space category information to obtain f _t ，f _t Representing whether the currently detected parking spot category based on the new top view is the same as the parking spot category of the example map, f _t Also known as a parking spot category cost;

c. matching by using the similarity of the parking space numbers to obtain f _b ，f _b Representing whether the currently detected parking space number based on the new top view is similar to the parking space number of the example map; f (f) _b Also known as space number similarity cost;

||P _A -P _B ||＜ΔS....(4)

f＝ω _d f _d +ω _t f _t +ω _b f _b +ω _n f _n +ω _r f _r ....(5)

2. The method for creating the top view semantic object according to claim 1, wherein the step 2) is based on image data captured by a plurality of fisheye cameras collected by a vehicle end to generate a top view, and the method is implemented by correcting the top view in real time, and comprises the following steps:

1) Converting the fisheye camera coordinates to top view coordinates;

wherein the rightmost u and v represent coordinates on the fisheye camera by pi ^-1 Bracket transformation of fisheye camera coordinatesFor corrected image coordinates, pass through [ R ] _p t _p ]The inverse transformation of (1) converts the corrected image coordinates into top view coordinates, i.e. leaves the region within the marked range of the corrected image, [ x ] _p y _p ]Representing coordinates in top view;

p＝HP....(2)

A. a fisheye camera external parameter matrix under a flat pavement;

3) Generating a corresponding geometric lookup table;

4) Correcting the top view image information in real time when jolting;

3. The method for mapping semantic objects according to claim 1, wherein the step 3) is based on the real-time corrected top view to obtain the information of the enhanced feature parking space, and the specific process is as follows:

1) Obtaining a top view corrected in real time at each moment;

4. The method for constructing the graph based on the top view semantic object according to claim 1, wherein the step 4) is based on the enhanced feature parking space information, and the initialization of the graph is completed by combining the inertia measurement unit and the wheel speed meter information, and the method is specifically as follows:

1) Initializing and building a diagram: based on the enhanced feature parking space information, combining the inertia measurement unit and the wheel speed meter information to complete initialization map building, wherein the initialization map building is to acquire a key frame of a first observed parking space and build an example map by using the key frame of the first observed parking space; the key frame is a frame for observing the parking space for the first time, and comprises observed parking space information and vehicle pose information for holding the frame;

5. A method of mapping based on top view semantic objects according to claim 1, wherein f _d 、f _t 、f _b 、f _n 、f _r The calculation formula of (2) is as follows:

f _r ＝ω _nl f _nl +ω _mn f _mn +ω _at f _at ....(10)

6. The method for constructing a map based on a top view semantic object according to claim 1, wherein the step three 1) is performed by fusing corner points between an newly added parking space and adjacent parking spaces in the instance map,

the specific process is as follows:

marking the collinear relative relation of corner points between adjacent parking spaces in the process 2) of the third step,

marking the colinear relative relation of the corner points between the parking spaces in the example map according to the relative relation between the parking spaces in the new top view,

the specific process is as follows:

7. The method for constructing the map based on the top view semantic object according to claim 1, wherein the step three, the process 3) is to confirm the key frames with the parking space and the key frames without the parking space, reject the key frames, and optimize the map in the later stage, and the specific process is as follows:

1) Validating key frames

a. Determining key frame according to whether new parking space exists

b. determining key frames based on distance

the formula for inserting key frames according to distance is as follows:

||P _k+1 -P _k ||＞ΔP....(15)

||θ _k+1 -θ _k ||>Δθ....(16)

Wherein θ _k Representing the heading angle, θ, of the vehicle at time k _k+1 Represents the heading angle of the vehicle at the moment k+1, and delta theta represents the set heading angle threshold value ifThe absolute value of the difference between the heading angles at the moment K and the moment K+1 is larger than the heading angle threshold value, and a new key frame is confirmed;

2) Removing key frames:

L _min ＝min(L _min ，L)....(17)

L _max ＝max(L _max ，L)....(18)

wherein L is _min The minimum distance from the midpoint of the parking space corner or the parking space number detection frame in the top view to the center of the top view is the nearest observation; l (L) _max For the furthest observation, namely the maximum distance from the point of the parking space corner or the point of the parking space number detection frame in the top view to the center of the top view, L is the distance from the point of the parking space corner or the point of the parking space number detection frame in the top view to the center of the top view of the currently calculated key frame, and g is the observation score in the current key frame observation weight;

f＝w _b w _l (∑(w _c g _c )+∑(w _n g _n ))....(20)

wherein f represents the observation weight of the key frame, w _b 、w _i Weight coefficient representing the influence of vehicle jolt, illumination g _c 、g _n Represents the observation fraction of the corner point and the parking space number, w _c 、w _n Whether the representative corner and the parking space number are the nearest observations.

8. The method for constructing the map based on the top view semantic object according to claim 1, wherein the step three, step 4), the optimization of the local map is performed by utilizing the key frames with parking spaces, and each parking space position of the adjacent parking spaces in the map and three-point collineation and image scale factors of the adjacent parking spaces are optimized; each parking space position of the adjacent parking spaces comprises a parking space corner point of each parking space and a parking space number detection frame midpoint;

The specific expression is as follows:

the above formula (21) is a difference value between the ith parking space coordinate information and the ith parking space coordinate information of the example map in the j-th key frame, and the information is converted into world coordinate system parking space coordinate information, wherein the difference value is a difference value under the world coordinate system, and specifically is: the 1 st item on the right of the equal sign in the formula (21) is world coordinate information of the i-th parking space in the example map, and the initial value of the value is coordinate information under the world coordinate system, wherein the observation information with highest observation scores in the observation results of the same parking space of the plurality of reserved key frames in the step three process 3) is converted into the coordinate information under the world coordinate system; the data in the bracket on the right 2 th item of the equal sign of the formula (21) is that the ith parking space observed by the jth key frame in the vehicle coordinate system is converted into coordinate information in the world coordinate system, and the difference value between the coordinate information of the ith parking space and the coordinate information of the ith parking space in the jth key frame observed by the example map can be obtained by subtracting the 2 nd item from the 1 st item on the right of the equal sign of the formula (21), and the specific steps are as follows:

to be used forThe upper formula (23) and the formula (24) respectively represent vectors formed by 2 corner points of adjacent parking spaces, and the formula (22) represents the cross multiplication between the two vectors of the formula (23) and the formula (24); if the vector cross multiplication result approaches zero, the three points tend to be collinear; wherein i and i-1 represent the ith and parking spaces and the ith-1 parking space;

the equation (25) above is the total optimization function, the first part on the right of the equal sign is to sum the squares of the position errors observed by the current parking space for the different key frames of the equation (21) and then multiply the squares by Λ _ijl ，Λ _iji For assigning weights, Λ _ijl Is a diagonal matrix in which elements are present only at diagonal lines, the elements of the matrix being observation weights corresponding to the observation information used in formula (21); the second part on the right of the equal sign is to sum squares of three-point collinear errors observed by the current parking space by different key frames of the formula (22) and then multiply by lambda _k ，Λ _k To onlyThe corner lines are provided with elements, the elements of the corner lines are all 1, and weight distribution is carried out on the square of each three-point collineation error.

9. The method for mapping based on top view semantic objects according to claim 1, wherein the third step further comprises a process 5): and (3) carrying out loop detection by utilizing the key frames with the parking space information, optimizing the pose of the key frames in the map by utilizing the key frames with the parking space information and the key frames without the parking space, and optimizing the parking space corner points and the midpoint of the parking space number detection frame in the map again: the loop detection comprises the following steps: searching for a parking space with a similar number or a similar distance to the parking space in the map, matching and associating the parking space in the sliding window with an example parking space in the map by using a KM algorithm, and if a certain amount of matching exists, considering that loop-back occurs, wherein the specific process is as follows: