WO2022002039A1 - Procédé et dispositif de positionnement visuel sur la base d'une carte visuelle - Google Patents

Procédé et dispositif de positionnement visuel sur la base d'une carte visuelle Download PDF

Info

Publication number
WO2022002039A1
WO2022002039A1 PCT/CN2021/103073 CN2021103073W WO2022002039A1 WO 2022002039 A1 WO2022002039 A1 WO 2022002039A1 CN 2021103073 W CN2021103073 W CN 2021103073W WO 2022002039 A1 WO2022002039 A1 WO 2022002039A1
Authority
WO
WIPO (PCT)
Prior art keywords
positioning
current frame
frame
matching
current
Prior art date
Application number
PCT/CN2021/103073
Other languages
English (en)
Chinese (zh)
Inventor
李建禹
易雨亭
龙学雄
Original Assignee
杭州海康机器人技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 杭州海康机器人技术有限公司 filed Critical 杭州海康机器人技术有限公司
Publication of WO2022002039A1 publication Critical patent/WO2022002039A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/20—Instruments for performing navigational calculations

Definitions

  • the present application relates to the field of visual navigation, and in particular, to a visual positioning method and device based on a visual map.
  • Visual navigation is to collect images of the surrounding environment through a camera device, and perform calculations based on the images to complete the position determination and path recognition of the mobile robot.
  • the constructed environment map Taking the map based on the ground texture as an example, when the mobile robot moves through a texture point, according to the feature registration of the current image and the map collected at the texture point, the pose of the current mobile robot can be calculated, and the ground texture-based method can be performed. positioning navigation.
  • the present application provides a visual localization method based on a visual map to improve the robustness of visual localization.
  • a visual positioning method based on a visual map provided by this application is implemented as follows:
  • feature point extraction is performed to obtain the feature points of the current frame
  • the feature points in the current frame are matched with the map points in the map to obtain the matching feature points
  • the pose of the current frame is calculated according to the matching feature points, and the positioning result is obtained.
  • a random sampling consensus algorithm is used to determine the best matching feature point set
  • the positioning The state is determined according to the positioning result of the previous frame and the current number of frames that have failed to locate continuously; the positioning state includes an uninitialized state, a successful positioning state, and a relocation state;
  • the positioning strategy includes initialization positioning in an uninitialized state, normal positioning in a successful positioning state, relocation in a relocation state, and conversion relationships between the positioning states;
  • the conversion relationship between the various positioning states includes:
  • the current positioning state is the uninitialized state
  • the initialization positioning if the initialization positioning is successful, it will switch to the positioning successful state, and if the initialization positioning fails, the current uninitialized state will be maintained;
  • the current positioning status is the positioning successful status
  • the normal positioning if the normal positioning is successful, the current positioning successful status will be maintained, and if the normal positioning fails, it will switch to the positioning lost status;
  • the relocation When the current positioning state is the positioning loss state, if the relocation is successful, it will switch to the positioning success state. If the relocation fails, determine whether the number of consecutive failed positioning frames exceeds the set frame number threshold, or determine whether the current frame is different from the last positioning. Whether the distance between the successful poses exceeds the set first distance threshold, if it exceeds the set number of frames threshold or the set first distance threshold, it will switch to the uninitialized state, if it does not exceed the set number of frames threshold or the set first distance threshold, then keep the current positioning loss state;
  • the determining of the positioning strategy according to the current positioning state includes generating positioning logic according to each positioning state and the conversion relationship between the respective positioning states; wherein, the logical content of the positioning logic includes: executing the current frame indicated by the corresponding positioning strategy The positioning process, the corresponding positioning strategy is the positioning strategy under the current positioning state;
  • Matching the feature points in the current frame with the map points in the map based on the positioning strategy including, based on the positioning logic, according to the positioning process indicated by the positioning strategy in each state, determining the candidate map range in the map for the current frame , to match the feature points in the current frame with the map points in the candidate map range.
  • generating the positioning logic according to each positioning state and the conversion relationship between the respective positioning states including,
  • the current frame will be positioned normally
  • the current frame will be initialized and positioned.
  • the current frame is relocated.
  • initializing and positioning the current frame further comprising,
  • the current frame is the first frame, it is determined that the current positioning state is an uninitialized state,
  • the current frame positioning is recorded successfully, and the status is switched to the positioning success state. If the current frame positioning fails, the current frame positioning failure is recorded, and the current uninitialized state is maintained;
  • the current positioning status is the positioning successful status.
  • the current frame positioning is recorded successfully, and the current positioning success status is maintained; if the current frame positioning fails, the current frame positioning failure is recorded, and the status is switched to the positioning loss state;
  • the current frame is initialized and positioned, further comprising,
  • the current continuous positioning failure frame number exceeds the frame number threshold, or the distance between the current frame and the most recent successful positioning pose exceeds the set first distance threshold, it is determined that the current positioning state is an uninitialized state,
  • the current frame positioning success is recorded, and the status is switched to the positioning success state. If the current frame initialization positioning fails, the current frame positioning failure is recorded, and the current uninitialized state is maintained;
  • the current continuous positioning failure frame number does not exceed the frame number threshold or the distance between the current frame and the most recent successful positioning does not exceed the set first distance threshold, then the current frame is relocated, further comprising,
  • the current positioning state is a positioning loss state.
  • the current frame relocation If the current frame relocation is successful, record the current frame positioning success, and switch to the positioning success state; if the current frame relocation fails, record the current frame positioning failure, keep the current positioning loss state, and return to the judgment whether the current continuous positioning failure frame number is Exceeds the set number of frames threshold, or whether the distance between the current frame and the last successful positioning pose exceeds the set first distance threshold.
  • the candidate map range in the map for the current frame including,
  • map points in the map are used as the first candidate map range, or the auxiliary information is used to obtain the first candidate map range; the map points within the first candidate map range are screened by brute force matching, If the feature matching degree of every two map points exceeds the set second matching threshold, then randomly delete one of the map points to obtain the revised first candidate map range;
  • the pose prediction is performed on the current frame according to the inter-frame motion information from the previous frame to the current frame, and the predicted pose of the current frame is obtained; according to the predicted pose of the current frame, the second frame in the map is determined. candidate map range;
  • the pose prediction is performed on the current frame to obtain the predicted pose of the current frame, including,
  • the first method Obtain the frame pose transformation from the previous frame to the current frame through the wheel odometer or inertial measurement element, and obtain the predicted pose of the current frame based on the inter-frame pose transformation and the positioning result of the previous frame. ;
  • the second method obtain the inter-frame pose transformation from the previous frame to the current frame through the visual odometry, and obtain the predicted pose of the current frame based on the inter-frame pose transformation and the positioning result of the previous frame;
  • the third method According to the historical frame for which the positioning result has been obtained, predict the inter-frame pose transformation from the previous frame to the current frame, and obtain the predicted pose of the current frame based on the inter-frame pose transformation and the positioning result of the previous frame ;
  • the fourth method adopt at least two of the first method, the second method, and the third method to obtain the first predicted pose of the current frame respectively, and obtain at least two first predicted poses;
  • the Kalman filter is used to filter the at least two first predicted poses to obtain a filtered second predicted pose, and the second predicted pose is used as the final predicted pose of the current frame; or, nonlinear optimization is used. method, optimize based on the at least two first predicted poses, obtain an optimized second predicted pose, and use the second predicted pose as the final predicted pose of the current frame;
  • the objective function of nonlinear optimization is the sum of various error terms obtained according to various methods, wherein, using a nonlinear optimization method to optimize based on the at least two first predicted poses, including: The pose, the first predicted pose of the current frame obtained in different ways, and the pose transformation between frames are used as initial values, and are substituted into the objective function of nonlinear optimization to solve the position when the objective function of nonlinear optimization achieves the minimum value. pose as the second predicted pose.
  • determining the second candidate map range in the map according to the predicted pose of the current frame including,
  • the first neighborhood of the center is determined as the second candidate map range
  • the second neighborhood of the center is determined as the third candidate map range
  • the range of the second neighborhood is greater than the range of the first neighborhood.
  • a random sampling consistency algorithm is used to determine the best matching feature point set, including,
  • the current pose is calculated, and the fitted pose estimation of the current matching feature point subset is obtained,
  • For each matching feature point in the matching feature point set according to the spatial position of the projected point of the matching feature point, determine whether the distance between the projected point of the matching feature point in the current frame and the map point matched by the matching feature point in the map is less than If the set second distance threshold is less than the set second distance threshold, then determine that the matching feature point is an inner point; repeatedly perform the described judgment of the projection point of the matching feature point in the current frame and the matching feature point in the map Whether the distance of the matched map points is less than the set second distance threshold, until all matching feature points in the current matching feature point set have been judged as interior points;
  • Described judging whether the best matching feature point set satisfies the conditions for the current frame pose calculation including,
  • At least two or more best matching feature point sets are respectively given weights that measure the matching degree between the current frame and the candidate map range; the weights are based on the number of matching feature points in the best matching feature point set and the number of feature points extracted from the current frame. , the distribution of feature points, one of the initial number of matching feature points and the number of matching feature points in the best matching feature point set or any combination thereof is determined,
  • the set weight threshold and maximum weight it is judged whether the best matching feature point set satisfies the conditions for the current frame pose calculation.
  • the method further includes:
  • the non-linear optimization based on sliding window is used to calculate the pose of the current frame;
  • the optimized variable is the pose of each image frame in the sliding window, the sliding window includes the current frame, and
  • the optimization constraints are the feature points of the current frame and the previous key Inter-frame matching constraints between frame feature points, and/or map matching constraints between current frame feature points and map points in the map,
  • the inter-frame matching error and/or the map matching error are minimized, and the optimized pose of the current frame is obtained as the positioning result;
  • the map matching constraint is: the error between the pixel position of the first matching map point back-projected on the current frame and the pixel position of the first matching feature point matching the first matching map point in the current frame, or, in the current frame
  • the first matching feature point is : the feature point in the current frame is matched with the map point in the map to obtain the successfully matched feature point
  • the first matching map point is: the map point successfully matched by the first matching feature point;
  • the inter-frame matching constraints are: the first matching feature point in the current frame is projected to the spatial position in the world coordinate system and the second matching feature point matching the first matching feature point in the previous key frame of the current frame is projected to the world The error between the spatial positions in the coordinate system, or, the second matching map point matching the second matching feature point is back-projected to the pixel position of the current frame and the second matching map point is back-projected to the pixel of the previous key frame error between positions;
  • the method After the current image is acquired and the current frame is obtained, feature point extraction is performed based on the current frame, and before the feature points of the current frame are obtained, the method includes performing image preprocessing on the current frame.
  • the least squares method is used to minimize the inter-frame matching error and/or the map matching error to obtain the optimized current frame pose, including,
  • the objective function for optimization is: a first result obtained by weighting the sum of map matching errors of all the first matching feature points of all frames in the current sliding window with the first weight, and , the second result obtained by weighting the sum of the inter-frame matching errors of all the second matching map points between all frames between each frame in the current sliding window and its previous key frame with the second weight, the accumulated information about the first the sum of the result and the second result;
  • map matching error initial value Take the map matching error obtained according to the pose of the current frame, the spatial position information of the first matching map point, the camera internal reference, and the pixel coordinates of the first matching feature point matching the first matching map point in the current frame, as the map matching error initial value
  • the iterative solution makes the pose of the current frame when the objective function used for optimization achieves the minimum value, and the optimized current pose is obtained;
  • the method further includes:
  • the current frame is a key frame: the number of the first matching feature points in the current frame is less than the first threshold, and the number of the second matching feature points in the current frame is less than the second threshold;
  • the current frame is deleted in the sliding window. If the current frame is a key frame, it is judged whether the number of frames in the current sliding window reaches the set first frame threshold. If the frame threshold is set, the earliest key frame added in the sliding window will be deleted. If the set first frame threshold is not reached, the earliest key frame added in the sliding window will not be deleted;
  • the performing image preprocessing on the current frame includes,
  • the current frame is de-distorted to obtain a de-distorted image
  • the foreground image is used as the current frame after image preprocessing. If it is not uniform, the foreground image is stretched to obtain the current frame after image preprocessing.
  • performing stretching processing on the foreground image includes:
  • the pixel value of the foreground image is less than or equal to the set minimum gray value, the pixel value of the foreground image is the minimum value within the pixel value range;
  • the pixel value of the foreground image is greater than the minimum gray value and less than the set maximum gray value, the pixel value that is proportional to the maximum value of the pixel is used as the pixel value of the foreground image; the ratio is the pixel value of the foreground image and the minimum value. The ratio of the difference between the gray values to the difference between the maximum gray value and the minimum gray value;
  • the pixel value of the foreground image is greater than or equal to the maximum gray value, the pixel value of the foreground image is the maximum value within the range of pixel values;
  • feature point extraction is performed to obtain the feature points of the current frame, including,
  • the feature points in the grid are arranged in descending order according to the response value of the feature points, the first Q feature points are retained, and the filtered feature points are obtained; among them, Q is based on the feature points in the target image frame.
  • the number and the set upper limit of the total number of feature points, and the total number of grid feature points are determined;
  • Feature descriptors are calculated separately for each feature point after screening.
  • a visual positioning device based on a visual map provided by the application includes:
  • the image acquisition module collects the current image and obtains the current frame
  • the feature extraction module based on the current frame, performs feature point extraction to obtain the feature points of the current frame;
  • the positioning module determines the positioning strategy according to the current positioning state. Based on the positioning strategy, the feature points in the current frame are matched with the map points in the map to obtain matching feature points. When the matching feature points meet the requirements for the current frame pose calculation. When conditions are met, the pose of the current frame is calculated according to the matching feature points, and the positioning result is obtained.
  • the present application also provides a computer-readable storage medium, where a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of any of the above-mentioned visual map-based visual positioning methods are implemented.
  • the present application also provides a mobile robot, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement any of the above-mentioned visual map-based visual positioning methods A step of.
  • the present application also provides a computer program product containing instructions, when the computer program product containing instructions is run on a computer, the computer program product can cause the computer to execute any of the above-mentioned steps of the visual map-based visual positioning method.
  • the present application also provides a computer program, which, when running on a computer, causes the computer to execute the steps of any of the above-mentioned visual map-based visual positioning methods.
  • the visual positioning method based on the visual map provided by the present application performs feature matching and pose calculation of the current frame according to the positioning strategy determined by the current positioning state, and does not need to set accurate calibration points; and, compared with the prior art , because different positioning strategies are used for positioning in different positioning states, so that positioning for various positioning states can be achieved, so it has better robustness to unmapped areas or when positioning fails; at the same time, Compared with the prior art positioning scheme of setting calibration points, only images passing through the calibration points can be positioned more accurately, and the positioning process of this scheme can continue to match with the map without significant jumps in the positioning results.
  • FIG. 1 is a schematic flowchart of visual positioning in this embodiment.
  • Figure 2 is a schematic diagram of feature point screening.
  • FIG. 3 is a schematic diagram of a transition relationship between positioning states.
  • FIG. 4 is a schematic diagram of positioning logic.
  • FIG. 5 is a schematic flow chart of initializing positioning.
  • FIG. 6 is a schematic diagram of map point screening.
  • FIG. 7 is a schematic flow chart of normal positioning.
  • FIG. 8 is a schematic flowchart of a relocation.
  • 9a and 9b are a schematic diagram of determining a candidate map range.
  • FIG. 10 is a schematic diagram of a visual positioning device based on a visual map of the present application.
  • FIG. 11 is a schematic diagram of an image preprocessing module.
  • FIG. 12 is a schematic diagram of map matching constraints and inter-frame matching constraints of the current frame in an image coordinate system.
  • FIG. 13 is a schematic diagram of map matching constraints and inter-frame matching constraints of the current frame in the world coordinate system.
  • the present application determines the positioning logic according to different positioning strategies adopted in different positioning states, so that the positioning state is converted when the positioning logic is satisfied, wherein the positioning strategies are different in the processing of pose prediction and feature matching, so that the overall positioning Robustness is increased.
  • the following description will be based on the visual positioning based on the visual map, wherein the visual map is a feature map that has been constructed in advance, and the map has map points with three-dimensional spatial information, that is, the world coordinates and descriptions of the feature points are stored in the map.
  • Sub-information that is, map points in the map can also be called feature points in the map
  • the descriptor information of any feature point is the information used to describe the image features of the feature point, which is used in the feature matching process, such as Color features, texture features, etc., wherein the descriptor information may also be called feature descriptors or descriptors.
  • the visual map is a texture map constructed based on the collected ground texture information, which can be a two-dimensional or three-dimensional point cloud map with feature descriptors, and the coverage of the texture map can be Continuous coverage can also be discrete coverage.
  • the following takes a 3D point cloud map as an example.
  • FIG. 1 is a schematic flowchart of the visual positioning in this embodiment. After loading the texture map, the mobile robot performs the following steps:
  • Step 101 collecting a current image to obtain a current frame
  • Step 102 Perform image preprocessing on the current frame to make the texture in the image prominent, including but not limited to one or more of optional processes such as image de-distortion, image filtering, and image enhancement.
  • This step 102 is an optional step based on image quality. For example, whether to add image preprocessing is determined according to whether the current frame is over-distorted and whether the texture is significant. It should be noted that if it is determined based on the image quality that the current frame does not need to be added to image preprocessing, after the current frame is collected, the feature points of the current frame can be directly extracted to obtain the feature points of the current frame. The process of extracting the feature points to obtain the feature points of the current frame is the same as the process of extracting the feature points of the preprocessed current frame.
  • Step 1021 perform de-distortion processing on the current frame according to the distortion coefficient of the camera, to obtain a de-distorted image I(u, v), where u and v represent pixel coordinates.
  • Step 1022 determine whether the pixel value of each pixel in the de-distorted image is greater than the set first pixel threshold, if it is greater than the set first pixel threshold, then perform an inversion operation on the pixel whose pixel value is greater than the first pixel threshold , then filter the de-distorted image after the inversion operation, if it is not greater than the set first pixel threshold, directly perform image filtering on the de-distorted image I (u, v) to obtain the background image I b (u , v).
  • the so-called inversion operation refers to: using 255 to subtract the original pixel value as the new image pixel value, the inverted image is compared with the original image, the black and white is reversed, the light and shade are reversed. on the contrary.
  • Step 1023 the image distortion spent subtracting the background image, foreground image to obtain I f (u, v), wherein, the foreground image equation is expressed as:
  • I f (u, v) I (u, v) -I b (u, v)
  • Step 1024 it is determined foreground image I f (u, v) whether or not the pixel value distribution, even if, then the foreground image as the current frame after pretreatment, if not uniform, the foreground image is stretched to obtain pre- the current frame after;
  • whether the pixel values of any image are evenly distributed can determine whether the pixel values of the image are distributed in the entire grayscale range, so as to prevent the contrast from being reduced due to the image being too dark or overexposed.
  • an image grayscale histogram is calculated, the number of pixels in each grayscale interval is counted, and if the pixel grayscale values are concentrated in a larger or smaller grayscale interval, the image is determined.
  • the pixel values of the image are not uniformly distributed; if there are pixel distributions in each grayscale interval, it is determined that the pixel values of the image are uniformly distributed.
  • stretching the foreground image may be:
  • the pixel value of the foreground image is the minimum value within the pixel value range, that is, the pixel value is 0; where the minimum gray value and the maximum gray value can be A preset gray value threshold; or, the minimum gray value may be the minimum value among the gray values of the foreground image pixels, and the maximum gray value may be the maximum value among the gray values of the foreground image pixels.
  • the ratio is: the ratio of the difference between the pixel value of the foreground image and the minimum gray value and the difference between the maximum gray value and the minimum gray value, that is, using the difference between the pixel value of the foreground image and the minimum gray value
  • the difference is divided by the difference between the maximum gray value and the minimum gray value to obtain the ratio.
  • the pixel value of the foreground image is the maximum value within the pixel value range, for example, the maximum value of the pixel is 255.
  • I min is the minimum gray value
  • I max is the maximum gray value.
  • the pixel value ranges from 0 to 255.
  • Step 103 Extract the image feature points in the preprocessed current frame, and calculate the descriptor based on the feature points to obtain the feature points and descriptors of the current frame, that is, extract the feature points in the preprocessed current frame, and based on the feature points
  • the point calculation is based on the descriptor of the feature point, so as to obtain the descriptor of the feature point of the current frame.
  • the descriptor form is the same as the descriptor form of the map point in the map.
  • features such as ORB (Oriented FAST and Rotated BRIEF), Scale Invariant Feature Transform (SIFT), SURF (Speeded Up Robust Features) and the like can be used.
  • ORB Oriented FAST and Rotated BRIEF
  • the feature point of the current frame is also the ORB feature point.
  • ORB is a feature point optimized based on FAST (Features from Accelerated Segment Test) feature point and BRIEF descriptor, which adds the calculation of FAST direction and the anti-rotation invariance of Brief.
  • the image feature points in the preprocessed current frame are extracted according to the manner of forming with the feature points of the constructed map.
  • ORB feature points may be used.
  • the collected ground texture images are usually scale-stable, multi-scale features do not need to be constructed to enhance the scale invariance of features, so the construction of pyramid images can be abandoned, and feature extraction based on the current frame image is equivalent to that based on the source image.
  • the pyramid image is also called an image pyramid, and the so-called image pyramid is a collection of images of different resolutions derived from the same image.
  • the extracted feature points can be screened.
  • Figure 2 is a schematic diagram of feature point screening.
  • the FAST feature is used as an example to introduce the feature point screening process. Specifically: after the FAST (Features from Accelerated Segment Test) feature is extracted, the current frame can be divided into a predetermined number of grids, and each grid is filtered. The Q feature points with the highest FAST response value are retained, and the screened feature points are obtained. Among them, Q is determined according to the number of feature points in one frame of target image, the upper limit of the total number of feature points set, and the total number of frame points in the grid, and the number of feature points retained by different grids is different.
  • FAST Features from Accelerated Segment Test
  • the determination of Q is expressed mathematically as:
  • the FAST response value refers to the score value extracted by the FAST corner point, that is, the score of the point becoming a FAST corner point, and the higher the score, the more significant the FAST feature is.
  • other types of feature points also have similar score values (response values), which are used to measure the significance of the point as a feature point.
  • FAST can be regarded as an algorithm for extracting corner points.
  • FAST corner points are defined as: if a pixel is significantly different from enough pixels in its surrounding neighborhood, the pixel may be a corner point; among them, the corner point is
  • the points with particularly prominent attributes in certain aspects are the isolated points with the largest or smallest intensity on some attributes, the end points of the line segments, etc., which play a very important role in the understanding and analysis of image graphics; While retaining the important features of image graphics, it can effectively reduce the amount of information data, make the information content very high, effectively improve the speed of calculation, and be conducive to the reliable matching of images, making real-time processing possible.
  • Step 104 in view of the fact that in the process of positioning, if the ground texture of the current position has not been mapped, or the ground texture has changed, then the current frame and the map point cannot be properly matched, and then there will be a frame positioning loss. Case. Therefore, in the positioning process, different positioning strategies are used for positioning in the current frame according to different current positioning states. which is:
  • the positioning state includes an uninitialized state, a successful positioning state, and a relocation state
  • the positioning strategies corresponding to each positioning state include: initialized positioning in an uninitialized state , the normal positioning in the successful positioning state, the relocation in the relocation state, and the transition relationship between the various states.
  • the relocation state may also be referred to as a location loss state.
  • FIG. 3 is a schematic diagram of a transition relationship between positioning states. in,
  • the current positioning state is the uninitialized state
  • the initialization positioning if the initialization positioning is successful, it will switch to the positioning successful state, and if the initialization positioning fails, the current uninitialized state will be maintained;
  • the current positioning status is the positioning successful status
  • the normal positioning if the normal positioning is successful, the current positioning successful status will be maintained, and if the normal positioning fails, it will switch to the positioning lost status;
  • the current positioning state is the positioning loss state
  • the relocation if the relocation is successful, it will switch to the positioning successful state. If the relocation fails, it is judged whether the number of frames of continuous positioning failure exceeds the set threshold of the number of frames, or whether the current frame and the last successful positioning are successful. Whether the distance between the poses exceeds the set first distance threshold, if it exceeds the set frame number threshold or the set first distance threshold, it will switch to the uninitialized state, if it does not exceed the set frame number threshold or the set first distance threshold, the current positioning loss state is maintained.
  • the current frame when using the current frame for positioning, if the current positioning state is uninitialized, the current frame will be initialized and positioned. If the initialization and positioning is successful, the current positioning state will be converted to the positioning successful state. If the initialization and positioning fails. , keep the current uninitialized state; if the current positioning state is the positioning successful state, the current frame will be positioned normally; if the normal positioning is successful, the current positioning successful state will be maintained; if the normal positioning fails, the current positioning state will be converted to the positioning lost state; If the current positioning state is the positioning loss state, the current frame will be relocated. If the relocation is successful, it will be converted to the positioning successful state.
  • the relocation fails, it will be judged whether the number of consecutive failed frames exceeds the set frame number threshold. Or, determine whether the distance between the current frame and the last successful positioning pose exceeds the set first distance threshold, and if it exceeds the set frame number threshold or the set first distance threshold, the current positioning state transitions to In the uninitialized state, if the set frame number threshold or the set first distance threshold is not exceeded, the current positioning loss state is maintained.
  • FIG. 4 is a schematic diagram of positioning logic, including,
  • Step 1041 determine whether the current frame is the first frame
  • the current frame is the first frame, it means that there is no positioning result of the previous frame, it is determined that the current positioning state is an uninitialized state, and initialization positioning is performed. Switch to the positioning success state. If the positioning fails, record the current frame positioning failure and keep the current uninitialized state; that is, if the current frame is the first frame, it is determined that the current positioning state is the uninitialized state, and the current frame is uninitialized. The initialization positioning in the state, and based on the positioning result of the current frame, according to the conversion relationship between the positioning states, the current positioning state is updated.
  • step 1042 If it is not the first frame, go to step 1042.
  • Step 1042 determine whether the positioning of the previous frame is successful
  • the positioning of the previous frame it is determined that the current positioning state is the successful positioning state, and normal positioning is performed. If the positioning is successful, that is, the matching in Figure 1 is successful, the current frame positioning is recorded successfully, and the current positioning successful state is maintained. If the positioning fails, Record the positioning failure of the current frame, and switch to the positioning loss state; that is, if the positioning of the previous frame is successful, the current positioning status is determined as the positioning successful state, and the normal positioning in the positioning successful state is performed for the current frame, and based on the current frame The positioning result, according to the conversion relationship between each positioning state, update the current positioning state;
  • step 1043 If the positioning of the previous frame fails, go to step 1043.
  • Step 1043 judging whether the current continuous positioning failure frame number exceeds the frame number threshold, or whether the distance between the current frame and the most recent successful positioning pose exceeds the set first distance threshold;
  • the current positioning state is determined to be an uninitialized state, and initialization positioning is performed.
  • the positioning is successful, that is, the matching in Figure 1 is successful, record the current frame positioning success, and switch to the positioning success state; if the positioning fails, record the current frame positioning failure, and keep the current uninitialized state; that is, determine the current state as uninitialized In the initialization state, the initialization positioning in the uninitialized state is performed on the current frame, and based on the positioning results of the current frame, the current positioning state is updated according to the conversion relationship between the positioning states; if the current number of consecutive failed positioning frames does not exceed the frame number threshold, Or, if the distance between the current frame and the pose with the most recent successful positioning does not exceed the set first distance threshold, determine that the current state is a repositioning state, and execute the repositioning.
  • the positioning is successful, that is, the matching in Figure 1 is successful. Then record the current frame positioning success, and switch to the positioning success state. If the positioning fails, record the current frame positioning failure, keep the current positioning loss state, and return to step 1043; Relocation in the state of location loss is performed, and based on the location result of the current frame, the current location state is updated according to the conversion relationship between each location state.
  • Initial positioning is a positioning strategy performed when the first frame or the number of consecutive failed positioning frames exceeds the threshold. Since there is no positioning result of the previous frame at this time, accurate pose prediction cannot be performed.
  • the first candidate map range is usually obtained by searching map points in the global map or using auxiliary information, and comparing the feature points of the current frame with the map points in the global map or the first candidate map range. Map points are matched. If the matching is successful, the pose of the current frame is calculated according to the matching feature points, so as to obtain the positioning result of the current frame.
  • all map points in the map can also be used as the first candidate map range, or the auxiliary information can be used to obtain the first candidate map range;
  • the points are screened by violent matching. If the feature matching degree of each two map points exceeds the set second matching threshold, one of the map points will be randomly deleted to obtain the revised first candidate map range; then, the current frame The feature points in are matched with the map points within the first candidate map range. If the matching is successful, the pose of the current frame is calculated according to the matching feature points, so as to obtain the positioning result of the current frame.
  • the so-called brute force matching is: pairwise matching, that is, the possibility of each matching is calculated and screened.
  • Normal positioning is a positioning strategy performed when the positioning of the previous frame is successful.
  • the pose prediction of the current frame can be used to predict the pose of the current frame more accurately. Therefore, when performing feature matching, it can be determined according to the first neighborhood of the predicted pose.
  • the feature points of the current frame are matched with the map points in the second candidate map range. If the matching is successful, the pose of the current frame is calculated according to the matching feature points, thereby obtaining the positioning result of the current frame.
  • Relocation is a positioning strategy performed when the previous frame positioning is lost.
  • the last frame that is not lost and the current frame is used as a reference frame, and the pose of the current frame is predicted according to the reference frame.
  • the frame closest to the current frame is used as the reference frame.
  • the pose prediction of the current frame is performed to obtain the predicted pose of the current frame.
  • the third candidate map range is determined according to the second neighborhood of the predicted pose, and the feature points of the current frame and the map points in the third candidate map range are matched by the method of brute force matching. Guaranteed robustness of relocation, of course, is not limited to brute force matching. If the relocation matching is successful, the pose of the current frame is calculated according to the matching feature points to obtain the positioning result of the current frame.
  • the range of the first neighborhood is smaller than the range of the second neighborhood.
  • the pose of the current frame is the pose of the mobile robot that collected the current frame
  • the positioning result of the current frame is the current positioning result of the mobile robot.
  • the pose of the mobile robot includes a position and an attitude.
  • pose prediction and feature matching can be used as: There are two steps in the process of matching the feature points in the frame with the map points in the map to obtain the matching feature points.
  • the initialization positioning only includes the processing flow of feature matching and pose calculation.
  • the feature matching method is different from the feature matching method in normal positioning and relocation.
  • the feature matching is as follows: the feature points in the current frame are compared with the map. It is a link in the process of matching the map points in to obtain the matching feature points.
  • the three positioning methods of initial positioning, normal positioning and repositioning are different matching methods; and when matching When the feature points meet the conditions for the pose calculation of the current frame, in the three positioning modes of initial positioning, normal positioning and repositioning, the pose of the current frame is calculated according to the matching feature points, and the specific implementation process of obtaining the positioning result can be the same.
  • step 105 performing graph optimization on the pose of the current frame that has been successfully positioned, so as to optimize the positioning result and improve the accuracy of the positioning.
  • the so-called graph optimization can also be called pose graph optimization, which is used to optimize the positioning results.
  • the purpose of positioning optimization is to make the output positioning results more accurate and smooth, and improve the accuracy and robustness of the entire system.
  • the positioning optimization adopts a nonlinear optimization based on a sliding window, the optimized variable is the pose of each image frame in the sliding window, the current frame is added to the sliding window, and the optimization constraints are the inter-frame matching constraints, and /or map matching constraints, using the LM method (least square method) to minimize the inter-frame matching error and/or map projection error to obtain the optimized pose of the current frame.
  • the map projection error is also called map matching error, and the error obtained by the map matching constraint is the map matching error, and the error obtained by the inter-frame matching constraint is the inter-frame matching error.
  • the map matching constraint is: the error between the pixel position of the first matching map point back-projected to the current frame and the pixel position of the first matching feature point matching the first matching map point in the current frame, that is, two pixel positions
  • the two pixel positions are respectively: the pixel position of the first matching map point back-projected on the current frame, and the pixel position of the first matching feature point matching the map point in the current frame; or, the current
  • the two spatial positions are: the spatial position where the first matching feature point in the current frame is projected to the world coordinate system, and the first matching map point matched by the first matching feature point is in the world coordinate system.
  • the spatial position of the first matching feature point is: the feature point in the current frame is matched with the map point in the map to obtain a successful matching feature point;
  • the first matching map point is: the first matching feature point is successfully matched the map point;
  • the inter-frame matching constraints are: the first matching feature point in the current frame is projected to the spatial position in the world coordinate system and the second matching feature point matching the first matching feature point in the previous key frame of the current frame is projected to the world
  • the error between the spatial positions in the coordinate system, that is, the error between two spatial positions, the two spatial positions are respectively: the spatial position where the first matching feature point in the current frame is projected to the world coordinate system, and the current In the previous key frame of the frame, the second matching feature point that matches the first matching feature point is projected to the spatial position in the world coordinate system; or, the second matching map point that matches the second matching feature point is back-projected to the current
  • the error between the pixel position of the frame and the pixel position of the second matching map point back-projected to the previous key frame, that is, the error between two pixel positions, the two pixel positions are: and the second matching feature point
  • the matched second matching map point is back-projected to the pixel position of the current frame, and the second matching map point is
  • the previous keyframe is the keyframe that is closest to the collection time of the current frame among the set keyframes.
  • FIG. 12 is a schematic diagram of map matching constraints and inter-frame matching constraints of the current frame.
  • the second matching feature point is a subset of the first matching feature point set
  • the second matching map point matching the second matching feature point is: a subset of the first matching map point set matching with the first matching feature point .
  • the error between the pixel position of the first matching map point back-projected to the current frame and the pixel position of the first matching feature point matched by the first map point constitutes a map matching constraint
  • the second matching map point is back-projected to the current frame.
  • the error between the pixel position and the pixel position of the last key frame of the back-projected current frame of the second matching map point constitutes an inter-frame matching constraint.
  • map matching constraints and the inter-frame matching constraints described in the embodiments of the present application are determined in the image coordinate system, the map matching constraints and the inter-frame matching constraints can also be determined in the world coordinate system.
  • FIG. 13 is a schematic diagram of map matching constraints and inter-frame matching constraints of the current frame in the world coordinate system.
  • the error between the spatial position of the first matching feature point in the current frame projected to the world coordinate system and the spatial position of the first matching map point matching the first matching feature point in the world coordinate system constitutes a map matching constraint , that is, the error between two spatial positions constitutes a map matching constraint
  • the two spatial positions are: the first matching feature point in the current frame is projected to the spatial position in the world coordinate system, and the first matching feature point matches The spatial position of the first matching map point in the world coordinate system; the spatial position of the first matching feature point in the current frame projected to the world coordinate system matches the first matching feature point in the previous key frame of the current frame.
  • the error between the second matching feature point projected to the spatial position in the world coordinate system constitutes the inter-frame matching constraint, that is, the error between the two spatial positions constitutes the inter-frame matching constraint, and the two spatial positions are: the current frame
  • the first matching feature point in is projected to the spatial position in the world coordinate system
  • the second matching feature point matching the first matching feature point in the previous key frame of the current frame is projected to the spatial position in the world coordinate system.
  • the spatial position of the matching feature point projected into the world coordinate system is obtained according to the camera model, according to the camera internal parameters, the pixel position of the matching feature point, and the pose of the frame where the matching feature point is located.
  • the first matching feature point is a feature point in the best matching feature point set.
  • the map matching constraint is: the first matching feature point in the current frame is projected to the spatial position in the world coordinate system and the first matching map point matching the first matching feature point is in the world.
  • the error between the spatial positions in the coordinate system, the inter-frame matching constraints are: the second matching map point matching the second matching feature point is back-projected to the pixel position of the current frame and the second matching map point is back-projected to the previous The error between the pixel positions of the keyframes, correspondingly,
  • the inter-frame matching error and/or map matching error is minimized, and the optimized current frame pose is obtained, including constructing an objective function for optimization.
  • the objective function for optimization is: taking the current sliding window The first result obtained by the sum of the map matching errors of all the first matching feature points of all the frames in the current sliding window, and/or the inter-frame matching of all the second matching map points of all the frames in the current sliding window and its previous key frame
  • the second result obtained by the sum of errors; that is, the objective function used for optimization can be: the first result obtained by the sum of the map matching errors of all the first matching feature points of all frames in the current sliding window , it can also be: the second result obtained by the sum of the inter-frame matching errors of all the frames in the current sliding window and all the second matching map points of the previous key frame, or can also be: the above-mentioned first result and the second result sum of results.
  • the map matching errors of all the first matching feature points in any frame are: The error between the spatial positions of the first matching map points in the point matching in the world coordinate system; and based on the above inter-frame matching constraints, it can be known that any frame is matched with all the second matching map points of the previous key frame.
  • the error is: the error between the back-projection of the second matching map point matching the second matching feature point to the pixel position of the frame and the back-projection of the second matching map point to the pixel position of the previous key frame, where, here
  • the second matching feature point of is the feature point matching the first matching feature point of the frame in the previous key frame of the frame.
  • e ik-map is the map matching error between the pixel position of the first matching map point k back-projected on the current frame i and the pixel position of the first matching feature point k that matches the map point in the current frame i;
  • p ik is the pixel coordinate of the first matching feature point k in the current frame i
  • K is the camera internal parameter matrix,
  • X k is the three-dimensional coordinate of the first matching map point k in the map, R i and t i are obtained through the first matching feature point the pose of the current frame i.
  • e ijm-frame is the back-projection error of the second matching map point m between the current frame i and the previous key frame j, that is, the matching error between frames
  • X m is the second matching feature in the current frame i and the previous key frame j
  • R i, t i is the current pose frame i
  • R j, t j is the pose of the previous key frame j
  • I is the total number of frames in the sliding window
  • K is the total number of first matching feature points in the current frame
  • j is the previous key frame of each frame in the sliding window
  • M is the total number of second matching map points back projected in the current frame .
  • the iterative solution makes the current frame pose when the objective function obtains the minimum value, and the pose is taken as the current positioning result.
  • the objective function for optimization is: weighted with the first weight.
  • the first result obtained by the sum of the map matching errors of all the first matching feature points of all frames in the current sliding window, and the second weighted weighting of all the second key frames between each frame in the current sliding window and its previous key frame
  • the objective function used for optimization is mathematically expressed as:
  • ⁇ 1 and ⁇ 2 are weights.
  • the pose of all frames in the constraint frame can also be used as a variable to be optimized for optimization.
  • the current sliding window is optimized, that is, whether the current frame can be used as a key frame and whether it can exist in the sliding window is identified, so as to further improve the accuracy of positioning, that is, to improve the accuracy of subsequent positioning using image frames.
  • the current frame is determined as a key frame when one of the following conditions is met: the number of first matching feature points in the current frame is less than the first threshold, or the number of second matching feature points in the current frame is less than the second threshold.
  • the current frame is deleted in the sliding window. If the current frame is a non-key frame, the current frame is deleted in the sliding window. If the current frame is a key frame, it is judged whether the number of frames in the current sliding window reaches the set first frame threshold, and if it reaches the first frame threshold , the earliest added keyframe in the sliding window is deleted, and if the first frame threshold is not reached, the earliest added keyframe in the sliding window is not deleted.
  • step 104 Each positioning strategy involved in step 104 will be described below.
  • FIG. 5 is a schematic flowchart of initializing positioning. After image preprocessing and feature point extraction of the current frame, the initial positioning includes,
  • Step 501 Match the feature points of the current frame with the map points in the map.
  • a first matching feature point is obtained, and the obtained first matching feature point constitutes a first matching feature point set; in this way, for the first matching feature point set;
  • the spatial position information of the first matching map point matched with the first matching feature point is determined as the spatial position information of the first matching feature point, so as to obtain the two-dimensional feature point of the current frame and the three-dimensional feature point in the map. matching of feature points.
  • the matching can be performed as follows: for any feature point of the current frame, calculate whether the matching degree between the feature point in the current frame and the map point descriptor in the map is less than the set first matching threshold, if it is less than the set first matching threshold.
  • the determined first matching threshold is determined to match, and if it is not less than the set first matching threshold, it is determined to be mismatched; wherein, the matching degree can be described by the Hamming distance, and the matching threshold is the Hamming distance threshold;
  • the map points in the map can be all map points in the map, and then the brute force matching method is used for matching, that is, the possibility of each matching is calculated when pairwise matching is performed;
  • the auxiliary information is used to obtain the candidate map range, that is, the above-mentioned first candidate map range, and the feature points in the current frame are matched with the map points in the first candidate map range, for example , and set easily identifiable positions as candidate map ranges, such as the starting and ending positions of paths, turning positions, and crossing positions.
  • the auxiliary information can be position information of easily identifiable positions.
  • FIG. 6 is a schematic diagram of map point screening, wherein the dotted circles are map points removed by screening, and the solid circles are candidate map positions.
  • Step 501 is repeated until all feature points of the current frame are matched.
  • Step 502 in view of the fact that there may be some mismatched points when only the descriptors of the feature points are used to match the feature points, preferably, a certain method can be used to screen the best first matching feature points to improve the accuracy of matching the feature points. , thereby improving the accuracy of the pose of the current frame. Therefore, in this step 502, based on the first matching feature point, a random sampling consistency (RANSAC) algorithm can be used to determine the best matching feature point set, which specifically includes:
  • Step 5021 from the first matching feature point set obtained in step 501, randomly select the matching feature points for calculating the pose estimation of the current frame, and obtain the current matching feature point subset;
  • Step 5022 based on the mapping of the spatial position information and the pixel position information established by the matching feature points in the current matching feature point subset, calculate the current pose, thereby obtaining the fitting pose estimation of the matching feature point subset; that is, Calculate the current position based on the mapping, which is a mapping between the spatial position information and the pixel position information established by using the matching feature points in the current matching feature point subset; and, the calculated current pose can be used as the matching feature point subset. Fitting pose estimation;
  • calculating the current pose includes but is not limited to the following methods: perspective N-point positioning PNP (2D-3D) method, 2-dimensional nearest neighbor iterative 2D-ICP (2D-2D), 3-dimensional nearest neighbor iterative 3D-ICP (3D-3D), the homography matrix H(2D-2D).
  • the product of the homography matrix and the spatial position coordinate matrix corresponds to the pixel coordinate matrix, which is expressed mathematically as:
  • the degree of freedom of the homography matrix is 8, and the value of each element in the homography matrix can be obtained through the correspondence between the spatial positions of the four first matching feature points and the pixel positions.
  • singular value (SVD) decomposition By performing singular value (SVD) decomposition on the homography matrix, the corresponding rotation matrix R and translation vector t can be obtained, and the fitted pose estimation can be obtained.
  • Step 5023 since the fitting pose estimation is obtained from the matching feature points in the subset, in order to consider whether other matching feature points in the first matching feature set also conform to the currently calculated fitted pose estimation, it is necessary to calculate the point rate.
  • the spatial positions of all feature points in the current frame are obtained according to the fitted pose estimation and the camera internal parameters, and the spatial positions of the projection points of all the feature points are obtained.
  • two-dimensional pixel coordinate points can be mapped to three-dimensional coordinate points.
  • this mapping is called projection; otherwise, from three-dimensional coordinate points to two-dimensional coordinate points, this mapping is called inverse projection. That is, the spatial position of the feature point refers to the spatial position of the projection point of the feature point.
  • the feature point itself exists in the image coordinate system, and the spatial position of the projection point can be obtained by projection.
  • Projection points exist in the world coordinate system.
  • step 5023 all the first matching feature points in the current frame are projected to three-dimensional spatial positions, that is, projected into a map, as the spatial positions of the projected points. Therefore, for any first matching feature point i in the current frame, its three-dimensional space coordinates can be obtained.
  • p i is the pixel coordinate of the first matching feature point i in the current frame
  • R and t are the current fitting pose estimation
  • X i is the three-dimensional space coordinate of the first matching feature point i projected to the map, that is, the projection point Three-dimensional space coordinates
  • K is the camera internal parameter matrix.
  • Step 5024 for each first matching feature point in the first matching feature point set, determine whether the distance between the projection point of the first matching feature point in the current frame and the map point matched by the first matching feature point in the map is less than If the set second distance threshold is smaller than the set second distance threshold, it is determined that the first matching feature point is an inner point, wherein, in the judgment process, the spatial position of the projection point of the first matching feature point can be to judge.
  • Step 5024 is repeatedly performed until all the first matching feature points are determined whether they are interior points.
  • Step 5025 Count the number of current inliers, and use the ratio of the current number of inliers to the number of first matching feature points as the inlier rate; the larger the ratio, the higher the inlier rate, which means the higher the degree of fitting, the better the fitting pose. The better the estimation, the better the randomly selected matching feature points.
  • Step 5026 determine whether the number of inliers currently counted is the largest in the previous iterations, if it is the largest in the previous iterations, then take the set formed by the current inliers as the current best matching feature point set, and then execute step 5027, if is not the largest in previous iterations, the current best matching feature point set is not updated, but step 5027 is directly executed,
  • Step 5027 determine whether the end condition is reached, if the end condition is met, then execute step 503, if the end condition is not met, return to step 5021, so as to randomly select a subset of matching feature points for fitting pose estimation, so as to carry out the fitting method.
  • the cycle of the pose estimation to the best matching feature point set confirmation that is, the cycle of executing S5021-S5027;
  • the end condition includes at least one of the following conditions:
  • the number of iterations satisfies the preset conditions; in order to make at least one random selection during the iterative loop process under the condition of confidence ⁇ , so that the selected m points are all interior points, which is beneficial to the process of the loop. , the best value of the fitted pose estimate can be obtained at least once. Therefore, the number of iterations ⁇ should satisfy the following conditions:
  • m is the size of the subset, that is, the number of matching feature points in the subset; the confidence level is generally set in the range of 0.95 to 0.99.
  • ⁇ is the interior point rate. In general, ⁇ is usually unknown, so the proportion of interior points under the worst condition can be taken, or the proportion under the worst condition can be set in the initial state, and then with the number of iterations, the Update to the current maximum inlier rate.
  • ⁇ represents the expectation of the number of times that "the subsets are all interior points" in the i rounds.
  • Step 503 judging whether the best matching feature point set satisfies the conditions for the current frame pose calculation
  • the pose calculation of the current frame is performed, and the calculation of the current pose includes but is not limited to the following methods: perspective N-point positioning PNP (2D-3D) method, 2-dimensional nearest neighbor iteration 2D-ICP(2D-2D), 3D nearest neighbor iterative 3D-ICP(3D-3D), homography matrix H(2D-2D);
  • the current frame positioning fails. It should be noted that in the case of not using the best matching feature point set, that is, based on the positioning strategy, the feature points in the current frame are matched with the map points in the map, and after the matching feature points are obtained, you can directly Determine whether the matching feature points meet the conditions for the current frame pose calculation, and when the judgment result is yes, calculate the pose of the current frame according to the matching feature points, and obtain the positioning result, wherein the current frame is calculated according to the matching feature points.
  • the methods of pose include but are not limited to the following methods: perspective N-point positioning PNP (2D-3D) method, 2-dimensional nearest neighbor iterative 2D-ICP (2D-2D), 3-dimensional nearest neighbor iterative 3D-ICP (3D-3D) ), the homography matrix H(2D-2D).
  • one of the embodiments in view of the different pose calculation methods required by different pose calculation conditions are different, taking the homography matrix H(2D-2D) as an example, to solve the pose at least There are 4 matching feature points, therefore, it is judged whether the number of matching feature points in the best matching feature point set satisfies the conditions for pose calculation.
  • each best matching feature point set is given a weight to measure the matching degree between the current frame and the candidate map range; the weight can also be based on the current frame.
  • the set weight threshold and maximum weight determine the unique best matching feature point set, so as to determine that the best matching feature point set meets the conditions for pose calculation; for example, adopt the weight threshold and the unique maximum weight Combined, that is, filter out the best matching feature point set whose weight is greater than the weight threshold, and select the unique best matching feature point set from the best matching feature point set according to the principle that the difference between the largest weight and the next largest weight is the largest.
  • the best matching feature point set determine the unique best matching feature point set, so as to determine that the best matching feature point set meets the conditions for pose calculation; for example, adopt the weight threshold and the unique maximum weight Combined, that is, filter out the best matching feature point set whose weight is greater than the weight threshold, and select the unique best matching feature point set from the best matching feature point set according to the principle that the difference between the largest weight and the next largest weight is the largest.
  • FIG. 7 is a schematic flow chart of normal positioning. After image preprocessing and feature point extraction of the current frame, normal positioning includes,
  • Step 701 in view of the fact that the positioning result of the previous frame has been obtained, according to the inter-frame motion information from the previous frame to the current frame, perform pose prediction on the current frame to obtain the predicted pose, so as to use the predicted pose to determine the second candidate. map range, thereby improving the efficiency of matching;
  • Pose prediction methods include,
  • Embodiment 1 Obtain the inter-frame pose transformation from the previous frame to the current frame through a wheeled odometer or an inertial measurement unit (IMU), and obtain the current frame based on the positioning result of the previous frame and the inter-frame pose transformation. Predict the pose.
  • a wheeled odometer or an inertial measurement unit IMU
  • the inter-frame pose transformation from the previous frame to the current frame is obtained through visual odometry (VO), and the predicted pose of the current frame is obtained based on the positioning result of the previous frame and the inter-frame pose transformation.
  • VO visual odometry
  • This implementation requires only image information, no additional inertial information.
  • Embodiment 3 According to several historical frames for which positioning results have been obtained, predict the inter-frame pose transformation from the previous frame to the current frame, and obtain the predicted position of the current frame based on the positioning results of the previous frame and the inter-frame pose transformation. posture. This implementation does not rely on any information about the current frame.
  • Embodiment 4 at least two of Embodiments 1 to 3 are used to obtain the first predicted poses of the current frame respectively, and at least two first predicted poses are obtained; Kalman filter is used to analyze the at least two first predicted poses. A predicted pose is filtered to obtain a filtered second predicted pose, and the second predicted pose is used as the final predicted pose of the current frame, or, a nonlinear optimization method is used, based on the at least two first predicted poses The pose is optimized to obtain the optimized second predicted pose, and the second predicted pose is used as the final predicted pose of the current frame;
  • the objective function of nonlinear optimization is constructed according to the various error terms obtained in various embodiments, and the mathematical expression is:
  • e ijs represents the error term when the implementation s is adopted
  • T i represents the pose of the previous frame i, that is, the positioning result of the previous frame i
  • T j represents the first predicted pose of the current frame j
  • ⁇ T ij represents the inter-frame position between the previous frame i and the current frame j Pose transformation
  • ⁇ j is the Lie algebra representation of the predicted pose of the current frame
  • S is the total number of implementations adopted.
  • the pose T i of the previous frame i, the first predicted pose T j of the current frame j obtained by different implementations, and the inter-frame pose transformation ⁇ T ij are substituted into the objective function as initial values, and the solution makes the objective function obtain the minimum value. posture at the time.
  • Step 702 according to the predicted pose of the current frame, determine the second candidate map range, match the feature points in the current frame with the map points in the second candidate map range, when the matching is successful, obtain the third matching feature point, Then perform step 703,
  • one embodiment is, as shown in FIG. 9a, taking the map position determined by the predicted pose of the current frame as the center, and determining the first neighborhood of the center as the second candidate map range; matching
  • the method can use brute force matching, that is, violently calculate the descriptor distance between all feature points in the current frame and the feature points in the candidate map range, and select the feature point with the smallest descriptor distance as the matching feature point. This method is suitable for use when the confidence of the predicted pose is low or the number of feature points to be matched is small.
  • the second candidate map range of the feature point is determined, that is, according to the predicted pose of the current frame. and the pixel coordinates of the feature point, calculate the projection point position of the feature point projected into the map, and take the first neighborhood of the projection point as the center as the second candidate map range of the feature point, calculate the current frame feature point and the first
  • the calculation formula used to calculate the position of the projection point projected from the feature point to the map may be the formula about p i shown in the relevant description of step 5023 expression.
  • the first neighborhood may be a map area covered by a set radius with the center as the center.
  • Step 702 is repeated until all feature points of the current frame are matched.
  • Step 703 based on the third matching feature point, adopt the random sampling consistency algorithm to determine the best matching feature point set, this step 703 is the same as step 502,
  • Step 704 determine whether the number of matching feature points in the best matching feature point set satisfies the conditions for the current frame pose calculation, if the conditions for the current frame pose calculation are met, then perform the current frame pose calculation , if the conditions for the current frame pose calculation are not met, it is determined that the current frame positioning fails.
  • This step 703 is the same as step 503 .
  • FIG. 8 is a schematic flowchart of a relocation. After image preprocessing and feature point extraction of the current frame, relocation includes,
  • Step 801 since the positioning of the previous frame fails, according to the historical frame of the successful positioning, trace the latest frame in the historical frame as a reference frame, and perform pose prediction on the current frame according to the inter-frame motion information from the reference frame to the current frame, and obtain: Predicting the pose, so as to use the predicted pose to determine the third candidate map range, thereby improving the efficiency of matching;
  • the pose prediction method is the same as the pose prediction method in step 701, and only the reference frame needs to be regarded as the previous frame.
  • Step 802 according to the predicted pose of the current frame, determine the third candidate map range, match the feature points in the current frame with the map points in the third candidate map range, when the matching is successful, obtain the fourth matching feature point, Then step 803 is executed,
  • an implementation manner is to take the map position determined by the predicted pose of the current frame as the center, and determine the second neighborhood of the center as the third candidate map range; the matching method can be brute force matching. , to improve the robustness of relocation.
  • the third candidate map range of the feature point is determined according to the projection range of the feature point, that is, according to the predicted pose of the current frame and the pixels of the feature point Coordinates, calculate the position of the projection point projected from the feature point of the current frame to the map, the second neighborhood centered on the position of the projection point is used as the third candidate map range of the feature point, and calculate the feature point of the current frame and the third candidate map range In the matching of map points within, the feature point pair with the smallest descriptor distance is selected as the matching feature point.
  • the second neighborhood may be a map range covered by a set radius with the center as the center.
  • Step 802 is repeated until all feature points of the current frame are matched.
  • Step 803 based on the fourth matching feature point, adopt the random sampling consistency algorithm to determine the best matching feature point set, this step 803 is the same as step 502,
  • Step 804 determine whether the number of matching feature points in the best matching feature point set satisfies the conditions for pose calculation, if it meets the conditions for pose calculation, then perform the pose calculation of the current frame, if not The condition used for pose calculation, it is determined that the current frame positioning fails.
  • the LM optimization method is used to solve the problem.
  • the visual positioning method provided in this embodiment does not need to transform the ground for positioning using texture maps, and can use natural textures for positioning, and the cost is low;
  • the positioning logic determined by the positioning strategy enhances the adaptability in the case of local texture changes or lack of local textures. Whether it is a texture map or a non-textured visual map, there is no need to pre-set accurate calibration points, image acquisition is easy, and positioning During the process, it can continue to match with the map without large jumps. In some areas where the map is built, it can be located on any path, not limited to the path of the map, which improves the robustness of the positioning process; different Different positioning strategies in pose prediction and feature matching processing not only improve the accuracy of positioning, but also help to improve the efficiency of positioning.
  • FIG. 10 is a schematic diagram of a visual positioning device based on a visual map of the present application.
  • the device includes,
  • the image acquisition module 1001 collects the current image to obtain the current frame
  • the feature extraction module 100 based on the current frame, performs feature point extraction to obtain the feature points of the current frame;
  • the positioning module 1003 determines a positioning strategy according to the current positioning state; based on the positioning strategy, a candidate map range in the map is determined for the current frame, and the feature points in the current frame are matched with the map points within the candidate range to obtain matching feature points.
  • the matching feature points meet the conditions for solving the pose of the current frame, the pose of the current frame is calculated according to the matching feature points, and the positioning result is obtained.
  • the positioning module 1003 includes:
  • the positioning state sub-module 1004 determines the current positioning state according to the positioning result of the previous frame and the current number of frames with continuous positioning failures,
  • the positioning strategy sub-module 1005 determines a positioning strategy according to the current positioning state, and the positioning strategy includes: initialization positioning in an uninitialized state, normal positioning in a successful positioning state, relocation in a relocation state, and between each positioning state conversion relationship;
  • the positioning logic sub-module 1006 generates positioning logic according to each positioning state and the conversion relationship between each positioning state; wherein, the logical content of the positioning logic includes: performing the positioning process indicated by the corresponding positioning strategy on the current frame, the The corresponding positioning strategy is the positioning strategy in the current positioning state;
  • Matching and positioning sub-module 1007 based on the positioning logic, according to the positioning process indicated by the positioning strategy in each state, determine the candidate map range in the map for the current frame, and match the feature points in the current frame with the map points within the candidate range , obtain the matching feature points, and calculate the pose of the current frame according to the matching feature points;
  • the pose graph optimization sub-module 1008 performs graph optimization on the pose of the current frame.
  • the positioning module 1003 is further configured to, after obtaining the matching feature points, use a random sampling consistency algorithm to determine the best matching feature point set based on the matching feature points, and determine whether the best matching feature point set satisfies the requirements for The conditions for the current frame pose calculation, when the best matching feature point set does not meet the conditions for the current frame pose calculation, it is determined that the current frame positioning fails.
  • the conversion relationship between the various positioning states includes:
  • the current positioning state is the uninitialized state
  • the initialization positioning if the initialization positioning is successful, it will switch to the positioning successful state, and if the initialization positioning fails, the current uninitialized state will be maintained;
  • the current positioning status is the positioning successful status
  • the normal positioning if the normal positioning is successful, the current positioning successful status will be maintained, and if the normal positioning fails, it will switch to the positioning lost status;
  • the relocation When the current positioning state is the positioning loss state, if the relocation is successful, it will switch to the positioning success state. If the relocation fails, determine whether the number of frames of continuous positioning failure exceeds the set threshold of the number of frames, or determine whether the current frame is different from the most recent positioning. Whether the distance between the successful poses exceeds the set first distance threshold, if it exceeds the set number of frames threshold or the set first distance threshold, it will switch to the uninitialized state, if it does not exceed the set number of frames threshold or the set first distance threshold, the current position loss state is maintained.
  • the matching and positioning sub-module 1007 includes:
  • the candidate map determination unit 1009 determines the candidate map range in the map for the current frame according to the positioning process indicated by the positioning strategy in each state,
  • the feature matching unit 1010 matches the feature points in the current frame with the map points in the candidate area,
  • the pose calculation unit 1011 calculates the pose of the current frame according to the matching feature points.
  • the positioning logic sub-module 1006 is specifically used for:
  • the current frame will be positioned normally
  • the current frame will be initialized and positioned.
  • the current frame is relocated.
  • the positioning logic sub-module 1006 is further configured to determine that the current positioning state is an uninitialized state according to the current frame being the first frame before initializing and positioning the current frame if the current frame is the first frame,
  • the current frame positioning success is recorded, and the status is switched to the positioning success state. If the current frame initialization positioning fails, the current frame positioning failure is recorded, and the current uninitialized state is maintained;
  • the positioning logic sub-module 1006 is also used to determine that the current positioning state is a positioning success state according to the successful positioning of the previous frame before performing normal positioning on the current frame if the positioning of the previous frame is successful, and if the current frame is successfully positioned normally, Then record the current frame positioning success, keep the current positioning successful state, if the current frame fails to locate normally, record the current frame positioning failure, and switch to the positioning loss state;
  • the positioning logic sub-module 1006 is also used for:
  • the current continuous positioning failure frame number exceeds the frame number threshold, or the distance between the current frame and the most recent successful positioning pose exceeds the set first distance threshold, it is determined that the current positioning state is an uninitialized state,
  • the current frame positioning success is recorded, and the status is switched to the positioning success state. If the current frame initialization positioning fails, the current frame positioning failure is recorded, and the current uninitialized state is maintained;
  • the positioning logic sub-module 1006 is also used for:
  • the current frame is relocated. If the current frame number of consecutive positioning failures does not exceed the frame number threshold or the distance between the current frame and the last successful positioning pose does not exceed the set first distance threshold, the current frame is relocated.
  • the current positioning state is a positioning loss state
  • the current frame relocation If the current frame relocation is successful, record the current frame positioning success, and switch to the positioning success state; if the current frame relocation fails, record the current frame positioning failure, keep the current positioning loss state, and return to the judgment whether the current continuous positioning failure frame number is Exceeds the set number of frames threshold, or whether the distance between the current frame and the last successful positioning pose exceeds the set first distance threshold.
  • the matching and positioning sub-module 1007 is specifically used for:
  • map points in the map are used as the first candidate map range, or the auxiliary information is used to obtain the first candidate map range; the map points within the first candidate map range are screened by brute force matching, If the feature matching degree of every two map points exceeds the set second matching threshold, then randomly delete one of the map points to obtain the revised first candidate map range;
  • the pose prediction is performed on the current frame according to the inter-frame motion information from the previous frame to the current frame, and the predicted pose of the current frame is obtained; according to the predicted pose of the current frame, the second frame in the map is determined. candidate map range;
  • the matching positioning sub-module 1007 performs pose prediction on the current frame according to the inter-frame motion information from the previous frame to the current frame, and obtains the predicted pose of the current frame, including:
  • the first method Obtain the frame pose transformation from the previous frame to the current frame through the wheel odometer or inertial measurement element, and obtain the predicted pose of the current frame based on the inter-frame pose transformation and the positioning result of the previous frame. ;or
  • the second method obtain the inter-frame pose transformation from the previous frame to the current frame through the visual odometry, and obtain the predicted pose of the current frame based on the inter-frame pose transformation and the positioning result of the previous frame; or
  • the third method According to the historical frame for which the positioning result has been obtained, predict the inter-frame pose transformation from the previous frame to the current frame, and obtain the predicted pose of the current frame based on the inter-frame pose transformation and the positioning result of the previous frame ;or,
  • the fourth method adopt at least two of the first method, the second method, and the third method to obtain the first predicted pose of the current frame respectively, and obtain at least two first predicted poses;
  • the Kalman filter is used to filter the at least two first predicted poses to obtain a filtered second predicted pose, and the second predicted pose is used as the final predicted pose of the current frame; or, nonlinear optimization is used. method, optimize based on the at least two first predicted poses, obtain an optimized second predicted pose, and use the second predicted pose as the final predicted pose of the current frame;
  • the objective function of nonlinear optimization is the sum of various error terms obtained according to various methods, wherein, using a nonlinear optimization method to optimize based on the at least two first predicted poses, including: The pose, the first predicted pose of the current frame obtained in different ways, and the pose transformation between frames are used as initial values, and are substituted into the objective function of nonlinear optimization to solve the position when the objective function of nonlinear optimization achieves the minimum value. pose as the second predicted pose.
  • the matching and positioning sub-module 1007 is specifically used for:
  • the first neighborhood of the center is determined as the second candidate map range
  • the second neighborhood of the center is determined as the third candidate map range
  • the range of the second neighborhood is greater than the range of the first neighborhood.
  • the positioning module 1003 is specifically used for:
  • the current pose is calculated, and the fitted pose estimation of the current matching feature point subset is obtained,
  • For each matching feature point in the matching feature point set according to the spatial position of the projected point of the matching feature point, determine whether the distance between the projected point of the matching feature point in the current frame and the map point matched by the matching feature point in the map is less than If the set second distance threshold is less than the set second distance threshold, then determine that the matching feature point is an inner point; repeatedly perform the described judgment of the projection point of the matching feature point in the current frame and the matching feature point in the map Whether the distance of the matched map points is less than the set second distance threshold, until all matching feature points in the current matching feature point set have been judged as interior points;
  • Described judging whether the best matching feature point set satisfies the conditions for the current frame pose calculation including,
  • At least two or more best matching feature point sets are respectively given weights that measure the matching degree between the current frame and the candidate map range; the weights are based on the number of matching feature points in the best matching feature point set and the number of feature points extracted from the current frame. , the distribution of feature points, one of the initial number of matching feature points and the number of matching feature points in the best matching feature point set or any combination thereof is determined,
  • the set weight threshold and maximum weight it is judged whether the best matching feature point set satisfies the conditions for the current frame pose calculation.
  • the positioning module 1003 is also used to calculate the pose of the current frame according to the matching feature points, and after obtaining the positioning result, it also includes:
  • the non-linear optimization based on sliding window is used to calculate the pose of the current frame;
  • the optimized variable is the pose of each image frame in the sliding window, the sliding window includes the current frame, and
  • the optimization constraints are the feature points of the current frame and the previous key Inter-frame matching constraints between frame feature points, and/or map matching constraints between current frame feature points and map points in the map,
  • the inter-frame matching error and/or the map matching error are minimized, and the optimized pose of the current frame is obtained as the positioning result;
  • the map matching constraint is: the error between the pixel position of the first matching map point back-projected on the current frame and the pixel position of the first matching feature point matching the first matching map point in the current frame, or, in the current frame
  • the first matching feature point is : the feature point in the current frame is matched with the map point in the map to obtain the successfully matched feature point
  • the first matching map point is: the map point successfully matched by the first matching feature point;
  • the inter-frame matching constraints are: the first matching feature point in the current frame is projected to the spatial position in the world coordinate system and the second matching feature point matching the first matching feature point in the previous key frame of the current frame is projected to the world The error between the spatial positions in the coordinate system, or, the second matching map point matching the second matching feature point is back-projected to the pixel position of the current frame and the second matching map point is back-projected to the pixel of the previous key frame error between positions;
  • the image acquisition module 1001 is further configured to collect the current image, and after obtaining the current frame, perform feature point extraction based on the current frame, and perform image preprocessing on the current frame before obtaining the feature points of the current frame.
  • the positioning module 1003 is specifically used for:
  • the objective function for optimization is: a first result obtained by weighting the sum of map matching errors of all the first matching feature points of all frames in the current sliding window with the first weight, and , the second result obtained by weighting the sum of the inter-frame matching errors of all the second matching map points between all frames between each frame in the current sliding window and its previous key frame with the second weight, the accumulated information about the first the sum of the result and the second result;
  • map matching error initial value Take the map matching error obtained according to the pose of the current frame, the spatial position information of the first matching map point, the camera internal reference, and the pixel coordinates of the first matching feature point matching the first matching map point in the current frame, as the map matching error initial value
  • the iterative solution makes the pose of the current frame when the objective function used for optimization achieves the minimum value, and the optimized current pose is obtained;
  • the positioning module is also used to obtain the optimized pose of the current frame
  • the current frame is a key frame: the number of the first matching feature points in the current frame is less than the first threshold, and the number of the second matching feature points in the current frame is less than the second threshold;
  • the current frame is deleted in the sliding window. If the current frame is a key frame, it is judged whether the number of frames in the current sliding window reaches the set first frame threshold, and if it reaches the set first frame threshold If the threshold is one frame, the earliest added keyframe in the sliding window will be deleted. If the set first frame threshold is not reached, the earliest added keyframe in the sliding window will not be deleted;
  • the image preprocessing performed by the image acquisition module 1001 on the current frame includes:
  • the current frame is de-distorted to obtain a de-distorted image
  • the foreground image is used as the current frame after image preprocessing. If it is not uniform, the foreground image is stretched to obtain the current frame after image preprocessing.
  • Stretching the foreground image includes:
  • the pixel value of the foreground image is less than or equal to the set minimum gray value, the pixel value of the foreground image is the minimum value within the pixel value range;
  • the pixel value of the foreground image is greater than the minimum gray value and less than the set maximum gray value, the pixel value that is proportional to the maximum value of the pixel is used as the pixel value of the foreground image; the ratio is the pixel value of the foreground image and the minimum value. The ratio of the difference between the gray values to the difference between the maximum gray value and the minimum gray value;
  • the pixel value of the foreground image is greater than or equal to the maximum gray value, the pixel value of the foreground image is the maximum value within the range of pixel values;
  • feature point extraction is performed to obtain the feature points of the current frame, including,
  • the feature points in the grid are arranged in descending order according to the response value of the feature points, the first Q feature points are retained, and the filtered feature points are obtained; among them, Q is based on the feature points in the target image frame.
  • the number and the set upper limit of the total number of feature points, and the total number of grid feature points are determined;
  • Feature descriptors are calculated separately for each feature point after screening.
  • the visual positioning device further includes an image preprocessing module 1012 for preprocessing the image; preferably, see FIG. 11 , which is a schematic diagram of the image preprocessing module.
  • the image preprocessing module 1012 includes,
  • the image de-distortion sub-module performs de-distortion processing on the source image frame according to the distortion coefficient of the camera to obtain a de-distorted image
  • the image filtering sub-module performs image filtering on the dedistorted image to obtain the background image
  • Image difference sub-module subtract the background image from the distorted image to obtain the foreground image
  • Image stretching sub-module which stretches the foreground image to obtain the target image frame.
  • the image filtering sub-module, the image difference sub-module, and the image stretching sub-module can be used to enhance the image texture.
  • the present application also provides a mobile robot, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the above-mentioned visual map-based visual positioning method.
  • the memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk storage.
  • RAM Random Access Memory
  • NVM non-Volatile Memory
  • the memory may also be at least one storage device located away from the aforementioned processor.
  • the above-mentioned processor can be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; it can also be a digital signal processor (Digital Signal Processing, DSP), dedicated integrated Circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • CPU Central Processing Unit
  • NP Network Processor
  • DSP Digital Signal Processing
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • FPGA Field-Programmable Gate Array
  • Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned visual positioning method based on a visual map are implemented.
  • Embodiments of the present invention also provide a computer program product containing instructions, which, when running on a computer, causes the computer to execute the steps of the above-mentioned visual positioning method based on a visual map.
  • An embodiment of the present invention further provides a computer program, which, when running on a computer, causes the computer to execute the steps of the above-mentioned visual positioning method based on a visual map.
  • a computer program which, when running on a computer, causes the computer to execute the steps of the above-mentioned visual positioning method based on a visual map.

Landscapes

  • Engineering & Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Automation & Control Theory (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne un procédé de positionnement visuel sur la base d'une carte visuelle. Le procédé consiste à : collecter une image courante et obtenir une trame courante (101) ; sur la base de la trame courante, exécuter une extraction de points caractéristiques pour obtenir des points caractéristiques de la trame courante (103) ; et déterminer une stratégie de positionnement conformément à un état de positionnement courant, mettre en correspondance les points caractéristiques de la trame courante avec des points de carte d'une carte sur la base de la stratégie de positionnement pour obtenir des points caractéristiques mis en correspondance, et lorsque les points caractéristiques mis en correspondance satisfont des conditions de calcul de pose de la trame courante, calculer une pose de la trame courante conformément aux points caractéristiques mis en correspondance pour obtenir un résultat de positionnement (104). Selon le procédé, il est inutile de définir un point d'étalonnage ayant une position précise, une robustesse relativement bonne est obtenue par rapport à une zone exempte de construction de carte ou lors d'un échec de positionnement, et une mise en correspondance continue avec la carte peut être effectuée lors du processus de positionnement sans saut important.
PCT/CN2021/103073 2020-06-30 2021-06-29 Procédé et dispositif de positionnement visuel sur la base d'une carte visuelle WO2022002039A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010618519.1 2020-06-30
CN202010618519.1A CN111780763B (zh) 2020-06-30 2020-06-30 一种基于视觉地图的视觉定位方法、装置

Publications (1)

Publication Number Publication Date
WO2022002039A1 true WO2022002039A1 (fr) 2022-01-06

Family

ID=72759967

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/103073 WO2022002039A1 (fr) 2020-06-30 2021-06-29 Procédé et dispositif de positionnement visuel sur la base d'une carte visuelle

Country Status (2)

Country Link
CN (1) CN111780763B (fr)
WO (1) WO2022002039A1 (fr)

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114111803A (zh) * 2022-01-26 2022-03-01 中国人民解放军战略支援部队航天工程大学 一种室内卫星平台的视觉导航方法
CN114577215A (zh) * 2022-03-10 2022-06-03 山东新一代信息产业技术研究院有限公司 一种移动机器人的特征地图更新方法、设备及介质
CN114782495A (zh) * 2022-06-16 2022-07-22 西安中科立德红外科技有限公司 一种多目标跟踪方法、系统及计算机存储介质
CN114898084A (zh) * 2022-04-18 2022-08-12 荣耀终端有限公司 视觉定位方法、设备和存储介质
CN115049847A (zh) * 2022-06-21 2022-09-13 上海大学 一种基于orb描述子的特征点局部邻域特征匹配方法
CN115439536A (zh) * 2022-08-18 2022-12-06 北京百度网讯科技有限公司 视觉地图更新方法、装置及电子设备
CN115451996A (zh) * 2022-08-30 2022-12-09 华南理工大学 一种面向室内环境的单应视觉里程计方法
CN115493612A (zh) * 2022-10-12 2022-12-20 中国第一汽车股份有限公司 一种基于视觉slam的车辆定位方法及装置
CN115752476A (zh) * 2022-11-29 2023-03-07 重庆长安汽车股份有限公司 一种基于语义信息的车辆地库重定位方法、装置、设备和介质
CN116147618A (zh) * 2023-01-17 2023-05-23 中国科学院国家空间科学中心 一种适用动态环境的实时状态感知方法及系统
CN116778141A (zh) * 2023-08-28 2023-09-19 深圳联友科技有限公司 一种基于orb算法的图片快速识别定位的方法
WO2024001849A1 (fr) * 2022-06-28 2024-01-04 中兴通讯股份有限公司 Procédé et appareil de détermination de pose basés sur la localisation visuelle, et dispositif électronique
CN117419690A (zh) * 2023-12-13 2024-01-19 陕西欧卡电子智能科技有限公司 一种无人船的位姿估计方法、装置及介质
CN117710469A (zh) * 2024-02-06 2024-03-15 四川大学 一种基于rgb-d传感器的在线稠密重建方法及系统
WO2024103708A1 (fr) * 2022-11-15 2024-05-23 中兴通讯股份有限公司 Procédé de positionnement, dispositif terminal, serveur et support de stockage

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111524192B (zh) * 2020-04-20 2023-10-03 阿波罗智能技术(北京)有限公司 车载相机外参的标定方法、装置、系统及存储介质
CN111780763B (zh) * 2020-06-30 2022-05-06 杭州海康机器人技术有限公司 一种基于视觉地图的视觉定位方法、装置
CN112509027B (zh) * 2020-11-11 2023-11-21 深圳市优必选科技股份有限公司 重定位方法、机器人及计算机可读存储介质
CN112488007B (zh) * 2020-12-04 2023-10-13 深圳市优必选科技股份有限公司 视觉定位方法、装置、机器人及存储介质
CN113160270A (zh) * 2021-02-24 2021-07-23 广州视源电子科技股份有限公司 视觉地图生成方法、装置、终端及存储介质
CN112950710A (zh) * 2021-02-24 2021-06-11 广州极飞科技股份有限公司 位姿确定方法、装置、电子设备和计算机可读存储介质
CN112950715B (zh) * 2021-03-04 2024-04-30 杭州迅蚁网络科技有限公司 无人机的视觉定位方法、装置、计算机设备和存储介质
CN112990003B (zh) * 2021-03-11 2023-05-19 深圳市无限动力发展有限公司 图像序列重定位判断方法、装置和计算机设备
CN113034595B (zh) * 2021-03-19 2024-06-07 浙江商汤科技开发有限公司 视觉定位的方法和相关装置、设备、存储介质
CN113239072A (zh) * 2021-04-27 2021-08-10 华为技术有限公司 一种终端设备定位方法及其相关设备
CN113252033B (zh) * 2021-06-29 2021-10-15 长沙海格北斗信息技术有限公司 基于多传感器融合的定位方法、定位系统及机器人
CN115267796B (zh) * 2022-08-17 2024-04-09 深圳市普渡科技有限公司 定位方法、装置、机器人和存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106885574A (zh) * 2017-02-15 2017-06-23 北京大学深圳研究生院 一种基于重跟踪策略的单目视觉机器人同步定位与地图构建方法
CN107990899A (zh) * 2017-11-22 2018-05-04 驭势科技（北京）有限公司 一种基于slam的定位方法和系统
US20200166944A1 (en) * 2016-08-29 2020-05-28 Trifo, Inc. Fault-Tolerance to Provide Robust Tracking for Autonomous and Non-Autonomous Positional Awareness
CN111750864A (zh) * 2020-06-30 2020-10-09 杭州海康机器人技术有限公司 一种基于视觉地图的重定位方法和装置
CN111780763A (zh) * 2020-06-30 2020-10-16 杭州海康机器人技术有限公司 一种基于视觉地图的视觉定位方法、装置
CN112541970A (zh) * 2020-11-30 2021-03-23 北京华捷艾米科技有限公司 一种集中式协同SlAM中的重定位方法及装置

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104732518B (zh) * 2015-01-19 2017-09-01 北京工业大学 一种基于智能机器人地面特征的ptam改进方法
US10054445B2 (en) * 2016-05-16 2018-08-21 Northrop Grumman Systems Corporation Vision-aided aerial navigation
CN107390245B (zh) * 2017-07-13 2020-07-10 广东小天才科技有限公司 一种定位方法、装置、设备及存储介质
CN109544636B (zh) * 2018-10-10 2022-03-15 广州大学 一种融合特征点法和直接法的快速单目视觉里程计导航定位方法
CN109633725B (zh) * 2018-10-31 2021-03-30 百度在线网络技术（北京）有限公司 定位初始化的处理方法、装置及可读存储介质
CN111322993B (zh) * 2018-12-13 2022-03-04 杭州海康机器人技术有限公司 一种视觉定位方法和装置
CN110108258B (zh) * 2019-04-09 2021-06-08 南京航空航天大学 一种单目视觉里程计定位方法
CN110361005B (zh) * 2019-06-26 2021-03-26 达闼机器人有限公司 定位方法、定位装置、可读存储介质及电子设备
CN111156984B (zh) * 2019-12-18 2022-12-09 东南大学 一种面向动态场景的单目视觉惯性slam方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200166944A1 (en) * 2016-08-29 2020-05-28 Trifo, Inc. Fault-Tolerance to Provide Robust Tracking for Autonomous and Non-Autonomous Positional Awareness
CN106885574A (zh) * 2017-02-15 2017-06-23 北京大学深圳研究生院 一种基于重跟踪策略的单目视觉机器人同步定位与地图构建方法
CN107990899A (zh) * 2017-11-22 2018-05-04 驭势科技（北京）有限公司 一种基于slam的定位方法和系统
CN111750864A (zh) * 2020-06-30 2020-10-09 杭州海康机器人技术有限公司 一种基于视觉地图的重定位方法和装置
CN111780763A (zh) * 2020-06-30 2020-10-16 杭州海康机器人技术有限公司 一种基于视觉地图的视觉定位方法、装置
CN112541970A (zh) * 2020-11-30 2021-03-23 北京华捷艾米科技有限公司 一种集中式协同SlAM中的重定位方法及装置

Cited By (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114111803A (zh) * 2022-01-26 2022-03-01 中国人民解放军战略支援部队航天工程大学 一种室内卫星平台的视觉导航方法
CN114577215A (zh) * 2022-03-10 2022-06-03 山东新一代信息产业技术研究院有限公司 一种移动机器人的特征地图更新方法、设备及介质
CN114577215B (zh) * 2022-03-10 2023-10-27 山东新一代信息产业技术研究院有限公司 一种移动机器人的特征地图更新方法、设备及介质
CN114898084A (zh) * 2022-04-18 2022-08-12 荣耀终端有限公司 视觉定位方法、设备和存储介质
CN117036663A (zh) * 2022-04-18 2023-11-10 荣耀终端有限公司 视觉定位方法、设备和存储介质
CN114898084B (zh) * 2022-04-18 2023-08-25 荣耀终端有限公司 视觉定位方法、设备和存储介质
CN114782495A (zh) * 2022-06-16 2022-07-22 西安中科立德红外科技有限公司 一种多目标跟踪方法、系统及计算机存储介质
CN115049847A (zh) * 2022-06-21 2022-09-13 上海大学 一种基于orb描述子的特征点局部邻域特征匹配方法
CN115049847B (zh) * 2022-06-21 2024-04-16 上海大学 一种基于orb描述子的特征点局部邻域特征匹配方法
WO2024001849A1 (fr) * 2022-06-28 2024-01-04 中兴通讯股份有限公司 Procédé et appareil de détermination de pose basés sur la localisation visuelle, et dispositif électronique
CN115439536B (zh) * 2022-08-18 2023-09-26 北京百度网讯科技有限公司 视觉地图更新方法、装置及电子设备
CN115439536A (zh) * 2022-08-18 2022-12-06 北京百度网讯科技有限公司 视觉地图更新方法、装置及电子设备
CN115451996A (zh) * 2022-08-30 2022-12-09 华南理工大学 一种面向室内环境的单应视觉里程计方法
CN115451996B (zh) * 2022-08-30 2024-03-29 华南理工大学 一种面向室内环境的单应视觉里程计方法
CN115493612A (zh) * 2022-10-12 2022-12-20 中国第一汽车股份有限公司 一种基于视觉slam的车辆定位方法及装置
WO2024103708A1 (fr) * 2022-11-15 2024-05-23 中兴通讯股份有限公司 Procédé de positionnement, dispositif terminal, serveur et support de stockage
CN115752476A (zh) * 2022-11-29 2023-03-07 重庆长安汽车股份有限公司 一种基于语义信息的车辆地库重定位方法、装置、设备和介质
CN116147618B (zh) * 2023-01-17 2023-10-13 中国科学院国家空间科学中心 一种适用动态环境的实时状态感知方法及系统
CN116147618A (zh) * 2023-01-17 2023-05-23 中国科学院国家空间科学中心 一种适用动态环境的实时状态感知方法及系统
CN116778141A (zh) * 2023-08-28 2023-09-19 深圳联友科技有限公司 一种基于orb算法的图片快速识别定位的方法
CN116778141B (zh) * 2023-08-28 2023-12-22 深圳联友科技有限公司 一种基于orb算法的图片快速识别定位的方法
CN117419690A (zh) * 2023-12-13 2024-01-19 陕西欧卡电子智能科技有限公司 一种无人船的位姿估计方法、装置及介质
CN117419690B (zh) * 2023-12-13 2024-03-12 陕西欧卡电子智能科技有限公司 一种无人船的位姿估计方法、装置及介质
CN117710469A (zh) * 2024-02-06 2024-03-15 四川大学 一种基于rgb-d传感器的在线稠密重建方法及系统
CN117710469B (zh) * 2024-02-06 2024-04-12 四川大学 一种基于rgb-d传感器的在线稠密重建方法及系统

Also Published As

Publication number Publication date
CN111780763B (zh) 2022-05-06
CN111780763A (zh) 2020-10-16

Similar Documents

Publication Publication Date Title
WO2022002039A1 (fr) Procédé et dispositif de positionnement visuel sur la base d'une carte visuelle
WO2022002150A1 (fr) Procédé et dispositif permettant de construire une carte de nuage de points visuels
CN111780764B (zh) 一种基于视觉地图的视觉定位方法、装置
US11798132B2 (en) Image inpainting method and apparatus, computer device, and storage medium
Labbé et al. Cosypose: Consistent multi-view multi-object 6d pose estimation
CN111750864B (zh) 一种基于视觉地图的重定位方法和装置
CN109035304B (zh) 目标跟踪方法、介质、计算设备和装置
Whelan et al. Deformation-based loop closure for large scale dense RGB-D SLAM
JP4964159B2 (ja) ビデオのフレームのシーケンスにおいてオブジェクトを追跡するコンピュータに実装される方法
Touati et al. An energy-based model encoding nonlocal pairwise pixel interactions for multisensor change detection
US8503760B2 (en) System and method for real-time object recognition and pose estimation using in-situ monitoring
CN110097584B (zh) 结合目标检测和语义分割的图像配准方法
Chen et al. A hierarchical model incorporating segmented regions and pixel descriptors for video background subtraction
JP2013508844A (ja) 画像のシーケンス内のオブジェクトのリアルタイム表現のハイブリッド型追跡のための方法、コンピュータプログラム、および装置
CN109544592B (zh) 针对相机移动的运动目标检测算法
JP2010238226A (ja) 物体を追跡するための方法及びシステム
WO2019080743A1 (fr) Procédé et appareil de détection de cible, et dispositif informatique
Deng et al. Plgslam: Progressive neural scene represenation with local to global bundle adjustment
JP2019185787A (ja) 地理的地域内のコンテナのリモート決定
Di et al. A unified framework for piecewise semantic reconstruction in dynamic scenes via exploiting superpixel relations
Zhao et al. Learning probabilistic coordinate fields for robust correspondences
CN113112479A (zh) 基于关键区块提取的渐进式目标检测方法和装置
CN116894876A (zh) 基于实时图像的6-dof的定位方法
CN117036404A (zh) 一种单目热成像同时定位与建图方法和系统
US20220311910A1 (en) Corner detection method and corner detection device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21833195

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21833195

Country of ref document: EP

Kind code of ref document: A1