CN113034584A - Mobile robot visual positioning method based on object semantic road sign - Google Patents

Mobile robot visual positioning method based on object semantic road sign Download PDF

Info

Publication number
CN113034584A
CN113034584A CN202110411557.4A CN202110411557A CN113034584A CN 113034584 A CN113034584 A CN 113034584A CN 202110411557 A CN202110411557 A CN 202110411557A CN 113034584 A CN113034584 A CN 113034584A
Authority
CN
China
Prior art keywords
map
landmark
coordinate system
road sign
graph
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110411557.4A
Other languages
Chinese (zh)
Other versions
CN113034584B (en
Inventor
何力
林旭滨
杨益枘
管贻生
张宏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong University of Technology
Original Assignee
Guangdong University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong University of Technology filed Critical Guangdong University of Technology
Priority to CN202110411557.4A priority Critical patent/CN113034584B/en
Publication of CN113034584A publication Critical patent/CN113034584A/en
Application granted granted Critical
Publication of CN113034584B publication Critical patent/CN113034584B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a mobile robot visual positioning method based on object semantic road signs, which comprises the steps of firstly constructing an off-line object road sign global map, then constructing an on-line object road sign local map, then carrying out mathematical expression on the object road sign local map and the object road sign global map by taking a map structure as a mathematical model, taking a sub-map matching problem as a map matching mathematical equation, realizing registration estimation of the object road sign local map in the object road sign global map by using a sub-map matching solving result, and finally realizing the visual positioning of a mobile robot. The invention carries out the visual positioning of the robot based on the semantic road signs of the object, utilizes the property that the geometric structure of the three-dimensional space has the visual angle invariance, gets rid of the limitation requirement of the positioning system on the motion posture of the robot, improves the robustness of the positioning system on the external environment conditions, and finally improves the autonomy and the flexibility of the mobile robot.

Description

Mobile robot visual positioning method based on object semantic road sign
Technical Field
The invention relates to the technical field of mobile robots, in particular to a mobile robot visual positioning method based on object semantic road signs.
Background
The autonomous navigation is a core technical problem in the technical field of mobile robots, and an autonomous navigation system generally comprises three main modules of positioning, path planning, control and the like, wherein the positioning module is used for inputting information of a real-time control module and a local path planning module and plays a vital role in the autonomous navigation system as an information feedback link. The real-time performance, the accuracy and the robustness of the positioning module are decisive factors for the performance of the autonomous navigation system.
The existing positioning system based on the visual sensor mainly depends on the matching of key points of visual features, namely, the matching and the association of the key points of the features in the current frame image and the key points in the feature map database are carried out. And under the condition of successful matching, estimating the three-dimensional pose of a camera coordinate system corresponding to the current frame image by using the camera projection geometric model by using the three-dimensional coordinates of the key points in the map database, namely equivalently estimating the pose of the sensor moving platform, thereby realizing the positioning function. In the positioning process, visual feature key point matching is a premise of accurate pose estimation, and the matching of the feature key points is mainly based on a description vector extraction algorithm and depends on the local pixel intensity distribution characteristics of an image essentially. Under the condition that texture information and illumination conditions of an external environment are strictly the same, the local pixel intensity distribution characteristics of an image can be influenced by a camera capturing visual angle, so that in the same scene, feature point matching fails due to inconsistency between an observation angle and a map construction process, and further a robot positioning system cannot work reliably.
Disclosure of Invention
Aiming at the problems of insufficient illumination invariance and visual angle invariance of an extraction and matching algorithm of visual feature points in the existing mobile robot global positioning technology based on a visual scheme, the invention provides a mobile robot visual positioning method based on object semantic road signs, aiming at getting rid of the limitation requirement of a positioning system on the motion posture of a robot, improving the robustness of the positioning system on external environment conditions and finally improving the autonomy and flexibility of the mobile robot.
In order to achieve the purpose, the technical scheme provided by the invention is as follows:
a mobile robot visual positioning method based on object semantic road signs comprises the following steps:
s1, constructing an offline object landmark global map;
s2, constructing an online object road sign local map;
s3, performing mathematical expression on the object landmark local map and the object landmark global map by taking the graph structure as a mathematical model, taking a sub-map matching problem as a map matching mathematical equation, realizing registration estimation of the object landmark local map in the object landmark global map by using a sub-map matching solution result, and finally realizing the visual positioning of the mobile robot.
Further, the specific process of constructing the offline object landmark global map in step S1 is as follows:
s1-1, acquiring an input image, and preprocessing the input image;
s1-2, realizing multi-target object detection by using the trained convolutional neural network, finally outputting in the form of a 2D rectangular envelope frame, and outputting the category vectors c of all target objects in the imageiAnd the position x of its envelope frame in the pixel coordinate systemi，yiAnd size information wi，hiI.e. { ci,xi,yi,wi,hi,i∈N}，wiIs height, hiIs the height;
s1-3, performing pixel level registration on the current image and the previous frame image by using an optical flow method, and estimating the motion pose of the camera by minimizing the gray value cost function of the registered pixels between two frames:
Figure BDA0003024340110000021
formula (1)) Wherein (·) the symbolic representation transforms 6-dimensional vectors in lie algebra se (3) into a 4x4 matrix form, i.e.
Figure BDA0003024340110000022
And exp (·) is an exponential mapping to lie algebra SE (3), i.e. exp (ξ ^) epsilon SE (3) is a rigid motion pose of an adjacent frame, and SE (3) is a special Euclidean group. P1And P2For projection matrices of corresponding cameras, Xi,1And Xi,2For the representation of the corresponding three-dimensional points in space in the corresponding camera coordinate system, I1(. and I)2(·) is the pixel gray value of the corresponding pixel point;
s1-4, enveloping the object by using the dual ellipsoid curve as object road sign map expression, firstly carrying out inscribed ellipse fitting on the image object detection frame and obtaining the dual matrix form of ellipse, and regarding the view i, at the camera attitude (R) of the view ii，ti) Dual ellipsoid curved surface Q in three-dimensional space under the condition that the sum camera internal reference matrix K is obtained*Projected contour in this view
Figure BDA0003024340110000031
By projecting a matrix Pi＝K·[Ri ti]Has the following relationship:
Figure BDA0003024340110000032
in the formula (2), scalar quantity
Figure BDA0003024340110000033
Indicating that the equations are equivalent in one dimension of phase difference.
S1-5, in the homogeneous coordinate system, the dual form Q of ellipsoid*Is a symmetric matrix of 4X4, and the dual form C of the ellipse*Is a symmetrical matrix of 3X3, i.e. Q*And C*There are 10 and 6 independent elements, respectively; equation (2) is a quadratic equation with PiPerforming quadratic form vector expression, and recording as BiLinear expression of primitive equationThe composition is as follows:
Figure BDA0003024340110000034
s1-6, for multiple views, simultaneously establishing multiple equations (3) to construct a linear equation set, solving the overconstrained equation to be a linear least squares problem, and solving through SVD to obtain variables
Figure BDA0003024340110000035
Restoring the elliptic matrix into a symmetric matrix to obtain an ellipsoid primal-dual form:
Figure BDA0003024340110000036
and reconstructing an object envelope ellipsoid to form an object landmark global map.
Further, in the step S1-1, the preprocessing process includes: filtering and denoising, converting a gray scale image and scaling the size to adapt to the input dimension of the neural network.
Further, the process of constructing the on-line object landmark local map in step S2 is as follows:
after the visual image is preprocessed, object detection and data association are carried out; then, a position and pose tracking is carried out by using an optical flow method, namely, the function of a visual odometer is realized, and the visual odometer integrates the position and pose optimized at the last moment to eliminate the accumulated error to the maximum extent; and then constructing a local object road sign map by using multiple views, adopting a sliding window type local map scale control strategy, carrying out rigid body transformation on an object road sign envelope surface coordinate system, expressing under a camera coordinate system, and finally forming the scale-controllable object road sign local map related to the camera coordinate system.
Further, the specific process of step S3 is as follows:
matching an object landmark local map and an object landmark global map, wherein the matching problem is a sub-graph matching problem in mathematics, and the object landmark global map is represented as a graph structure G (V, E), wherein V is a node and comprises a three-dimensional coordinate and a label of each landmark i, and E is an edge and represents the mutual three-dimensional relationship between two landmarks; similarly, the local map is represented as a graph structure H ═ V ', E', V 'is a node, and E' is an edge;
the map matching problem is specifically defined as: given a graph G with n nodes (V, E) and a graph H with m nodes (V ', E'), node correspondences are sought in graph G and graph H:
X∈{0,1}n×m
to maximize consistency of graph attributes and structure:
Figure BDA0003024340110000041
wherein v represents i of graph G1Node and i of graph H2A consistency measure of the nodes, the measure derived from the three-dimensional scale consistency of the signposts and the consistency of the semantic tags; d represents i of diagram G1-j1Edge and i of graph H2-j2A consistency measure of the edge, the measure derived from three-dimensional positional distance consistency between different landmarks in the map; matching nodes with the same label in the two maps by utilizing semantic information, reducing the number of optimization iterations, finally obtaining an optimal matching solution, and determining the one-to-one correspondence of the road signs between the two maps;
after the matching of the object landmark local map to the object landmark global map is completed, the pose propagation is carried out by taking the landmark three-dimensional information in the object landmark global map as a reference, namely the transformation T of the global map landmark three-dimensional information-local map landmark three-dimensional information-reference coordinate system of the local map-camera coordinate system in the world coordinate system on the matchingwcE, SE (3), and finally obtaining the pose of the robot at the current moment relative to a map world coordinate system:
Twr＝TwcTcr∈SE(3)
in the formula, TCRE SE (3) is the transformation matrix of the robot with respect to the camera coordinate system, TWCE SE (3) is the transformation moment of the camera coordinate system relative to the world coordinate systemThe matrix is obtained by multiplying the transformation matrix and the transformation matrix according to the chain rule of the transformation matrix and performing matrix operation to obtain the transformation matrix T of the robot relative to the world coordinate systemWRE SE (3), namely the positioning posture information of the robot.
Compared with the prior art, the principle and the advantages of the scheme are as follows:
according to the scheme, an off-line object road sign global map is constructed, an on-line object road sign local map is constructed, then a graph structure is taken as a mathematical model to perform mathematical expression on the object road sign local map and the object road sign global map, a sub-graph matching problem is taken as a map matching mathematical equation, a sub-map matching solving result is used for realizing registration estimation of the object road sign local map in the object road sign global map, and finally visual positioning of the mobile robot is realized. According to the scheme, the robot is visually positioned based on the object semantic road signs, the property that the geometric structure of a three-dimensional space has visual angle invariance is utilized, the limitation requirement of a positioning system on the motion posture of the robot is eliminated, the robustness of the positioning system on external environment conditions is improved, and finally the autonomy and flexibility of the mobile robot are improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the services required for the embodiments or the technical solutions in the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic flow chart of a mobile robot vision positioning method based on semantic object landmarks according to the present invention;
FIG. 2 is a global map of object landmarks based on an ellipsoid envelope surface representation in an embodiment;
fig. 3 is a local map of object signposts under the sliding window strategy in the embodiment.
Detailed Description
The invention will be further illustrated with reference to specific examples:
as shown in fig. 1, the method for visually positioning a mobile robot based on semantic object landmarks in this embodiment includes the following steps:
s1, constructing an offline object landmark global map, wherein the process is as follows:
s1-1, acquiring an input image, and preprocessing the input image, wherein the preprocessing comprises the following steps: filtering and denoising, converting a gray level image and scaling the size to adapt to the input dimension of the neural network;
s1-2, realizing multi-target object detection by using the trained convolutional neural network, finally outputting in the form of a 2D rectangular envelope frame, and outputting the category vectors c of all target objects in the imageiAnd the position x of its envelope frame in the pixel coordinate systemi，yiAnd size information wi，hiI.e. { ci,xi,yi,wi,hi,i∈N}，wiIs height, hiIs the height;
s1-3, performing pixel level registration on the current image and the previous frame image by using an optical flow method, and estimating the motion pose of the camera by minimizing the gray value cost function of the registered pixels between two frames:
Figure BDA0003024340110000061
in equation (1) (. cndot.). Lambda-symbology transforms a 6-dimensional vector in lie algebra se (3) into a 4x4 matrix form, i.e.
Figure BDA0003024340110000062
And exp (·) is an exponential mapping to lie algebra SE (3), i.e. exp (ξ ^) epsilon SE (3) is a rigid motion pose of an adjacent frame, and SE (3) is a special Euclidean group. P1And P2For projection matrices of corresponding cameras, Xi,1And Xi,2For the representation of the corresponding three-dimensional points in space in the corresponding camera coordinate system, I1(. and I)2(·) is the pixel gray value of the corresponding pixel point;
s1-4, enveloping the object by using dual ellipsoid curves as the object road sign mapExpressing that firstly, the image object detection frame is subjected to inscribed ellipse fitting to obtain an elliptic dual matrix form, and for the view i, the view i is in the camera attitude (R)i，ti) Dual ellipsoid curved surface Q in three-dimensional space under the condition that the sum camera internal reference matrix K is obtained*Projected contour in this view
Figure BDA0003024340110000063
By projecting a matrix Pi＝K·[Ri ti]Has the following relationship:
Figure BDA0003024340110000064
in the formula (2), scalar quantity
Figure BDA0003024340110000065
Indicating that the equations are equivalent in one dimension of phase difference.
S1-5, in the homogeneous coordinate system, the dual form Q of ellipsoid*Is a symmetric matrix of 4X4, and the dual form C of the ellipse*Is a symmetrical matrix of 3X3, i.e. Q*And C*There are 10 and 6 independent elements, respectively; equation (2) is a quadratic equation with PiPerforming quadratic form vector expression, and recording as BiThe original equation is linearly expressed as:
Figure BDA0003024340110000071
s1-6, for multiple views, simultaneously establishing multiple equations (3) to construct a linear equation set, solving the overconstrained equation to be a linear least squares problem, and solving through SVD to obtain variables
Figure BDA0003024340110000072
Restoring the elliptic matrix into a symmetric matrix to obtain an ellipsoid primal-dual form:
Figure BDA0003024340110000073
and reconstructing an object envelope ellipsoid to form an object landmark global map, as shown in fig. 2.
S2, constructing an online object road sign local map;
the step is the same as the step S1, and object detection and data association are carried out after the visual image is preprocessed; then, a position and pose tracking is carried out by using an optical flow method, namely, the function of a visual odometer is realized, and the visual odometer integrates the position and pose optimized at the last moment to eliminate the accumulated error to the maximum extent; and then constructing a local object landmark map by using multiple views, performing rigid body transformation on an object landmark envelope surface coordinate system by using a sliding window type local map scale control strategy, expressing the object landmark envelope surface coordinate system in a camera coordinate system, and finally forming the scale-controllable object landmark local map associated with the camera coordinate system, as shown in fig. 3.
And S3, matching the object landmark local map with the object landmark global map, and finally realizing the visual positioning of the mobile robot.
The specific process of the step is as follows:
matching an object landmark local map and an object landmark global map, wherein the matching problem is a sub-graph matching problem in mathematics, and the object landmark global map is represented as a graph structure G (V, E), wherein V is a node and comprises a three-dimensional coordinate and a label of each landmark i, and E is an edge and represents the mutual three-dimensional relationship between two landmarks; similarly, the local map is represented as a graph structure H ═ V ', E', V 'is a node, and E' is an edge;
the map matching problem is specifically defined as: given a graph G with n nodes (V, E) and a graph H with m nodes (V ', E'), node correspondences are sought in graph G and graph H:
X∈{0,1}n×m
to maximize consistency of graph attributes and structure:
Figure BDA0003024340110000081
wherein v represents i of graph G1Node and i of graph H2A consistency measure of the nodes, the measure derived from the three-dimensional scale consistency of the signposts and the consistency of the semantic tags; d represents i of diagram G1-j1Edge and i of graph H2-j2A consistency measure of the edge, the measure derived from three-dimensional positional distance consistency between different landmarks in the map; matching nodes with the same label in the two maps by utilizing semantic information, reducing the number of optimization iterations, finally obtaining an optimal matching solution, and determining the one-to-one correspondence of the road signs between the two maps;
after the matching of the object landmark local map to the object landmark global map is completed, the pose propagation is carried out by taking the landmark three-dimensional information in the object landmark global map as a reference, namely the transformation T of the global map landmark three-dimensional information-local map landmark three-dimensional information-reference coordinate system of the local map-camera coordinate system in the world coordinate system on the matchingwcE, SE (3), and finally obtaining the pose of the robot at the current moment relative to a map world coordinate system:
Twr＝TwcTcr∈SE(3)
in the formula, TCRE SE (3) is the transformation matrix of the robot with respect to the camera coordinate system, TWCE SE (3) is a transformation matrix of the camera coordinate system relative to the world coordinate system, and the transformation matrix T of the robot relative to the world coordinate system can be obtained by multiplying the two according to a transformation matrix chain rule and performing matrix operationWRE SE (3), namely the positioning posture information of the robot.
The embodiment carries out robot visual positioning based on the object semantic road sign, utilizes the property that a geometric structure of a three-dimensional space has visual angle invariance, breaks away from the limitation requirement of a positioning system on the motion posture of the robot, improves the robustness of the positioning system on external environment conditions, and finally improves the autonomy and flexibility of the mobile robot.
The above-mentioned embodiments are merely preferred embodiments of the present invention, and the scope of the present invention is not limited thereto, so that variations based on the shape and principle of the present invention should be covered within the scope of the present invention.

Claims (5)

1. A mobile robot visual positioning method based on object semantic road signs is characterized by comprising the following steps:
s1, constructing an offline object landmark global map;
s2, constructing an online object road sign local map;
s3, performing mathematical expression on the object landmark local map and the object landmark global map by taking the graph structure as a mathematical model, taking a sub-map matching problem as a map matching mathematical equation, realizing registration estimation of the object landmark local map in the object landmark global map by using a sub-map matching solution result, and finally realizing the visual positioning of the mobile robot.
2. The visual positioning method for mobile robot based on semantic object signposts of claim 1, wherein the specific process of constructing the offline global map of object signposts in step S1 is as follows:
s1-1, acquiring an input image, and preprocessing the input image;
s1-2, realizing multi-target object detection by using the trained convolutional neural network, finally outputting in the form of a 2D rectangular envelope frame, and outputting the category vectors c of all target objects in the imageiAnd the position x of its envelope frame in the pixel coordinate systemi，yiAnd size information wi，hiI.e. { ci,xi,yi,wi,hi,i∈N}，wiIs height, hiIs the height;
s1-3, performing pixel level registration on the current image and the previous frame image by using an optical flow method, and estimating the motion pose of the camera by minimizing the gray value cost function of the registered pixels between two frames:
Figure FDA0003024340100000011
in the formula (1), the reaction mixture is,(. SP) symbolic representation transforms 6-dimensional vectors in lie algebra se (3) into a 4x4 matrix form, i.e.
Figure FDA0003024340100000012
Exp (·) is an exponential mapping to lie algebra SE (3), namely exp (ξ ^) epsilon SE (3) is a rigid motion pose of an adjacent frame, and SE (3) is a special Euclidean group; p1And P2For projection matrices of corresponding cameras, Xi,1And Xi,2For the representation of the corresponding three-dimensional points in space in the corresponding camera coordinate system, I1(. and I)2(·) is the pixel gray value of the corresponding pixel point;
s1-4, enveloping the object by using the dual ellipsoid curve as object road sign map expression, firstly carrying out inscribed ellipse fitting on the image object detection frame and obtaining the dual matrix form of ellipse, and regarding the view i, at the camera attitude (R) of the view ii，ti) Dual ellipsoid curved surface Q in three-dimensional space under the condition that the sum camera internal reference matrix K is obtained*Projected contour in this view
Figure FDA0003024340100000026
By projecting a matrix Pi＝K·[Ri ti]Has the following relationship:
Figure FDA0003024340100000021
in the formula (2), scalar quantity
Figure FDA0003024340100000022
Showing that the equations are equivalent in one scale of phase difference;
s1-5, in the homogeneous coordinate system, the dual form Q of ellipsoid*Is a symmetric matrix of 4X4, and the dual form C of the ellipse*Is a symmetrical matrix of 3X3, i.e. Q*And C*There are 10 and 6 independent elements, respectively; equation (2) is a quadratic equation with PiPerforming quadratic form vector expressionIs BiThe original equation is linearly expressed as:
Figure FDA0003024340100000023
s1-6, for multiple views, simultaneously establishing multiple equations (3) to construct a linear equation set, solving the overconstrained equation to be a linear least squares problem, and solving through SVD to obtain variables
Figure FDA0003024340100000024
Restoring the elliptic matrix into a symmetric matrix to obtain an ellipsoid primal-dual form:
Figure FDA0003024340100000025
and reconstructing an object envelope ellipsoid to form an object landmark global map.
3. The method for visually positioning a mobile robot based on semantic landmarks of objects as recited in claim 2, wherein said preprocessing step S1-1 comprises: filtering and denoising, converting a gray scale image and scaling the size to adapt to the input dimension of the neural network.
4. The visual positioning method for mobile robot based on semantic object signposts of claim 1, wherein the step S2 comprises the following steps:
after the visual image is preprocessed, object detection and data association are carried out; then, a position and pose tracking is carried out by using an optical flow method, namely, the function of a visual odometer is realized, and the visual odometer integrates the position and pose optimized at the last moment to eliminate the accumulated error to the maximum extent; and then constructing a local object road sign map by using multiple views, adopting a sliding window type local map scale control strategy, carrying out rigid body transformation on an object road sign envelope surface coordinate system, expressing under a camera coordinate system, and finally forming the scale-controllable object road sign local map related to the camera coordinate system.
5. The visual positioning method for mobile robot based on semantic object road sign of claim 1, wherein the specific process of step S3 is as follows:
matching an object landmark local map and an object landmark global map, wherein the matching problem is a sub-graph matching problem in mathematics, and the object landmark global map is represented as a graph structure G (V, E), wherein V is a node and comprises a three-dimensional coordinate and a label of each landmark i, and E is an edge and represents the mutual three-dimensional relationship between two landmarks; similarly, the local map is represented as a graph structure H ═ V ', E', V 'is a node, and E' is an edge;
the map matching problem is specifically defined as: given a graph G with n nodes (V, E) and a graph H with m nodes (V ', E'), node correspondences are sought in graph G and graph H:
X∈{0,1}n×m
to maximize consistency of graph attributes and structure:
Figure FDA0003024340100000031
wherein v represents i of graph G1Node and i of graph H2A consistency measure of the nodes, the measure derived from the three-dimensional scale consistency of the signposts and the consistency of the semantic tags; d represents i of diagram G1-j1Edge and i of graph H2-j2A consistency measure of the edge, the measure derived from three-dimensional positional distance consistency between different landmarks in the map; matching nodes with the same label in the two maps by utilizing semantic information, reducing the number of optimization iterations, finally obtaining an optimal matching solution, and determining the one-to-one correspondence of the road signs between the two maps;
after the matching of the object landmark local map to the object landmark global map is completed, the pose transmission is carried out by taking the landmark three-dimensional information in the object landmark global map as the referenceBroadcasting, i.e. transformation T of the global map landmark three-dimensional information-local map landmark three-dimensional information-reference coordinate system of local map-camera coordinate system in the world coordinate system on matchwcE, SE (3), and finally obtaining the pose of the robot at the current moment relative to a map world coordinate system:
TWR＝TWCTCR∈SE(3)
in the formula, TCRE SE (3) is the transformation matrix of the robot with respect to the camera coordinate system, TWCE SE (3) is a transformation matrix of the camera coordinate system relative to the world coordinate system, and the transformation matrix T of the robot relative to the world coordinate system can be obtained by multiplying the two according to a transformation matrix chain rule and performing matrix operationWRE SE (3), namely the positioning posture information of the robot.
CN202110411557.4A 2021-04-16 2021-04-16 Mobile robot visual positioning method based on object semantic road sign Active CN113034584B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110411557.4A CN113034584B (en) 2021-04-16 2021-04-16 Mobile robot visual positioning method based on object semantic road sign

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110411557.4A CN113034584B (en) 2021-04-16 2021-04-16 Mobile robot visual positioning method based on object semantic road sign

Publications (2)

Publication Number Publication Date
CN113034584A true CN113034584A (en) 2021-06-25
CN113034584B CN113034584B (en) 2022-08-30

Family

ID=76457968

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110411557.4A Active CN113034584B (en) 2021-04-16 2021-04-16 Mobile robot visual positioning method based on object semantic road sign

Country Status (1)

Country Link
CN (1) CN113034584B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115655262A (en) * 2022-12-26 2023-01-31 广东省科学院智能制造研究所 Deep learning perception-based multi-level semantic map construction method and device
CN115683129A (en) * 2023-01-04 2023-02-03 苏州尚同墨方智能科技有限公司 Long-term repositioning method and device based on high-definition map

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104374395A (en) * 2014-03-31 2015-02-25 南京邮电大学 Graph-based vision SLAM (simultaneous localization and mapping) method
CN106441319A (en) * 2016-09-23 2017-02-22 中国科学院合肥物质科学研究院 System and method for generating lane-level navigation map of unmanned vehicle
CN111462135A (en) * 2020-03-31 2020-07-28 华东理工大学 Semantic mapping method based on visual S L AM and two-dimensional semantic segmentation
CN111780771A (en) * 2020-05-12 2020-10-16 驭势科技（北京）有限公司 Positioning method, positioning device, electronic equipment and computer readable storage medium

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104374395A (en) * 2014-03-31 2015-02-25 南京邮电大学 Graph-based vision SLAM (simultaneous localization and mapping) method
CN106441319A (en) * 2016-09-23 2017-02-22 中国科学院合肥物质科学研究院 System and method for generating lane-level navigation map of unmanned vehicle
CN111462135A (en) * 2020-03-31 2020-07-28 华东理工大学 Semantic mapping method based on visual S L AM and two-dimensional semantic segmentation
CN111780771A (en) * 2020-05-12 2020-10-16 驭势科技（北京）有限公司 Positioning method, positioning device, electronic equipment and computer readable storage medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
LI HE ET.AL: "Local Feature Descriptor for Image Matching: A Survey", 《IEEE ACCESS》 *
丁帅华 等: "基于局部子图匹配的SLAM方法", 《机器人》 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115655262A (en) * 2022-12-26 2023-01-31 广东省科学院智能制造研究所 Deep learning perception-based multi-level semantic map construction method and device
CN115683129A (en) * 2023-01-04 2023-02-03 苏州尚同墨方智能科技有限公司 Long-term repositioning method and device based on high-definition map

Also Published As

Publication number Publication date
CN113034584B (en) 2022-08-30

Similar Documents

Publication Publication Date Title
CN107741234B (en) Off-line map construction and positioning method based on vision
Wan et al. Teaching robots to do object assembly using multi-modal 3d vision
CN110462343A (en) The automated graphics for vehicle based on map mark
CN112258618A (en) Semantic mapping and positioning method based on fusion of prior laser point cloud and depth map
CN113034584B (en) Mobile robot visual positioning method based on object semantic road sign
CN109325979B (en) Robot loop detection method based on deep learning
CN112734852A (en) Robot mapping method and device and computing equipment
CN103192397A (en) Off-line visual programming method and system for robot
CN114332360A (en) Collaborative three-dimensional mapping method and system
CN110260866A (en) A kind of robot localization and barrier-avoiding method of view-based access control model sensor
US6175648B1 (en) Process for producing cartographic data by stereo vision
CN112629520A (en) Robot navigation and positioning method, system, equipment and storage medium
CN115272596A (en) Multi-sensor fusion SLAM method oriented to monotonous texture-free large scene
CN116518984B (en) Vehicle road co-location system and method for underground coal mine auxiliary transportation robot
Zhen et al. LiDAR-enhanced structure-from-motion
CN113570662A (en) System and method for 3D localization of landmarks from real world images
KR20230003803A (en) Automatic calibration through vector matching of the LiDAR coordinate system and the camera coordinate system
Giordano et al. 3D structure identification from image moments
CN111145267A (en) IMU (inertial measurement unit) assistance-based 360-degree panoramic view multi-camera calibration method
CN115239822A (en) Real-time visual identification and positioning method and system for multi-module space of split type flying vehicle
CN111784798B (en) Map generation method and device, electronic equipment and storage medium
CN115496873A (en) Monocular vision-based large-scene lane mapping method and electronic equipment
Madsen et al. A robustness analysis of triangulation-based robot self-positioning
Leishman et al. Robust Motion Estimation with RBG-D Cameras
CN115147344A (en) Three-dimensional detection and tracking method for parts in augmented reality assisted automobile maintenance

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB03 Change of inventor or designer information
CB03 Change of inventor or designer information

Inventor after: Lin Xubin

Inventor after: He Li

Inventor after: Yang Yinen

Inventor after: Guan Yisheng

Inventor after: Zhang Hong

Inventor before: He Li

Inventor before: Lin Xubin

Inventor before: Yang Yinen

Inventor before: Guan Yisheng

Inventor before: Zhang Hong

GR01 Patent grant
GR01 Patent grant