CN109579825B - Robot positioning system and method based on binocular vision and convolutional neural network - Google Patents
Robot positioning system and method based on binocular vision and convolutional neural network Download PDFInfo
- Publication number
- CN109579825B CN109579825B CN201811416964.9A CN201811416964A CN109579825B CN 109579825 B CN109579825 B CN 109579825B CN 201811416964 A CN201811416964 A CN 201811416964A CN 109579825 B CN109579825 B CN 109579825B
- Authority
- CN
- China
- Prior art keywords
- landmark
- mobile robot
- parallax
- neural network
- convolutional neural
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/005—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 with correlation of navigation data from several sources, e.g. map or contour matching
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
- G01C11/04—Interpretation of pictures
- G01C11/30—Interpretation of pictures by triangulation
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
- G01C11/36—Videogrammetry, i.e. electronic processing of video signals from a single source or from different sources to give parallax or range information
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
- G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
- G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
- G01C21/165—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/20—Instruments for performing navigational calculations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
Abstract
The invention discloses a robot positioning system and a positioning method based on binocular vision and a convolutional neural network. The system comprises a mobile robot, wherein the mobile robot is provided with an inertial sensor module, a binocular vision module and an upper computer which are respectively connected with a sensor control module. The motion state of the mobile robot is obtained through the inertial sensor module, the image of the surrounding environment is obtained through the binocular vision module, the convolutional neural network is used for carrying out stereo matching, and the position of the landmark is obtained through calculation. The sensor control module is used for carrying out operation control on the inertial sensor module and the binocular vision module, receiving data of the inertial sensor module and the binocular vision module, and then transmitting the obtained data to the upper computer for data processing to obtain the position of the mobile robot. The proposed method enables efficient self-positioning of a mobile robot.
Description
Technical Field
The invention relates to the field of robot instant positioning and Mapping (SLAM), in particular to a robot positioning system and a positioning method based on binocular vision and a convolutional neural network.
Background
For an autonomous mobile robot, a feasible scheme is provided for realizing autonomous navigation in a real sense by an instant positioning and mapping (SLAM) technology in a complex environment with unknown or no prior map. SLAM techniques can obtain position estimates of themselves in a map of the surrounding environment while the map is being built in real time.
The mobile robot acquires the environmental information through a sensor, wherein the sensor mainly comprises a laser sensor, a vision sensor and the like. The laser sensor has large volume, high energy consumption and high price, and is not suitable for robots with smaller sizes. In addition, the laser sensor has a problem that it is difficult to extract angular or linear features when the sensor is highly confused or crowded, and the sensor has a perceptual drift. The perceived resolution is low and it is difficult to correlate the observed features with known features. The visual sensor can obtain abundant information, and is convenient to load and low in price. The method has good performance in the aspects of extraction and matching of environment characteristic landmarks, representation and management of environment maps and the like, and can well solve the data association problem in SLAM by using vision. Compared with a stereoscopic vision system, monocular vision has the disadvantages that the uncertainty in direction measurement is large, the depth recovery is difficult, and the application occasions are limited to a certain extent because the obtained image information is less.
Binocular vision is a method of passively perceiving a distance using a computer by simulating the principle of human vision. Observing an object from two or more points, acquiring images under different viewing angles, and calculating the offset between pixels according to the matching relation of the pixels between the images by using a triangulation principle to acquire the three-dimensional information of the object. The depth information of the object is obtained, and the actual distance between the object and the camera, the three-dimensional size of the object and the actual distance between two points can be calculated.
Convolutional neural networks are a particular type of artificial neural network that has become one of the most commonly used tools in the field of speech analysis and image recognition today. The convolutional neural network optimizes the network structure by fully utilizing the characteristics of locality and the like contained in data by combining local sensing regions, shared weights, and spatial or temporal down-sampling. The structure of a convolutional neural network is a special multi-layered perceptron that is highly invariant to translation, scaling, tilting, rotation, or other forms of transformations.
The Extended Kalman Filtering (EKF) has the problems of poor real-time performance, low precision, easy interference and the like in the process of simultaneously positioning and mapping (SLAM) a mobile robot.
Disclosure of Invention
The invention aims to provide a robot positioning system and a positioning method based on binocular vision and a convolutional neural network, aiming at the problems and the defects of the prior art. The invention adopts the extended Kalman filtering algorithm to enhance the robustness of robot positioning and improve the positioning precision.
In order to achieve the purpose, the invention adopts the following technical scheme:
the utility model provides a robot positioning system based on binocular vision and convolution neural network, the system includes mobile robot, mobile robot carries with inertial sensor module, binocular vision module and the host computer that are connected with sensor control module respectively. The motion state of the mobile robot is obtained through the inertial sensor module, the image of the surrounding environment is obtained through the binocular vision module, and the position of the landmark is obtained through stereo matching by utilizing the convolutional neural network. The sensor control module is used for carrying out operation control on the inertial sensor module and the binocular vision module, receiving data of the inertial sensor module and the binocular vision module, and then transmitting the obtained data to the upper computer for data processing to obtain the position of the mobile robot. The proposed method enables efficient self-positioning of a mobile robot.
Further, the binocular vision module is composed of a binocular stereo camera, the inertial sensor module is composed of an inertial sensor, the sensor control module is composed of a single chip microcomputer, and the upper computer is a notebook computer.
In order to achieve the above object, the present invention adopts another technical solution as follows.
A positioning method of a robot positioning system based on binocular vision and a convolutional neural network comprises the following steps:
(1) acquiring the motion state of the mobile robot, including acceleration and angular velocity, through an inertial sensor;
(2) acquiring surrounding images through a binocular stereo camera and calculating the position of a landmark according to the surrounding images;
2.1 acquiring binocular images of the surrounding environment by using a binocular camera;
2.2 image correction
The binocular camera is difficult to obtain a binocular vision model completely in a parallel alignment state, at the moment, binocular correction operation is needed, and the image pair obtained by the binocular camera is subjected to re-projection to enable the image pair to meet the parallel alignment state. The image correction corrects two images, which are not in fact in coplanar line alignment, to be in coplanar line alignment.
The binocular correction is to respectively eliminate distortion and line alignment of the left view and the right view according to monocular internal reference data (focal length, imaging origin, distortion coefficient) and binocular relative position relationship (rotation matrix and translation vector) obtained after the cameras are calibrated, so that the imaging origin coordinates of the left view and the right view are consistent, the optical axes of the two cameras are parallel, the left imaging plane and the right imaging plane are coplanar, and the epipolar lines are aligned in a line manner. The left and right views are adjusted to be in an ideal form of complete parallel alignment, so that any point on one image and the corresponding point on the other image have the same line number, and the corresponding point can be matched only by one-dimensional search on the line.
Binocular correction is typically achieved using the standard binocular correction algorithm, the Bouguet algorithm.
The principle of the Bouguet algorithm is mainly to maximize the observation area when the reprojection distortion is minimized. Under the condition of obtaining a relative relation matrix of the two cameras, namely a rotation matrix R and a translation vector T, the left camera and the right camera are respectively rotated by half, namely the rotation matrix R is decomposed into R l And r r Thus, the optical axes of the two cameras can point to the vector and the direction of the original main optical axis in parallel. Figure rotated at this timeThe image planes are coplanar and not aligned. To achieve row alignment, i.e. pole to infinity and pole horizontal alignment, a transformation matrix R needs to be constructed rect Setting:
wherein R is rect Is a transformation matrix, e 1 、e 2 、e 3 Three vectors are formed.
Due to the matrix R rect Transform the pole of the left camera to infinity and epipolar horizontal, so vector e 1 The direction is the direction of the plane between the two camera principal points:
wherein the content of the first and second substances,
vector e 2 Only the sum vector e 1 Orthogonal is sufficient.
The direction orthogonal to the main optical axis is generally selected by calculating the vector e 1 And the main optical axis direction (0,0,1), normalized to:
wherein, T x 、T y 、T z Obtained by matrix T decomposition.
Third vector e 3 And vector e 1 And e 2 Quadrature, by cross product:
e 3 ＝e 1 ×e 2
the correction matrix of the binocular camera can be obtained through the rotation matrix and the transformation matrix of the left camera and the right camera, and an ideal parallel optical axis binocular vision model can be obtained.
Wherein R is rect For transforming the matrix, r l ,r r Is a synthetic rotation matrix, R l ,R r Is an overall rotation matrix.
2.3, carrying out stereo matching by using a convolutional neural network;
2.3.1 calculating each parallax matching cost for the binocular image by using a convolutional neural network;
the essence of stereo matching is to compare the similarity of two pixel points in a binocular image, and the matching cost is the mathematical expression of the similarity. The lower the matching cost between two pixel points is, the more similar the two pixel points are, and the higher the matching degree is. The method comprises the steps of utilizing a block-based image matching method, using a convolutional neural network to find out accurate corresponding relations among blocks, dividing an image into a plurality of blocks, comparing the similarity of the blocks to try to find out corresponding pixel points, and then scoring the similarity of different degrees according to the corresponding relations of different blocks. The negative value of the similarity is defined as the cost of stereo matching for subsequent cost aggregation and disparity calculation. The specific steps of binocular stereo matching cost calculation based on the convolutional neural network method can be summarized as follows:
1) extracting different feature information of the left image block and the right image block by using 4 layers of convolutional neural network layers respectively;
2) and connecting the left and right characteristic information, and classifying and judging the characteristics by utilizing a 4-layer full-connection layer network. Wherein the loss function uses a cross-entropy cost function:
tlog(s)+(1-t)log(1-s)
wherein s is the output of the similarity comparison network, t is a sample mark, and t is 1 when the input is a positive sample and t is 0 when the input is a negative sample;
3) outputting judgment results of similarity and dissimilarity in a similarity scoring mode for subsequent stereo matching;
4) and constructing a cost function by utilizing the inverse proportion of the similarity score to obtain the matching cost.
2.3.2 cost polymerization
The optimal parallax is selected at the matching cost of one pixel point, and the method is easily influenced by noise in the image, so that the stability of the algorithm is poor. In the local matching algorithm, the matching cost of all pixel points in the surrounding neighborhood (aggregation window) is taken as the center to be matched, and the overlapping statistics is carried out. For a parallax to be selected, the aggregation window is a two-dimensional plane; and for all parallaxes to be selected in the parallax search range, the aggregation window is a three-dimensional space.
The aggregation of the matching costs is equivalent to the convolution of an aggregation window with a to-be-selected disparity plane in the disparity space, as shown in the following formula.
In the formula, C A (p, d) represents the aggregate cost of the reference point p at the disparity d to be selected, C 0 (p, d) represents the original matching cost of the reference point p at the disparity d to be selected, and w (p, d) represents the aggregated window weight of the point p at the disparity d plane. The simplest window weight selection method is to take an average value, and assuming that the aggregation window is a square window with a fixed size of 5 × 5, the window weight is set to 1/25.
2.3.3 parallax calculation
In the parallax calculation, the local matching algorithm and the global matching algorithm have a large difference. The local matching algorithm focuses on two steps of matching cost calculation and cost aggregation, and the final disparity value is easy to establish, and a Winner Take All strategy (WTA) is generally adopted. The WTA strategy is to select a disparity d with the optimal matching cost in all the search disparity ranges as the final disparity of the point p to be matched.
d p ＝arg minC(p,d),d∈D
In the formula, d p And D represents a parallax searching range.
Of these four steps, the disparity calculation is the key step of the global matching algorithm. Most global matching algorithms establish an energy function first, and then optimize the energy function by continuously adjusting the parallax value to obtain the final parallax.
E(d)＝E data (d)+λE smooth (d)
In the formula, E data (d) The data item indicates how similar two pixels are when the disparity is d. E smooth (d) For smoothing terms, used to encode smoothing hypotheses in a global matching algorithm, usually E smooth (d) Only the difference between the disparities of the neighboring pixels is measured.
2.3.4 parallax refinement to obtain the final parallax value
The local matching algorithm and the global matching algorithm directly obtain integer parallax values distributed in a search parallax range, and can meet the requirement of occasions with low precision requirements. However, for applications requiring precision, such parallax often does not work well. On the basis of the initial parallax, the parallax can be further refined by adopting mathematical methods such as sub-pixel enhancement and the like to obtain the parallax value at the sub-pixel level. Meanwhile, mismatching points in the disparity map are removed by using methods such as left and right consistency check, median filtering and the like, and a more accurate disparity value is obtained.
And 2.4 obtaining the position of the landmark according to the parallax value by utilizing the triangulation principle.
Let space point P (X) c ,Y c ,Z c ) The image coordinates projected in the left and right cameras are p l (u l ,v l ) And p r (u r ,v r ) Since the cameras are set in a parallel alignment, v is equal to v l ＝v r 。ΔPO l O r And Δ Pp l p r Similarly, the following relationship is obtained:
simplifying to obtain:
wherein, B is the base length, namely the distance between the optical centers of the two cameras; f is the focal length; u ═ d l u r I.e. parallax. The conversion relationship between the camera coordinates and the image coordinates is as follows:
the conversion coordinate relation of the vision model with the parallel optical axis can be obtained as follows:
(3) and carrying out augmentation extended Kalman filtering processing according to the acquired data.
3.1 prediction Process
And predicting the state vector and covariance of the mobile robot at the current moment through the pose vector and covariance of the mobile robot at the previous moment.
3.2 Observation Process
And observing to obtain different landmark positions, and updating the pose and the system error correction parameters of the mobile robot by using Kalman gain and an innovation matrix.
3.3 data Association
Judging an observation result, and entering a state augmentation stage if the observed landmark is a new landmark; if the observed landmark is a known landmark, the method proceeds to step.
3.4 State augmentation Process
When the mobile robot observes a new landmark, after verification, the position state of the newly observed landmark needs to be added into the system state vector.
3.5 update phase
And updating the new system state vector and the covariance matrix by applying an augmentation model.
(4) And updating the map, and drawing and updating the position of the mobile robot according to the result of the augmented extended Kalman filtering.
The invention has the following advantages:
1. the invention uses the binocular camera to collect the image, and the accuracy of the image acquisition is high
2. And the stereo matching is carried out by using a convolution neural network method, so that an accurate disparity map can be obtained, a more accurate landmark position is obtained, and the position estimation accuracy of the mobile robot is improved.
3. The invention positions the mobile robot by using the method of the extended Kalman filtering, thereby improving the positioning precision.
Drawings
FIG. 1 is a system diagram of a robot positioning system based on binocular vision and a convolutional neural network;
FIG. 2 is a flow chart of a method for a robot positioning system based on binocular vision and convolutional neural networks;
FIG. 3 is a flow chart of stereo matching;
FIG. 4 is a flowchart of a landmark acquisition portion;
FIG. 5 is a partial flow diagram of an augmented extended Kalman filter.
Detailed Description
The invention is described in detail below with reference to the figures and examples.
Fig. 1 is a schematic system diagram of a robot positioning system based on binocular vision and a convolutional neural network. The utility model provides a robot positioning system based on binocular vision and convolution neural network includes mobile robot, mobile robot carries with inertial sensor module, binocular vision module and the host computer that is connected with sensor control module respectively.
The inertial sensor acquires the motion state of the mobile robot, the adopted inertial sensor can be an MPU6050 or an MPU6500, and the inertial sensor module and the sensor control module are connected through a USB data line. The three-axis gyroscope and the three-axis accelerometer are integrated in the inertial sensor, and when the robot carrying the inertial sensor module moves, the chip can detect and output the angular velocity and the acceleration in real time according to the detection axis and the direction specified by the inertial sensor. Binocular vision module comprises binocular camera, can be ZED binocular camera or bumblebee2 binocular camera, and binocular camera passes through the USB data line and the controller module is connected.
The sensor control module adopts STM32 as a core processor, and is connected with a computer through an RS232 serial port and carries out real-time communication. The sensor control module is used for carrying out operation control on the inertial sensor module and the binocular vision module, receiving data of the inertial sensor module and the binocular vision module, and then transmitting the obtained data to the upper computer for data processing to obtain the position of the mobile robot.
The upper computer is a notebook computer, and the position of the mobile robot can be drawn and displayed according to the result of the extended Kalman filtering. The mobile robot is a small-sized turtbelto mobile robot, is small in size and high in flexibility, and can be loaded with an inertial sensor module, a binocular vision module, a sensor control module and a notebook computer.
The invention relates to a positioning method of a robot positioning system based on binocular vision and a convolutional neural network, which is characterized in that reference is made to a flow chart of the positioning method of the robot positioning system based on the binocular vision and the convolutional neural network in figure 2; fig. 3 is a perspective matching flowchart; fig. 4 is a flowchart of a landmark acquisition part; referring to fig. 5, a partial flowchart of the extended kalman filter is shown, which includes the following steps:
(1) prediction
The dynamic equation of the mobile robot is as follows:
in the formula: f (-) is a dynamic model equation of the mobile robot, dt is a control time step length, and L is a wheel base distance of a front axle and a rear axle. At the moment of k-1, the pose vector of the mobile robot
The control vector of the mobile robot from k-1 to k is
As the velocity correction parameter, γ is the heading angle.
Then, at time k, the state space and covariance predictions are:
in the formula
Is a pose vector of the mobile robot, and f (-) is a dynamic model equation of the mobile robot
In the formula: v ^ f x ，▽f u Pose vector for mobile robot motion model relative to mobile robot
And the Jacobian (Jacobian) determinant of the control vector u (k); q k The process noise is assumed to be gaussian noise with a mean of zero as the error covariance of the process noise.
(2) Observation of
Assuming that the signpost is stationary in the environment,
is the ith landmark position observed by the mobile robot, and the distance and angle between the inertial sensor return sensor and the landmark are loaded by the mobile robot, the observation model can be expressed as:
S(k)＝▽h·P - (K)·(▽h) T +R k
in the above formula, S (k) is an innovation matrix, and ^ h is an observation model about a state space
Jacobian's equation of (P) - (k) Is a covariance matrix, which is now processed as follows:
S new (k)＝0.5·(S(k)+S(k) T )
S Chol (k)＝chol(S new (k))
above formula S new (k) Symmetrizing the information matrix, formula S Chol (k) The method shows that Cholesky decomposition is carried out on the newly generated symmetrical innovation matrix, and the processing result is more stable in value in the recursive operation process than that of simply applying the innovation matrix.
Kalman gain K k The change is then:
K k ＝P - (k)·(▽h) T ·S -1 Chol (k)·(S -1 Chol (k)) T
in the above formula, K k Is Kalman gain, P - (k) Is a covariance matrix
The recursive update equation is:
P + (k)＝(I-K k ·▽h)·P - (k)
in the above formula: h is the observation model with respect to the state space
Jacobian, R k To observe the error covariance of the noise, the observed noise is assumed to be gaussian with a mean of zero.
(3) Data association, namely judging an observation result, and if the observed landmark is a new landmark, entering the step (4); if the observed landmark is a known landmark, step (5) is entered.
(4) State augmentation
When the mobile robot observes a new landmark, after verification, the position state of the newly observed landmark needs to be added into the system state vector.
(5) And updating the new system state vector and the covariance matrix by applying an augmentation model. Newly observed landmarks
The position can be expressed as:
Then the system state space containing the new landmark
Comprises the following steps:
corresponding covariance matrix P * (k) Comprises the following steps:
wherein:
in the formula:
is g (-) a Jacobian determinant on the pose vector and the observation vector of the mobile robot.
While the invention has been described with reference to a specific embodiment, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (3)
1. A positioning method of a robot positioning system based on binocular vision and a convolutional neural network comprises the following steps that the positioning system comprises a mobile robot, and the mobile robot is provided with an inertial sensor module, a binocular vision module and an upper computer which are respectively connected with a sensor control module; the method is characterized by comprising the following steps:
(1) acquiring data through an inertial sensor and a binocular vision sensor, acquiring the acceleration and the angular velocity of the mobile robot through the inertial sensor, and acquiring images around the mobile robot through the binocular vision sensor;
(2) stereo matching calculation is carried out through the acquired image by utilizing a convolutional neural network to obtain the position of the landmark;
(3) according to the motion state of the mobile robot obtained by the inertial sensor and the obtained landmark position, carrying out augmentation Kalman filtering processing to obtain the position of the mobile robot;
(4) drawing a motion map according to the position change of the mobile robot;
the step (2) of obtaining the position of the landmark by stereo matching calculation of the acquired image by using a convolutional neural network comprises the following specific steps:
(1) acquiring an environment image around the robot by using a binocular camera;
(2) correcting the binocular images, and re-projecting the image pairs acquired by the binocular camera to enable the image pairs to meet a parallel alignment state;
(3) stereo matching is carried out on the collected images by using a convolutional neural network to obtain a parallax value;
(4) calculating to obtain the position of the landmark according to the parallax value and the triangulation principle;
the specific method for carrying out stereo matching on the acquired image by utilizing the convolutional neural network comprises the following steps:
(1) calculating parallax matching cost of the binocular images by using a convolutional neural network;
(2) cost aggregation, namely aggregating the matching cost in a support window by a summing and averaging method to obtain the accumulated cost of a point on the image at a specific parallax;
(3) calculating the parallax, selecting a point with the optimal accumulated cost in the parallax search range as a corresponding matching point, wherein the parallax corresponding to the point is the required parallax;
(4) and (5) refining the parallax, and optimizing the obtained parallax image to obtain a final parallax value.
2. The positioning method according to claim 1, wherein the step (3) of performing extended kalman filter processing according to the acquired data includes:
(1) predicting the state matrix and the covariance matrix of the robot at the current moment through the state matrix and the covariance matrix of the robot at the previous moment;
(2) in the observation process, the landmark of the surrounding environment is observed, and the state of the robot and the system error correction parameters are updated through Kalman gain and the innovation matrix;
(3) data association, namely judging the relationship between the observation landmark and the known landmark according to the observation result, if the observation landmark is a new landmark, entering the step (4), and if the observation landmark is the known landmark, entering the step (5);
(4) in the state augmentation process, if a new landmark is observed, the position of the observed new landmark is added into a system state vector;
(5) and updating the new system state matrix and covariance if the observed landmark is a known landmark.
3. The positioning method according to claim 1, wherein the binocular vision module is composed of binocular stereo cameras; the inertial sensor module consists of an inertial sensor; the sensor control module consists of a single chip microcomputer; the upper computer is a notebook computer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811416964.9A CN109579825B (en) | 2018-11-26 | 2018-11-26 | Robot positioning system and method based on binocular vision and convolutional neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811416964.9A CN109579825B (en) | 2018-11-26 | 2018-11-26 | Robot positioning system and method based on binocular vision and convolutional neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109579825A CN109579825A (en) | 2019-04-05 |
| CN109579825B true CN109579825B (en) | 2022-08-19 |
Family
ID=65924557
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811416964.9A Active CN109579825B (en) | 2018-11-26 | 2018-11-26 | Robot positioning system and method based on binocular vision and convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109579825B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110118556A (en) * | 2019-04-12 | 2019-08-13 | 浙江工业大学 | A kind of robot localization method and device based on covariance mixing together SLAM |
| CN110189366B (en) * | 2019-04-17 | 2021-07-06 | 北京迈格威科技有限公司 | Laser coarse registration method and device, mobile terminal and storage medium |
| CN110519582A (en) * | 2019-08-16 | 2019-11-29 | 哈尔滨工程大学 | A kind of crusing robot data collection system and collecting method |
| US11531107B2 (en) * | 2019-11-19 | 2022-12-20 | Volvo Car Corporation | Long range LIDAR-based speed estimation |
| CN111174781B (en) * | 2019-12-31 | 2022-03-04 | 同济大学 | Inertial navigation positioning method based on wearable device combined target detection |
| CN111325794B (en) * | 2020-02-23 | 2023-05-26 | 哈尔滨工业大学 | Visual simultaneous localization and map construction method based on depth convolution self-encoder |
| CN111504331B (en) * | 2020-04-29 | 2021-09-14 | 杭州环峻科技有限公司 | Method and device for positioning panoramic intelligent vehicle from coarse to fine |
| CN111913484B (en) * | 2020-07-30 | 2022-08-12 | 杭州电子科技大学 | Path planning method of transformer substation inspection robot in unknown environment |
| CN114862955B (en) * | 2022-07-07 | 2022-09-02 | 诺伯特智能装备（山东）有限公司 | Rapid visual positioning method for industrial robot |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102042835A (en) * | 2010-11-05 | 2011-05-04 | 中国海洋大学 | Autonomous underwater vehicle combined navigation system |
| CN102288176A (en) * | 2011-07-07 | 2011-12-21 | 中国矿业大学(北京) | Coal mine disaster relief robot navigation system based on information integration and method |
| CN103983263A (en) * | 2014-05-30 | 2014-08-13 | 东南大学 | Inertia/visual integrated navigation method adopting iterated extended Kalman filter and neural network |
| CN108648161A (en) * | 2018-05-16 | 2018-10-12 | 江苏科技大学 | The binocular vision obstacle detection system and method for asymmetric nuclear convolutional neural networks |
| CN108734143A (en) * | 2018-05-28 | 2018-11-02 | 江苏迪伦智能科技有限公司 | A kind of transmission line of electricity online test method based on binocular vision of crusing robot |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11526744B2 (en) * | 2016-07-09 | 2022-12-13 | Doxel, Inc. | Monitoring construction of a structure |
-
2018
- 2018-11-26 CN CN201811416964.9A patent/CN109579825B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102042835A (en) * | 2010-11-05 | 2011-05-04 | 中国海洋大学 | Autonomous underwater vehicle combined navigation system |
| CN102288176A (en) * | 2011-07-07 | 2011-12-21 | 中国矿业大学(北京) | Coal mine disaster relief robot navigation system based on information integration and method |
| CN103983263A (en) * | 2014-05-30 | 2014-08-13 | 东南大学 | Inertia/visual integrated navigation method adopting iterated extended Kalman filter and neural network |
| CN108648161A (en) * | 2018-05-16 | 2018-10-12 | 江苏科技大学 | The binocular vision obstacle detection system and method for asymmetric nuclear convolutional neural networks |
| CN108734143A (en) * | 2018-05-28 | 2018-11-02 | 江苏迪伦智能科技有限公司 | A kind of transmission line of electricity online test method based on binocular vision of crusing robot |
Non-Patent Citations (2)
| Title |
|---|
| "Deep Convolutional Neural Networks for pedestrian detection";Tomè, D.1 等;《Signal Processing: Image Communication》;20161231;第47卷(第7期);正文第482-489页 * |
| 卷积神经网络双目视觉路面障碍物检测;胡颖等;《计算机工程与设计》;20181016(第10期);正文第3278-3282页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109579825A (en) | 2019-04-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109579825B (en) | Robot positioning system and method based on binocular vision and convolutional neural network | |
| CN110125928B (en) | Binocular inertial navigation SLAM system for performing feature matching based on front and rear frames | |
| EP2833322B1 (en) | Stereo-motion method of three-dimensional (3-D) structure information extraction from a video for fusion with 3-D point cloud data | |
| CN107590827A (en) | A kind of indoor mobile robot vision SLAM methods based on Kinect | |
| CN112304307A (en) | Positioning method and device based on multi-sensor fusion and storage medium | |
| CN109029433A (en) | Join outside the calibration of view-based access control model and inertial navigation fusion SLAM on a kind of mobile platform and the method for timing | |
| US20170161901A1 (en) | System and Method for Hybrid Simultaneous Localization and Mapping of 2D and 3D Data Acquired by Sensors from a 3D Scene | |
| KR100855657B1 (en) | System for estimating self-position of the mobile robot using monocular zoom-camara and method therefor | |
| CN108519102B (en) | Binocular vision mileage calculation method based on secondary projection | |
| CN106595659A (en) | Map merging method of unmanned aerial vehicle visual SLAM under city complex environment | |
| CN107677274B (en) | Unmanned plane independent landing navigation information real-time resolving method based on binocular vision | |
| US11062475B2 (en) | Location estimating apparatus and method, learning apparatus and method, and computer program products | |
| CN110322507B (en) | Depth reprojection and space consistency feature matching based method | |
| CN104281148A (en) | Mobile robot autonomous navigation method based on binocular stereoscopic vision | |
| CN111998862A (en) | Dense binocular SLAM method based on BNN | |
| Alcantarilla et al. | Large-scale dense 3D reconstruction from stereo imagery | |
| Xian et al. | Fusing stereo camera and low-cost inertial measurement unit for autonomous navigation in a tightly-coupled approach | |
| Yong-guo et al. | The navigation of mobile robot based on stereo vision | |
| CN116468786B (en) | Semantic SLAM method based on point-line combination and oriented to dynamic environment | |
| CN116128966A (en) | Semantic positioning method based on environmental object | |
| Aggarwal | Machine vision based SelfPosition estimation of mobile robots | |
| CN115578417A (en) | Monocular vision inertial odometer method based on feature point depth | |
| Billy et al. | Adaptive SLAM with synthetic stereo dataset generation for real-time dense 3D reconstruction | |
| CN113554754A (en) | Indoor positioning method based on computer vision | |
| Chen et al. | Binocular vision localization based on vision SLAM system with multi-sensor fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |