CN111862120B - Monocular SLAM scale recovery method - Google Patents
Monocular SLAM scale recovery method Download PDFInfo
- Publication number
- CN111862120B CN111862120B CN202010711561.8A CN202010711561A CN111862120B CN 111862120 B CN111862120 B CN 111862120B CN 202010711561 A CN202010711561 A CN 202010711561A CN 111862120 B CN111862120 B CN 111862120B
- Authority
- CN
- China
- Prior art keywords
- calibration box
- wall
- monocular
- scale
- slam
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method for recovering monocular SLAM scale, which comprises the following steps: manufacturing a calibration box, wherein a rectangular through hole is formed in the calibration box in a penetrating manner; placing a calibration box at the visual field of a monocular camera, and modeling a target object after the monocular camera passes through a through hole of the calibration box; the monocular camera acquires images of the inner wall of the calibration box, and the upper and lower images of the inner wall of the calibration box in the previous key frames generated by SLAM are segmented in areas; respectively obtaining three-dimensional map point coordinates of upper and lower images of the inner wall of the calibration box according to the corresponding relation between the feature points and the three-dimensional map points in the SLAM; fitting the coordinates of the three-dimensional map points on the upper surface and the lower surface of the inner wall of the calibration box according to the constraint condition of the two planes, and calculating to obtain the distances between the upper surface and the lower surface of the inner wall of the calibration box in the three-dimensional map; calculating a scale factor; the scale factor F is used to recover the entire three-dimensional map scale. The monocular SLAM device can automatically recover the size of monocular SLAM in the human intestinal environment, and is good in effect and high in accuracy.
Description
Technical Field
The invention relates to the technical field of surgical robot navigation, in particular to a method for recovering monocular SLAM scale.
Background
In the process of taking a picture by a monocular camera, the actual distance from the object to the camera cannot be known only by a single picture, the distance from the object to the camera is solved in monocular vision SLAM by triangulating the matched characteristic points in the adjacent key frames, the measured distance is only a numerical value, no unit exists, and the scale has uncertainty, so that the structural framework of the object is restored only, and the size of the object in practical significance is not restored.
Currently, other sensors are mainly added in a large scene environment to restore the scale, such as an IMU, a GPS, a laser radar and the like; the absolute scale can be recovered with higher accuracy by using the scale information provided by the additional sensor, and such methods cannot be used in the intestinal environment due to the limitation of the sensor size. The method for reconstructing the three-dimensional structure of the abdominal cavity scene based on the laparoscope is characterized in that a calibrated scale with scales is placed at the abdominal cavity, and the scale recovery is manually carried out, but is inconvenient in actual operation; still other schemes for reconstructing an endoscopic scene eventually do not recover scale information for the scene.
Disclosure of Invention
The invention aims to provide a method for recovering the monocular SLAM scale, which can solve the problem of uncertainty of the monocular SLAM scale in the intestinal environment of a human body.
In order to solve the technical problems, the invention provides a method for recovering the monocular SLAM scale, which comprises the following steps:
s1, manufacturing a calibration box, wherein a rectangular through hole is formed in the calibration box in a penetrating manner, and the distance between the upper surface and the lower surface of the wall of the through hole is d i ′；
S2, placing a calibration box at the visual field of the monocular camera, and modeling a target object after the monocular camera passes through a through hole of the calibration box;
s3, the monocular camera performs image acquisition on the inner wall of the calibration box, and upper and lower images of the inner wall of the calibration box in the previous several key frames generated by SLAM are segmented in areas;
s4, respectively obtaining three-dimensional map point coordinates of the upper image and the lower image of the inner wall of the calibration box according to the corresponding relation between the characteristic points in the SLAM and the three-dimensional map points;
s5, fitting the coordinates of the three-dimensional map points on the upper surface and the lower surface of the inner wall of the calibration box by using the constraint condition of the two planes to obtain the distance d between the upper surface and the lower surface of the inner wall of the calibration box in the three-dimensional map by calculation i ；
S6, according to d i ' and d i Calculating a scale factor F;
s7, restoring the whole three-dimensional map scale by using the scale factor F.
Preferably, the area division in S3 is defined by four ridges on the inner wall of the calibration box.
Preferably, the method for using four edges of the inner wall of the calibration box as boundaries specifically includes:
detecting edge information of key frames of the previous frames;
obtaining a linear edge in the edge information;
and connecting the straight lines on the same ridge line end to end for merging.
Preferably, the edge information of key frames of the previous frames is detected through a canny operator, a Roberts operator, a Prewitt operator or a Sobel operator.
Preferably, the obtaining the straight line edge in the edge information specifically includes:
and obtaining the linear edges in the edge information by using a hough linear detection algorithm or a Freeman linear detection algorithm, and simultaneously setting a threshold value to filter out the short linear edges.
Preferably, in S3, the key frame image is the first five frames.
Preferably, the step S6 specifically includes: f=d i ′/d i
Preferably, the step S7 specifically includes multiplying the coordinates of all three-dimensional map points constructed by the monocular SLAM by a scale factor F.
The invention has the beneficial effects that:
the method utilizes the auxiliary of the calibration box to automatically recover the scale of the monocular SLAM in the human intestinal environment, has good effect and high precision, is suitable for a narrow environment, and can solve the problem that the scale of the monocular SLAM is difficult to recover in the narrow human body cavity environment.
Drawings
FIG. 1 is a flow chart of the present invention;
FIG. 2 is an external structure of the calibration box;
FIG. 3 is an internal view of a calibration box;
FIG. 4 is a simulation experiment scenario; .
FIG. 5 is a contour extracted from a key frame of SLAM by a canny operator;
FIG. 6 is a straight line segment detected by Hough;
FIG. 7 shows the upper and lower surfaces of the calibration box obtained by region division;
fig. 8 is a SLAM sparse feature point map after scale recovery.
Detailed Description
The present invention will be further described with reference to the accompanying drawings and specific examples, which are not intended to be limiting, so that those skilled in the art will better understand the invention and practice it.
Initializing the map size is completed in ORB-SLAM2 with the median value of the map point depths generated by the first frame as unit size 1. To restore the scale, the size (with unit information) of some objects in the real scene and the corresponding size (without unit information) of the objects in the map must be known in advance, and the scale factor is calculated, so that the scale information of all objects in the scene is restored.
Referring to FIG. 1, the invention discloses a method for monocular SLAM scale recovery, which comprises the following steps:
s1, manufacturing a calibration box, wherein a rectangular through hole is formed in the calibration box in a penetrating manner, and the distance between the upper surface and the lower surface of the wall of the through hole is d i ′。
As shown in fig. 2 and fig. 3, the inner wall of the calibration box is provided with features, the front and the back of the calibration box are square and non-closed cuboid calibration boxes, the external dimensions are 250mm by 51mm, and the internal dimensions are 250mm by 42mm.
S2, placing a calibration box at the visual field of the monocular camera, and modeling the target object after the monocular camera passes through the through hole of the calibration box. In this embodiment, the calibration box may be placed at the front end of the intestinal inlet, and the monocular camera enters the intestinal tract for modeling after passing through the calibration box. The monocular camera may be an endoscope.
S3, the monocular camera acquires images of the inner wall of the calibration box, the upper and lower images of the inner wall of the calibration box in the first few key frames generated by SLAM are segmented in areas, and the first five key frames are preferable because if the number of map points generated is less than five, the number of map points is increased, errors are increased, and after the number of map points exceeds five, the endoscope acquires the scene of the intestinal tract, so that the area segmentation is not facilitated.
The step S3 of region segmentation takes four edge lines of the inner wall of the calibration box as boundaries, and specifically comprises the following steps:
detecting edge information of key frames of the previous frames through a canny operator, a Roberts operator, a Prewitt operator or a Sobel operator;
obtaining a linear edge in the edge information by using a hough linear detection algorithm or a Freeman linear detection algorithm, and simultaneously setting a threshold value to filter out a short linear edge;
and connecting and combining the ends of the straight lines detected on one ridge line according to the coordinate range of the four ridge lines.
As shown in fig. 4, in an experimental scenario, a calibration box is placed at the front end of the intestinal inlet, and an endoscope passes through the calibration box and then enters the intestinal tract to perform intestinal modeling.
As shown in fig. 6, is a straight line segment detected by Hough. And setting a length threshold in the detection process, and removing the straight line segment with shorter length.
As shown in fig. 7, the upper and lower surfaces of the calibration box obtained by the region division are divided, and the blue line is the boundary of the region division. And dividing the upper plane and the lower plane by using the straight line segment detected by hough according to the coordinate range.
S4, respectively obtaining three-dimensional map point coordinates of the upper and lower images of the inner wall of the calibration box according to the corresponding index relation between the feature points and the three-dimensional map points in the SLAM;
s5, fitting the coordinates of the three-dimensional map points on the upper surface and the lower surface of the inner wall of the calibration box by using the constraint condition of the two planes to obtain the distance d between the upper surface and the lower surface of the inner wall of the calibration box in the three-dimensional map by calculation i ；
S6, according to d i ' and d i Calculating scale factor F, f=d i '/d i ；
S7, recovering the scale of the whole three-dimensional map by using the scale factor F, and multiplying the scale factor F by all three-dimensional map point coordinates constructed by the monocular SLAM.
As shown in fig. 8, the SLAM sparse feature point map after scale recovery. The unit is cm.
The above-described embodiments are merely preferred embodiments for fully explaining the present invention, and the scope of the present invention is not limited thereto. Equivalent substitutions and modifications will occur to those skilled in the art based on the present invention, and are intended to be within the scope of the present invention. The protection scope of the invention is subject to the claims.
Claims (9)
1. A method for monocular SLAM scale recovery, comprising the steps of:
s1, manufacturing a calibration box, wherein a rectangular through hole is formed in the calibration box in a penetrating manner, and the distance between the upper surface and the lower surface of the wall of the through hole is d i ′；
S2, placing a calibration box at the visual field of the monocular camera, placing the calibration box at the front end of an intestinal inlet, and modeling the intestinal tract after the monocular camera passes through the through hole of the calibration box;
s3, the monocular camera performs image acquisition on the inner wall of the calibration box, and upper and lower images of the inner wall of the calibration box in the previous several key frames generated by SLAM are segmented in areas;
s4, respectively obtaining three-dimensional map point coordinates of the upper image and the lower image of the inner wall of the calibration box according to the corresponding relation between the characteristic points in the SLAM and the three-dimensional map points;
s5, fitting the coordinates of the three-dimensional map points on the upper surface and the lower surface of the inner wall of the calibration box by using the constraint condition of the two planes to obtain the distance d between the upper surface and the lower surface of the inner wall of the calibration box in the three-dimensional map by calculation i ；
S6, according to d i ' and d i Calculating a scale factor F;
s7, restoring the whole three-dimensional map scale by using the scale factor F.
2. The method for monocular SLAM scale restoration of claim 1, wherein the region segmentation in S3 is bounded by four ridges of the inner wall of the calibration box.
3. The method for monocular SLAM scale restoration of claim 2, wherein the boundaries of four ridges on the inner wall of the calibration box specifically include:
detecting edge information of key frames of the previous frames;
obtaining a linear edge in the edge information;
and connecting the straight lines on the same ridge line end to end for merging.
4. The method for monocular SLAM scale restoration of claim 3, wherein the edge information of key frames of the previous frames is detected by a canny operator, a Roberts operator, a Prewitt operator, or a Sobel operator.
5. The method for monocular SLAM scale recovery of claim 3, wherein obtaining the straight line edges in the edge information specifically comprises:
and obtaining the linear edges in the edge information by using a hough linear detection algorithm or a Freeman linear detection algorithm, and simultaneously setting a threshold value to filter out the short linear edges.
6. The method for monocular SLAM scale restoration of claim 1, wherein in S3, the key frame image is the first five frames.
7. The method for monocular SLAM scale recovery of claim 1, wherein S6 specifically comprises: f=d i ′/d i 。
8. The method for monocular SLAM scale restoration of claim 1, wherein S7 specifically comprises multiplying all three-dimensional map point coordinates of the monocular SLAM construction by a scale factor F.
9. A processor for running a program, wherein the program when run performs the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010711561.8A CN111862120B (en) | 2020-07-22 | 2020-07-22 | Monocular SLAM scale recovery method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010711561.8A CN111862120B (en) | 2020-07-22 | 2020-07-22 | Monocular SLAM scale recovery method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111862120A CN111862120A (en) | 2020-10-30 |
| CN111862120B true CN111862120B (en) | 2023-07-11 |
Family
ID=72949208
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010711561.8A Active CN111862120B (en) | 2020-07-22 | 2020-07-22 | Monocular SLAM scale recovery method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111862120B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112348869A (en) * | 2020-11-17 | 2021-02-09 | 的卢技术有限公司 | Method for recovering monocular SLAM scale through detection and calibration |
| CN113779012B (en) * | 2021-09-16 | 2023-03-07 | 中国电子科技集团公司第五十四研究所 | Monocular vision SLAM scale recovery method for unmanned aerial vehicle |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106803270A (en) * | 2017-01-13 | 2017-06-06 | 西北工业大学深圳研究院 | Unmanned aerial vehicle platform is based on many key frames collaboration ground target localization method of monocular SLAM |
| CN106920279A (en) * | 2017-03-07 | 2017-07-04 | 百度在线网络技术（北京）有限公司 | Three-dimensional map construction method and device |
| CN108090954A (en) * | 2017-12-15 | 2018-05-29 | 南方医科大学南方医院 | Abdominal cavity environmental map based on characteristics of image rebuilds the method with laparoscope positioning |
| CN110533587A (en) * | 2019-07-03 | 2019-12-03 | 浙江工业大学 | A kind of SLAM method of view-based access control model prior information and map recovery |
| CN111028280A (en) * | 2019-12-09 | 2020-04-17 | 西安交通大学 | # -shaped structured light camera system and method for performing scaled three-dimensional reconstruction of target |
| CN111141295A (en) * | 2019-12-20 | 2020-05-12 | 南京航空航天大学 | Automatic map recovery method based on monocular ORB-SLAM |
| CN111145238A (en) * | 2019-12-12 | 2020-05-12 | 中国科学院深圳先进技术研究院 | Three-dimensional reconstruction method and device of monocular endoscope image and terminal equipment |
| CN111402429A (en) * | 2020-06-08 | 2020-07-10 | 成都索贝数码科技股份有限公司 | Scale reduction and three-dimensional reconstruction method, system, storage medium and equipment |
-
2020
- 2020-07-22 CN CN202010711561.8A patent/CN111862120B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106803270A (en) * | 2017-01-13 | 2017-06-06 | 西北工业大学深圳研究院 | Unmanned aerial vehicle platform is based on many key frames collaboration ground target localization method of monocular SLAM |
| CN106920279A (en) * | 2017-03-07 | 2017-07-04 | 百度在线网络技术（北京）有限公司 | Three-dimensional map construction method and device |
| CN108090954A (en) * | 2017-12-15 | 2018-05-29 | 南方医科大学南方医院 | Abdominal cavity environmental map based on characteristics of image rebuilds the method with laparoscope positioning |
| CN110533587A (en) * | 2019-07-03 | 2019-12-03 | 浙江工业大学 | A kind of SLAM method of view-based access control model prior information and map recovery |
| CN111028280A (en) * | 2019-12-09 | 2020-04-17 | 西安交通大学 | # -shaped structured light camera system and method for performing scaled three-dimensional reconstruction of target |
| CN111145238A (en) * | 2019-12-12 | 2020-05-12 | 中国科学院深圳先进技术研究院 | Three-dimensional reconstruction method and device of monocular endoscope image and terminal equipment |
| CN111141295A (en) * | 2019-12-20 | 2020-05-12 | 南京航空航天大学 | Automatic map recovery method based on monocular ORB-SLAM |
| CN111402429A (en) * | 2020-06-08 | 2020-07-10 | 成都索贝数码科技股份有限公司 | Scale reduction and three-dimensional reconstruction method, system, storage medium and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111862120A (en) | 2020-10-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108717712B (en) | Visual inertial navigation SLAM method based on ground plane hypothesis | |
| CN110264567B (en) | Real-time three-dimensional modeling method based on mark points | |
| Zhou et al. | Seamless fusion of LiDAR and aerial imagery for building extraction | |
| CN109472828B (en) | Positioning method, positioning device, electronic equipment and computer readable storage medium | |
| Tamas et al. | Targetless calibration of a lidar-perspective camera pair | |
| CN107016705A (en) | Ground level estimation in computer vision system | |
| CN111862120B (en) | Monocular SLAM scale recovery method | |
| CN110675455B (en) | Natural scene-based self-calibration method and system for vehicle body looking-around camera | |
| CN111260715B (en) | Depth map processing method, small obstacle detection method and system | |
| JP4568845B2 (en) | Change area recognition device | |
| CN111798505A (en) | Monocular vision-based dense point cloud reconstruction method and system for triangularized measurement depth | |
| CN112179373A (en) | Measuring method of visual odometer and visual odometer | |
| CN109493378B (en) | Verticality detection method based on combination of monocular vision and binocular vision | |
| Dhiman et al. | A multi-frame stereo vision-based road profiling technique for distress analysis | |
| Bhowmick et al. | Mobiscan3D: A low cost framework for real time dense 3D reconstruction on mobile devices | |
| Paudel et al. | 2D-3D camera fusion for visual odometry in outdoor environments | |
| CN109902695B (en) | Line feature correction and purification method for image pair linear feature matching | |
| Siddique et al. | 3d object localization using 2d estimates for computer vision applications | |
| Kochi et al. | 3D modeling of architecture by edge-matching and integrating the point clouds of laser scanner and those of digital camera | |
| Douret et al. | A multi-cameras 3d volumetric method for outdoor scenes: a road traffic monitoring application | |
| CN115880643A (en) | Social distance monitoring method and device based on target detection algorithm | |
| JP3966419B2 (en) | Change area recognition apparatus and change recognition system | |
| CN114494582A (en) | Three-dimensional model dynamic updating method based on visual perception | |
| Förstner et al. | Automatic orientation and recognition in highly structured scenes | |
| Yang et al. | Road detection by RANSAC on randomly sampled patches with slanted plane prior |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |