CN112102347B - Step detection and single-stage step height estimation method based on binocular vision - Google Patents
Step detection and single-stage step height estimation method based on binocular vision Download PDFInfo
- Publication number
- CN112102347B CN112102347B CN202011301586.7A CN202011301586A CN112102347B CN 112102347 B CN112102347 B CN 112102347B CN 202011301586 A CN202011301586 A CN 202011301586A CN 112102347 B CN112102347 B CN 112102347B
- Authority
- CN
- China
- Prior art keywords
- line segment
- information
- detection
- image
- coordinate system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000001514 detection method Methods 0.000 title claims abstract description 32
- 238000012216 screening Methods 0.000 claims abstract description 7
- 239000011159 matrix material Substances 0.000 claims description 8
- 238000003708 edge detection Methods 0.000 claims description 6
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 239000000284 extract Substances 0.000 abstract 1
- 238000005516 engineering process Methods 0.000 description 4
- 238000013507 mapping Methods 0.000 description 4
- 230000007547 defect Effects 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 238000005286 illumination Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G06T5/90—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10044—Radar image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
- G06T2207/20061—Hough transform
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The invention discloses a binocular vision-based step detection and single-stage step height estimation method, which converts a picture into a gray-scale image by utilizing dynamic white balance and graying, and extracts edge information on the basis of the gray-scale image; carrying out line detection from the edge information by using Hough transform, obtaining a straight line set in the image, and screening out a straight line set which is approximately horizontal according to a slope; clustering the horizontal line segment set by using the distance information of the middle points of the line segments, and extracting potential classes related to steps from different classes; carrying out subdivision clustering and regression of step straight lines on the screened step straight line set by using a least square method; and calculating the depth information of the step area according to the binocular parallax, reversely reducing the positions of the pixel points in the world coordinate system by using the depth information and the internal and external parameters of the camera, and further calculating the height of the single-stage step. Based on the method provided by the invention, the detection of the step position, the estimation of the horizontal distance of the step and the height of the single-stage step can be realized only by using binocular vision information.
Description
Technical Field
The invention belongs to the field of computer vision, and particularly relates to a step detection and single-stage step height estimation method based on binocular vision.
Background
In a plurality of scenes such as park navigation, endowment service, smart home and the like, an intelligent agent such as a robot providing service generally needs to face a process of going up and down steps. Common methods for step detection are typically three-dimensional data methods based on lidar and two-dimensional data methods based on images.
The step detection algorithm based on the laser radar acquires three-dimensional data of a scene by using the laser radar or a depth camera, and generally adopts a technology of combining plane detection and graph segmentation. Since three-dimensional data acquired by a laser radar or a depth camera is generally sparse data, an upsampling technology is adopted to increase the density of the three-dimensional data. And then, a horizontal plane and a vertical plane are obtained by adopting a plane detection technology based on a normal vector, and the positions of the steps are screened out from the detected plane by utilizing a graph segmentation technology according to the alternating attribute of the horizontal plane and the vertical plane of the steps. The method has the advantages that the height of a single-stage step can be obtained after the step position is obtained by utilizing three-dimensional data; the defect is that the calculated amount of the three-dimensional data is large, and the three-dimensional data is not suitable for being deployed on a movable intelligent body represented by a robot and the like.
Image-based step detection algorithms use monocular or binocular cameras to acquire two-dimensional data of a scene, typically using techniques related to edge detection and line detection. The color image is first converted into a gray scale image and object edge information is extracted from the gray scale image using the Canny edge detection algorithm. And then, acquiring straight line information in the image from the edge information by using a Hough transform equal line detection algorithm, and determining the position of the step from the previous straight line set according to the horizontal attribute of the step. The method has less calculation amount and lower requirement on hardware equipment, but the method has difficulty in calculating the height of a single-stage step on the basis of step detection.
Disclosure of Invention
The invention aims to provide a step detection and single-stage step height estimation method based on binocular vision, aiming at the defects of the prior art.
The above object of the present invention is achieved by the following technical solutions: a step detection and single-stage step height estimation method based on binocular vision comprises the following steps:
s1, acquiring an RGB image by using a binocular camera, and adjusting the brightness of the RGB image by using a dynamic white balance algorithm;
s2, converting the RGB image after brightness adjustment into a gray scale image, and extracting edge information of the gray scale image by using an edge detection algorithm;
s3, extracting line segment information from the edge information by using a line detection algorithm, and screening horizontal line segments from the line segment set according to the slope to form an original horizontal line segment set;
s4, clustering in the original horizontal line segment set according to the vertical distance relationship of the middle points of the line segments, and screening out a potential line segment set forming a step;
s5, carrying out subdivision clustering and regression of a step linear equation on the potential line segment set by using a least square method;
s6, constructing a pixel depth map of the monocular image by using a disparity map of a binocular image of the binocular camera, and performing information completion on pixel points with incomplete depth information by using the color and position information of pixels in the RGB image;
and S7, projecting the step pixels in the monocular image into a world coordinate system by using the internal and external parameters of the binocular camera and the depth information of the depth map, and calculating the height of the single-step according to the three-dimensional information in the world coordinate system.
Further, in step S1, the brightness of the image is adjusted using a perfect reflection white balance algorithm.
Further, in step S2, the RGB image is converted into a Gray scale map Gray using the following formula, and edge information of an object in the image is extracted using Canny operator on the basis of the Gray scale map:
where R, G, B respectively represent data of three color channels of an RGB image.
Further, in step S3, the hough transform is used to detect the line segment information from the edge information, resulting in a line segment set satisfying the length requirement.
Further, in step S4, coarse clustering is performed on the original horizontal line segment set according to the y-axis distance of the midpoint of the line segment, and a potential line segment set constituting a step is selected according to the number of line segments included in each class and the class center.
Further, in step S5, in the potential line segment set constituting the step, the midpoint (x) is determined0，y0) X of0Sequencing, and performing subdivision clustering by using a least square method, specifically: first, from the set of potential line segments, the first line segment is selected to calculate its equation of a straight line y = kx + b, and then the midpoint of the second line segment (x) is calculated0，y0) Distance y to the equation of a straight line0-kx0-b |, if the distance is less than a set threshold, the two segments belong to the same class, the two segments are deleted from the set of potential segments, and the linear equation y = kx + b is refitted using the least squares method from the mid-points of the two segments; all segments are traversed in sequence until the set of potential segments is empty.
Further, in step S7, the mapping method of the step pixel projection to the world coordinate system is as follows:
wherein, P uv Is a coordinate point of a pixel coordinate system, and u and v represent coordinates under the pixel coordinate system; k represents camera internal parameters, and P represents pixel points of a camera coordinate system; x, Y and Z represent coordinates in a camera coordinate system; r represents a rotation matrix, Xw,Yw,ZwRepresenting coordinates in the world coordinate system and t representing the translation matrix.
Further, in step S7, n points are selected from each of two adjacent step straight lines in the world coordinate system
And
calculating the height h of the single-stage step:
where d is the pixel depth, fyFocal length in the y-axis direction of the camera, cyIs the principal point of the y-axis direction of the camera.
The invention has the beneficial effects that: the method comprises the steps of obtaining a stable original horizontal line set from an image through a dynamic white balance, edge detection and line detection algorithm, determining the approximate position of a step by combining coarse clustering of a midpoint distance and subdivision clustering of a least square method, and fitting a linear expression of the step; and converting the pixel coordinate system and the world coordinate system by utilizing the internal and external parameters of the camera, and finally estimating the height of the single-stage step in the world coordinate system. The invention has small calculated amount and is suitable for being deployed on movable intelligent bodies represented by robots and the like; the invention can realize the detection of the step position only by using binocular vision information, and can calculate the height of a single-stage step on the basis of step detection.
Drawings
FIG. 1 is a flow chart of the present invention.
Detailed Description
As shown in FIG. 1, the invention relates to a binocular vision-based step detection and single-stage step height estimation method, which comprises the following steps:
s1, acquiring image data by using a binocular camera, and adjusting the brightness of the RGB image by using a dynamic white balance algorithm (perfect reflection white balance); the contrast of the image is enhanced, and the influence of the illumination intensity on the brightness of the image can be overcome.
S2, converting the RGB image with the brightness adjusted in the step S1 into a Gray level map Gray by using the following formula, and extracting the edge information of the object in the image by using an edge detection algorithm (Canny operator) on the basis of the Gray level map:
where R, G, B respectively represent data of three color channels of an RGB image.
S3, extracting line segment information from the edge information obtained in the step S2 by using a line detection algorithm (Hough transform) to obtain a line segment set meeting the length requirement; and screening out horizontal line segments from the line segment set according to the slope to form an original horizontal line segment set.
S4, the midpoint of each line segment in the original horizontal line segment set obtained in step S3 is (x)0，y0) And carrying out preliminary clustering according to the y-axis distance between the midpoints, and screening out a potential line segment set forming the step according to the number of line segments contained in each class and the position of the class center. The class center is the midpoint (x) of the line segment of the same class0，y0) Average value of (a). The edge lines of the steps are approximately parallel horizontal straight lines, but in the detection process, the horizontal straight lines which do not belong to the steps can be detected; the straight lines belonging to the steps are usually gathered together, and the horizontal straight lines not belonging to the steps are usually relatively large in distance; therefore, the step straight line and the non-step horizontal straight line can be roughly distinguished according to the distance of the middle point of each horizontal line segment.
S5, the potential line segment set obtained in the step S4 is based on the midpoint (x)0，y0) X of0Sequencing the line segments from small to large, and performing secondary clustering and regression of a step linear equation on the potential line segment set by using a least square method, wherein the method specifically comprises the following steps: firstly, selecting a line segment ranked first from a potential line segment set to calculate a straight line equation y = kx + b; the midpoint (x) of the second line segment is then calculated0，y0) Distance y to the equation of a straight line0-kx0-b |, if the distance is less than a set threshold, the two segments belong to the same class (same straight line), the two segments are deleted from the set of potential segments, and the straight line equation y = kx + b is refitted using the least squares method from the midpoints of the two segments; all segments are traversed in sequence until the set of potential segments is empty.
S6, constructing a pixel depth map of the monocular camera by using the disparity map of the binocular image of the binocular camera, and completing depth information of information residual defects in the depth map by using a pixel segmentation mode according to the color and position information of pixels in the RGB image.
And S7, projecting the step pixel coordinate system in the monocular image into the world coordinate system by using the internal and external parameters of the binocular camera and the pixel depth information of the depth map obtained in the step S6, and calculating the height of the single step according to the height information of the step area in the three-dimensional information obtained by projecting the step area in the world coordinate system. The mapping method comprises the following steps:
wherein, P uv As pixel coordinate system coordinate points, u, v denote pixel coordinatesCoordinates under a standard system; k represents camera internal parameters, and P represents pixel points of a camera coordinate system; x, Y and Z represent coordinates in a camera coordinate system; r represents a rotation matrix, Xw,Yw,ZwRepresenting coordinates in the world coordinate system and t representing the translation matrix.
In this embodiment, a binocular image acquired by a binocular camera is used as an object to perform step detection and height estimation process description. Setting pixel depth
The internal reference of the camera is
The rotation matrix is
The translation matrix is
Focal length of camera
And principal point
。
In this embodiment, according to step S1, after the binocular vision image is acquired, the brightness of the image is adjusted using a dynamic white balance algorithm such as perfect reflection white balance, and the influence of the illumination intensity on the image quality is reduced.
In this embodiment, according to steps S2 to S3, Canny operator is used to extract the edge information of the image on the basis of the gray scale map, and Hough transform is used to screen out a line set from the edge information, from which an approximately horizontal line set is separated according to the slope attribute of the line.
In this embodiment, according to steps S4 to S5, coarse clustering and fine clustering are performed using the midpoint distance of the straight line, and the least square method is used to find the line segment sets constructing the steps from the original straight line set, and the straight line equation of the steps is fitted according to these line segment sets.
In this embodiment, according to step S6, a binocular disparity map is used to construct a pixel depth map of a left view, and pixel points lacking depth information in the depth map are filled up according to the color and distance information of pixels in the RGB image.
In this embodiment, according to step S7, a mapping relationship between a pixel coordinate system and a world coordinate system is constructed using the internal and external parameters of the left eye camera, and the points of the step linear equation constructed in step S5 are mapped to the world coordinate system, so as to obtain three-dimensional information of a step region, and height information of a single step is estimated using the three-dimensional information. Selecting n points from two adjacent step straight lines
And
. The vertical height in the camera space according to the mapping relation is as follows:
. The height h of the single-stage step is as follows:
Claims (7)
1. a step detection and single-stage step height estimation method based on binocular vision is characterized by comprising the following steps:
s1, acquiring an RGB image by using a binocular camera, and adjusting the brightness of the RGB image by using a dynamic white balance algorithm;
s2, converting the RGB image after brightness adjustment into a gray scale image, and extracting edge information of the gray scale image by using an edge detection algorithm;
s3, extracting line segment information from the edge information by using a line detection algorithm, and screening horizontal line segments from the line segment set according to the slope to form an original horizontal line segment set;
s4, clustering in the original horizontal line segment set according to the vertical distance relationship of the middle points of the line segments, and screening out a potential line segment set forming a step;
s5, carrying out subdivision clustering and regression of a step linear equation on the potential line segment set by using a least square method;
s6, constructing a pixel depth map of the monocular image by using a disparity map of a binocular image of the binocular camera, and performing information completion on pixel points with incomplete depth information by using the color and position information of pixels in the RGB image;
s7, projecting step pixels in the monocular image into a world coordinate system by using the internal and external parameters of the binocular camera and the depth information of the depth map, and calculating the height of the single-stage step according to the three-dimensional information in the world coordinate system;
in step S5, the potential line segment set constituting the step is centered on the midpoint (x)0，y0) X of0Sequencing, and performing subdivision clustering by using a least square method, specifically: firstly, from the potential line segment set, selecting the first line segment to calculate its straight line equation y-kx + b, and then calculating the midpoint (x) of the second line segment0，y0) Distance y to the equation of a straight line0-kx0-b |, if the distance is less than a set threshold, the two segments belong to the same class, the two segments are deleted from the set of potential segments, and the linear equation y ═ kx + b is refitted using the least squares method from the mid-points of the two segments; all segments are traversed in sequence until the set of potential segments is empty.
2. The binocular vision-based step detection and single-stage step height estimation method of claim 1, wherein in step S1, the brightness of the image is adjusted using a perfect reflection white balance algorithm.
3. The binocular vision based step detection and single-stage step height estimation method of claim 1, wherein in step S2, the RGB image is converted into a Gray map Gray using the following formula, and edge information of an object in the image is extracted using a Canny operator on the basis of the Gray map:
Gray＝0.299*R+0.587*G+0.114*B
where R, G, B respectively represent data of three color channels of an RGB image.
4. The binocular vision based step detection and single-stage step height estimation method of claim 1, wherein in step S3, line segment information is detected from the edge information using hough transform, resulting in a line segment set satisfying length requirements.
5. The binocular vision-based step detection and single-level step height estimation method of claim 1, wherein in step S4, coarse clustering is performed on the original horizontal line segment set according to the y-axis distance of the midpoint of the line segments, and a set of potential line segments constituting the step is selected according to the number of line segments included in each class and the class center.
6. The binocular vision-based step detection and single-step height estimation method of claim 1, wherein in step S7, the step pixels are projected to the world coordinate system as follows:
wherein, PuvIs a coordinate point of a pixel coordinate system, and u and v represent coordinates under the pixel coordinate system; k represents camera internal parameters, and P represents pixel points of a camera coordinate system; x, Y and Z represent coordinates in a camera coordinate system; r represents a rotation matrix, Xw,Yw,ZwRepresenting coordinates in the world coordinate system and t representing the translation matrix.
7. The binocular vision-based step detection and single-stage step height estimation method of claim 6, wherein in step S7, n points { xy ] are selected from each of two adjacent step straight lines in a world coordinate system11 xy12 … xy1nAnd { xy }21 xy22 … xy2n}, calculatingHeight h of the single step:
where d is the pixel depth, fyFocal length in the y-axis direction of the camera, cyIs the principal point of the y-axis direction of the camera.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011301586.7A CN112102347B (en) | 2020-11-19 | 2020-11-19 | Step detection and single-stage step height estimation method based on binocular vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011301586.7A CN112102347B (en) | 2020-11-19 | 2020-11-19 | Step detection and single-stage step height estimation method based on binocular vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112102347A CN112102347A (en) | 2020-12-18 |
| CN112102347B true CN112102347B (en) | 2021-02-12 |
Family
ID=73785538
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011301586.7A Active CN112102347B (en) | 2020-11-19 | 2020-11-19 | Step detection and single-stage step height estimation method based on binocular vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112102347B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112529903B (en) * | 2021-02-03 | 2022-01-28 | 德鲁动力科技(成都)有限公司 | Stair height and width visual detection method and device and robot dog |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202582535U (en) * | 2012-04-19 | 2012-12-05 | 刘瑜 | Active visual ranging system of movable platform |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102324099B (en) * | 2011-09-05 | 2013-07-03 | 广东工业大学 | Step edge detection method oriented to humanoid robot |
| CN108876799B (en) * | 2018-06-12 | 2021-07-02 | 杭州视氪科技有限公司 | Real-time step detection method based on binocular camera |
| CN110631529A (en) * | 2019-09-29 | 2019-12-31 | 苏州卡杰斯机电设备有限公司 | Step height detection mechanism |
-
2020
- 2020-11-19 CN CN202011301586.7A patent/CN112102347B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202582535U (en) * | 2012-04-19 | 2012-12-05 | 刘瑜 | Active visual ranging system of movable platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112102347A (en) | 2020-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110569704B (en) | Multi-strategy self-adaptive lane line detection method based on stereoscopic vision | |
| CN109360235B (en) | Hybrid depth estimation method based on light field data | |
| CN110264416B (en) | Sparse point cloud segmentation method and device | |
| CN109903331B (en) | Convolutional neural network target detection method based on RGB-D camera | |
| JP7205613B2 (en) | Image processing device, image processing method and program | |
| TW201724026A (en) | Generating a merged, fused three-dimensional point cloud based on captured images of a scene | |
| CN109410264B (en) | Front vehicle distance measuring method based on laser point cloud and image fusion | |
| US8867826B2 (en) | Disparity estimation for misaligned stereo image pairs | |
| US7822224B2 (en) | Terrain map summary elements | |
| CN105809716B (en) | Foreground extraction method integrating superpixel and three-dimensional self-organizing background subtraction method | |
| CN113034568A (en) | Machine vision depth estimation method, device and system | |
| CN111027415B (en) | Vehicle detection method based on polarization image | |
| CN110738731A (en) | 3D reconstruction method and system for binocular vision | |
| CN113205604A (en) | Feasible region detection method based on camera and laser radar | |
| CN117036641A (en) | Road scene three-dimensional reconstruction and defect detection method based on binocular vision | |
| CN112102347B (en) | Step detection and single-stage step height estimation method based on binocular vision | |
| CN114463303A (en) | Road target detection method based on fusion of binocular camera and laser radar | |
| CN113569896A (en) | Computer-implemented method for 3D positioning of objects based on image and depth data | |
| CN117058183A (en) | Image processing method and device based on double cameras, electronic equipment and storage medium | |
| CN109242900B (en) | Focal plane positioning method, processing device, focal plane positioning system and storage medium | |
| CN114935316B (en) | Standard depth image generation method based on optical tracking and monocular vision | |
| CN113808185B (en) | Image depth recovery method, electronic device and storage medium | |
| CN112308776B (en) | Method for solving occlusion and error mapping image sequence and point cloud data fusion | |
| CN111630569B (en) | Binocular matching method, visual imaging device and device with storage function | |
| Brunken et al. | Incorporating Plane-Sweep in Convolutional Neural Network Stereo Imaging for Road Surface Reconstruction. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |