CN114140507A - Depth estimation method, device and equipment integrating laser radar and binocular camera - Google Patents

Depth estimation method, device and equipment integrating laser radar and binocular camera Download PDF

Info

Publication number
CN114140507A
CN114140507A CN202111261395.7A CN202111261395A CN114140507A CN 114140507 A CN114140507 A CN 114140507A CN 202111261395 A CN202111261395 A CN 202111261395A CN 114140507 A CN114140507 A CN 114140507A
Authority
CN
China
Prior art keywords
radar
disparity map
binocular camera
image information
fusing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111261395.7A
Other languages
Chinese (zh)
Inventor
李恩
徐光耀
杨国栋
梁自泽
景奉水
龙晓宇
陈铭浩
田雨农
郭锐
李勇
刘海波
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Institute of Automation of Chinese Academy of Science
State Grid Shandong Electric Power Co Ltd
Original Assignee
Institute of Automation of Chinese Academy of Science
State Grid Shandong Electric Power Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Institute of Automation of Chinese Academy of Science, State Grid Shandong Electric Power Co Ltd filed Critical Institute of Automation of Chinese Academy of Science
Priority to CN202111261395.7A priority Critical patent/CN114140507A/en
Publication of CN114140507A publication Critical patent/CN114140507A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/06—Topological mapping of higher dimensional structures onto lower dimensional surfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/97—Determining parameters from multiple pictures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10032—Satellite or aerial image; Remote sensing
    • G06T2207/10044—Radar image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20228—Disparity calculation for image-based rendering

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a depth estimation method, a device and equipment for fusing a laser radar and a binocular camera, wherein the method comprises the following steps: acquiring scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera; projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map; carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map; and fusing the up-sampling radar disparity map and the image information to obtain a target disparity map, so that the method can effectively adapt to scenes with difficult control of conditions such as outdoor illumination, object textures and the like, and can effectively improve matching accuracy.

Description

Depth estimation method, device and equipment integrating laser radar and binocular camera
Technical Field
The invention relates to the technical field of three-dimensional reconstruction, in particular to a depth estimation method, device and equipment integrating a laser radar and a binocular camera.
Background
Many tasks require the acquisition of depth information of the environment, especially for mobile robotic and computer vision applications, such as three-dimensional reconstruction, SLAM, autopilot, and robotic path planning. Depth information measurement is a fundamental problem in three-dimensional vision. In recent years, more and more scholars have started to focus on this area and have proposed several new approaches. These methods are mainly divided into two categories: a conventional method and a deep learning method. Deep learning methods can also be divided into supervised methods and unsupervised methods. The supervised approach requires a large amount of real depth data as the ground truth. These ground truths require high precision structured light sensors or lidar to acquire, which is very costly. However, the unsupervised approach does not require these ground truths, and the basic idea is to use photometric errors for guiding the trained left and right image neural networks.
However, the unsupervised approach also requires a large number of high quality stereo images to train the parameters. On the other hand, there are some conventional techniques to achieve depth estimation, but they generally have some limitations. For example, a disparity map can be obtained quickly by an RGB-D camera, while the quality of the view difference depends on the outdoor environment. Since sunlight may affect the structured light sensor of an RGB-D camera. At the same time, light also has an effect on the stereo algorithm. For example, a partial image may be over-exposed for several reasons. In addition, the stereo algorithm has certain requirements on scenes in the picture. For example, a scene needs to have rich texture and appropriate lighting conditions.
Therefore, the binocular stereo matching method has relatively low matching accuracy in scenes where conditions such as challenging outdoor lighting and textures of objects are difficult to control.
Disclosure of Invention
The invention provides a depth estimation method, a depth estimation device and depth estimation equipment integrating a laser radar and a binocular camera, which are used for overcoming the defects of dependence of a binocular stereo matching method on target textures, illumination change and other factors in the prior art and effectively improving matching accuracy.
The invention provides a depth estimation method fusing a laser radar and a binocular camera, which comprises the following steps:
acquiring scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera;
calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera;
projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map;
carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map;
and fusing the up-sampling radar disparity map with the image information to obtain a target disparity map.
According to the depth estimation method fusing the laser radar and the binocular camera, before the bilinear interpolation is carried out on the radar disparity map to obtain the up-sampling radar disparity map, the depth estimation method further comprises the following steps:
and identifying abnormal projection points in the radar disparity map, and clearing the abnormal projection points.
According to the depth estimation method fusing the laser radar and the binocular camera, provided by the invention, the identification of the abnormal projection point in the radar disparity map comprises the following steps:
completing the scanning line of the laser radar;
and when the radar depth of a target scanning line is greater than the first radar depth of a first scanning line and the second radar depth of a second scanning line which are adjacent to the scanning line, determining that the target scanning line is composed of abnormal projection points.
According to the depth estimation method fusing the laser radar and the binocular camera, provided by the invention, the calibration of the scene depth information and the image information is carried out to obtain the pose transformation matrix of the laser radar and the binocular camera, and the method comprises the following steps:
calibrating a first spatial relative position relation of a first camera and a second camera of the binocular camera according to the image information;
calibrating a second spatial relative position relation between the laser radar and the binocular camera according to the scene depth information and the image information;
and constructing a pose transformation matrix of the laser radar and the binocular camera according to the first spatial relative position relation and the second spatial relative position relation.
According to the depth estimation method fusing the laser radar and the binocular camera, provided by the invention, the method for fusing the up-sampling radar disparity map and the image information to obtain the target disparity map comprises the following steps:
determining a tilt window model parameter according to the up-sampling radar disparity map and the image information;
determining an inclined plane according to the inclined window model parameters, and determining a matching cost determination rule of any pixel in the inclined plane;
and determining the disparity map with the minimized cost as a target disparity map according to the matching cost determination rule.
According to the depth estimation method fusing the laser radar and the binocular camera, the method for determining the disparity map with the minimized cost as the target disparity map according to the matching cost determination rule comprises the following steps:
carrying out random initialization on the inclined plane to obtain an initialized plane model;
and performing parallax transmission based on the initialized plane model, and determining a parallax image corresponding to the plane with the minimum matching cost when the parallax transmission reaches a preset number as a target parallax image.
According to the depth estimation method fusing the laser radar and the binocular camera, after parallax propagation, the method further comprises the following steps:
and optimizing the plane with the minimum matching cost when the parallax transmission reaches the preset times based on the predetermined maximum parallax and normal vector change range.
The invention also provides a depth estimation device fusing the laser radar and the binocular camera, which comprises the following components:
the system comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring scene depth information and image information through a radar camera, and the radar camera comprises a laser radar and a binocular camera;
the calibration module is used for calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera;
the projection module is used for projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map;
the up-sampling module is used for carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map;
and the fusion module is used for fusing the up-sampling radar disparity map with the image information to obtain a target disparity map.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the steps of the depth estimation method for fusing the laser radar and the binocular camera.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method for depth estimation fusing a lidar and a binocular camera as described in any of the above.
The present invention also provides a computer program product comprising a computer program which, when executed by a processor, carries out the steps of the method of depth estimation incorporating a lidar and a binocular camera as described in any one of the above.
The invention provides a depth estimation method, a depth estimation device and depth estimation equipment integrating a laser radar and a binocular camera, wherein the method comprises the following steps: acquiring scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera; projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map; carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map; the up-sampling radar disparity map and the image information are fused to obtain a target disparity map, the target disparity map can effectively adapt to scenes with difficult control of conditions such as outdoor illumination and object textures, can effectively improve matching accuracy, can quickly and robustly estimate depth values in a complex environment, and has high practicability and engineering value.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a depth estimation method for fusing a laser radar and a binocular camera according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a mounting position relationship of a radar camera provided in an embodiment of the present invention;
FIG. 3 is a schematic diagram of the definition of radar scan lines and points between scan lines provided by an embodiment of the present invention;
FIG. 4 is a schematic diagram of a tilt window and a parallel window provided by an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a depth estimation device incorporating a laser radar and a binocular camera according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The depth estimation method, device and equipment for fusing the laser radar and the binocular camera according to the present invention are described below with reference to fig. 1 to 6.
Fig. 1 is a schematic flow chart of a depth estimation method for fusing a laser radar and a binocular camera according to an embodiment of the present invention; fig. 2 is a schematic diagram of a mounting position relationship of a radar camera provided in an embodiment of the present invention; FIG. 3 is a schematic diagram of the definition of radar scan lines and points between scan lines provided by an embodiment of the present invention; fig. 4 is a schematic diagram of a tilt window and a parallel window provided by an embodiment of the present invention.
As shown in fig. 1, a depth estimation method fusing a laser radar and a binocular camera provided in an embodiment of the present invention includes the following steps:
101. scene depth information and image information are collected through a radar camera, and the radar camera comprises a laser radar and a binocular camera.
Specifically, the radar camera refers to a laser radar and a binocular camera, wherein the laser radar can be a mechanical rotation type multi-line laser radar, a solid multi-line laser radar can also be used, and a single line laser radar can also be used to generate a plurality of scanning lines through registration. As shown in fig. 2 for the installation position relation schematic diagram of LiDAR and binocular camera that this embodiment provided, LiDAR represents LiDAR, and Zed represents binocular camera, and wherein, binocular camera and LiDAR's relative position can change, and according to the different visual angles of binocular camera, the position change range is also different, as long as radar data can project to required position can. The specific manner of collecting the scene depth information may be that the laser radar determines the distance between the object to be measured and the test point by emitting a laser beam to the object to be measured, receiving a reflected wave of the laser beam, and recording the time difference. The binocular camera is composed of two ordinary cameras, and the scene depth is obtained by utilizing an optical principle and a triangulation principle.
102. And calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera.
Specifically, the scene depth information and the image information are calibrated to obtain a pose transformation matrix of the laser radar and the binocular camera, and the pose transformation matrix can be divided into two parts, one part is used for calibrating the binocular camera, and the other part is used for calibrating the radar camera. The calibration of the binocular camera may be to calibrate a first spatial relative position relationship between a first camera and a second camera of the binocular camera according to the image information, that is, to obtain image information of the two cameras of the binocular camera, and then calculate an internal parameter, a lens distortion parameter, and a spatial relative position relationship between the two cameras through a calibration algorithm. The process of calibrating the radar camera may be to calibrate a second spatial relative position relationship between the laser radar and the binocular camera according to the scene depth information and the image information, that is, to obtain a spatial relative position relationship between the laser radar and the left (right) camera, that is, the first camera and the second camera, through an algorithm according to the scene depth information and the image information. And then constructing a pose transformation matrix of the laser radar and the binocular camera according to the first space relative position relation and the second space relative position relation.
103. And projecting the scene depth information onto an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map.
Specifically, the radar disparity map can be obtained by projecting radar points onto an image plane of image information of the camera through a relative position relation of a radar camera, internal parameters of the camera and distortion parameters based on a pose transformation matrix.
104. And carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map.
After the radar disparity map is obtained, bilinear interpolation is carried out on the radar disparity map to fill gaps among the scanning lines, and the up-sampling radar disparity map is obtained. The formula of bilinear interpolation is shown as formula (1):
Figure BDA0003325859290000071
where π (x, y) is the disparity value at coordinate (x, y). y isdownAnd yupIs the point on the radar scan line in the y-axis direction of (x, y). As shown in FIG. 3, ydownIs the y-axis coordinate on the scan line below point (x, y). In the same way, yupIs the y-axis coordinate on the scan line above point (x, y). If there is no radar scan line above or below a pixel, then a global maximum is assigned to the pixel.
105. And fusing the up-sampling radar disparity map with the image information to obtain a target disparity map.
After up-sampling is carried out to obtain an up-sampling radar disparity map, the up-sampling radar disparity map and RGB information of the image are fused to obtain an optimal disparity map, namely a target disparity map. The specific mode can be that the parameters of the tilt window model are determined according to the up-sampling radar disparity map and the image information; determining a tilt plane according to the tilt window model parameters, wherein the tilt window is formed by (n) pixel coordinates (x, y), parallax d and a normal vector n of the pointx,ny,nz) Determining a unique inclined plane
Figure BDA0003325859290000081
If the normal vector n is 0, the tilted window is degenerated into a parallel window; the plane can be expressed as formula (2)
Figure BDA0003325859290000082
Wherein (x, y, d) is a point on the parallax plane,
Figure BDA0003325859290000083
and
Figure BDA0003325859290000084
are three parameters of the inclined plane; these three parameters are determined by the following equations (3) (4) (5):
Figure BDA0003325859290000085
Figure BDA0003325859290000086
Figure BDA0003325859290000087
as shown in fig. 4, for comparing the tilt window and the parallel window, it can be clearly understood from fig. 4 that the tilt window has a better adaptation effect to the tilt plane. Wherein the slow support window represents a tilt window, and the front-parallel window represents a parallel window.
After all the parameters of the inclined plane are determined, determining a matching cost determination rule of any pixel in the inclined plane, which can define the matching cost of the pixel p in the plane, and the formula is shown in (6):
Figure BDA0003325859290000088
wherein,
Figure BDA0003325859290000089
is the plane of the pixel p, wpIs a square window centered on pixel p, (p ', d') is the pixels and disparity within the window of pixel p.
The formula ω (p, p') can be expressed as formula (7):
Figure BDA0003325859290000091
wherein | Ip-Ip′II is L of pixel p and pixel p' in RGB space1Distance, gamma is a custom parameter, and in the algorithm, the value of gamma is set to 10;
formula (II)
Figure BDA0003325859290000092
Represented by formula (8):
ρ(p′,q)＝α·min(‖Ip′-Iq‖,τcol)+β·min(|πp′-d′|,τdisp) (8)
wherein q represents
Figure BDA0003325859290000093
Representing the same name point in another view; min () represents the minimum value out of two values; alpha and beta are the ratio of color to parallaxThe sum of them is 1; tau iscolAnd τdispRespectively, color and disparity truncation costs; when shielding exists, the matching cost is prevented from being overlarge by truncation cost; II | Ip′-IqII denotes L of pixel p' and pixel q in RGB space1A distance; l pip′-d ' | represents the disparity of pixel p ' in the upsampled disparity map and the pixel p ' is in-plane
Figure BDA0003325859290000094
Absolute difference of the intermediate parallax.
Then, the disparity map with the minimized cost can be determined as the target disparity map according to the matching cost determination rule.
According to the matching cost determination rule, determining the disparity map with the minimized cost as a target disparity map comprises the following steps: randomly initializing the inclined plane to obtain an initialized plane model, that is, after the inclined plane model is determined, the parameters x, y, d, n of the modelx,ny,nzNeeds to be determined; if there is a disparity d ' in the upsampled disparity map for pixel p ' (x, y), assigning the value of d ' to d; if no such value exists, assigning a value between the maximum and minimum disparity to d; simultaneously selecting a random unit normal vector to assign to nx,ny,nz。
Then, carrying out parallax transmission based on the initialized plane model, distributing a plane to each pixel, and searching a plane with the minimum matching cost on each pixel; the adjacent pixels may have similar planes, and a plane with lower matching cost can be found by expanding the planes of the adjacent pixels; there are many ways of propagation, such as left to right, top to bottom, top left to bottom right, etc.; the propagation mode used by the invention can be expressed as follows: in odd-numbered passes, the propagation direction propagates from the upper left to the lower right, and in even-numbered passes, the propagation direction propagates from the lower right to the upper left. And during an odd number of times, if
Figure BDA0003325859290000101
Figure BDA0003325859290000102
Then
Figure BDA0003325859290000103
Figure BDA0003325859290000104
In the even-numbered process, the pixel p needs to be associated with pupAnd pleftComparing; and if the specified times are finished, the whole process is ended, namely the disparity map corresponding to the plane with the minimum matching cost when the disparity propagation reaches the preset times is determined as the target disparity map.
In order to ensure optimization of the disparity map, after disparity propagation, the method further includes: and optimizing the plane with the minimum matching cost when the parallax propagation reaches the preset times based on the predetermined maximum parallax and normal vector change range. Specifically, after the disparity propagation process is performed, in order to further reduce the matching cost, plane optimization needs to be performed; the optimization plane is positioned between the two propagation processes, and a new plane with lower aggregation cost is provided for the propagation processes as much as possible; in the process, a plane needs to use a point (x, y, d) and a normal vector (n)x,ny,nz) Represents; at the same time
Figure BDA0003325859290000105
And
Figure BDA0003325859290000106
a range of variation defined as the maximum disparity and normal vectors; if the pixel is located between two radar scan lines, then
Figure BDA0003325859290000107
If not, then
Figure BDA0003325859290000108
Is the maximum parallax of the image; in a similar manner to that described above,
Figure BDA0003325859290000109
is also the maximum allowable range of the vector; from
Figure BDA00033258592900001010
In randomly selecting deltadD' ═ d + Δ is calculatedd；
Figure BDA00033258592900001011
Is also at
Figure BDA00033258592900001012
A random value within the range, and
Figure BDA00033258592900001013
can be expressed as
Figure BDA00033258592900001014
Wherein the unit () functions as a unitization; now a new plane
Figure BDA00033258592900001015
Is determined if
Figure BDA00033258592900001016
Then plane surface
Figure BDA00033258592900001017
Is used to replace
Figure BDA00033258592900001018
This is an iterative process that, during the first calculation,
Figure BDA00033258592900001019
is arranged as
Figure BDA0003325859290000111
And is
Figure BDA0003325859290000112
During the course of each iteration of the process,
Figure BDA0003325859290000113
is updated to
Figure BDA0003325859290000114
And is
Figure BDA0003325859290000115
If it is not
Figure BDA0003325859290000116
Optimized plane
Figure BDA0003325859290000117
Then this indicates completion; and then continuing to perform parallax transmission processing to obtain a final target parallax image.
The invention provides a depth estimation method fusing a laser radar and a binocular camera, which comprises the following steps: acquiring scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera; projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map; carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map; the up-sampling radar disparity map and the image information are fused to obtain a target disparity map, the target disparity map can effectively adapt to scenes with difficult control of conditions such as outdoor illumination and object textures, can effectively improve matching accuracy, can quickly and robustly estimate depth values in a complex environment, has high practicability and engineering value, can overcome the defects that a traditional method is sensitive to noise and a depth learning method is relatively dependent on a data set, and can be suitable for outdoor scenes and other large scenes.
Further, in this embodiment, before performing bilinear interpolation on the radar disparity map to obtain the up-sampling radar disparity map, the method may further include: and identifying abnormal projection points in the radar disparity map, and clearing the abnormal projection points. In the radar projection process, an abnormal value clearing module is needed to clear the abnormal value, because the relative position relationship between the camera and the radar can cause wrong projection points. Identifying abnormal projection points in the radar disparity map, wherein the scanning lines of the laser radar are supplemented firstly because the radar points on the scanning lines of the radar are discontinuous; when the radar depth of the target scanning line is greater than the first radar depth of the first scanning line and the second radar depth of the second scanning line which are adjacent to the scanning lines, the target scanning line is determined to be composed of abnormal projection points, namely when one radar scanning line with a larger depth value appears between the two scanning lines with a shallower radar depth, the scanning line is considered to be composed of the abnormal projection points, and the scanning line is eliminated, so that abnormal values in the parallax image projected by the radar can be effectively eliminated.
Based on the same general inventive concept, the present application also provides a depth estimation device fusing a laser radar and a binocular camera, which is described below, and the depth estimation device fusing the laser radar and the binocular camera described below and the depth estimation method fusing the laser radar and the binocular camera described above may be referred to correspondingly.
Fig. 5 is a schematic structural diagram of a depth estimation device that integrates a laser radar and a binocular camera according to an embodiment of the present invention.
As shown in fig. 5, a depth estimation apparatus fusing a laser radar and a binocular camera according to an embodiment of the present invention includes:
the acquisition module 51 is used for acquiring scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera;
the calibration module 52 is configured to calibrate the scene depth information and the image information to obtain pose transformation matrices of the laser radar and the binocular camera;
the projection module 53 is configured to project the scene depth information onto an image plane of the image information according to the pose transformation matrix, so as to obtain a radar disparity map;
an upsampling module 54, configured to perform bilinear interpolation on the radar disparity map to obtain an upsampled radar disparity map;
and the fusion module 55 is configured to fuse the up-sampling radar disparity map with the image information to obtain a target disparity map.
The depth estimation device fusing the laser radar and the binocular camera provided by the embodiment of the invention comprises: acquiring scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera; projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map; carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map; the up-sampling radar disparity map and the image information are fused to obtain a target disparity map, the target disparity map can effectively adapt to scenes with difficult control of conditions such as outdoor illumination and object textures, can effectively improve matching accuracy, can quickly and robustly estimate depth values in a complex environment, and has high practicability and engineering value.
Further, the present embodiment further includes an exception handling module, configured to:
and identifying abnormal projection points in the radar disparity map, and clearing the abnormal projection points.
Further, the embodiment further includes an exception handling module, specifically configured to:
completing the scanning line of the laser radar;
and when the radar depth of a target scanning line is greater than the first radar depth of a first scanning line and the second radar depth of a second scanning line which are adjacent to the scanning line, determining that the target scanning line is composed of abnormal projection points.
Further, the calibration module 52 in this embodiment is specifically configured to:
calibrating a first spatial relative position relation of a first camera and a second camera of the binocular camera according to the image information;
calibrating a second spatial relative position relation between the laser radar and the binocular camera according to the scene depth information and the image information;
and constructing a pose transformation matrix of the laser radar and the binocular camera according to the first spatial relative position relation and the second spatial relative position relation.
Further, the fusion module 55 in this embodiment is specifically configured to:
determining a tilt window model parameter according to the up-sampling radar disparity map and the image information;
determining an inclined plane according to the inclined window model parameters, and determining a matching cost determination rule of any pixel in the inclined plane;
and determining the disparity map with the minimized cost as a target disparity map according to the matching cost determination rule.
Further, the fusion module 55 in this embodiment is further specifically configured to:
carrying out random initialization on the inclined plane to obtain an initialized plane model;
and performing parallax transmission based on the initialized plane model, and determining a parallax image corresponding to the plane with the minimum matching cost when the parallax transmission reaches a preset number as a target parallax image.
Further, the fusion module 55 in this embodiment is further specifically configured to:
and optimizing the plane with the minimum matching cost when the parallax transmission reaches the preset times based on the predetermined maximum parallax and normal vector change range.
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
As shown in fig. 6, the electronic device may include: a processor (processor)610, a communication Interface (Communications Interface)620, a memory (memory)630 and a communication bus 640, wherein the processor 610, the communication Interface 620 and the memory 630 communicate with each other via the communication bus 640. Processor 610 may invoke logic instructions in memory 630 to perform a method of depth estimation that merges lidar and a binocular camera, the method comprising: acquiring scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera; projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map; carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map; and fusing the up-sampling radar disparity map with the image information to obtain a target disparity map.
In addition, the logic instructions in the memory 630 may be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the depth estimation method for fusing a lidar and a binocular camera, which is provided by the above methods, the method comprising: acquiring scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera; projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map; carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map; and fusing the up-sampling radar disparity map with the image information to obtain a target disparity map.
In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a depth estimation method for fusing a lidar and a binocular camera provided by the above methods, the method including: acquiring scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera; projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map; carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map; and fusing the up-sampling radar disparity map with the image information to obtain a target disparity map.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A depth estimation method fusing a laser radar and a binocular camera is characterized by comprising the following steps:
acquiring scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera;
calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera;
projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map;
carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map;
and fusing the up-sampling radar disparity map with the image information to obtain a target disparity map.
2. The method for estimating depth by fusing lidar and a binocular camera according to claim 1, wherein before the bilinear interpolation is performed on the radar disparity map to obtain the up-sampled radar disparity map, the method further comprises:
and identifying abnormal projection points in the radar disparity map, and clearing the abnormal projection points.
3. The method for depth estimation by fusing a lidar and a binocular camera according to claim 2, wherein the identifying the abnormal projection point in the radar disparity map comprises:
completing the scanning line of the laser radar;
and when the radar depth of a target scanning line is greater than the first radar depth of a first scanning line and the second radar depth of a second scanning line which are adjacent to the scanning line, determining that the target scanning line is composed of abnormal projection points.
4. The depth estimation method fusing a lidar and a binocular camera according to claim 1, wherein the calibrating the scene depth information and the image information to obtain a pose transformation matrix of the lidar and the binocular camera comprises:
calibrating a first spatial relative position relation of a first camera and a second camera of the binocular camera according to the image information;
calibrating a second spatial relative position relation between the laser radar and the binocular camera according to the scene depth information and the image information;
and constructing a pose transformation matrix of the laser radar and the binocular camera according to the first spatial relative position relation and the second spatial relative position relation.
5. The method for depth estimation by fusing a lidar and a binocular camera according to claim 1, wherein the fusing the up-sampling radar disparity map with the image information to obtain a target disparity map comprises:
determining a tilt window model parameter according to the up-sampling radar disparity map and the image information;
determining an inclined plane according to the inclined window model parameters, and determining a matching cost determination rule of any pixel in the inclined plane;
and determining the disparity map with the minimized cost as a target disparity map according to the matching cost determination rule.
6. The method for depth estimation by fusing a lidar and a binocular camera according to claim 5, wherein the determining the disparity map with the minimized cost as the target disparity map according to the matching cost determination rule comprises:
carrying out random initialization on the inclined plane to obtain an initialized plane model;
and performing parallax transmission based on the initialized plane model, and determining a parallax image corresponding to the plane with the minimum matching cost when the parallax transmission reaches a preset number as a target parallax image.
7. The depth estimation method combining a lidar and a binocular camera according to claim 6, wherein after the performing the disparity propagation, further comprising:
and optimizing the plane with the minimum matching cost when the parallax transmission reaches the preset times based on the predetermined maximum parallax and normal vector change range.
8. A depth estimation device fusing a laser radar and a binocular camera, comprising:
the system comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring scene depth information and image information through a radar camera, and the radar camera comprises a laser radar and a binocular camera;
the calibration module is used for calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera;
the projection module is used for projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map;
the up-sampling module is used for carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map;
and the fusion module is used for fusing the up-sampling radar disparity map with the image information to obtain a target disparity map.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of the method of depth estimation fusing a lidar and a binocular camera according to any of claims 1 to 7.
10. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor, performs the steps of the method for depth estimation fusing a lidar and a binocular camera according to any one of claims 1 to 7.
CN202111261395.7A 2021-10-28 2021-10-28 Depth estimation method, device and equipment integrating laser radar and binocular camera Pending CN114140507A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111261395.7A CN114140507A (en) 2021-10-28 2021-10-28 Depth estimation method, device and equipment integrating laser radar and binocular camera

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111261395.7A CN114140507A (en) 2021-10-28 2021-10-28 Depth estimation method, device and equipment integrating laser radar and binocular camera

Publications (1)

Publication Number Publication Date
CN114140507A true CN114140507A (en) 2022-03-04

Family

ID=80395652

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111261395.7A Pending CN114140507A (en) 2021-10-28 2021-10-28 Depth estimation method, device and equipment integrating laser radar and binocular camera

Country Status (1)

Country Link
CN (1) CN114140507A (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130278631A1 (en) * 2010-02-28 2013-10-24 Osterhout Group, Inc. 3d positioning of augmented reality information
CN109035309A (en) * 2018-07-20 2018-12-18 清华大学苏州汽车研究院（吴江） Pose method for registering between binocular camera and laser radar based on stereoscopic vision
CN111325794A (en) * 2020-02-23 2020-06-23 哈尔滨工业大学 Visual simultaneous localization and map construction method based on depth convolution self-encoder
CN112233163A (en) * 2020-12-14 2021-01-15 中山大学 Depth estimation method and device for laser radar stereo camera fusion and medium thereof
CN112902874A (en) * 2021-01-19 2021-06-04 中国汽车工程研究院股份有限公司 Image acquisition device and method, image processing method and device and image processing system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130278631A1 (en) * 2010-02-28 2013-10-24 Osterhout Group, Inc. 3d positioning of augmented reality information
CN109035309A (en) * 2018-07-20 2018-12-18 清华大学苏州汽车研究院（吴江） Pose method for registering between binocular camera and laser radar based on stereoscopic vision
CN111325794A (en) * 2020-02-23 2020-06-23 哈尔滨工业大学 Visual simultaneous localization and map construction method based on depth convolution self-encoder
CN112233163A (en) * 2020-12-14 2021-01-15 中山大学 Depth estimation method and device for laser radar stereo camera fusion and medium thereof
CN112902874A (en) * 2021-01-19 2021-06-04 中国汽车工程研究院股份有限公司 Image acquisition device and method, image processing method and device and image processing system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
李海标;时君;田春月;: "一种基于深度特征的室外环境下激光地图辅助视觉定位方法", 科学技术与工程, no. 13, 8 May 2020 (2020-05-08) *

Similar Documents

Publication Publication Date Title
CN109461181B (en) Depth image acquisition method and system based on speckle structured light
US10237532B2 (en) Scan colorization with an uncalibrated camera
US10867407B2 (en) System and methods for extrinsic calibration of cameras and diffractive optical elements
CN109660783B (en) Virtual reality parallax correction
US20160292882A1 (en) Method for estimating the speed of movement of a camera
US9437034B1 (en) Multiview texturing for three-dimensional models
CN110119679B (en) Object three-dimensional information estimation method and device, computer equipment and storage medium
CN111563921A (en) Underwater point cloud acquisition method based on binocular camera
CN111123242B (en) Combined calibration method based on laser radar and camera and computer readable storage medium
CN109640066B (en) Method and device for generating high-precision dense depth image
US9147279B1 (en) Systems and methods for merging textures
CN110648274A (en) Fisheye image generation method and device
US12052406B2 (en) Display apparatus and operating method of the same
US11651506B2 (en) Systems and methods for low compute high-resolution depth map generation using low-resolution cameras
CN110619660A (en) Object positioning method and device, computer readable storage medium and robot
JP2020042503A (en) Three-dimensional symbol generation system
CN109118533B (en) Depth information processing method, device and equipment
CN112184793B (en) Depth data processing method and device and readable storage medium
CN117237789A (en) Method for generating texture information point cloud map based on panoramic camera and laser radar fusion
CN111866493B (en) Image correction method, device and equipment based on head-mounted display equipment
JP7315216B2 (en) Corrected Distance Calculation Device, Corrected Distance Calculation Program, and Corrected Distance Calculation Method
CN114140507A (en) Depth estimation method, device and equipment integrating laser radar and binocular camera
CN117058183A (en) Image processing method and device based on double cameras, electronic equipment and storage medium
CN113658260B (en) Robot pose calculation method, system, robot and storage medium
CN112487893B (en) Three-dimensional target identification method and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination