CN114140507A - Depth estimation method, device and device for fusion of lidar and binocular camera - Google Patents

Abstract

The invention provides a depth estimation method, a device and equipment for fusing a laser radar and a binocular camera, wherein the method comprises the following steps: acquiring scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera; calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera; projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map; carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map; and fusing the up-sampling radar disparity map and the image information to obtain a target disparity map, so that the method can effectively adapt to scenes with difficult control of conditions such as outdoor illumination, object textures and the like, and can effectively improve matching accuracy.

Description

Depth estimation method, device and device for fusion of lidar and binocular camera

technical field

The present invention relates to the technical field of three-dimensional reconstruction, and in particular, to a depth estimation method, device and device integrating laser radar and binocular camera.

Background technique

Many tasks require obtaining depth information of the environment, especially mobile robotics and computer vision applications such as 3D reconstruction, SLAM, autonomous driving, and robotic path planning. Depth information measurement is a fundamental problem in 3D vision. In recent years, more and more scholars have begun to pay attention to this field and proposed several new methods. These methods are mainly divided into two categories: traditional methods and deep learning methods. Deep learning methods can also be divided into supervised methods and unsupervised methods. Supervised methods require a large amount of real depth data as ground truth. These ground truths require high-precision structured light sensors or lidars to obtain, and the cost is very high. However, unsupervised methods do not require these ground truths, and the basic idea is to use the photometric error to guide the training of left and right image neural networks.

But unsupervised methods also require a large number of high-quality stereo images to train parameters. On the other hand, there are some conventional techniques for implementing depth estimation, but they usually have some limitations. For example, an RGB-D camera can quickly get a disparity map, and the quality of the disparity depends on the outdoor environment. Because sunlight affects the structured light sensor of an RGB-D camera. At the same time, light also has an impact on the stereo algorithm. For example, parts of the image may be overexposed for a number of reasons. In addition, the stereo algorithm has certain requirements for the scene in the picture. For example, the scene needs to have rich textures and suitable lighting conditions.

Therefore, the matching accuracy of the binocular stereo matching method is relatively low for challenging outdoor scenes where conditions such as lighting and object texture are difficult to control.

SUMMARY OF THE INVENTION

The present invention provides a depth estimation method, device and equipment integrating laser radar and binocular camera, which are used to solve the defects of the binocular stereo matching method in the prior art, such as dependence on target texture and illumination changes, and effectively improve the matching accuracy. Spend.

The present invention provides a depth estimation method that integrates lidar and binocular cameras, including:

Collect scene depth information and image information through a radar camera, the radar camera includes a lidar and a binocular camera;

calibrating the scene depth information and the image information to obtain the pose transformation matrix of the lidar and the binocular camera;

According to the pose transformation matrix, project the scene depth information onto the image plane of the image information to obtain a radar disparity map;

performing bilinear interpolation on the radar disparity map to obtain an up-sampled radar disparity map;

The up-sampled radar disparity map is fused with the image information to obtain a target disparity map.

According to a depth estimation method for fusion of lidar and binocular camera provided by the present invention, before performing bilinear interpolation on the radar disparity map to obtain the up-sampled radar disparity map, the method further includes:

Identify abnormal projection points in the radar disparity map, and remove the abnormal projection points.

According to a depth estimation method that integrates lidar and binocular cameras provided by the present invention, the identifying abnormal projection points in the radar disparity map includes:

Completing the scan lines of the lidar;

When the radar depth of the target scan line is greater than the first radar depth of the first scan line and the second radar depth of the second scan line adjacent to the scan line, it is determined that the target scan line is composed of abnormal projection points.

According to a depth estimation method for fusion of lidar and binocular camera provided by the present invention, the scene depth information and the image information are calibrated to obtain the pose transformation of the lidar and the binocular camera matrix, including:

According to the image information, calibrating the first relative positional relationship in space between the first camera and the second camera of the binocular camera;

According to the scene depth information and the image information, calibrating the second spatial relative position relationship between the lidar and the binocular camera;

According to the first spatial relative position relationship and the second spatial relative position relationship, a pose transformation matrix of the lidar and the binocular camera is constructed.

According to a depth estimation method for fusing a lidar and a binocular camera provided by the present invention, the upsampling radar disparity map and the image information are fused to obtain a target disparity map, including:

determining the oblique window model parameters according to the up-sampled radar disparity map and the image information;

Determine an inclined plane according to the inclined window model parameters, and determine a matching cost determination rule for any pixel in the inclined plane;

According to the matching cost determination rule, the disparity map that minimizes the cost is determined as the target disparity map.

According to a depth estimation method for fusion of lidar and binocular camera provided by the present invention, determining the disparity map that minimizes the cost as the target disparity map according to the matching cost determination rule, including:

Perform random initialization on the inclined plane to obtain an initialization plane model;

Based on the initialized plane model, disparity propagation is performed, and when the disparity propagation reaches a preset number of times, the disparity map corresponding to the plane with the smallest matching cost is determined as the target disparity map.

According to a depth estimation method for fusion of lidar and binocular camera provided by the present invention, after the parallax propagation is performed, the method further includes:

Based on the predetermined maximum disparity and normal vector variation ranges, the plane with the smallest matching cost is optimized when the disparity propagation reaches a preset number of times.

The present invention also provides a depth estimation device integrating lidar and binocular camera, including:

an acquisition module, used for acquiring scene depth information and image information through a radar camera, where the radar camera includes a lidar and a binocular camera;

a calibration module, configured to calibrate the scene depth information and the image information to obtain the pose transformation matrix of the lidar and the binocular camera;

a projection module, configured to project the scene depth information onto the image plane of the image information according to the pose transformation matrix to obtain a radar disparity map;

an up-sampling module for performing bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map;

A fusion module, configured to fuse the up-sampled radar disparity map with the image information to obtain a target disparity map.

The present invention also provides an electronic device, including a memory, a processor and a computer program stored in the memory and running on the processor, the processor implements the fusion lidar as described in any one of the above when the processor executes the program and steps of the depth estimation method for binocular cameras.

The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, realizes the depth estimation method of fusion laser radar and binocular camera as described in any one of the above. step.

The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned depth estimation methods for fusion of lidar and binocular cameras.

The present invention provides a depth estimation method, device and device for fusing a laser radar and a binocular camera. The method includes: collecting scene depth information and image information through a radar camera, the radar camera including a laser radar and a binocular camera; The scene depth information and the image information are calibrated to obtain the pose transformation matrix of the lidar and the binocular camera; according to the pose transformation matrix, the scene depth information is projected to the image of the image information On the plane, a radar disparity map is obtained; bilinear interpolation is performed on the radar disparity map to obtain an up-sampled radar disparity map; the up-sampled radar disparity map is fused with the image information to obtain a target disparity map, which can effectively It can effectively improve the matching accuracy, and can quickly and robustly estimate the depth value in complex environments, which has strong practicability and engineering value.

Description of drawings

In order to explain the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are the For some embodiments of the invention, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

1 is a schematic flowchart of a depth estimation method for fusing a lidar and a binocular camera provided by an embodiment of the present invention;

2 is a schematic diagram of an installation position relationship of a radar camera provided by an embodiment of the present invention;

3 is a schematic diagram of the definition of a radar scan line and a point between the scan lines provided by an embodiment of the present invention;

4 is a schematic diagram of an inclined window and a parallel window provided by an embodiment of the present invention;

5 is a schematic structural diagram of a depth estimation device for fusing a lidar and a binocular camera provided by an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention. , not all examples. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The following describes a depth estimation method, apparatus, and device for integrating lidar and binocular cameras of the present invention with reference to FIGS. 1-6 .

1 is a schematic flowchart of a depth estimation method for fusing a lidar and a binocular camera provided by an embodiment of the present invention; FIG. 2 is a schematic diagram of an installation position relationship of a radar camera provided by an embodiment of the present invention; FIG. 3 is provided by an embodiment of the present invention. A schematic diagram of the definition of a radar scan line and a point between the scan lines; FIG. 4 is a schematic diagram of an inclined window and a parallel window provided by an embodiment of the present invention.

As shown in FIG. 1 , a depth estimation method for fusing a lidar and a binocular camera provided by an embodiment of the present invention includes the following steps:

101. Collect scene depth information and image information through a radar camera, where the radar camera includes a lidar and a binocular camera.

Specifically, a radar camera refers to a lidar and a binocular camera. The lidar can be a mechanical rotating multi-line lidar, a solid-state multi-line lidar, or a single-line lidar to generate multiple scans through registration. Wire. Figure 2 shows a schematic diagram of the installation position relationship between the lidar and the binocular camera provided in this embodiment. LiDAR represents the lidar, and Zed represents the binocular camera. The relative positions of the binocular camera and the lidar can be changed. Depending on the angle of view of the camera, the range of position variation is also different, as long as the radar data can be projected to the desired position. The specific way to collect scene depth information may be that the lidar transmits a laser beam to the measured object, receives the reflected wave of the laser beam, and records the time difference to determine the distance between the measured object and the test point. The binocular camera is composed of two ordinary cameras, and uses the principle of optics and triangulation to obtain the depth of the scene.

102. Calibrate the scene depth information and image information to obtain the pose transformation matrix of the lidar and the binocular camera.

Specifically, the scene depth information and image information are calibrated to obtain the pose transformation matrix of the lidar and the binocular camera, which can be divided into two parts, one is the calibration of the binocular camera, and the other is the calibration of the radar camera. The calibration of the binocular camera can be based on the image information, calibrating the first relative positional relationship between the first camera and the second camera of the binocular camera, that is, to obtain the image information of the two cameras of the binocular camera, and then through the calibration algorithm Calculate the internal parameters of the binocular camera, the lens distortion parameters and the relative spatial relationship between the two cameras. The process of calibrating the radar camera can be, according to the scene depth information and image information, calibrating the second space relative position relationship between the lidar and the binocular camera, that is, according to the scene depth information and image information, obtain the lidar and left by algorithm through the algorithm. (Right) The spatial relative positional relationship between the cameras, namely the first camera and the second camera. Then, according to the first spatial relative position relationship and the second spatial relative position relationship, the pose transformation matrix of the lidar and the binocular camera is constructed.

103. Project the scene depth information onto the image plane of the image information according to the pose transformation matrix to obtain a radar disparity map.

Specifically, based on the pose transformation matrix, the radar disparity map can be obtained by projecting the radar points onto the image plane of the image information of the camera through the relative position relationship of the radar camera, the internal parameters of the camera and the distortion parameters.

104. Perform bilinear interpolation on the radar disparity map to obtain an up-sampled radar disparity map.

After the radar disparity map is obtained, bilinear interpolation is performed on the radar disparity map to fill the gap between the scan lines, and the up-sampled radar disparity map is obtained. Among them, the formula of bilinear interpolation is shown in formula (1):

where π(x, y) is the disparity value at the coordinates (x, y). y _down and y _up are (x, y) points on the radar scan line in the y-axis direction. As shown in Figure 3, y _down is the y-axis coordinate on the scan line below the point (x,y). Similarly, y _up is the y-axis coordinate on the scan line above the point (x,y). If there is no radar scan line above or below a pixel, the pixel is assigned a global maximum value.

105. Fusion of the up-sampled radar disparity map and image information to obtain a target disparity map.

如果法向量n为0，则倾斜窗口退化为平行窗口；平面可以被表示为公式(2)After up-sampling to obtain the up-sampled radar disparity map, the up-sampled radar disparity map and the RGB information of the image are fused to obtain the optimal disparity map, that is, the target disparity map. The specific method may be: according to the up-sampled radar disparity map and image information, determine the parameters of the oblique window model; according to the parameters of the oblique window model, determine the oblique plane, and the oblique window is composed of pixel coordinates (x, y), disparity d and the point The normal vector n=(n _x , _ny ,n _z ) determines the unique inclined plane

If the normal vector n is 0, the slanted window degenerates into a parallel window; the plane can be expressed as Equation (2)

和

是倾斜平面的三个参数；这三个参数被下面的公式(3)(4)(5)确定：where (x, y, d) is a point on the parallax plane,

and

are the three parameters of the inclined plane; these three parameters are determined by the following equations (3)(4)(5):

As shown in FIG. 4 , for the comparison between the inclined window and the parallel window, it can be clearly understood from FIG. 4 that the inclined window has a better adaptation effect to the inclined plane. Among them, the slanted support window represents a slanted window, and the front-parallel window represents a parallel window.

After all the parameters of the inclined plane are determined, the matching cost determination rule of any pixel in the inclined plane is determined, and the matching cost of the pixel p in the plane can be defined. The formula is shown in (6):

是像素p所在平面，w_p是以像素p为中心的一个方形窗口，(p′,d′)是像素p窗口内的像素和视差。in,

is the plane where the pixel p is located, w _p is a square window centered on the pixel p, and (p', d') is the pixel and disparity within the window of the pixel p.

The formula ω(p, p′) can be expressed as formula (7):

Among them, ‖I _p -I _p' ‖ is the L ₁ distance between pixel p and pixel p' in RGB space, γ is a custom parameter, in this algorithm, the value of γ is set to 10;

表示为式(8)：formula

Expressed as formula (8):

ρ(p′,q)=α·min(‖I _p′ -I _q ‖,τ _col )+β·min(|π _p′ -d′|,τ _disp ) (8)

表示在另一幅视图中同名点；min()表示在两个值中去最小值；α和β是颜色和视差所占比例的大小，它们的和是1；τ_col和τ_disp分别是颜色和视差的截断代价；当存在遮挡时，截断代价防止匹配代价过大；‖I_p′-I_q‖表示像素p′和像素q在RGB空间的L₁距离；|π_p′-d′|表示像素p′在上采样视差图中的视差和像素p′在平面

中视差的绝对差值。Among them, q represents

Indicates the point with the same name in another view; min() indicates the minimum value among the two values; α and β are the proportions of color and disparity, and their sum is 1; τ _col and τ _disp are color, respectively and the truncation cost of disparity; when there is occlusion, the truncation cost prevents the matching cost from being too large; ‖I _p' -I _q ‖ represents the L ₁ distance between pixel p' and pixel q in RGB space; |π _p' -d'| represents the disparity of pixel p' in the upsampled disparity map and the pixel p' in the plane

The absolute difference in parallax.

Then, according to the matching cost determination rule, the disparity map that minimizes the cost can be determined as the target disparity map.

Among them, according to the matching cost determination rule, the disparity map that minimizes the cost is determined as the target disparity map, including: randomly initializing the oblique plane to obtain an initialization plane model, that is, after the oblique plane model is determined, the parameters of the model x, y ,d,n _x , _ny ,n _z need to be determined; if pixel p′(x,y) has disparity d′ in the upsampled disparity map, assign the value of d′ to d; if there is no such value, the value between the maximum and minimum disparity is assigned to d; at the same time, a random unit normal vector is selected and assigned to n _x , n _y , and n _z .

则

在偶数次过程中，像素p则需要与p_up和p_left进行比较；如果已经完成指定的次数，则整个过程结束，即确定视差传播达到预设次数时匹配代价最小的平面对应的视差图作为目标视差图。Then, based on the initialized plane model, disparity propagation is performed, each pixel is assigned a plane, and then the plane with the least matching cost on each pixel is found; adjacent pixels may have similar planes, by expanding the planes of adjacent pixels , a plane with a lower matching cost can be found; there are many ways of propagation, such as from left to right, from top to bottom, from top left to bottom right, etc. The present invention uses the propagation mode can be expressed as: in the process of odd number of propagation , the propagation direction propagates from the upper left to the lower right, and in the even-numbered propagation process, the propagation direction propagates from the lower right to the upper left. and during odd number of times, if

but

In the even-numbered process, the pixel p needs to be compared with p _up and p _left ; if the specified number of times has been completed, the whole process ends, that is, the disparity map corresponding to the plane with the least matching cost when the disparity propagation reaches the preset number of times is determined as Object disparity map.

和

被定义为最大的视差和法向量的变化范围；若像素位于两条雷达扫描线之间，则

如果不是，则

为图像的最大视差；相似的，

也是向量的最大允许范围；从

中随机选取Δ_d，计算出d′＝d+Δ_d；

也是在

范围内的一个随机值，并且

可以表示为

其中unit()的作用是单位化；现在一个新的平面

的所有参数都被确定，如果

则平面

被用来代替

这是一个迭代的过程，在第一次计算过程中，

被设置成

并且

在每次的迭代过程中，

被更新为

并且

如果

优化平面

则表示已经完成；然后继续进行视差传播处理，得到最终的目标视差图。In order to ensure the optimization of the disparity map, after the disparity propagation is performed, the method further includes: based on the predetermined maximum disparity and normal vector variation ranges, optimizing the plane with the least matching cost when the disparity propagation reaches a preset number of times. Specifically, after the parallax propagation process, in order to further reduce the matching cost, the plane needs to be optimized; the optimized plane is located between the two propagation processes, and a new plane with lower aggregation cost is provided for the propagation process as much as possible; In this process, the plane needs to be represented by a point (x, y, d) and a normal vector (n _x , n _y , n _z ); at the same time

and

is defined as the maximum disparity and normal vector variation range; if the pixel lies between two radar scan lines, then

If not, then

is the maximum disparity of the image; similarly,

is also the maximum allowable range for a vector; from

Randomly select Δ _d from , and calculate d′=d+Δ _d ;

also in

a random value in the range, and

It can be expressed as

The role of unit() is unitization; now a new plane

All parameters of are determined if

then plane

be used instead

This is an iterative process, in the first calculation process,

is set to

and

During each iteration,

was updated to

and

if

Optimize plane

It means that it has been completed; then continue the disparity propagation processing to obtain the final target disparity map.

The present invention provides a depth estimation method integrating lidar and binocular camera, comprising: collecting scene depth information and image information through a radar camera, the radar camera including lidar and a binocular camera; The image information is calibrated to obtain the pose transformation matrix of the lidar and the binocular camera; according to the pose transformation matrix, the scene depth information is projected onto the image plane of the image information to obtain the radar disparity map; perform bilinear interpolation on the radar disparity map to obtain an up-sampled radar disparity map; fuse the up-sampled radar disparity map with the image information to obtain a target disparity map, which can effectively adapt to outdoor lighting and For scenes where the texture of objects and other conditions are difficult to control, it can effectively improve the accuracy of matching, and can quickly and robustly estimate the depth value in complex environments, which has strong practicability and engineering value, and can also overcome the noise caused by traditional methods. The disadvantages of sensitive and deep learning methods relying on data sets can be applied to large scenes such as outdoor.

Further, in this embodiment, before obtaining the up-sampled radar disparity map by performing bilinear interpolation on the radar disparity map, the method may further include: identifying abnormal projection points in the radar disparity map, and removing the abnormal projection points. In the process of radar projection, due to the relative positional relationship between the camera and the radar, wrong projection points may be caused, so an outlier removal module is required to remove them. Among them, the abnormal projection points in the radar disparity map are identified. Specifically, since the radar points on the radar scan line are discontinuous, the scan line of the lidar is first completed; when the radar depth of the target scan line is greater than and scan line When the first radar depth of the first scan line and the second radar depth of the second scan line are adjacent to the line, it is determined that the target scan line is composed of abnormal projection points, that is, when a scan line with a shallower radar depth appears in the middle When a radar scan line with a large depth value is found, the scan line is considered to be composed of abnormal projection points, and the scan line is cleared, so that the abnormal values in the disparity map of the radar projection can be effectively eliminated.

Based on the same general inventive concept, the present application also protects a depth estimation device that fuses lidar and binocular camera. The depth estimation device that fuses lidar and binocular camera provided by the present invention is described below. The depth estimation device of the binocular camera and the depth estimation method of the fusion lidar and the binocular camera described above can be referred to each other correspondingly.

FIG. 5 is a schematic structural diagram of a depth estimation apparatus integrating a lidar and a binocular camera provided by an embodiment of the present invention.

As shown in FIG. 5, an embodiment of the present invention provides a depth estimation device that integrates a lidar and a binocular camera, including:

an acquisition module 51, configured to acquire scene depth information and image information through a radar camera, where the radar camera includes a lidar and a binocular camera;

A calibration module 52, configured to calibrate the scene depth information and the image information to obtain the pose transformation matrix of the lidar and the binocular camera;

a projection module 53, configured to project the scene depth information onto the image plane of the image information according to the pose transformation matrix, to obtain a radar disparity map;

an up-sampling module 54, configured to perform bilinear interpolation on the radar disparity map to obtain an up-sampled radar disparity map;

The fusion module 55 is configured to fuse the up-sampled radar disparity map with the image information to obtain a target disparity map.

An embodiment of the present invention provides a depth estimation device integrating a lidar and a binocular camera, including: collecting scene depth information and image information through a radar camera, the radar camera including a lidar and a binocular camera; The information and the image information are calibrated to obtain the pose transformation matrix of the lidar and the binocular camera; according to the pose transformation matrix, the scene depth information is projected onto the image plane of the image information, Obtain a radar disparity map; perform bilinear interpolation on the radar disparity map to obtain an up-sampled radar disparity map; fuse the up-sampled radar disparity map with the image information to obtain a target disparity map, which can effectively adapt to outdoor For scenes with difficult-to-control conditions such as lighting and object texture, it can effectively improve the matching accuracy, and can quickly and robustly estimate the depth value in complex environments, which has strong practical and engineering value.

Further, this embodiment also includes an exception handling module for:

Identify abnormal projection points in the radar disparity map, and remove the abnormal projection points.

Further, this embodiment also includes an exception handling module, which is specifically used for:

Completing the scan lines of the lidar;

When the radar depth of the target scan line is greater than the first radar depth of the first scan line and the second radar depth of the second scan line adjacent to the scan line, it is determined that the target scan line is composed of abnormal projection points.

Further, the calibration module 52 in this embodiment is specifically used for:

According to the image information, calibrating the first relative positional relationship in space between the first camera and the second camera of the binocular camera;

According to the scene depth information and the image information, calibrating the second spatial relative position relationship between the lidar and the binocular camera;

According to the first spatial relative position relationship and the second spatial relative position relationship, a pose transformation matrix of the lidar and the binocular camera is constructed.

Further, the fusion module 55 in this embodiment is specifically used for:

determining the oblique window model parameters according to the up-sampled radar disparity map and the image information;

Determine an inclined plane according to the inclined window model parameters, and determine a matching cost determination rule for any pixel in the inclined plane;

According to the matching cost determination rule, the disparity map that minimizes the cost is determined as the target disparity map.

Further, the fusion module 55 in this embodiment is specifically also used for:

Perform random initialization on the inclined plane to obtain an initialization plane model;

Based on the initialized plane model, disparity propagation is performed, and when the disparity propagation reaches a preset number of times, the disparity map corresponding to the plane with the smallest matching cost is determined as the target disparity map.

Further, the fusion module 55 in this embodiment is specifically also used for:

Based on the predetermined maximum disparity and normal vector variation ranges, the plane with the smallest matching cost is optimized when the disparity propagation reaches a preset number of times.

FIG. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

As shown in FIG. 6 , the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 pass through The communication bus 640 accomplishes the mutual communication. The processor 610 can invoke the logic instructions in the memory 630 to execute a depth estimation method fused with a lidar and a binocular camera, the method comprising: collecting scene depth information and image information through a radar camera, the radar camera including a lidar and a binocular camera. camera; calibrate the scene depth information and the image information to obtain the pose transformation matrix of the lidar and the binocular camera; according to the pose transformation matrix, project the scene depth information to the On the image plane of the image information, a radar disparity map is obtained; bilinear interpolation is performed on the radar disparity map to obtain an up-sampling radar disparity map; the up-sampling radar disparity map is fused with the image information to obtain a target Parallax map.

In addition, the above-mentioned logic instructions in the memory 630 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

In another aspect, the present invention also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can Execute the depth estimation method for fusion of lidar and binocular camera provided by the above methods, the method includes: collecting scene depth information and image information through a radar camera, the radar camera includes lidar and a binocular camera; The depth information and the image information are calibrated to obtain the pose transformation matrix of the lidar and the binocular camera; according to the pose transformation matrix, the scene depth information is projected onto the image plane of the image information to obtain a radar disparity map; perform bilinear interpolation on the radar disparity map to obtain an up-sampled radar disparity map; and fuse the up-sampled radar disparity map with the image information to obtain a target disparity map.

In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored, and the computer program is implemented by a processor to execute the fusion laser radar and binocular camera provided by the above methods when the computer program is executed. A depth estimation method, comprising: collecting scene depth information and image information through a radar camera, wherein the radar camera includes a lidar and a binocular camera; calibrating the scene depth information and the image information to obtain the lidar and the pose transformation matrix of the binocular camera; according to the pose transformation matrix, project the scene depth information on the image plane of the image information to obtain a radar disparity map; perform a bi-line on the radar disparity map interpolate to obtain an up-sampled radar disparity map; and fuse the up-sampled radar disparity map with the image information to obtain a target disparity map.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic A disc, an optical disc, etc., includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A depth estimation method fusing a laser radar and a binocular camera is characterized by comprising the following steps:

acquiring scene depth information and image information through a radar camera, wherein the radar camera comprises a laser radar and a binocular camera;

calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera;

projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map;

carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map;

and fusing the up-sampling radar disparity map with the image information to obtain a target disparity map.

2. The method for estimating depth by fusing lidar and a binocular camera according to claim 1, wherein before the bilinear interpolation is performed on the radar disparity map to obtain the up-sampled radar disparity map, the method further comprises:

and identifying abnormal projection points in the radar disparity map, and clearing the abnormal projection points.

3. The method for depth estimation by fusing a lidar and a binocular camera according to claim 2, wherein the identifying the abnormal projection point in the radar disparity map comprises:

completing the scanning line of the laser radar;

and when the radar depth of a target scanning line is greater than the first radar depth of a first scanning line and the second radar depth of a second scanning line which are adjacent to the scanning line, determining that the target scanning line is composed of abnormal projection points.

4. The depth estimation method fusing a lidar and a binocular camera according to claim 1, wherein the calibrating the scene depth information and the image information to obtain a pose transformation matrix of the lidar and the binocular camera comprises:

calibrating a first spatial relative position relation of a first camera and a second camera of the binocular camera according to the image information;

calibrating a second spatial relative position relation between the laser radar and the binocular camera according to the scene depth information and the image information;

and constructing a pose transformation matrix of the laser radar and the binocular camera according to the first spatial relative position relation and the second spatial relative position relation.

5. The method for depth estimation by fusing a lidar and a binocular camera according to claim 1, wherein the fusing the up-sampling radar disparity map with the image information to obtain a target disparity map comprises:

determining a tilt window model parameter according to the up-sampling radar disparity map and the image information;

determining an inclined plane according to the inclined window model parameters, and determining a matching cost determination rule of any pixel in the inclined plane;

and determining the disparity map with the minimized cost as a target disparity map according to the matching cost determination rule.

6. The method for depth estimation by fusing a lidar and a binocular camera according to claim 5, wherein the determining the disparity map with the minimized cost as the target disparity map according to the matching cost determination rule comprises:

carrying out random initialization on the inclined plane to obtain an initialized plane model;

and performing parallax transmission based on the initialized plane model, and determining a parallax image corresponding to the plane with the minimum matching cost when the parallax transmission reaches a preset number as a target parallax image.

7. The depth estimation method combining a lidar and a binocular camera according to claim 6, wherein after the performing the disparity propagation, further comprising:

and optimizing the plane with the minimum matching cost when the parallax transmission reaches the preset times based on the predetermined maximum parallax and normal vector change range.

8. A depth estimation device fusing a laser radar and a binocular camera, comprising:

the system comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring scene depth information and image information through a radar camera, and the radar camera comprises a laser radar and a binocular camera;

the calibration module is used for calibrating the scene depth information and the image information to obtain a pose transformation matrix of the laser radar and the binocular camera;

the projection module is used for projecting the scene depth information to an image plane of the image information according to the pose transformation matrix to obtain a radar disparity map;

the up-sampling module is used for carrying out bilinear interpolation on the radar disparity map to obtain an up-sampling radar disparity map;

and the fusion module is used for fusing the up-sampling radar disparity map with the image information to obtain a target disparity map.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of the method of depth estimation fusing a lidar and a binocular camera according to any of claims 1 to 7.

10. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor, performs the steps of the method for depth estimation fusing a lidar and a binocular camera according to any one of claims 1 to 7.

Images