US11350073B2 - Disparity image stitching and visualization method based on multiple pairs of binocular cameras - Google Patents

Abstract

The present invention discloses a disparity image stitching and visualization method based on multiple pairs of binocular cameras. A calibration algorithm is used to solve the positional relationship between binocular cameras, and the prior information is used to solve a homography matrix between images; internal parameters and external parameters of the cameras are used to perform camera coordinate system transformation of depth images; the graph cut algorithm has high time complexity and depends on the number of nodes in a graph; the present invention divides the images into layers, and solutions are obtained layer by layer and iterated; then the homography matrix is used to perform image coordinate system transformation of the depth images, and a stitching seam is synthesized to realize seamless panoramic depth image stitching; and finally, depth information of a disparity image is superimposed on a visible light image.

Description

TECHNICAL FIELD

The present invention belongs to the field of image processing and computer vision, and particularly relates to a method comprising the steps of calculating a homography matrix between images through external parameters (a rotation vector R and a translation vector T) between cameras, finding an optimal stitching seam between images by graph cut, using R, T, the homography matrix and an optimal transition area to stitch disparity images, and finally fusing the disparity images and visible light images for display.

BACKGROUND

At present, driverless technology is developing rapidly, and safety needs to be ensured for driverless technology. However, by simply using visible light images, it is not possible to well locate an obstacle, obtain the distance of the obstacle, and accurately locate the vehicle of one's own. With the improvement of the technology for obtaining disparity images based on binocular cameras, disparity images are also used as basic data in the field of driverless technology. However, limited by the accuracy of the disparity images, the field angle of the binocular cameras is small, and a single pair of binocular cameras cannot provide sufficient environmental information for the vehicle of one's own. The larger the field angle of a vehicle is, the more complete the information obtained will be, and the higher the guarantee of driving safety will be. In order to make the disparity images have a wider range of field angle, it is necessary to stitch multiple disparity images together. Currently, the following two methods are mainly used for stitching disparity images:

1. Stitching Method Using Feature Points

This method is to extract feature matching points between images, then solve a rotation vector R and a translation vector T between cameras, and finally stitch disparity images according to R and T. The advantages of this method are that the stitching effect is relatively good, the use is flexible, and the method can be used in most scenes; and the disadvantages are that the calculation complexity is high and the method cannot meet the high real-time requirements of driverless technology.

2. Stitching Method Using Camera Calibration

This method is to obtain external parameters R and T between cameras by using checkers, and then stitch disparity images. This method has a small amount of stitching calculation and high real-time performance, but it is easy to produce stitching seams during a disparity image stitching process, which makes the stitching effect poor.

The disparity image stitching process is divided into two processes: camera coordinate transformation and image coordinate transformation. The transformation of a camera coordinate system requires the use of internal parameters K of cameras and external parameters RT between cameras to calculate in a three-dimensional coordinate system; and the transformation of an image coordinate system requires the use of a homography matrix H between camera images and an optimal transition area of visible light images for stitching. An image coordinate system transformation process requires pre-registration, and it takes a lot of time to calculate the external parameters and the homography matrix between cameras by matching feature points. Experiments show that camera angles are fixed, the positional relationship RT between cameras and the internal parameters K of cameras can be calculated by a calibration algorithm, and the homography matrix between two images can be derived through RT and the internal parameters K of cameras, and through the relationship between a global coordinate system and an image coordinate system, so that the feature point matching time is omitted by the prior information. Image registration is completed and solutions are obtained by a graph cut algorithm. As the graph cut algorithm is time-consuming, in order to achieve real-time performance, the images need to be processed layer by layer to reduce the calculation complexity of graph cut. An optimal stitching seam obtained based on the images is used to seamlessly stitch the disparity images after image coordinate system transformation. Finally, disparity image information is superimposed on visible light images to facilitate the observation of the depth information of the environment.

SUMMARY

To overcome the defects in the prior art, the present invention provides a disparity image stitching and visualization method based on multiple pairs of binocular cameras: a homography matrix between images is pre-solved based on the prior information (i.e., the positional relationship R and T between cameras), the traditional graph cut algorithm is improved to increase the efficiency of the graph cut algorithm and then is used for stitching disparity images, and the disparity images are fused with visible light images to make it more convenient to observe the depth of the environment. A stitching process requires image information and depth image information obtained by each binocular camera.

The present invention has the following specific technical solution:

A disparity image stitching and visualization method based on multiple pairs of binocular cameras, comprising the following steps:

Step 1): calibrating internal parameters and external parameters of each binocular camera; the internal parameters K include a focal length focus and optical center coordinates C_x, C_y; the external parameters include a rotation matrix R and a translation vector T; obtaining a baseline length baseline of each binocular camera by calibration; and obtaining visible light images and disparity images of two binocular cameras;

Step 2): calculating homography matrix: calculating a homography matrix H according to the internal parameters and external parameters of each binocular camera, the placement angle between the cameras, and a scene plane distance d; selecting an appropriate value of d according to actual conditions, with the value range thereof being 8-15 in in general; and the specific steps are as follows:

2-1) Imaging a plane in a scene by two binocular cameras, and assuming that the unit normal vector of the plane in the coordinate system of the first binocular camera is N, and the distance from the plane to the center (coordinate origin) of the first binocular camera (i.e., the scene plane distance) is d, then the plane n is expressed as:
N ^T C ₁ =d  (1)
Wherein C₁ is the three-dimensional coordinate of a three-dimensional point P in the coordinate system of the first binocular camera, and the coordinate of the three-dimensional point P in the coordinate system of the second binocular camera is C₂, then the relationship between C₁ and C₂ is:
C ₂ =RC ₁ +T  (2)

Formula (2) is further expressed as:

$\begin{array}{cc}{C}_2={\mathrm{RC}}_1+T\frac{1}{d}{N}^T{C}_1=\left(R+T\frac{1}{d}{N}^T\right)C_1=H^{\prime }{C}_1& \left(3\right)\end{array}$
Wherein R and T are respectively a rotation vector and a translation vector from the first binocular camera to the second binocular camera;

2-2) Transforming C₁ and C₂ in step 2-1) from the internal parameters of the cameras into an image coordinate system:
c ₁ =K ₁ C ₁  (4)
c ₂ =K ₂ C ₂  (5)
It can be obtained from formulas (3), (4) and (5) that:

$\begin{array}{cc}{c}_2=K_1\left(R+T\frac{1}{d}{N}^T\right){K_2}^{-1}{c}_1=K_1{H}^{\prime }{K_2}^{-1}{c}_1& \left(6\right)\end{array}$
Finally, a calculation formula of the homography matrix calculated by the internal parameters and the external parameters is obtained:

$\begin{array}{cc}H=K_1{H}^{\prime }{K_2}^{-1}=K_1\left(R+T\frac{1}{d}{N}^T\right){K_2}^{-1}& \left(7\right)\\ H=\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right]& \left(8\right)\end{array}$
Wherein c₁ is a corresponding coordinate of C₁ in the coordinate system of an imaging plane, and c₂ is a corresponding coordinate of C₂ in the coordinate system of the imaging plane; K₁ is the internal parameters of the first binocular camera; K₂ is the internal parameters of the second binocular camera; and the finally obtained transformation matrix H is a 3*3 matrix, and a₁₁-a₃₃ represent specific values.

Step 3): using the internal parameters of the binocular cameras and the external parameters between the binocular cameras obtained in step 1) and step 2) to perform camera coordinate system transformation of the disparity images; and the specific steps are as follows:

3-1) Using the internal parameters K₁ (i.e., the baseline length baseline₁ and the focal length focus₁) of the first binocular camera to restore the disparity images to a point cloud in the coordinate system of the first binocular camera, and the calculation formulas of the three-dimensional coordinates C₁ (X₁, Y₁, Z₁) of the point cloud are as follows:

$\begin{array}{cc}{Z}_1=\frac{{\mathrm{baseline}}_1*{\mathrm{focus}}_1}{{\mathrm{disparity}}_1}& \left(9\right)\\ X_1=\frac{\left(x_1-C_x\right)*{\mathrm{baseline}}_1}{{\mathrm{disparity}}_1}& \left(10\right)\\ Y_1=\frac{\left(y_1-C_y\right)*{\mathrm{focus}}_1}{{\mathrm{disparity}}_1}& \left(11\right)\end{array}$
Wherein x₁ and y₁ are the pixel coordinates of the first binocular camera; disparity is a disparity value;

3-2) Using the external parameters R and T from the first binocular camera to the second binocular camera to transform the camera coordinate system of the point cloud and obtain the three-dimensional coordinates of the point cloud in the coordinate system of the second binocular camera; and the coordinate transformation formula is as follows:

$\begin{array}{cc}\left(\begin{array}{c}{X}_2\\ Y_2\\ Z_2\end{array}\right)=R\left(\begin{array}{c}{X}_1\\ Y_1\\ Z_1\end{array}\right)+T& \left(12\right)\end{array}$

3-3) Using the internal parameters K₂ (i.e., the baseline length baseline₂ and the focal length focus₂) of the second binocular camera to restore the point cloud to the disparity images, at this time, only Z₂ is needed to calculate the disparity value in the coordinate system of the second binocular camera, and the calculation formula is as follows:

$\begin{array}{cc}{\mathrm{disparity}}_2=\frac{{\mathrm{baseline}}_2*{\mathrm{focus}}_2}{Z_2}& \left(13\right)\end{array}$

Step 4): building overlapping area model: using the homography matrix H between images obtained in step 2) to calculate an overlapping area ROI of images, and modeling the overlapping area; and the specific steps of building a mathematical model are as follows:

4-1) For the pixels of two images in the overlapping area, calculating the second norm of the RGB pixels corresponding to the overlapping area of the two images, and building t-links; the calculation formulas of the second norm are as follows:
e(p,q)=∥p—p′∥+∥q−q′∥  (14)
∥p−p′∥=(R _p −R _p′)²+(G _p −G _p′)²+(B _p −B _p′)²  (15)
∥q−q′∥=(R _q −R _q′)²+(G _q −G _q′)²+(B _q −B _q′)²  (16)

Wherein e(⋅) is a weight function, p is a source image, q is a target image, p is the pixel value of one point in the image p, p′ is the pixel value of a p adjacent point, q is the pixel value of one point in the target image, q′ is the pixel value of a q adjacent point, R_p is the value of R channel at p point, R_p′ is the value of R channel at p′ point, G_p is the value of G channel at p point, G_p′ is the value of G channel at p′ point, B_p is the value of B channel at p point, B_p′, is the value of B channel at p′ point, R_q is the value of R channel at q point, R_q′ is the value of R channel at q′ point, G_q is the value of G channel at q point, G_q′ is the value of G channel at q′ point, B_q is the value of B channel at q point, and B_q′ is the value of B channel at q′ point;

4-2) Finding an optimal stitching line for the built model, and solving (the stitching seam) by graph cut; an energy function is defined as:
E(f)=Σ_p,q∈N S _p,q(l _p ,l _qi)+Σ_p∈P D _p(I _p)  (17)
Wherein S_p,q is a smoothing term representing the cost of assigning a pair of pixels (p, q) in the overlapping area to (l_p, l_q), l_p is a label assigned to the pixel p, l_q is a label assigned to the pixel q, and D_P is a data term representing the cost of marking the pixel p in the overlapping area as l_p;

Step 5): dividing each image into blocks with a size of B₁*B₂, taking the divided blocks as nodes of a graph, performing graph cut to find a local optimal solution, then continuing to divide each node corresponding to an optimal stitching line corresponding to B₁*B₂ until a final block size is equal to a pixel value, and finally and approximately finding a global optimal solution by finding the local optimal solution each time;

Step 6): using the homography matrix H to perform image coordinate system transformation of the disparity images; seamlessly stitching the optimal stitching line in step 5); and the specific steps of disparity image stitching are as follows:

6-1) Transforming the disparity image of the first binocular camera into the image coordinate system of the second binocular camera:

$\begin{array}{cc}w\left(\begin{array}{c}{x}_2\\ y_2\\ 1\end{array}\right)=H\left(\begin{array}{c}{x}_1\\ y_1\\ 1\end{array}\right)& \left(18\right)\end{array}$
Wherein x₁ and y₁ are the coordinates in the image coordinate system of the first binocular camera; x₂ and y₂ are the coordinates in the image coordinate system of the second binocular camera; and w is a normalization coefficient;

6-2) Stitching image: comparing the positions of the first binocular image after image coordinate system transformation and the second binocular image corresponding to an optimal stitching seam, and merging two visible light images and two disparity images respectively;

When the number of binocular cameras is greater than two, repeating steps 3)-6) to obtain disparity images with a larger field angle.

Step 7): adding the stitched disparity information to the visible light images; and the specific steps are as follows:

7-1) Converting the disparity images to color images, replacing disparity information with color information, and using different colors to represent different depths;

7-2) Superimposing and fusing the color images obtained from the disparity images and the visible light images, and the superposition method is a weighted average method:
Fused image=k*visible light image+(1−k)*color image  (19)

Wherein k is a weight coefficient.

The present invention has the following beneficial effects: the present invention realizes large-field-angle panoramic disparity image display; the algorithm of the present invention achieves real-time performance, and realizes large-disparity seamless panoramic disparity image stitching and visualization. The present invention has the following advantages: (1) the program has low requirements on memory and hardware, and can achieve real-time performance on Nvidia TX2; (2) the program is simple and easy to implement; (3) after obtained, the prior information can be directly passed in as parameters to be used as default values; (4) the optimal stitching seam obtained from the images is applied to disparity image stitching to achieve seamless stitching; and (5) disparity image information is superimposed on visible light images.

The present invention makes full use of the prior information of the images and reduces the time of image registration. The method proposed has good scalability; panoramic display of multiple pairs of cameras can be realized by simply inputting R, T and internal parameters K of cameras, and manually setting d value; and the disparity image information is superimposed on the visible light images to display the depth information of the environment more intuitively.

DESCRIPTION OF DRAWINGS

FIG. 1 is a flow chart of the present invention.

FIG. 2 is a system structure diagram of binocular cameras of an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention proposes a disparity image stitching and visualization method based on multiple pairs of binocular cameras, and will be described in detail below in combination with drawings and embodiments.

The present invention uses multiple pairs of horizontally placed binocular cameras as an imaging system to perform multi-viewpoint image collection, wherein K₁ is the internal parameters of the first binocular camera, and K₂ is the internal parameters of the second binocular camera. The resolution of each binocular camera is 1024*768, the video frame rate is greater than 20 frames per second, and a system reference structure is shown in FIG. 2. The spatial transformation relationship R and T between each pair of binocular cameras is calculated on this basis, and the homography matrix H between images is calculated through R, T and the distance d of the imaging plane; the horizontal translation of each image is calculated by taking an intermediate image as a benchmark; and finally, the calculated parameters are used as inputs for stitching and visualization. The specific process is as follows:

1) System calibration and data collection

1-1) Calibrating each pair of binocular cameras to obtain the internal parameters including focal length and optical center and the external parameters including rotation and translation of each pair of binocular cameras;

1-2) Connecting each pair of binocular cameras to multiple computers, and using a router to control and conduct synchronous data collection;

1-3) Using special customized calibration boards to collect images at the same time; paying attention to ensure that the positional relationship between the binocular cameras is consistent during the collection process and keep the calibration boards fixed; and rotating the calibration boards to collect 10-15 groups of images for each pair of binocular cameras according to the actual conditions.

2) Calculating homography matrix between image transformations

2-1) Imaging a plane in a scene by two cameras, and assuming that the unit normal vector of the plane in the coordinate system of the first camera is N, and the distance from the plane to the center (coordinate origin) of the first camera is d, then the plane π can be expressed as:
N ^T C ₁ =d
Wherein C₁ is the coordinate of a three-dimensional point P in the coordinate system of the first camera, X₁ and the coordinate of the three-dimensional point P in the coordinate system of the second camera is C₂, then the relationship between the two is:

$C_2=R*C_1+T$ $C_2={\mathrm{RC}}_1+T\frac{1}{d}{N}^T{C}_1=\left(R+T\frac{1}{d}{N}^T\right)C_1=H^{\prime }{C}_1$
2-2) Obtaining the homography matrix obtained in step 2-1) from the coordinate system of the first camera, and the homography matrix needs to be transformed into the coordinate system of the imaging plane:
c ₁ =K ₁ C ₁
c ₂ =K ₂ C ₂
H=K ₁ H′K ₂ ⁻¹
The value of d in the above formula can be set manually, and the rest are fixed values. In this way, the homography matrix H from the first binocular camera to the second binocular camera is obtained.

3) Using the internal parameters of the binocular cameras and the external parameters between the binocular cameras obtained in steps 1) and 2) to perform camera coordinate system transformation of the disparity images;

3-1) Using the internal parameters K₁, etc. of the first pair of binocular cameras to restore the disparity images to a point cloud in the coordinate system of the first camera:

$Z_1=\frac{{\mathrm{baseline}}_1*{\mathrm{focus}}_1}{{\mathrm{disparity}}_1}$ $X_1=\frac{\left(x_1-C_x\right)*{\mathrm{baseline}}_1}{{\mathrm{disparity}}_1}$ $Y_1=\frac{\left(y_1-C_y\right)*{\mathrm{focus}}_1}{{\mathrm{disparity}}_1}$

3-2) Using R and T from the first binocular camera to the second binocular camera to transform the camera coordinate system of the point cloud:

$\left(\begin{array}{c}{X}_2\\ Y_2\\ Z_2\end{array}\right)=R\left(\begin{array}{c}{X}_1\\ Y_1\\ Z_1\end{array}\right)+T$

Using the internal parameters K₂ of an intermediate viewpoint binocular camera to restore the point cloud to the disparity images, at this time, only Z₂ is needed to obtain the disparity images, and the calculation formula is as follows:

${\mathrm{disparity}}_2=\frac{{\mathrm{baseline}}_2*{\mathrm{focus}}_2}{Z_2}$
Calculating overlapping area of images and solving optimal stitching seam by modeling: first, calculating an overlapping area ROI through the homography matrix between images, and then building an overlapping area model; and the specific steps are as follows:

4-1) Calculating the size of the overlapping area by the homography matrix between images:

Taking the four vertices (0, 0), (img.cols, 0), (img.cols, img.rows) and (0, img.rows) of an image, calculating the transformed coordinates, the transformed upper left corner coordinate is for the stitched image, and the homography transformation matrix H is:

$H=\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right]$

The calculation formulas are:

$x=\frac{x^{\prime }}{w^{\prime }}=\frac{a_{11}u+a_{12}v+a_{13}}{a_{31}u+a_{32}v+a_{33}}$ $y=\frac{y^{\prime }}{w^{\prime }}=\frac{a_{21}u+a_{22}u+a_{23}}{a_{31}u+a_{32}u+a_{33}}$
Wherein x is the x-coordinate of the source image p point after perspective transformation, y is the y-coordinate of the source image p point after perspective transformation, u is the x-coordinate of the source image p point, and v is the y-coordinate of the source image p point;

4-2) Building an energy model (seam-driven image stitching), and constructing an energy function of a graph cut algorithm:

$E\left(1\right)=\sum _{p\in P}{D}_p\left(l_p\right)+\sum _{\left(p,q\right)\in N}{S}_{p,q}\left(l_p,l_q\right)$
Wherein the data term D_p(l_p) represents the assigned value of pixel p in the overlapping area:

$\{\begin{array}{cc}{D}_p\left(1\right)=0D_p\left(0\right)=\mu & \mathrm{if}p\in \partial I_0\bigcap \partial P\\ D_p\left(0\right)=0D_p\left(1\right)=\mu & \mathrm{if}p\in \partial I_1\bigcap \partial P\\ D_p\left(0\right)=D_p\left(1\right)=0& \mathrm{otherwise}\end{array}\hspace{1em}$

In order to avoid marking errors, pi is set to be a very large number;

S_p,q(l_p, l_q) is a smoothing term;
S _p,q(l _p ,l _q)=I _*(p)+I _*(q)
I _*(p)=∥I ₀(⋅)−I ₁(⋅)∥₂

5) After the model is built, obtaining a solution by graph cut, and the result is an optimal stitching seam. It can be known that the construction of the energy function is very important for the result of the stitching seam.

5-1) As the operation time of graph cut is related to the number of nodes in a graph, and the algorithm complexity is relatively high, only by down-sampling or stratifying the overlapping area to reduce the number of nodes in the constructed graph, and making the local optimal solution obtained by this method be approximately equal to the global optimal solution, can the real-time performance of the algorithm meet requirements.

5-2) In addition, the parallelization of the graph cut algorithm can also achieve a further acceleration effect. (Fast graphcut on GPU CVPR2008) 6) The specific steps of disparity image stitching are as follows:

6-1) Transforming the depth image of the first binocular camera into the image coordinate system of the second binocular camera:

$W\left(\begin{array}{c}{x}_2\\ y_2\\ 1\end{array}\right)=H\left(\begin{array}{c}{x}_1\\ y_1\\ 1\end{array}\right)$

6-2) Stitching image: comparing the positions of the disparity image after image coordinate system transformation and an intermediate disparity image corresponding to the optimal stitching seam, and merging the two disparity images.

Completing the disparity image stitching of one pair of binocular cameras by steps 1)-6), and repeating steps 1)-6) to complete the disparity image stitching of the second pair of binocular cameras (e.g., the second and third binocular cameras).

7) Adding the stitched disparity information to the visible light images:

7-1) Converting the disparity images to color images, replacing disparity information with color information, and using different colors to represent different depths, wherein the color images calculated from the disparity images includes but is not limited to pseudo-color images and rainbow images;

7-2) Superimposing and fusing the color images obtained from the disparity images and the visible light images, and the superposition method is a weighted average method:
Fused image=k*visible light image+(1−k)*color image

k is a weight coefficient; when k is relatively large (1-*0.5), visible light information can be observed more clearly; and when k is relatively small (0.5-0), more depth information can be observed.

Claims (10)

The invention claimed is:

1. A disparity image stitching and visualization method based on multiple pairs of binocular cameras, comprising the following steps:

step 1): calibrating internal parameters and external parameters of each binocular camera; the internal parameters K include a focal length focus and optical center coordinates C_x, C_y; the external parameters include a rotation matrix R and a translation vector T; obtaining a baseline length baseline of each binocular camera by calibration; and obtaining visible light images and disparity images of two binocular cameras;

step 2): calculating homography matrix: calculating a homography matrix H according to the internal parameters and external parameters of each binocular camera, the placement angle between the cameras, and a scene plane distance d; and the value range of d is 8-15 m;

step 3): using the internal parameters of the binocular cameras and the external parameters between the binocular cameras obtained in step 1) and step 2) to perform camera coordinate system transformation of the disparity images;

step 4): building overlapping area model: using the homography matrix H between images obtained in step 2) to calculate an overlapping area ROI of images, and modeling the overlapping area;

step 5): dividing each image into blocks with a size of B₁*B₂, taking the divided blocks as nodes of a graph, performing graph cut to find a local optimal solution, then continuing to divide each node corresponding to an optimal stitching line corresponding to B₁*B₂ until a final block size is equal to a pixel value, and finally and approximately finding a global optimal solution by finding the local optimal solution each time;

step 6): using the homography matrix H to perform image coordinate system transformation of the disparity images; seamlessly stitching the optimal stitching line in step 5); when the number of binocular cameras is greater than two, repeating steps 3)-6) to obtain disparity images with a larger field angle; and

step 7): adding the stitched disparity information to the visible light images.

2. The disparity image stitching and visualization method based on multiple pairs of binocular cameras according to claim 1, wherein the specific steps of calculating homography matrix described in step 2) are as follows:

2-1) imaging a plane in a scene by two binocular cameras, and assuming that the unit normal vector of the plane in the coordinate system of the first binocular camera is N, and the distance from the plane to the center of the first binocular camera (i.e., the scene plane distance) is d, then the plane 7C is expressed as:

N ^T C ₁ =d  (1)

wherein C₁ is the three-dimensional coordinate of a three-dimensional point P in the coordinate system of the first binocular camera, and the coordinate of the three-dimensional point P in the coordinate system of the second binocular camera is C₂, then the relationship between C₁ and C₂ is:

C ₂ =RC ₁ +T  (2)

formula (2) is further expressed as:

$\begin{array}{cc}{C}_2={\mathrm{RC}}_1+T\frac{1}{d}{N}^T{C}_1=\left(R+T\frac{1}{d}{N}^T\right)C_1=H^{\prime }{C}_1& \left(3\right)\end{array}$

wherein R and T are respectively a rotation vector and a translation vector from the first binocular camera to the second binocular camera;

2-2) transforming C₁ and C₂ in step 2-1) from the internal parameters of the cameras into the image coordinate system:

c ₁ =K ₁ C ₁  (4)

c ₂ =K ₂ C ₂  (5)

it can be obtained from formulas (3), (4) and (5) that:

$\begin{array}{cc}{c}_2=K_1\left(R+T\frac{1}{d}{N}^T\right){K_2}^{-1}{c}_1=K_1{H}^{\prime }{K_2}^{-1}{c}_1& \left(6\right)\end{array}$

finally, a calculation formula of the homography matrix calculated by the internal parameters and the external parameters is obtained:

$\begin{array}{cc}H=K_1{H}^{\prime }{K_2}^{-1}=K_1\left(R+T\frac{1}{d}{N}^T\right){K_2}^{-1}& \left(7\right)\\ H=\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right]& \left(8\right)\end{array}$

wherein c₁ is a corresponding coordinate of C₁ in the coordinate system of an imaging plane, and c₂ is a corresponding coordinate of C₂ in the coordinate system of the imaging plane;

K₁ is the internal parameters of the first binocular camera; K₂ is the internal parameters of the second binocular camera; and the finally obtained transformation matrix H is a 3*3 matrix, and a₁₁-a₃₃ represent specific values.

3. A disparity image stitching and visualization method based on multiple pairs of binocular cameras according to claim 1, wherein the specific steps of step 3) are as follows:

3-1) using the internal parameters K₁ (i.e., the baseline length baseline₁ and the focal length focus₁) of the first binocular camera to restore the disparity images to a point cloud in the coordinate system of the first binocular camera, and the calculation formulas of the three-dimensional coordinates C₁ (X₁,Y₁, Z₁) of the point cloud are as follows:

$\begin{array}{cc}{Z}_1=\frac{{\mathrm{baseline}}_1*{\mathrm{focus}}_1}{{\mathrm{disparity}}_1}& \left(9\right)\\ X_1=\frac{\left(x_1-C_x\right)*{\mathrm{baseline}}_1}{{\mathrm{disparity}}_1}& \left(10\right)\\ Y_1=\frac{\left(y_1-C_y\right)*{\mathrm{focus}}_1}{{\mathrm{disparity}}_1}& \left(11\right)\end{array}$

wherein x₁ and y₁ are the pixel coordinates of the first binocular camera; disparity is a disparity value;

3-2) using the external parameters R and T from the first binocular camera to the second binocular camera to transform the camera coordinate system of the point cloud and obtain the three-dimensional coordinates of the point cloud in the coordinate system of the second binocular camera; and the coordinate transformation formula is as follows:

$\begin{array}{cc}\left(\begin{array}{c}{X}_2\\ Y_2\\ Z_2\end{array}\right)=R\left(\begin{array}{c}{X}_1\\ Y_1\\ Z_1\end{array}\right)+T& \left(12\right)\end{array}$

3-3) using the internal parameters K₂ (i.e., the baseline length baseline₂ and the focal length focus₂) of the second binocular camera to restore the point cloud to the disparity images, at this time, only Z₂ is needed to calculate the disparity value in the coordinate system of the second binocular camera, and the calculation formula is as follows:

$\begin{array}{cc}{\mathrm{disparity}}_2=\frac{{\mathrm{baseline}}_2*{\mathrm{focus}}_2}{Z_2}.& \left(13\right)\end{array}$

4. The disparity image stitching and visualization method based on multiple pairs of binocular cameras according to claim 1, wherein the specific steps of building overlapping area model described in step 4) are as follows:

4-1) for the pixels of two images in the overlapping area, calculating the second norm of the RGB pixels corresponding to the overlapping area of the two images, and building t-links; the calculation formulas of the second norm are as follows:

e(p,q)=∥p—p′∥+∥q−q′∥  (14)

∥p−p′∥=(R _p −R _p′)²+(G _p −G _p′)²+(B _p −B _p′)²  (15)

∥q−q′∥=(R _q −R _q′)²+(G _q −G _q′)²+(B _q −B _q′)²  (16)

Wherein e(⋅) is a weight function, p is a source image, q is a target image, p is the pixel value of one point in the image p, p′ is the pixel value of a p adjacent point, q is the pixel value of one point in the target image, q′ is the pixel value of a q adjacent point, R_p is the value of R channel at p point, R_p′ is the value of R channel at p′ point, G_p is the value of G channel at p point, G_p′ is the value of G channel at p′ point, B_p is the value of B channel at p point, B_p′, is the value of B channel at p′ point, R_q is the value of R channel at q point, R_q′ is the value of R channel at q′ point, G_q is the value of G channel at q point, G_q′ is the value of G channel at q′ point, B_q is the value of B channel at q point, and B_q′ is the value of B channel at q′ point;

4-2) Finding an optimal stitching line for the built model, and solving (the stitching seam) by graph cut; an energy function is defined as:

E(f)=Σ_p,q∈N S _p,q(l _p ,l _qi)+Σ_p∈P D _p(I _p)  (17)

wherein S_p,q is a smoothing term representing the cost of assigning a pair of pixels (p, q) in the overlapping area to (l_p,l_q), l_p is a label assigned to the pixel p, l_q is a label assigned to the pixel q, and D_P is a data term representing the cost of marking the pixel p in the overlapping area as l_p.

5. The disparity image stitching and visualization method based on multiple pairs of binocular cameras according to claim 3, wherein the specific steps of building overlapping area model described in step 4) are as follows:

4-1) for the pixels of two images in the overlapping area, calculating the second norm of the RGB pixels corresponding to the overlapping area of the two images, and building t-links; the calculation formulas of the second norm are as follows:

e(p,q)=∥p—p′∥+∥q−q′∥  (14)

∥p−p′∥=(R _p −R _p′)²+(G _p −G _p′)²+(B _p −B _p′)²  (15)

∥q−q′∥=(R _q −R _q′)²+(G _q −G _q′)²+(B _q −B _q′)²  (16)

Wherein e(⋅) is a weight function, p is a source image, q is a target image, p is the pixel value of one point in the image p, p′ is the pixel value of a p adjacent point, q is the pixel value of one point in the target image, q′ is the pixel value of a q adjacent point, R_p is the value of R channel at p point, R_p′ is the value of R channel at p′ point, G_p is the value of G channel at p point, G_p′ is the value of G channel at p′ point, B_p is the value of B channel at p point, B_p′, is the value of B channel at p′ point, R_q is the value of R channel at q point, R_q′ is the value of R channel at q′ point, G_q is the value of G channel at q point, G_q′ is the value of G channel at q′ point, B_q is the value of B channel at q point, and B_q′ is the value of B channel at q′ point;

4-2) Finding an optimal stitching line for the built model, and solving (the stitching seam) by graph cut; an energy function is defined as:

E(f)=Σ_p,q∈N S _p,q(l _p ,l _qi)+Σ_p∈P D _p(I _p)  (17)

wherein S_p,q is a smoothing term representing the cost of assigning a pair of pixels (p, q) in the overlapping area to (l_p,l_q), l_p is a label assigned to the pixel p, l_q is a label assigned to the pixel q, and Dp is a data term representing the cost of marking the pixel p in the overlapping area as l_p.

6. The disparity image stitching and visualization method based on multiple pairs of binocular cameras according to claim 1, wherein the specific steps of disparity image stitching described in step 6) are as follows:

6-1) transforming the disparity image of the first binocular camera into the image coordinate system of the second binocular camera:

$\begin{array}{cc}W\left(\begin{array}{c}{x}_2\\ y_2\\ 1\end{array}\right)=H\left(\begin{array}{c}{x}_1\\ y_1\\ 1\end{array}\right)& \left(18\right)\end{array}$

wherein x₁ and y₁ are the coordinates in the image coordinate system of the first binocular camera; x₂ and y₂ are the coordinates in the image coordinate system of the second binocular camera; and w is a normalization coefficient;

6-2) stitching image: comparing the positions of the first binocular image after image coordinate system transformation and the second binocular image corresponding to an optimal stitching seam, and merging two visible light images and two disparity images respectively.

7. The disparity image stitching and visualization method based on multiple pairs of binocular cameras according to claim 3, wherein the specific steps of disparity image stitching described in step 6) are as follows:

6-1) transforming the disparity image of the first binocular camera into the image coordinate system of the second binocular camera:

$\begin{array}{cc}W\left(\begin{array}{c}{x}_2\\ y_2\\ 1\end{array}\right)=H\left(\begin{array}{c}{x}_1\\ y_1\\ 1\end{array}\right)& \left(18\right)\end{array}$

wherein x₁ and y₁ are the coordinates in the image coordinate system of the first binocular camera; x₂ and y₂ are the coordinates in the image coordinate system of the second binocular camera; and w is a normalization coefficient;

6-2) stitching image: comparing the positions of the first binocular image after image coordinate system transformation and the second binocular image corresponding to an optimal stitching seam, and merging two visible light images and two disparity images respectively.

8. The disparity image stitching and visualization method based on multiple pairs of binocular cameras according to claim 4, wherein the specific steps of disparity image stitching described in step 6) are as follows:

6-1) transforming the disparity image of the first binocular camera into the image coordinate system of the second binocular camera:

$\begin{array}{cc}W\left(\begin{array}{c}{x}_2\\ y_2\\ 1\end{array}\right)=H\left(\begin{array}{c}{x}_1\\ y_1\\ 1\end{array}\right)& \left(18\right)\end{array}$

wherein x₁ and y₁ are the coordinates in the image coordinate system of the first binocular camera; x₂ and y₂ are the coordinates in the image coordinate system of the second binocular camera; and w is a normalization coefficient;

6-2) stitching image: comparing the positions of the first binocular image after image coordinate system transformation and the second binocular image corresponding to an optimal stitching seam, and merging two visible light images and two disparity images respectively.

9. The disparity image stitching and visualization method based on multiple pairs of binocular cameras according to claim 1, 2, 5, 7 or 8, wherein the specific steps of step 7) are as follows:

7-1) converting the disparity images to color images, replacing disparity information with color information, and using different colors to represent different depths;

7-2) superimposing and fusing the color images obtained from the disparity images and the visible light images, and the superposition method is a weighted average method:

Fused image=k*visible light image+(1−k)*color image  (19)

wherein k is a weight coefficient.

10. The disparity image stitching and visualization method based on multiple pairs of binocular cameras according to claim 6, wherein the specific steps of step 7) are as follows:

7-1) converting the disparity images to color images, replacing disparity information with color information, and using different colors to represent different depths;

7-2) superimposing and fusing the color images obtained from the disparity images and the visible light images, and the superposition method is a weighted average method:

Fused image=k*visible light image+(1−k)*color image  (19)

wherein k is a weight coefficient.

Images