CN111062900B - Binocular disparity map enhancement method based on confidence fusion - Google Patents

Binocular disparity map enhancement method based on confidence fusion Download PDF

Info

Publication number
CN111062900B
CN111062900B CN201911148748.5A CN201911148748A CN111062900B CN 111062900 B CN111062900 B CN 111062900B CN 201911148748 A CN201911148748 A CN 201911148748A CN 111062900 B CN111062900 B CN 111062900B
Authority
CN
China
Prior art keywords
image
disparity map
disparity
parallax
map
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911148748.5A
Other languages
Chinese (zh)
Other versions
CN111062900A (en
Inventor
赵春晖
苏梅梅
刘慧霞
胡劲文
田雪涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Northwestern Polytechnical University
Original Assignee
Northwestern Polytechnical University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Northwestern Polytechnical University filed Critical Northwestern Polytechnical University
Priority to CN201911148748.5A priority Critical patent/CN111062900B/en
Publication of CN111062900A publication Critical patent/CN111062900A/en
Application granted granted Critical
Publication of CN111062900B publication Critical patent/CN111062900B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a binocular disparity map enhancement method based on confidence fusion, which comprises the following steps of: s1, placing the picture right in front of the center of the binocular camera, and utilizing the double cameraShooting a target by a target camera to obtain a left image a and a right image b of a binocular image; s2, respectively carrying out stereogram distortion correction on the left image a and the right image b to obtain a stereogram corrected left image a1 and a stereogram corrected right image b 1; s3, performing disparity estimation on the left image a1 and the right image b1 by using a convolutional neural network algorithm to obtain a dense disparity map a 2; carrying out disparity map optimization processing on the dense disparity map a2 to obtain an optimized disparity map Ic(ii) a S4, carrying out parallax estimation on the stereogram corrected left image a1 and right image b1 by adopting an improved optical flow estimation algorithm to obtain a parallax image IO(ii) a S5, according to the confidence level, comparing the obtained disparity map IOAnd a disparity map IcAnd (5) performing fusion to obtain a final disparity map I (u).

Description

Binocular disparity map enhancement method based on confidence fusion
[ technical field ] A method for producing a semiconductor device
The invention belongs to the technical field of stereo matching in image processing, and particularly relates to a binocular disparity map enhancement method based on confidence fusion.
[ background of the invention ]
Binocular stereo vision is an important form of machine vision, and three-dimensional information of an object is acquired by combining two images according to the parallax principle. The stereo matching is a main technical means for obtaining three-dimensional object information from two-dimensional image information, and refers to projection points of any point on two or more cameras on a spatial scene, and the points are called corresponding points to obtain corresponding points between a left image plane and a right image plane. The process of finding stereo corresponding points is called disparity estimation process, so disparity estimation is an important basis for depth estimation.
The traditional parallax estimation method generally uses a large window to solve the problem in processing low-texture and depth discontinuous areas, which can cause blurring of important features, and the poor-texture and blurred surfaces cannot be matched consistently, so that errors caused by noise exist in parallax values.
[ summary of the invention ]
The invention aims to provide a binocular disparity map enhancement method based on confidence fusion, and the method is used for solving the problems that the accuracy of disparity values in low-texture areas and depth discontinuous areas cannot be improved and high-quality disparity maps cannot be generated in the prior art.
The invention adopts the following technical scheme: a binocular disparity map enhancement method based on confidence fusion comprises the following steps:
s1, placing the picture in front of the center of the binocular camera, and shooting a target by using the binocular camera to obtain a left picture a and a right picture b of the binocular picture; the camera coordinate system takes the center of the camera as an origin, x points to the right of the camera, y points to the upper part of the camera, and z points to the front of the camera;
s2, respectively carrying out stereogram distortion correction on the left image a and the right image b to obtain a stereogram corrected left image a1 and a stereogram corrected right image b 1;
s3, performing disparity estimation on the left image a1 and the right image b1 by using a convolutional neural network algorithm to obtain a dense disparity map a 2;
carrying out disparity map optimization processing on the dense disparity map a2 to obtain an optimized disparity map Ic；
S4, carrying out parallax estimation on the stereogram corrected left image a1 and right image b1 by adopting an improved optical flow estimation algorithm to obtain a parallax image IO；
S5, according to the confidence level, comparing the obtained disparity map IOAnd a disparity map IcAnd (5) performing fusion to obtain a final disparity map I (u).
Further, the specific method adopted in step S2 is as follows: co-planar left and right panels a and b; aligning the image lines of the left image a and the right image b; and (5) correcting the left image a and the right image b to obtain a stereo corrected output matrix, namely obtaining a corrected left image a1 and a corrected right image b 1.
Further, the specific method of step S3 is:
3.1, carrying out weight distribution on the left graph a1 and the right graph b1 by using a convolutional neural network, wherein each layer of neural network is subjected to homogenization treatment and a ReLU function; feature maps obtained after the left map a1 and the right map b1 pass through a convolutional neural network are sent to a relevant layer to calculate cost, and an initial disparity map is obtained after three basic processes of cost aggregation and disparity prediction;
step 3.2, performing left-right verification on the generated initial disparity map, updating the network weight, and updating the updating rule according to a gradient descent method;
3.3, repeating the step 3.1 and the step 3.2, after updating the network weight, continuing to predict the initial disparity map, and then checking out the pixels with high confidence coefficient to continue training;
step 3.4, after the training is finished, smoothing the predicted disparity map by a median filter to obtain a dense disparity map a 2;
step 3.5, filling points with edge parallax value of 0 in the dense parallax map a2, filling the parallax information with value of 0 by adopting an image fusion method, and fusing to obtain an optimized parallax map Ic(x,y)。
Further, the specific process of step S4 is:
step 4.1, acquiring a first derivative and a second derivative of the midpoint of the images a1 and b1 according to an optical flow equation; assuming that the pixel points of the fixed window have the same motion and have smaller displacement; for the left graph a1, define uaThe moving speed v of the pixel point in the x-axis directionaThe motion speed of the pixel point in the y-axis direction is obtained; wherein Ix，IyAnd ItThe derivatives of the image with respect to x-direction, y-direction, and time, respectively; in the right graph b1, u is definedbThe moving speed v of the pixel point in the x-axis directionbObtaining the derivative L of the pixel point in the image b1 for the x direction, the y direction and the time for the moving speed of the pixel point in the y-axis directionx、LyAnd Lt；
Step 4.2, obtaining a characteristic value of the Hessian matrix, and calculating a condition number of the Hessian matrix;
if the condition number is between 0.91 and 0.99, the pixel is a reliable point;
if the condition number is not between 0.91 and 0.99, the pixel is an unreliable point, and the unreliable point is rejected;
4.3, removing unreliable points by using a Hessian matrix, and taking the reciprocal of the condition number corresponding to the points as the weight of the points;
step 4.4, solving the optical flow field according to the weighted least square method [ u v]TAnd [ u 'v']TThe obtained optical flow field is approximate to a parallax value, and a parallax image I is obtainedO。
Further, the specific process of S5 is as follows:
step 5.1, in the disparity map IcAnd a disparity map IOFor a certain position (x)i,yi) In two parallax images IcAnd IOSelecting a fusion window W with a certain size, and calculating the confidence levels according to an SNR confidence measure method, wherein the confidence levels are calculated as shown in the specification,
si＝|dli-dri|，
Wherein d isliObtaining a disparity value, d, for the reference image as the primary imageriObtaining parallax for the image to be matched as the main image;
step 5.2, comparing the two disparity maps (x)i,yi) Selecting the confidence level with high confidence as the final parallax; according to the weight and the confidence level of each parallax in the fusion window, the total confidence level can be obtained as follows:
Figure BDA0002282949880000041
wherein the weight is
Figure BDA0002282949880000042
di(xc,yc) For blending the disparity values of the central pixels of the window, di(x, y) is the disparity value of the pixels in the fusion window, and the final disparity is defined as follows:
D＝argmax[ci(x,y,di)]，
and processing according to the steps to obtain a final fused disparity map I (u).
The invention has the beneficial effects that: aiming at the problems in stereo matching of images with low texture and discontinuous depth, stereo parallax is approximated through horizontal optical flow, meanwhile, the accuracy of parallax values of low texture and discontinuous depth areas is improved through fusion between the convolutional neural network-based parallax map and the improved optical flow image, and a high-quality parallax map is generated.
[ description of the drawings ]
Fig. 1 is a flow chart of a binocular disparity map enhancement method based on confidence fusion according to the present invention;
FIG. 2(a) is a left image captured by a binocular camera in the embodiment;
FIG. 2(b) is a right view of the binocular camera in the embodiment;
FIG. 2(c) is the resulting disparity map of the improved LK optical flow algorithm in an embodiment;
FIG. 2(d) is the resulting disparity map of the modified convolutional neural network in an embodiment;
fig. 2(e) is a disparity map obtained by using the binocular disparity map enhancement method based on confidence fusion according to the embodiment of the present invention;
fig. 2(f) is a disparity map obtained by using the SGBM method in the embodiment.
[ detailed description ] embodiments
The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
The invention provides a binocular disparity map enhancement method based on confidence fusion, which comprises the following steps of:
s1, placing the picture in front of the center of the binocular camera, and shooting a target by using the binocular camera to obtain a left picture a and a right picture b of the binocular picture; the camera coordinate system takes the center of the camera as an origin, x points to the right of the camera, y points to the upper of the camera, and z points to the front of the camera.
S2, respectively carrying out stereogram distortion correction on the left image a and the right image b to obtain a stereogram corrected left image a1 and a stereogram corrected right image b 1; the distortion correction process is that projection transformation matrixes of images are obtained by aiming at the two cameras obtained by calibration, so that the images obtained by the two cameras are coplanar and aligned in rows from a world coordinate system to a camera coordinate system, and epipolar alignment is realized;
the specific method comprises the following steps:
step 2.1, coplanar left graph a and right graph b: taking the upper left corner of the image as an origin, arranging the image rows as an x axis and arranging the image rows as a y axis, so that the focal planes of the two images become coplanar; dividing a camera rotation matrix R into a composite matrix of left and right cameras
Figure BDA0002282949880000051
To achieve coplanar image planes;
step 2.2, aligning the image lines of the left image a and the right image b: first a rotation matrix R of the direction of the translation vector is createde＝[e1,e2,e3]TWherein e is1＝T/||T||，T＝[Tx,Ty,Tz]，Tx,TyAnd TzTranslation vectors in the x, y and z directions respectively,
Figure BDA0002282949880000052
e3＝e1×e2(ii) a The row alignment matrix is obtained as follows:
Rl＝Rerl,Rr＝Rerr；
and 2.3, correcting the left image a and the right image b to obtain a three-dimensional corrected output matrix, wherein a specific formula is as follows:
Figure BDA0002282949880000061
Figure BDA0002282949880000062
Figure BDA0002282949880000063
in the above formula, the Q matrix is a reprojection matrix, which realizes the conversion between the world coordinate system and the pixel coordinate system, d is the parallax, f is the focal length, and the vertical and horizontal offset of the origin a of the left image is cxAnd cy，
Figure BDA0002282949880000064
Is used for shooting three-dimensional coordinates, c 'corresponding to an original image'xMain point representing right image b, when cx＝c′xIf so, correct correction is carried out;
according to the steps, whether the corresponding points on the two images are on the same polar line is finally judged, and the two corrected images obtained by the method are successfully corrected through verification, so that a left image a1 and a right image b1 of the corrected images are obtained;
s3, performing disparity estimation on the left image a1 and the right image b1 by using a convolutional neural network algorithm to obtain a dense disparity map a 2; carrying out disparity map optimization processing on the dense disparity map a2 to obtain an optimized disparity map Ic；
The specific method comprises the following steps:
3.1, carrying out weight distribution on the left graph a1 and the right graph b1 by using a convolutional neural network, wherein each layer of neural network is subjected to homogenization treatment and a ReLU function;
feature maps obtained after the left map a1 and the right map b1 pass through a convolutional neural network are sent to a relevant layer to calculate cost, and an initial disparity map is obtained after three basic processes of cost aggregation and disparity prediction;
step 3.2, performing left-right verification on the generated initial disparity map, updating the network weight, and updating the updating rule according to a gradient descent method, wherein the formula is as follows:
Figure BDA0002282949880000071
Figure BDA0002282949880000072
wherein J (w, beta; a1, b1) is a cost function, Pw,β() represents the predicted values of the network for the left graph a1 and the right graph b1, w, β represent the weight parameters of the neural network; α represents learning efficiency;
3.3, repeating the step 3.1 and the step 3.2, after updating the network weight, continuing to predict the initial disparity map, and then checking out the pixels with high confidence coefficient to continue training;
step 3.4, after the training is finished, smoothing the predicted disparity map by a median filter to ensure edge information, and obtaining a dense disparity map a 2;
step 3.5, filling points with edge parallax value of 0 in the dense parallax map a2, filling the parallax information with value of 0 by adopting an image fusion method, and fusing to obtain an optimized parallax map Ic(x,y)；
The fusion mode is as follows:
Figure BDA0002282949880000073
Figure BDA0002282949880000074
wherein, Ic(x, y) is the optimized disparity map; i isa1(x,y)、Ib1(x, y) respectively representing a disparity map obtained by matching the reference image with the adjacent left and right images; δ represents a threshold value; t is a translation vector in the camera's external reference matrix;
s4, carrying out parallax estimation on the stereogram corrected left image a1 and right image b1 by adopting an improved optical flow estimation algorithm to obtain a parallax image IO；
The specific process is as follows:
step 4.1: acquiring a first derivative and a second derivative of the point in the images a1 and b1 according to an optical flow equation; assuming that the pixel points of the fixed window have the same motion and have smaller displacement; for the left graph a1, define uaThe moving speed v of the pixel point in the x-axis directionaThe motion speed of the pixel point in the y-axis direction is obtained; wherein Ix，IyAnd ItThe derivatives of the image with respect to x-direction, y-direction, and time, respectively; in the right graph b1, u is definedbThe moving speed v of the pixel point in the x-axis directionbFor the moving speed of the pixel point in the y-axis direction, the derivative L of the pixel point in the image b1 with respect to the x-direction, the y-direction, and the time can be obtainedx、LyAnd Lt；
According to the basic optical flow equation:
Ixu+Iyv+It＝0 (4)，
respectively deriving x and y in formula (4):
Figure BDA0002282949880000081
Figure BDA0002282949880000082
defining the Hessian matrix as:
Figure BDA0002282949880000083
the same can be obtained
Figure BDA0002282949880000084
Step 4.2: obtaining an eigenvalue of the Hessian matrix, and calculating a condition number of the Hessian matrix, as shown in formula (8):
Figure BDA0002282949880000085
wherein λ ismaxAnd λminRespectively a Hessian matrix HKThe maximum eigenvalue and the minimum eigenvalue of the eigenvalue of (6) can be judged by the formula (8); if the condition number is large, corresponding to HKThe rank of the matrix is very small, and the calculated optical flow is unreliable; if the condition number is close to 1, corresponding to HKThe rank of the matrix is large, and the solution robustness of the equation (6) is good;
therefore, if the condition number is between 0.91-0.99, the pixel is a reliable point;
if the condition number is not between 0.91 and 0.99, the pixel is an unreliable point, and the unreliable point is rejected;
step 4.3: setting the weight of points in the image:
unreliable points are eliminated by using a Hessian matrix, reciprocals of condition numbers corresponding to the points are used as weights of the points, as shown in a formula (9),
Figure BDA0002282949880000091
step 4.4: optical flow field [ u v ] is solved according to weighted least squares]TAnd [ u 'v']TAs shown in formulas (10) and (11);
Figure BDA0002282949880000092
Figure BDA0002282949880000093
according to the processing of the steps, the obtained optical flow field is approximate to a parallax value, and a parallax image I is obtainedO。
S5, according to the confidence level, comparing the obtained disparity map IOAnd a disparity map IcAnd (5) performing fusion to obtain a final disparity map I (u).
The specific process is as follows:
step 5.1, in the disparity map IcAnd a disparity map IOFor a certain position (x)i,yi) In two parallax images IcAnd IOSelecting a fusion window W with a certain size, and calculating the confidence levels according to an SNR confidence measure method, wherein the confidence levels are calculated as follows:
si＝|dli-dri| (11)，
wherein d isliObtaining a disparity value, d, for the reference image as the primary imageriIs the image to be matched isObtaining parallax of the main image;
step 5.2, comparing the two disparity maps (x)i,yi) Selecting the confidence level with high confidence as the final parallax; according to the weight and the confidence level of each parallax in the fusion window, the total confidence level can be obtained as follows:
Figure BDA0002282949880000101
wherein the weight is
Figure BDA0002282949880000102
di(xc,yc) For blending the disparity values of the central pixels of the window, di(x, y) is the disparity value of the pixels in the fusion window, and the final disparity is defined as follows:
D＝argmax[ci(x,y,di)]，
and (4) obtaining the finally fused disparity map I (u) according to the processing of the steps, wherein the obtained disparity map can solve the problems of detail loss and large noise.
Since the convolutional neural network has a certain defect in understanding the image features, even though the problem of insufficient edge information is improved. However, the obtained disparity map still has the problem of detail loss. However, the optical flow estimation algorithm not only has a good effect on processing the motion information of objects in the image, but also is sensitive to other detailed information in the image, and the single optical flow algorithm has the problem of double images when estimating the image parallax. Therefore, in order to solve the problems of double images and lack of details of the disparity map, the two algorithms are fused, and the difference is made up by taking the advantages and the disadvantages, and the disparity map is obtained according to the fusion algorithm. Aiming at the problems in stereo matching of images with low textures and discontinuous depths, the invention approximates stereo parallax through horizontal optical flow, and simultaneously improves the accuracy of parallax values of low textures and discontinuous depth regions and generates a high-quality parallax image by fusing a convolutional neural network-based parallax image and an improved optical flow image.
The convolutional neural network is a popular algorithm for binocular disparity estimation at present, and many scholars make many innovations on network structure in order to improve training performance of the neural network and accelerate convergence rate. However, many approaches fail to compromise training performance with proficiency speed. For example, a method of increasing the number of feature extraction layers of an original network will bring a larger calculation scale. In order to obtain a high-quality parallax map in a simpler and more convenient mode, the method of the invention develops a new way, is not limited by the design on the neural network, skillfully combines the improved convolutional neural network with the LK optical flow estimation algorithm, and in the design of the method, under the condition of ensuring the rapid convergence of the convolutional neural network, the LK optical flow estimation algorithm is utilized to make up the problem of detail loss caused by the shortage of training samples of the convolutional neural network. The binocular disparity map enhancement method based on confidence fusion has the difficulty that the fusion mode of the two disparity maps is selected, and the two disparity maps are fused by a plurality of fusion methods and a plurality of experimental verifications, so that the method meets the effective information of the two disparity maps required by the method, and can achieve good disparity maps.
Example (b):
in order to verify the effectiveness of the method, a Tsukuba image is selected, and the traditional method SGBM is compared with the binocular disparity map enhancement method based on confidence fusion.
First, image correction is performed on the input left Tsukuba map of fig. 2(a) and right Tsukuba map of fig. 2(b) according to the method in the above step S2; and then performing disparity estimation on the corrected left Tsukuba map and right Tsukuba map by the improved convolutional neural network method described in step S3: the method comprises the steps of using a KITTI Stereo 2012 data set as a training set, using a Tsukuba graph as a test set, updating weights through optimizing a cost function through calculation of a convolution layer, and finally obtaining a disparity map IcAs shown in fig. 2 (d). The left Tsukuba map and the right Tsukuba map corrected by the method in the step S2 are processed by the LK optical flow estimation algorithm in the step S4 to obtain an optical flow field, and a disparity map I is obtainedOAs shown in fig. 2 (c). Then, the disparity map I is usedcAnd a disparity map IO. The disparity map i (u) is obtained by the memorable disparity map fusion according to the confidence level in step S5 in the present invention, and the result is shown in fig. 2 (e). Meanwhile, based on the same group of Tsukuba graphs, we perform disparity estimation based on a traditional algorithm SGBM (semi-global block matching) to obtain a disparity map as shown in fig. 2 (f).
Based on the proposed inventive method, a set of experiments was performed, and we compared the performance of the method based on the percentage of bad pixels represented by the percentage of pixels in the disparity map with disparity values different from the true values to the total pixels. According to the experimental data, the poor pixel rate of the final parallax result is 3.4% (Tsukuba graph), which is reduced by 2.08% (Tsukuba graph) compared to the conventional SGBM (semi-global block matching) method. From the experimental results, as shown in fig. 2(e), it can be seen that the edge region has no missing information and the contour in the image has no ghost phenomenon.

Claims (4)

1. A binocular disparity map enhancement method based on confidence fusion is characterized by comprising the following steps:
s1, placing the picture in front of the center of the binocular camera, and shooting a target by using the binocular camera to obtain a left picture a and a right picture b of the binocular picture; the camera coordinate system takes the center of the camera as an origin, x points to the right of the camera, y points to the upper part of the camera, and z points to the front of the camera;
s2, respectively carrying out stereogram distortion correction on the left image a and the right image b to obtain a stereogram corrected left image a1 and a stereogram corrected right image b 1;
s3, performing disparity estimation on the left image a1 and the right image b1 by adopting a convolutional neural network algorithm to obtain a dense disparity map a 2;
performing disparity map optimization processing on the dense disparity map a2 to obtain an optimized disparity map Ic；
S4, carrying out parallax estimation on the stereogram corrected left image a1 and right image b1 by adopting an improved optical flow estimation algorithm to obtain a parallax image IO；
The specific process of step S4 is as follows:
and 4. step 4.1. Acquiring a first derivative and a second derivative of the point in the images a1 and b1 according to an optical flow equation; the pixel points of the fixed window have the same motion and have smaller displacement; for the left graph a1, define uaThe moving speed v of the pixel point in the x-axis directionaThe motion speed of the pixel point in the y-axis direction is obtained; wherein Ix，IyAnd ItThe derivatives of the image with respect to x-direction, y-direction, and time, respectively; in the right graph b1, u is definedbThe moving speed v of the pixel point in the x-axis directionbObtaining the derivative L of the pixel point in the image b1 for the x direction, the y direction and the time for the moving speed of the pixel point in the y-axis directionx、LyAnd Lt；
Step 4.2, obtaining a characteristic value of the Hessian matrix, and calculating a condition number of the Hessian matrix;
if the condition number is between 0.91 and 0.99, the pixel is a reliable point;
if the condition number is not between 0.91 and 0.99, the pixel is an unreliable point, and the unreliable point is rejected;
4.3, removing unreliable points by using a Hessian matrix, and taking the reciprocal of the condition number corresponding to the points as the weight of the points;
step 4.4, solving the optical flow field according to the weighted least square method [ u v]TAnd [ u 'v']TThe obtained optical flow field is approximate to a parallax value, and a parallax image I is obtainedO；
S5, according to the confidence level, comparing the obtained disparity map IOAnd a disparity map IcAnd (5) performing fusion to obtain a final disparity map I (u).
2. The binocular disparity map enhancement method based on confidence fusion of claim 1, wherein the specific application method of the step S2 is as follows: co-planar the left and right panels a and b; aligning the image lines of the left image a and the right image b; and correcting the left image a and the right image b to obtain a three-dimensional corrected output matrix, namely obtaining a corrected left image a1 and a corrected right image b 1.
3. The binocular disparity map enhancement method based on confidence fusion according to claim 1 or 2, wherein the specific method of the step S3 is as follows:
3.1, carrying out weight distribution on the left graph a1 and the right graph b1 by using a convolutional neural network, wherein each layer of neural network is subjected to homogenization treatment and a ReLU function; feature maps obtained after the left map a1 and the right map b1 pass through a convolutional neural network are sent to a relevant layer to calculate cost, and an initial disparity map is obtained after three basic processes of cost aggregation and disparity prediction;
step 3.2, performing left-right verification on the generated initial disparity map, updating the network weight, and updating the updating rule according to a gradient descent method;
3.3, repeating the step 3.1 and the step 3.2, after updating the network weight, continuing to predict the initial disparity map, and then checking out the pixels with high confidence coefficient to continue training;
step 3.4, after the training is finished, smoothing the predicted disparity map by a median filter to obtain a dense disparity map a 2;
step 3.5, filling points with edge parallax value of 0 in the dense parallax map a2, filling the parallax information with value of 0 by adopting an image fusion method, and fusing to obtain an optimized parallax map Ic。
4. The binocular disparity map enhancement method based on confidence fusion according to claim 1 or 2, wherein the specific process of S5 is as follows:
step 5.1, in the disparity map IcAnd a disparity map IOFor a certain position (x)i,yi) In two parallax images IcAnd IOSelecting a fusion window W with a certain size, and calculating the confidence levels according to an SNR confidence measure method, wherein the confidence levels are calculated as shown in the specification,
si＝|dli-dri|，
Wherein d isliObtaining a disparity value, d, for the reference image as the primary imageriIs the image to be matched isObtaining parallax of the main image;
step 5.2, comparing the two disparity maps (x)i,yi) Selecting the confidence level with high confidence as the final parallax; according to the weight and the confidence level of each parallax in the fusion window, the total confidence level can be obtained as follows:
Figure FDA0002885543950000031
wherein the weight is
Figure FDA0002885543950000032
di(xc,yc) For blending the disparity values of the central pixels of the window, di(x, y) is the disparity value of the pixels in the fusion window, and the final disparity is defined as follows:
D＝arg max[ci(x,y,di)]，
and processing according to the steps to obtain a final fused disparity map I (u).
CN201911148748.5A 2019-11-21 2019-11-21 Binocular disparity map enhancement method based on confidence fusion Active CN111062900B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911148748.5A CN111062900B (en) 2019-11-21 2019-11-21 Binocular disparity map enhancement method based on confidence fusion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911148748.5A CN111062900B (en) 2019-11-21 2019-11-21 Binocular disparity map enhancement method based on confidence fusion

Publications (2)

Publication Number Publication Date
CN111062900A CN111062900A (en) 2020-04-24
CN111062900B true CN111062900B (en) 2021-02-12

Family

ID=70298636

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911148748.5A Active CN111062900B (en) 2019-11-21 2019-11-21 Binocular disparity map enhancement method based on confidence fusion

Country Status (1)

Country Link
CN (1) CN111062900B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230305572A1 (en) * 2021-12-29 2023-09-28 China University Of Mining And Technology Method for drivable area detection and autonomous obstacle avoidance of unmanned haulage equipment in deep confined spaces

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113436119B (en) * 2021-08-25 2021-12-28 上海海栎创科技股份有限公司 Binocular mobile phone preview real-time parallax image calculation method based on optical flow alignment

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2722816A3 (en) * 2012-10-18 2017-04-19 Thomson Licensing Spatio-temporal confidence maps
CN104021525B (en) * 2014-05-30 2017-02-08 西安交通大学 Background repairing method of road scene video image sequence
CN105956597A (en) * 2016-05-04 2016-09-21 浙江大学 Binocular stereo matching method based on convolution neural network
CN106600583B (en) * 2016-12-07 2019-11-01 西安电子科技大学 Parallax picture capturing method based on end-to-end neural network
US10733714B2 (en) * 2017-11-09 2020-08-04 Samsung Electronics Co., Ltd Method and apparatus for video super resolution using convolutional neural network with two-stage motion compensation
CN108520537B (en) * 2018-03-29 2020-02-18 电子科技大学 Binocular depth acquisition method based on luminosity parallax
CN108876836B (en) * 2018-03-29 2021-08-27 北京旷视科技有限公司 Depth estimation method, device and system and computer readable storage medium
US10878590B2 (en) * 2018-05-25 2020-12-29 Microsoft Technology Licensing, Llc Fusing disparity proposals in stereo matching
CN109377530B (en) * 2018-11-30 2021-07-27 天津大学 Binocular depth estimation method based on depth neural network
CN110096950B (en) * 2019-03-20 2023-04-07 西北大学 Multi-feature fusion behavior identification method based on key frame

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230305572A1 (en) * 2021-12-29 2023-09-28 China University Of Mining And Technology Method for drivable area detection and autonomous obstacle avoidance of unmanned haulage equipment in deep confined spaces
US11880208B2 (en) * 2021-12-29 2024-01-23 China University Of Mining And Technology Method for drivable area detection and autonomous obstacle avoidance of unmanned haulage equipment in deep confined spaces

Also Published As

Publication number Publication date
CN111062900A (en) 2020-04-24

Similar Documents

Publication Publication Date Title
CN102665086B (en) Method for obtaining parallax by using region-based local stereo matching
CN108470370B (en) Method for jointly acquiring three-dimensional color point cloud by external camera of three-dimensional laser scanner
CN106408513B (en) Depth map super resolution ratio reconstruction method
AU2016355215B2 (en) Methods and systems for large-scale determination of RGBD camera poses
CN105513064B (en) A kind of solid matching method based on image segmentation and adaptive weighting
CN108564617A (en) Three-dimensional rebuilding method, device, VR cameras and the panorama camera of more mesh cameras
CN109859105B (en) Non-parameter image natural splicing method
CN110853151A (en) Three-dimensional point set recovery method based on video
CN106023230B (en) A kind of dense matching method of suitable deformation pattern
CN110910431B (en) Multi-view three-dimensional point set recovery method based on monocular camera
CN113256698B (en) Monocular 3D reconstruction method with depth prediction
CN106780573B (en) A kind of method and system of panorama sketch characteristic matching precision optimizing
CN103426190B (en) The method and system of image reconstruction
CN111062900B (en) Binocular disparity map enhancement method based on confidence fusion
CN111047709A (en) Binocular vision naked eye 3D image generation method
CN110517309A (en) A kind of monocular depth information acquisition method based on convolutional neural networks
CN116310131A (en) Three-dimensional reconstruction method considering multi-view fusion strategy
CN115601406A (en) Local stereo matching method based on fusion cost calculation and weighted guide filtering
CN114332125A (en) Point cloud reconstruction method and device, electronic equipment and storage medium
CN117237546A (en) Three-dimensional profile reconstruction method and system for material-adding component based on light field imaging
CN114401391B (en) Virtual viewpoint generation method and device
CN108924434B (en) Three-dimensional high dynamic range image synthesis method based on exposure transformation
CN113077401A (en) Method for stereo correction based on viewpoint synthesis technology of novel network
US20220068018A1 (en) Method for 3d reconstruction of an object
CN111369435A (en) Color image depth up-sampling method and system based on self-adaptive stable model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant