CN117011560A - Coal mine underground image stereo matching method based on threshold and weight Census transformation - Google Patents
Coal mine underground image stereo matching method based on threshold and weight Census transformation Download PDFInfo
- Publication number
- CN117011560A CN117011560A CN202310950032.7A CN202310950032A CN117011560A CN 117011560 A CN117011560 A CN 117011560A CN 202310950032 A CN202310950032 A CN 202310950032A CN 117011560 A CN117011560 A CN 117011560A
- Authority
- CN
- China
- Prior art keywords
- value
- parallax
- pixel
- census
- threshold
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 68
- 239000003245 coal Substances 0.000 title claims abstract description 27
- 230000009466 transformation Effects 0.000 title claims abstract description 18
- 238000004364 calculation method Methods 0.000 claims abstract description 20
- 238000012545 processing Methods 0.000 claims abstract description 20
- 230000002776 aggregation Effects 0.000 claims abstract description 16
- 238000004220 aggregation Methods 0.000 claims abstract description 16
- 230000008569 process Effects 0.000 claims abstract description 9
- 230000000007 visual effect Effects 0.000 claims abstract description 5
- 238000004422 calculation algorithm Methods 0.000 claims description 48
- 238000003860 storage Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 15
- 239000011159 matrix material Substances 0.000 claims description 12
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 6
- 208000006440 Open Bite Diseases 0.000 claims description 4
- 230000004927 fusion Effects 0.000 claims description 4
- 241001116389 Aloe Species 0.000 claims description 3
- 235000011399 aloe vera Nutrition 0.000 claims description 3
- 239000003086 colorant Substances 0.000 claims description 3
- 238000001514 detection method Methods 0.000 claims description 3
- 238000002474 experimental method Methods 0.000 claims description 3
- 238000009499 grossing Methods 0.000 claims description 3
- 238000003384 imaging method Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000013519 translation Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 abstract description 5
- 238000005286 illumination Methods 0.000 abstract description 4
- 239000000428 dust Substances 0.000 abstract description 2
- 238000011426 transformation method Methods 0.000 abstract 1
- 150000003839 salts Chemical class 0.000 description 14
- 239000006002 Pepper Substances 0.000 description 13
- 235000002566 Capsicum Nutrition 0.000 description 12
- 241000722363 Piper Species 0.000 description 12
- 235000016761 Piper aduncum Nutrition 0.000 description 12
- 235000017804 Piper guineense Nutrition 0.000 description 12
- 235000008184 Piper nigrum Nutrition 0.000 description 12
- 238000010586 diagram Methods 0.000 description 8
- 230000000694 effects Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 239000002817 coal dust Substances 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000016776 visual perception Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/30—Noise filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a coal mine underground image stereo matching method based on threshold and weight Census transformation, which comprises the following steps: s1, acquiring image information through a binocular cradle head formed by two monocular cameras; s2, carrying out threshold processing on gray values of all pixels in the support window; s3, acquiring a matching cost value by an improved center point pixel calculation method; s4, obtaining a cost aggregation value by the improved dynamic cross domain; s5, using a WTA strategy to obtain a parallax value; s6, verifying the whole process on a visual system of the underground unmanned carrier vehicle. The method applies the Census transformation method based on the threshold value and the weight to underground coal mine sensing, realizes the autonomous obstacle avoidance and visual reconnaissance functions of the underground coal mine unmanned transport vehicle, reduces the influence of factors such as dust, unstable illumination conditions and the like on the stereo matching, and improves the accuracy of the stereo matching.
Description
Technical Field
The invention belongs to the technical field of coal mine safety production, and particularly relates to a coal mine underground image stereo matching method based on threshold and weight Census transformation.
Background
In the underground coal mine environment, matching cost calculation of binocular stereo matching is susceptible to coal dust and particulate matters, unstable illumination and long-time exposure of a vision sensor, so that salt and pepper noise is generated during image acquisition, and the accuracy of stereo matching is greatly reduced. Therefore, it is important to reduce the computational cost of matching for the visual system of the unmanned carrier vehicle in the coal mine.
By comparing the matching performance of different matching cost calculation methods under the salt and pepper noise, the method for calculating the matching cost based on the pixel gray value by utilizing the sum of the gray difference absolute values and the like is greatly influenced by the salt and pepper noise, and the method for calculating the matching cost based on Census transformation has stronger robustness on the salt and pepper noise, so that the method has important significance on the selection of the matching cost calculation method in the stereo matching of the underground unmanned vehicles of the coal mine. However, the conventional Census is too dependent on the center point pixel of the transform window, and is susceptible to noise. Aiming at the defect, a person combines the absolute value of gray difference and Census transformation, adds the cost value of AD and the cost value of Census in a normalized way when calculating the cost, not only considers the absolute value information of three color differences, but also carries out Census transformation on a central pixel, and fully utilizes the cross-correlation information among pixels. Researchers also improve the Census algorithm, adopt Census transformation based on the mean value, and take the mean value of the gray values of all pixel points in the window to replace the gray value of the central pixel when calculating the matching cost. The method avoids over-dependence on the central pixel point, but has strong dependence on the pixel point in the window, and under the condition of salt and pepper noise, the abnormal value can have larger influence on mean value calculation, so that mismatching is increased. In the underground coal mine environment, the salt and pepper noise can increase the error matching rate of stereo matching, damage the continuity of parallax, and reduce the stability and the robustness of matching. Therefore, research and improvement of the stereo matching algorithm aiming at the influence of the spiced salt noise are very important.
Therefore, in view of the above problems, it is necessary to provide a stereo matching algorithm based on threshold and weight Census transformation, so as to effectively reduce the influence of salt and pepper noise on stereo matching, reduce matching errors, improve matching precision, and provide reliable perception information for an intelligent decision system of an unmanned transportation vehicle.
Disclosure of Invention
The invention discloses a coal mine underground image three-dimensional matching algorithm based on threshold and weight Census transformation, which can provide a better three-dimensional matching algorithm for visual perception of an unmanned auxiliary transport vehicle aiming at factors such as high coal dust, unstable illumination and the like in the coal mine underground, reduces matching errors and improves matching precision.
In order to solve at least one of the above technical problems, according to an aspect of the present invention, there is provided a method for stereo matching of images under coal mine based on threshold and weight Census transformation, comprising the steps of:
s1, acquiring image information through a binocular cradle head formed by two monocular cameras;
s2, carrying out threshold processing on gray values of all pixels in the support window;
s3, obtaining a matching cost value by using an improved center point pixel calculation method;
s4, obtaining a cost aggregation value by using the improved dynamic cross domain;
s5, a WTA strategy is used for obtaining the parallax value, and then the parallax map and the parallax value are optimized by using a left-right consistency check method, an improved parallax filling method, a median filtering method and a sub-pixel refinement method.
Preferably, in S2, the threshold processing is performed on the gray values of all pixels in the support window, including the following steps:
s21, traversing gray values of all pixel points in a window, and finding out a maximum value and a minimum value;
s22, comparing the maximum value and the minimum value with a set maximum threshold value and a set minimum threshold value, and selecting the compared maximum value and minimum value as the threshold values;
s23, comparing the obtained threshold value with all pixels in the window, and removing the pixels which are larger than or equal to the maximum threshold value and smaller than or equal to the minimum threshold value in the window.
Preferably, in S3, the improved center point pixel calculation method obtains a matching cost value, including the following steps:
s31, selecting four diagonal pixels of upper left, lower left, upper right and lower right corresponding to the center point of the pixel in the processed window in the S2;
s32, respectively taking the pixel points with the step sizes of 1, 2 and 3 on each diagonal line according to the distance between the four diagonal line pixels and the central pixel point, wherein the total number of the pixel points is 12;
s33, respectively giving weights to the 12 pixel points according to step sizes 1, 2 and 3, wherein the weights are not given to the pixel points which are subjected to threshold processing in the window and are on the diagonal line, and the specific formula is as follows:
wherein I (a) represents the gray value of the pixel point on the diagonal line; x and y represent coordinates of a pixel point at the center of the window; h, w represents pixel point coordinates on the diagonal line;
s34, multiplying effective pixel points in the 12 pixel points by corresponding weights respectively, and dividing the effective pixel points by the sum of the effective weights to obtain a weighted central pixel point value, wherein the specific formula is as follows:
I(p 1 )＝(0.7*I(x±1,j±1)+0.2*I(x±2,j±2)+0.1*I(x±3,j±3))/4 (2)
in the middle ofI(p 1 ) Representing the weighted gray value of the center point, and I (x+ -I, j+ -j) represents the gray value on the diagonal line;
s35, comparing the calculated value of the central pixel point with other pixels in the window to generate a binary string, wherein the specific formula is as follows:
wherein T is Census (p) represents a binary string generated by window transformation of the center pixel p; n (N) p Then the neighborhood of pixel p is represented; i (p) is the gray value of the central pixel point p after weighted fusion treatment; i (q) is the gray value of the pixel q in the neighborhood;representing a bitwise connector;
performing exclusive or operation on binary strings of pixels corresponding to the left image and the right image, and counting the number of 1 in the binary strings of the exclusive or operation result to obtain a hamming distance, thereby obtaining a matching cost calculation value, wherein the specific formula is as follows:
C Census (p,q)＝Hamming(T(p),T(q)) (4)
wherein T (p) is a binary string generated by a left graph, T (q) is a binary string generated by a right graph, and the Hamming distance is the number of different corresponding bits of the two binary strings;
s36, in order to verify the advance of the Census algorithm based on the threshold and the weight in the matching cost calculation of the algorithm, respectively carrying out a comparison experiment on the traditional Census, the average Census and the Census method adopted in the text, adopting four groups of images Cones, teddy, dolls and Aloe in the Middlebury data set, comparing an initial matching cost graph and a standard parallax graph which are obtained by the traditional Census, the average Census and the Census method adopted in the text, and analyzing the mismatching rate of all the areas and the mismatching rate of the non-occlusion area;
s37, comparing the algorithm with a traditional Census algorithm and a Census algorithm based on a mean value, respectively calculating matching cost values under different noise, and eliminating interference of cost aggregation and parallax optimization.
Preferably, the specific steps of S4 are as follows:
s41, setting three different colors and distance thresholds, so as to generate three dynamic crisscross domains with different sizes for the same pixel point;
s42, calculating an average value of three dynamic cross domain cost aggregation values corresponding to the pixel point to serve as a final cost aggregation value of the pixel point.
Preferably, the specific steps of S5 are as follows:
s51, selecting a characteristic value with the smallest parallax value for each pixel point as the parallax value of the pixel point by using a WTA strategy according to the cost aggregation value in the S4;
s52, detecting error matching parallax points by adopting a left-right consistency detection method, and detecting abnormal points by comparing parallax values of the left graph and the right graph matching parallax points, wherein the calculation process is as follows:
|d l (p)-d r (p+d)|＞α (5)
wherein d l (p) is the disparity value of point p in the left matching graph, d r (p+d) is the disparity value corresponding to the point p in the right matching graph, and alpha is the judgment threshold;
s53, performing parallax filling on the mismatching area obtained after left-right consistency checking by adopting an improved 8-way interpolation method, taking the average value of ten nearest effective values of each method as the parallax value of the method, and then taking the 8-way average value as the parallax value of the mismatching area;
s54, performing median filtering smoothing processing on the parallax map by adopting a 3 multiplied by 3 sliding window to remove some isolated outliers in the parallax map, so as to obtain a final parallax map;
s55, carrying out sub-pixel refinement on the parallax value obtained through WTA, and obtaining the parallax value represented by the minimum value point of the quadratic curve as a pixel parallax value;
s56, comparing parallax graphs generated by the three algorithms for verifying the performance of the algorithm.
Preferably, the specific steps of S6 are as follows:
s61, acquiring a black-and-white checkerboard image by using a binocular camera holder of the underground unmanned transport vehicle, and calibrating a camera of the binocular camera holder;
s62, selecting 25 images with clear imaging and multi-angle shooting, and calibrating the images; correcting the image by using the obtained parameters, and obtaining an internal reference matrix L of the left camera, an internal reference matrix R of the right camera, a rotation matrix Q and a translation matrix T through calibration, namely:
T＝[-309.467 11.985 -22.362] (6)；
s63, the calibrated left and right images are subjected to the stereo matching algorithm, and a corresponding parallax image is obtained.
According to another aspect of the invention there is provided a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps in the method of stereo matching of images downhole in a coal mine based on threshold and weight Census transforms of the present invention.
According to a further aspect of the invention there is provided a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps in the method of stereo matching of images downhole in coal mine based on threshold and weight Census transforms of the invention when the program is executed.
Compared with the prior art, the invention has at least the following beneficial effects:
the three-dimensional matching algorithm based on the threshold and the weight Census transformation is applied to visual perception of the unmanned transport vehicle in the coal mine, so that the environment perception and the obstacle positioning of the unmanned transport vehicle are realized, a safe driving path is planned, three-dimensional matching can be performed in the coal mine more accurately, the defect of three-dimensional matching in the coal mine in the traditional three-dimensional matching algorithm is overcome, and the three-dimensional matching precision is improved.
The invention can effectively reduce the proportion of the salt and pepper noise in the window and reduce the influence of the salt and pepper noise on the stereo matching through the threshold processing. This helps to improve the accuracy and stability of the matching result; meanwhile, the influence of the abnormal value on the weighted fusion can be solved by the threshold processing, and misleading of the abnormal value on the matching result is avoided. The method is beneficial to improving the robustness of the algorithm to factors such as high dust, unstable illumination conditions and the like in the underground coal mine environment; and a larger weight is distributed to reliable pixel points through a weight strategy, so that the excessive dependence of the traditional algorithm on the center point of the Census transformation window is avoided. This helps to improve the matching accuracy and reduce the false matching rate.
Compared with a global matching algorithm, the method has lower calculation complexity and can more efficiently carry out stereo matching. This makes algorithms more viable in real-time applications and large-scale data processing.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following brief description of the drawings of the embodiments will make it apparent that the drawings in the following description relate only to some embodiments of the present invention and are not limiting of the present invention.
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a comparison chart of three algorithm cost calculations of the original chart;
FIG. 3 is a histogram of the mismatch rates of all regions under salt and pepper noise for three algorithms (noise density on the abscissa and mismatch rates of all regions on the ordinate);
FIG. 4 is a histogram of the mismatch ratio of non-occluded areas under salt and pepper noise for three algorithms (noise density on the abscissa and non-occluded areas mismatch ratio on the ordinate);
fig. 5 is a graph comparing experimental results under 10% salt-and-pepper noise;
FIG. 6 is a reprojection error map;
FIG. 7 is left and right images before correction;
FIG. 8 is a corrected left and right image;
fig. 9 is a view of three algorithm coal mine downhole disparities.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. It will be apparent that the described embodiments are some, but not all, embodiments of the invention.
Unless defined otherwise, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this invention belongs.
As shown in figures 1-9 of the drawings,
example 1:
the coal mine underground image stereo matching algorithm based on the threshold and weight Census transformation shown in fig. 1 comprises the following steps:
s1, acquiring image information through a binocular cradle head formed by two monocular cameras;
s2, carrying out threshold processing on gray values of all pixels in a support window, wherein the threshold processing comprises the following steps:
s21, traversing gray values of all pixel points in a window, and finding out a maximum value and a minimum value;
s22, comparing the maximum value and the minimum value with a set maximum threshold value and a set minimum threshold value, and selecting the compared maximum value and minimum value as the threshold values;
s23, comparing the obtained threshold value with all pixels in the window, and removing the pixels which are larger than or equal to the maximum threshold value and smaller than or equal to the minimum threshold value in the window.
S3, acquiring a matching cost value by an improved center point pixel calculation method, wherein the method comprises the following steps of:
s31, selecting four diagonal pixels of upper left, lower left, upper right and lower right corresponding to the center point of the pixel in the processed window in the S2;
s32, respectively taking the pixel points with the step sizes of 1, 2 and 3 on each diagonal line according to the distance between the four diagonal line pixels and the central pixel point, wherein the total number of the pixel points is 12;
s33, respectively giving weights to the 12 pixel points according to step sizes 1, 2 and 3, wherein the pixel points which are subjected to threshold processing in a window and are on an inclined diagonal line do not need to be given with weights, and the specific formula is as follows:
wherein I (a) represents the gray value of the pixel point on the diagonal line; x and y represent coordinates of a pixel point at the center of the window; h, w represents pixel coordinates on the diagonal.
S34, multiplying effective pixel points in the 12 pixel points by corresponding weights respectively, and dividing the effective pixel points by the sum of the effective weights to obtain a weighted central pixel point value, wherein the specific formula is as follows:
I(p 1 )＝(0.7*I(x±1,j±1)+0.2*I(x±2,j±2)+0.1*I(x±3,j±3))/4 (2)
wherein I (p) 1 ) Representing the weighted center point gray values, I (x±i, j±j) representing gray values on diagonal lines.
S35, comparing the calculated value of the central pixel point with other pixels in the window to generate a binary string, wherein the specific formula is as follows:
wherein T is Census (p) represents a binary string generated by window transformation of the center pixel p; n (N) p Then the neighborhood of pixel p is represented; i (p) is the gray value of the central pixel point p after weighted fusion treatment; i (q) is the gray value of the pixel q in the neighborhood;representing a bitwise connector.
Performing exclusive or operation on binary strings of pixels corresponding to the left image and the right image, and counting the number of 1 in the binary strings of the exclusive or operation result to obtain a hamming distance, thereby obtaining a matching cost calculation value, wherein the specific formula is as follows:
C Census (p,q)＝Hamming(T(p),T(q)) (4)
wherein T (p) is a binary string generated by the left graph, T (q) is a binary string generated by the right graph, and the Hamming distance is the number of different corresponding bits of the two binary strings.
S36, in order to verify the advancement of the threshold-based and weight Census algorithm in the algorithm in matching cost calculation, comparison experiments are respectively carried out on traditional Census, average Census and the Census method adopted in the process, four groups of images Cones, teddy, dolls and Aloe in the Middlebury data set are adopted, an initial matching cost graph and a standard parallax graph obtained by the three algorithms are compared, and the mismatching rate of all areas and the mismatching rate of non-occlusion areas are analyzed. The experimental results are shown in fig. 2, and fig. 2 is a diagram of the original left, a diagram of the real parallax, a traditional Census, a mean-based Census, and a Census used herein in this order. The error rates of all areas of the three algorithms are shown in table 1, and the mismatch rates of non-occlusion areas are shown in table 2.
S37, in order to further verify the noise resistance of the algorithm, the algorithm is compared with the traditional Census algorithm and the Census algorithm based on the mean value, the matching cost values under different noise are calculated respectively, and interference of cost aggregation and parallax optimization is eliminated.
And (3) respectively adding salt and pepper noise with the noise density of 0.01,0.05,0.1,0.15,0.2 to the test picture in the experimental process, calculating initial matching cost graphs of 4 images under different noise by using 3 algorithms, comparing the obtained initial matching cost graphs with a standard parallax graph, and calculating the mismatching rate of all areas and the mismatching rate of non-shielding areas in the 4 images by each algorithm under each noise, wherein the results are shown in tables 3 and 4.
S4, obtaining a cost aggregation value by the improved dynamic cross domain, wherein the method comprises the following steps of:
s41, setting three different colors and distance thresholds, so as to generate three dynamic crisscross domains with different sizes for the same pixel point;
s42, calculating an average value of three dynamic cross domain cost aggregation values corresponding to the pixel point to serve as a final cost aggregation value of the pixel point.
S5, a WTA strategy is used for obtaining a parallax value, and further, a left-right consistency check method, an improved parallax filling method, a median filtering method and a sub-pixel refinement method are used for optimizing the parallax map and the parallax value, and the method comprises the following steps:
s51, selecting a characteristic value with the smallest parallax value for each pixel point as the parallax value of the pixel point by using a WTA strategy according to the cost aggregation value in the S4;
s52, detecting error matching parallax points by adopting a left-right consistency detection method, and detecting abnormal points by comparing parallax values of the left graph and the right graph matching parallax points, wherein the calculation process is as follows:
|d l (p)-d r (p+d)|＞α (5)
wherein d l (p) is the disparity value of point p in the left matching graph, d r (p+d) is the disparity value corresponding to the point p in the right matching chart, and α is the determination threshold.
And S53, performing parallax filling on the mismatching area obtained after the left-right consistency check by adopting an improved 8-way interpolation method, taking the average value of ten nearest effective values of each method as the parallax value of the method, and then taking the 8-way average value as the parallax value of the mismatching area.
S54, performing median filtering smoothing processing on the parallax map by adopting a 3 multiplied by 3 sliding window to remove some isolated outliers in the parallax map, so as to obtain a final parallax map.
S55, carrying out sub-pixel refinement on the parallax value obtained through WTA, and obtaining the parallax value represented by the minimum value point of the quadratic curve as a pixel parallax value.
S56, in order to verify the performance of the algorithm, parallax graphs generated by the three algorithms are compared, and an effect graph is shown in FIG. 5. The method comprises the steps of displaying an original left image, a real parallax image, a left image of 0.1 spiced salt noise, a traditional Census initial matching cost image under noise, a mean-based Census initial matching cost image under noise, an algorithm initial matching cost image under noise, a final parallax image under noise and an algorithm mismatching pixel image under noise of four images from top to bottom.
S6, verifying the whole process on a visual system of the underground unmanned carrier vehicle.
S61, acquiring black and white checkerboard images by using a binocular camera holder of the underground unmanned transport vehicle, and calibrating a camera of the binocular camera holder. And (3) adopting black and white checkerboards as calibration boards, wherein the size of each checkerboard is 41mm, and obtaining the internal and external parameters of the camera through calibration.
S62, selecting 25 images with clear imaging and multi-angle shooting, calibrating the images, wherein the re-projection error calibrated by Matlab software is shown in figure 6, and the average error is 0.22, so that the practical standard is achieved. The image is corrected using the obtained parameters, the effects of which are shown in fig. 7 and 8. Obtaining an internal reference matrix L of the left camera, an internal reference matrix R of the right camera, a rotation matrix Q and a translation matrix T through calibration, namely:
T＝[-309.467 11.985 -22.362] (6)
s63, the calibrated left and right images are subjected to the stereo matching algorithm to obtain corresponding parallax images, and the effect is shown in FIG. 9. The left image obtained by using the binocular camera head is sequentially from left to right, and the traditional Census parallax image, the average Census parallax image and the algorithm parallax image are obtained.
Example 2:
the computer readable storage medium of the present embodiment has stored thereon a computer program which, when executed by a processor, implements the steps in the coal mine downhole image stereo matching method based on threshold and weight Census transform of embodiment 1.
The computer readable storage medium of the present embodiment may be an internal storage unit of the terminal, for example, a hard disk or a memory of the terminal; the computer readable storage medium of the present embodiment may also be an external storage device of the terminal, for example, a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, etc. provided on the terminal; further, the computer-readable storage medium may also include both an internal storage unit of the terminal and an external storage device.
The computer-readable storage medium of the present embodiment is used to store a computer program and other programs and data required for a terminal, and the computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
Example 3:
the computer device of this embodiment includes a memory, a processor, and a computer program stored on the memory and executable on the processor, which when executed implements the steps in the method for stereo matching of images downhole in coal mine based on threshold and weight Census transform of embodiment 1.
In this embodiment, the processor may be a central processing unit, or may be other general purpose processors, digital signal processors, application specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like, where the general purpose processor may be a microprocessor or the processor may also be any conventional processor, or the like; the memory may include read only memory and random access memory, and provide instructions and data to the processor, and a portion of the memory may also include non-volatile random access memory, e.g., the memory may also store information of the device type.
It will be appreciated by those skilled in the art that the embodiment(s) disclosure may be provided as a method, system, or computer program product. Thus, the present approach may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present aspects may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present aspects are described with reference to flowchart illustrations and/or block diagrams of methods and computer program products according to embodiments of the invention, it being understood that each flowchart illustration and/or block diagram illustration, and combinations of flowcharts and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions; these computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Those skilled in the art will appreciate that implementing all or part of the above-described methods in accordance with the embodiments may be accomplished by way of a computer program stored on a computer readable storage medium, which when executed may comprise the steps of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-only memory (ROM), a random access memory (RandomAccessMemory, RAM), or the like.
The examples of the present invention are merely for describing the preferred embodiments of the present invention, and are not intended to limit the spirit and scope of the present invention, and those skilled in the art should make various changes and modifications to the technical solution of the present invention without departing from the spirit of the present invention.
Claims (8)
1. The coal mine underground image stereo matching method based on threshold and weight Census transformation is characterized by comprising the following steps of:
s1, acquiring image information through a binocular cradle head formed by two monocular cameras;
s2, carrying out threshold processing on gray values of all pixels in the support window;
s3, acquiring a matching cost value by an improved center point pixel calculation method;
s4, obtaining a cost aggregation value by the improved dynamic cross domain;
s5, a WTA strategy is used for obtaining a parallax value, and then the parallax map and the parallax value are optimized by using a left-right consistency check method, an improved parallax filling method, a median filtering method and a sub-pixel refinement method;
s6, verifying the whole process on a visual system of the underground unmanned carrier vehicle.
2. The method according to claim 1, characterized in that the specific step of S2 is as follows:
s21, traversing gray values of all pixel points in a window, and finding out a maximum value and a minimum value;
s22, comparing the maximum value and the minimum value with a set maximum threshold value and a set minimum threshold value, and selecting the compared maximum value and minimum value as the threshold values;
s23, comparing the obtained threshold value with all pixels in the window, and removing the pixels which are larger than or equal to the maximum threshold value and smaller than or equal to the minimum threshold value in the window.
3. The method according to claim 2, characterized in that the specific step of S3 is as follows:
s31, selecting four diagonal pixels of upper left, lower left, upper right and lower right corresponding to the center point of the pixel in the processed window in the S2;
s32, respectively taking the pixel points with the step sizes of 1, 2 and 3 on each diagonal line according to the distance between the four diagonal line pixels and the central pixel point, wherein the total number of the pixel points is 12;
s33, respectively giving weights to the 12 pixel points according to step sizes 1, 2 and 3, wherein the weights are not given to the pixel points which are subjected to threshold processing in the window and are on the diagonal line, and the specific formula is as follows:
wherein I (a) represents the gray value of the pixel point on the diagonal line; x and y represent coordinates of a pixel point at the center of the window; h, w represents pixel point coordinates on the diagonal line;
s34, multiplying effective pixel points in the 12 pixel points by corresponding weights respectively, and dividing the effective pixel points by the sum of the effective weights to obtain a weighted central pixel point value, wherein the specific formula is as follows:
I(p 1 ＝(0.7*I(x±1,j±1)+0.2*I(x±2,j±2)+0.1*I(x±3,j±3))/4 (2)
wherein I (p) 1 ) Representing the weighted gray value of the center point, and I (x+ -I, j+ -j) represents the gray value on the diagonal line;
s35, comparing the calculated value of the central pixel point with other pixels in the window to generate a binary string, wherein the specific formula is as follows:
wherein T is Census (p) represents a binary string generated by window transformation of the center pixel p; n (N) p Then the neighborhood of pixel p is represented; i (p) is the gray value of the central pixel point p after weighted fusion treatment; i (q) is the gray value of the pixel q in the neighborhood;representing a bitwise connector;
performing exclusive or operation on binary strings of pixels corresponding to the left image and the right image, and counting the number of 1 in the binary strings of the exclusive or operation result to obtain a hamming distance, thereby obtaining a matching cost calculation value, wherein the specific formula is as follows:
C Census (p,q)＝Hamming(T(p),T(q)) (4)
wherein T (p) is a binary string generated by a left graph, T (q) is a binary string generated by a right graph, and the Hamming distance is the number of different corresponding bits of the two binary strings;
s36, in order to verify the advance of the Census algorithm based on the threshold and the weight in the matching cost calculation of the algorithm, respectively carrying out a comparison experiment on the traditional Census, the average Census and the Census method adopted in the text, adopting four groups of images Cones, teddy, dolls and Aloe in the Middlebury data set, comparing an initial matching cost graph and a standard parallax graph which are obtained by the traditional Census, the average Census and the Census method adopted in the text, and analyzing the mismatching rate of all the areas and the mismatching rate of the non-occlusion area;
s37, comparing the algorithm with a traditional Census algorithm and a Census algorithm based on a mean value, respectively calculating matching cost values under different noise, and eliminating interference of cost aggregation and parallax optimization.
4. A method according to claim 3, characterized in that the specific step S4 is as follows:
s41, setting three different colors and distance thresholds, so as to generate three dynamic crisscross domains with different sizes for the same pixel point;
s42, calculating an average value of three dynamic cross domain cost aggregation values corresponding to the pixel point to serve as a final cost aggregation value of the pixel point.
5. The method according to claim 4, wherein the specific step of S5 is as follows:
s51, selecting a characteristic value with the smallest parallax value for each pixel point as the parallax value of the pixel point by using a WTA strategy according to the cost aggregation value in the S4;
s52, detecting error matching parallax points by adopting a left-right consistency detection method, and detecting abnormal points by comparing parallax values of the left graph and the right graph matching parallax points, wherein the calculation process is as follows:
|d l (p)-d r (p+d)|＞α (5)
wherein d l (p) is the disparity value of point p in the left matching graph, d r (p+d) is the disparity value corresponding to the point p in the right matching graph, and alpha is the judgment threshold;
s53, performing parallax filling on the mismatching area obtained after left-right consistency checking by adopting an improved 8-way interpolation method, taking the average value of ten nearest effective values of each method as the parallax value of the method, and then taking the 8-way average value as the parallax value of the mismatching area;
s54, performing median filtering smoothing processing on the parallax map by adopting a 3 multiplied by 3 sliding window to remove some isolated outliers in the parallax map, so as to obtain a final parallax map;
s55, carrying out sub-pixel refinement on the parallax value obtained through WTA, and obtaining the parallax value represented by the minimum value point of the quadratic curve as a pixel parallax value;
s56, comparing parallax graphs generated by the three algorithms for verifying the performance of the algorithm.
6. The method according to claim 4, wherein the specific step of S6 is as follows:
s61, acquiring a black-and-white checkerboard image by using a binocular camera holder of the underground unmanned transport vehicle, and calibrating a camera of the binocular camera holder;
s62, selecting 25 images with clear imaging and multi-angle shooting, and calibrating the images; correcting the image by using the obtained parameters, and obtaining an internal reference matrix L of the left camera, an internal reference matrix R of the right camera, a rotation matrix Q and a translation matrix T through calibration, namely:
T＝[-309.467 11.985 -22.362] (6)；
s63, the calibrated left and right images are subjected to the stereo matching algorithm, and a corresponding parallax image is obtained.
7. A computer-readable storage medium having stored thereon a computer program, characterized by: the program when executed by a processor implements the steps in the method for stereo matching of images under coal mine based on threshold and weight Census transform as claimed in any one of claims 1 to 6.
8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the program, performs the steps in the method for stereo matching of images downhole in coal mine based on threshold and weight Census transform as claimed in any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310950032.7A CN117011560A (en) | 2023-07-31 | 2023-07-31 | Coal mine underground image stereo matching method based on threshold and weight Census transformation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310950032.7A CN117011560A (en) | 2023-07-31 | 2023-07-31 | Coal mine underground image stereo matching method based on threshold and weight Census transformation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117011560A true CN117011560A (en) | 2023-11-07 |
Family
ID=88570436
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310950032.7A Pending CN117011560A (en) | 2023-07-31 | 2023-07-31 | Coal mine underground image stereo matching method based on threshold and weight Census transformation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117011560A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117251087A (en) * | 2023-11-17 | 2023-12-19 | 济宁市金桥煤矿 | Coal mine safety simulation interaction method based on virtual reality |
| CN117788570A (en) * | 2024-02-26 | 2024-03-29 | 山东济矿鲁能煤电股份有限公司阳城煤矿 | Bucket wheel machine positioning method and system based on machine vision |
-
2023
- 2023-07-31 CN CN202310950032.7A patent/CN117011560A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117251087A (en) * | 2023-11-17 | 2023-12-19 | 济宁市金桥煤矿 | Coal mine safety simulation interaction method based on virtual reality |
| CN117251087B (en) * | 2023-11-17 | 2024-02-09 | 济宁市金桥煤矿 | Coal mine safety simulation interaction method based on virtual reality |
| CN117788570A (en) * | 2024-02-26 | 2024-03-29 | 山东济矿鲁能煤电股份有限公司阳城煤矿 | Bucket wheel machine positioning method and system based on machine vision |
| CN117788570B (en) * | 2024-02-26 | 2024-05-07 | 山东济矿鲁能煤电股份有限公司阳城煤矿 | Bucket wheel machine positioning method and system based on machine vision |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10909707B2 (en) | System and methods for measuring depth using an array of independently controllable cameras | |
| CN117011560A (en) | Coal mine underground image stereo matching method based on threshold and weight Census transformation | |
| US9070042B2 (en) | Image processing apparatus, image processing method, and program thereof | |
| TWI489418B (en) | Parallax Estimation Depth Generation | |
| CN110414385A (en) | A kind of method for detecting lane lines and system based on homography conversion and characteristic window | |
| CN112991420A (en) | Stereo matching feature extraction and post-processing method for disparity map | |
| CN111899295A (en) | Monocular scene depth prediction method based on deep learning | |
| CN111105452A (en) | High-low resolution fusion stereo matching method based on binocular vision | |
| US20130342741A1 (en) | Method And Apparatus For Correcting For Vignetting In An Imaging System | |
| Kong et al. | Local stereo matching using adaptive cross-region-based guided image filtering with orthogonal weights | |
| CN110188640B (en) | Face recognition method, face recognition device, server and computer readable medium | |
| Kallwies et al. | Triple-SGM: stereo processing using semi-global matching with cost fusion | |
| CN110120012B (en) | Video stitching method for synchronous key frame extraction based on binocular camera | |
| Lv et al. | Stereo matching algorithm based on HSV color space and improved census transform | |
| Damjanović et al. | Local stereo matching using adaptive local segmentation | |
| He et al. | A novel way to organize 3D LiDAR point cloud as 2D depth map height map and surface normal map | |
| CN113763449B (en) | Depth recovery method and device, electronic equipment and storage medium | |
| Quenzel et al. | Beyond photometric consistency: Gradient-based dissimilarity for improving visual odometry and stereo matching | |
| CN112509124B (en) | Depth map obtaining method and system, unmanned aerial vehicle orthogram generating method and medium | |
| CN113077504B (en) | Large scene depth map generation method based on multi-granularity feature matching | |
| Chen et al. | Stereo Matching Algorithm Based on HSV Color Space and Improved Census Transform | |
| Basaru et al. | Quantized census for stereoscopic image matching | |
| He et al. | DarkFeat: noise-robust feature detector and descriptor for extremely low-light RAW images | |
| US12002233B2 (en) | Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints | |
| Irijanti et al. | Local stereo matching algorithm: Using small-color census and sparse adaptive support weight |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |