CN110910438B - High-speed stereo matching algorithm for ultrahigh-resolution binocular image - Google Patents

High-speed stereo matching algorithm for ultrahigh-resolution binocular image Download PDF

Info

Publication number
CN110910438B
CN110910438B CN201811079109.3A CN201811079109A CN110910438B CN 110910438 B CN110910438 B CN 110910438B CN 201811079109 A CN201811079109 A CN 201811079109A CN 110910438 B CN110910438 B CN 110910438B
Authority
CN
China
Prior art keywords
parallax
matching
point
image
value
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201811079109.3A
Other languages
Chinese (zh)
Other versions
CN110910438A (en
Inventor
华春生
随心
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenyang Institute of Automation of CAS
Original Assignee
Shenyang Institute of Automation of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenyang Institute of Automation of CAS filed Critical Shenyang Institute of Automation of CAS
Priority to CN201811079109.3A priority Critical patent/CN110910438B/en
Publication of CN110910438A publication Critical patent/CN110910438A/en
Application granted granted Critical
Publication of CN110910438B publication Critical patent/CN110910438B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/50Depth or shape recovery
    • G06T7/55Depth or shape recovery from multiple images
    • G06T7/593Depth or shape recovery from multiple images from stereo images

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

The invention relates to a high-speed stereo matching algorithm of a binocular image with ultrahigh resolution, which comprises the steps of down-sampling left and right images of a binocular camera, obtaining a parallax range through matching, and setting a threshold value; coding each picture, and calculating matching cost; constructing a minimum spanning tree method according to NLCA, and aggregating matching cost; and determining the parallax by selecting the parallax value corresponding to the point with the minimum matching cost, and reserving the parallax according to the threshold values corresponding to different layer numbers. The invention determines the size of the left and right images to be matched according to the distance of the object, and for the object at the near part, the parallax is large, the occupied pixel points are more, and much matching time can be consumed, so that the matching is carried out by adopting the image with smaller size, and on the contrary, for the object at the far part, the matching is carried out by adopting the image with large size, so that the matching precision of the object at the far part can be ensured.

Description

High-speed stereo matching algorithm for ultrahigh-resolution binocular image
Technical Field
The invention relates to the field of binocular stereo vision, in particular to a high-speed stereo matching algorithm for an ultrahigh-resolution binocular image.
Background
The stereo matching is to search matching pixel points between two or more images of the same scene and different images shot at different visual angles, obtain a depth value by obtaining a parallax value between the matching pixel points, and solve three-dimensional information of an object. The stereo matching is widely applied to the fields of three-dimensional reconstruction, robot navigation, virtual reality and the like.
The stereo matching algorithm is mainly used for estimating the parallax value of a pixel point by establishing an energy cost function and minimizing the energy cost function. The essence of the stereo matching algorithm is an optimization solving problem, a reasonable energy function is established, some constraints are added, an optimization theory method is adopted to solve equations, corresponding matching points of the left and right pictures are finally obtained, and then the disparity value of the points is obtained. The stereo matching generally comprises four steps of matching cost calculation, cost aggregation, parallax calculation and parallax refinement. The matching cost calculation is generally to obtain the matching cost value of each point by calculating the gray level images of the left and right images, and specific algorithms include AD, SAD, census and the like. And obtaining a cost graph of the matching cost after the cost calculation. The number of cost maps is the search range of the disparity. The cost aggregation stage is a filtering stage, each matched cost graph is filtered, and algorithms such as CSCA, NLCA and the like which are popular at present can obtain good effects. In the parallax calculation stage, for a certain pixel point in the image imagery, a point with small matching cost of the point is searched according to the obtained matching cost graph. The final parallax refinement stage is a process of optimizing the obtained parallax, and mainly comprises the following steps: algorithms such as left-right consistency detection, region voting and the like rarely use the steps in practical application.
In conclusion, the important process of stereo matching is also the matching cost calculation and cost aggregation stage. However, most stereo matching only considers matching of a full-size whole image, which wastes a long time in ultrahigh resolution.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a high-speed stereo matching algorithm for a binocular image with ultrahigh resolution, and solves the technical problems that for a near object, a plurality of pixels are occupied on the image, the parallax is large, and the complete calculation costs a lot of time.
The technical scheme adopted by the invention for realizing the purpose is as follows:
a high-speed stereo matching algorithm for an ultrahigh-resolution binocular image comprises the following steps:
step 1: down-sampling left and right images of the binocular camera, matching on the image with the smallest size to obtain the parallax range of the whole image, and setting a threshold value for each layer according to the down-sampling times and the parallax range;
step 2: coding each picture through census transformation, and calculating matching costs of different scales according to different distances;
and step 3: according to the method for constructing the minimum spanning tree by NLCA, aggregation of matching cost is carried out, and a filtered matching cost matrix is obtained;
and 4, step 4: and determining the parallax by selecting the parallax value corresponding to the point with the minimum matching cost, and reserving the parallax according to the threshold values corresponding to different layer numbers.
The encoding process of each picture comprises the following steps:
selecting a point in the graph, drawing a rectangle of n x n by taking the point as the center, comparing each point except the center point in the rectangle with the center point, marking the gray value as 1 when the gray value is smaller than the center point, marking the gray value as 0 when the gray value is larger than the center point, and representing the pixel value of the center point by the obtained census sequence with the length of (n x n-1).
The calculation process of the matching cost comprises the following steps:
the matching cost of each point is calculated by comparing the hamming distance of the left and right images:
Figure BDA0001801448510000021
wherein x and y are the n-bit codes of the gray values of the pixel points of the left and right images,
Figure BDA0001801448510000022
expressing exclusive OR, I expressing a left parallax image, I, j expressing the position in the image, wherein each pixel point is the parallax value of the point, and C expressing the Hamming distance (namely the matching cost value); the smaller the Hamming distance is, the smaller the matching cost is, and the higher the similarity is.
The setting of the threshold for each layer according to the number of downsampling and the parallax range includes:
the number of the threshold values is the number of down-sampling times, the number of the down-sampling times is 1 more than the number of layers, and the size of the threshold value of each layer is selected according to the percentage of the parallax range.
The determining the parallax by selecting the parallax value corresponding to the point with the minimum matching cost includes the following processes:
Figure BDA0001801448510000031
wherein d (C)min(x(il,jl),y(ir,jr) ) represents the parallax value when the cost is minimum, I represents the left parallax image, I, j represents the position in the image, each pixel point is the parallax value of the point, and C represents the matching cost value.
The threshold reserved parallax according to different layer numbers is as follows:
the smaller the threshold value is, the larger the corresponding layer number is; and retains a disparity greater than a threshold.
The invention has the following beneficial effects and advantages:
1. the method is different from other stereo matching algorithms in that a matching strategy of an image pyramid is proposed and adopted during the calculation of the matching cost, and only the cost value without matching points is calculated during the calculation of the cost, so that the calculation time on the maximum graph can be greatly reduced through the calculation, and the matching speed can be improved;
2. the invention provides and uses a calculation strategy based on an image pyramid in the process of calculating the parallax, and calculates the near object on the left and right camera images with small scale. When calculating distant objects, first consider whether matching has already been performed on a small scale, and only points that have not been matched are calculated. This has the advantage of reducing computation time and also reduces interference from nearby objects at the time of matching.
Drawings
FIG. 1 is a block diagram of a process flow;
FIG. 2 is a flowchart of an algorithm for matching cost calculation;
fig. 3 is a flowchart of an algorithm for calculating parallax.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples.
In order to make the aforementioned objects, features and advantages of the present invention more comprehensible, embodiments accompanying the drawings are described in detail below. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, but rather should be construed as modified in the spirit and scope of the present invention as set forth in the appended claims.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
The present invention is described by taking the example of using three-level matching and performing two downsampling operations, and is not intended to limit the present invention.
As shown in fig. 1, firstly, down-sampling is performed on the pictures acquired by the left and right cameras, and the down-sampling is performed twice because three-layer matching is used in the application; selecting a disparity threshold θ12. And calculating the matching cost of each pixel, coding each picture by census transformation, and calculating the matching cost of different scales according to the difference of distance. And constructing a minimum spanning tree mode by using NLCA for aggregation of matching costs, wherein the weight of the edge of the minimum spanning tree is at least defined by the distance of matching cost vector characteristics of two pixel points connecting the edge. Calculating parallax, determining parallax by selecting corresponding parallax value of point with minimum matching cost, selecting different matching modes on different scales, and storing parallax larger than theta for object at the nearest position2A second layer preserving a parallax greater than theta1The last layer retains all points remaining.
In the matching cost calculation stage, as shown in fig. 2, the left and right camera pictures are subjected to gray scale processing, then a point is selected from the picture, each point except the central point in a rectangle with the point as the center, n is drawn out (the rectangle with the size of 9 × 9 is selected in this application), the gray value is smaller than the central point and is recorded as 1, the gray value larger than the central point is recorded as 0, and the obtained census sequence with the length of (n-1) represents the pixel value of the central point.
Determining the disparity values of points needing to be matched on the left and right images, if the disparity values are matched, not needing to be matched, and if the disparity values are not matched, calculating the similarity of each point by comparing the Hamming distances of the left and right images:
Figure BDA0001801448510000051
where x, y are both n-bit encodings of the gray scale values of the left and right image pixel points,
Figure BDA0001801448510000052
expressing exclusive OR, I expressing a left parallax image, I, j expressing the position in the image, wherein each pixel point is the parallax value of the point, and C expressing the Hamming distance (namely the matching cost value); the smaller the Hamming distance, the higher the similarity.
In the parallax calculation section, as shown in fig. 3, we first determine whether the point has already been matched, and if so, do not need to perform any further calculation. If not, selecting the parallax value corresponding to the minimum matching cost point, and calculating according to the following formula
Figure BDA0001801448510000053
In the above formula, d (C)min(x(il,jl),y(ir,jr) ) represents the parallax value when the cost is minimum, I represents the left parallax image, I, j represents the position in the image, each pixel point is the parallax value of the point, and C represents the matching cost value.
After matching, judging to determine whether the parallax value needs to be reserved, wherein the parallax of the object at the nearest position is larger than theta2A second layer preserving a parallax greater than theta1Point of (2), last layerAll points remaining are retained.

Claims (4)

1. A high-speed stereo matching algorithm of an ultrahigh-resolution binocular image is characterized in that: the method comprises the following steps:
step 1: down-sampling left and right images of the binocular camera, matching on the image with the smallest size to obtain the parallax range of the whole image, and setting a threshold value for each layer according to the down-sampling times and the parallax range;
step 2: coding each picture through census transformation, and calculating matching costs of different scales according to different distances;
the calculation process of the matching cost comprises the following steps:
the matching cost of each point is calculated by comparing the hamming distance of the left and right images:
Figure FDA0003486804980000011
wherein x and y are the n-bit codes of the gray values of the pixel points of the left and right images,
Figure FDA0003486804980000012
expressing exclusive OR, I expressing a left parallax image, I, j expressing the position in the image, wherein each pixel point is the parallax value of the point, and C expressing Hamming distance; the smaller the Hamming distance is, the smaller the matching cost is, and the higher the similarity is;
and step 3: constructing a minimum spanning tree method according to NLCA, and aggregating matching cost;
and 4, step 4: determining the parallax by selecting the parallax value corresponding to the point with the minimum matching cost, and reserving the parallax according to the threshold values corresponding to different layer numbers;
the determining the parallax by selecting the parallax value corresponding to the point with the minimum matching cost includes the following processes:
Figure FDA0003486804980000013
where x, y are both n-bit encodings of the gray values of the left and right image pixel points, d (C)min(x(il,jl),y(ir,jr) ) represents the parallax value when the cost is minimum, I represents the left parallax image, I, j represents the position in the image, each pixel point is the parallax value of the point, and C represents the matching cost value.
2. The high-speed stereo matching algorithm for the ultrahigh-resolution binocular image according to claim 1, wherein: the encoding process of each picture comprises the following steps:
selecting a point in the graph, drawing a rectangle of n x n by taking the point as the center, comparing each point except the center point in the rectangle with the center point, marking the gray value as 1 when the gray value is smaller than the center point, marking the gray value as 0 when the gray value is larger than the center point, and representing the pixel value of the center point by the obtained census sequence with the length of (n x n-1).
3. The high-speed stereo matching algorithm for the ultrahigh-resolution binocular image according to claim 1, wherein: the setting of the threshold for each layer according to the number of downsampling and the parallax range includes:
the number of the threshold values is the number of down-sampling times, the number of the down-sampling times is less than the number of layers by 1, and the threshold value of each layer is selected according to the percentage of the parallax range.
4. The high-speed stereo matching algorithm for the ultrahigh-resolution binocular image according to claim 1, wherein: the threshold reserved parallax according to different layer numbers is as follows:
the smaller the threshold value is, the larger the corresponding layer number is; and retains a disparity greater than a threshold.
CN201811079109.3A 2018-09-17 2018-09-17 High-speed stereo matching algorithm for ultrahigh-resolution binocular image Active CN110910438B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811079109.3A CN110910438B (en) 2018-09-17 2018-09-17 High-speed stereo matching algorithm for ultrahigh-resolution binocular image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811079109.3A CN110910438B (en) 2018-09-17 2018-09-17 High-speed stereo matching algorithm for ultrahigh-resolution binocular image

Publications (2)

Publication Number Publication Date
CN110910438A CN110910438A (en) 2020-03-24
CN110910438B true CN110910438B (en) 2022-03-22

Family

ID=69813259

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811079109.3A Active CN110910438B (en) 2018-09-17 2018-09-17 High-speed stereo matching algorithm for ultrahigh-resolution binocular image

Country Status (1)

Country Link
CN (1) CN110910438B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114708317A (en) * 2022-05-24 2022-07-05 北京中科慧眼科技有限公司 Matching cost matrix generation method and system based on binocular stereo matching

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101710423A (en) * 2009-12-07 2010-05-19 青岛海信网络科技股份有限公司 Matching search method for stereo image
CN101841730A (en) * 2010-05-28 2010-09-22 浙江大学 Real-time stereoscopic vision implementation method based on FPGA
CN104065954A (en) * 2014-07-03 2014-09-24 中国传媒大学 Method for quickly detecting parallax scope of high-definition stereoscopic video
CN104662896A (en) * 2012-09-06 2015-05-27 诺基亚技术有限公司 An apparatus, a method and a computer program for image processing
CN104820991A (en) * 2015-05-15 2015-08-05 武汉大学 Multi-soft-constraint stereo matching method based on cost matrix
CN104869387A (en) * 2015-04-19 2015-08-26 中国传媒大学 Method for acquiring binocular image maximum parallax based on optical flow method
CN105550675A (en) * 2016-02-02 2016-05-04 天津大学 Binocular pedestrian detection method based on optimization polymerization integration channel
CN106340036A (en) * 2016-08-08 2017-01-18 东南大学 Binocular stereoscopic vision-based stereo matching method
CN106846416A (en) * 2017-02-06 2017-06-13 遂昌县睿鼎科技服务有限公司 Unit beam splitting bi-eye passiveness stereo vision Accurate Reconstruction and subdivision approximating method
CN106991693A (en) * 2017-03-17 2017-07-28 西安电子科技大学 Binocular solid matching process based on fuzzy support weight
CN107103610A (en) * 2017-04-25 2017-08-29 国家测绘地理信息局卫星测绘应用中心 Stereo mapping satellite image matches suspicious region automatic testing method

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5405264B2 (en) * 2009-10-20 2014-02-05 任天堂株式会社 Display control program, library program, information processing system, and display control method
GB2506338A (en) * 2012-07-30 2014-04-02 Sony Comp Entertainment Europe A method of localisation and mapping

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101710423A (en) * 2009-12-07 2010-05-19 青岛海信网络科技股份有限公司 Matching search method for stereo image
CN101841730A (en) * 2010-05-28 2010-09-22 浙江大学 Real-time stereoscopic vision implementation method based on FPGA
CN104662896A (en) * 2012-09-06 2015-05-27 诺基亚技术有限公司 An apparatus, a method and a computer program for image processing
CN104065954A (en) * 2014-07-03 2014-09-24 中国传媒大学 Method for quickly detecting parallax scope of high-definition stereoscopic video
CN104869387A (en) * 2015-04-19 2015-08-26 中国传媒大学 Method for acquiring binocular image maximum parallax based on optical flow method
CN104820991A (en) * 2015-05-15 2015-08-05 武汉大学 Multi-soft-constraint stereo matching method based on cost matrix
CN105550675A (en) * 2016-02-02 2016-05-04 天津大学 Binocular pedestrian detection method based on optimization polymerization integration channel
CN106340036A (en) * 2016-08-08 2017-01-18 东南大学 Binocular stereoscopic vision-based stereo matching method
CN106846416A (en) * 2017-02-06 2017-06-13 遂昌县睿鼎科技服务有限公司 Unit beam splitting bi-eye passiveness stereo vision Accurate Reconstruction and subdivision approximating method
CN106991693A (en) * 2017-03-17 2017-07-28 西安电子科技大学 Binocular solid matching process based on fuzzy support weight
CN107103610A (en) * 2017-04-25 2017-08-29 国家测绘地理信息局卫星测绘应用中心 Stereo mapping satellite image matches suspicious region automatic testing method

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
A non-local cost aggregation method for stereo matching;Yang Q;《Computer Vision and Pattern Recognition》;20121231;1402-1409 *
Cross-Scale Cost Aggregation for Stereo Matching;Kang Zhang 等;《Computer Vision and Pattern Recognition》;20141231;1590-1597 *
基于图像处理的钢板表面缺陷成像优化与深度信息提取方法研究;刘源泂;《中国博士学位论文全文数据库 工程科技Ⅰ辑》;20120515(第05期);B022-3 *
计算机视觉中双目立体匹配技术的研究;荆鑫;《中国优秀硕士学位论文全文数据库 信息科技辑》;20130615(第06期);I138-1314 *

Also Published As

Publication number Publication date
CN110910438A (en) 2020-03-24

Similar Documents

Publication Publication Date Title
CN109377530B (en) Binocular depth estimation method based on depth neural network
CN111833393A (en) Binocular stereo matching method based on edge information
CN107329962B (en) Image retrieval database generation method, and method and device for enhancing reality
US20130170736A1 (en) Disparity estimation depth generation method
US8867826B2 (en) Disparity estimation for misaligned stereo image pairs
CN106530336B (en) Stereo matching method based on color information and graph cut theory
CN111105452B (en) Binocular vision-based high-low resolution fusion stereo matching method
CN112784874B (en) Binocular vision stereo matching method and device, electronic equipment and storage medium
CN112734837B (en) Image matching method and device, electronic equipment and vehicle
CN111489394A (en) Object posture estimation model training method, system, device and medium
CN113705796B (en) Optical field depth acquisition convolutional neural network based on EPI feature reinforcement
CN112907573B (en) Depth completion method based on 3D convolution
CN111508013A (en) Stereo matching method
CN111553296B (en) Two-value neural network stereo vision matching method based on FPGA
CN115329111B (en) Image feature library construction method and system based on point cloud and image matching
CN112287824A (en) Binocular vision-based three-dimensional target detection method, device and system
Xu et al. High-speed stereo matching algorithm for ultra-high resolution binocular image
CN115222889A (en) 3D reconstruction method and device based on multi-view image and related equipment
CN111914913A (en) Novel stereo matching optimization method
CN110889868B (en) Monocular image depth estimation method combining gradient and texture features
CN114419102B (en) Multi-target tracking detection method based on frame difference time sequence motion information
CN113313740B (en) Disparity map and surface normal vector joint learning method based on plane continuity
CN113034666B (en) Stereo matching method based on pyramid parallax optimization cost calculation
CN110910438B (en) High-speed stereo matching algorithm for ultrahigh-resolution binocular image
CN111402429B (en) Scale reduction and three-dimensional reconstruction method, system, storage medium and equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant