WO2016065579A1 - 一种全局视差估计方法和系统 - Google Patents

一种全局视差估计方法和系统 Download PDF

Info

Publication number
WO2016065579A1
WO2016065579A1 PCT/CN2014/089924 CN2014089924W WO2016065579A1 WO 2016065579 A1 WO2016065579 A1 WO 2016065579A1 CN 2014089924 W CN2014089924 W CN 2014089924W WO 2016065579 A1 WO2016065579 A1 WO 2016065579A1
Authority
WO
WIPO (PCT)
Prior art keywords
point
image
matching
points
pixel
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2014/089924
Other languages
English (en)
French (fr)
Inventor
彭祎
王荣刚
王振宇
高文
董胜富
王文敏
赵洋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Peking University Shenzhen Graduate School
Original Assignee
Peking University Shenzhen Graduate School
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Peking University Shenzhen Graduate School filed Critical Peking University Shenzhen Graduate School
Priority to PCT/CN2014/089924 priority Critical patent/WO2016065579A1/zh
Publication of WO2016065579A1 publication Critical patent/WO2016065579A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding

Definitions

  • the present application relates to the field of stereo matching image processing, and in particular, to a global disparity estimation method and system.
  • the user can only passively view the images captured by the camera, and can not view the different viewpoints from other perspectives, while the multi-view video allows the user to view from multiple viewpoints, enhancing Interactivity and 3D sensory effects have broad application prospects in stereo TV, video conferencing, auto navigation, virtual reality and other fields.
  • the stronger interactivity and sensory effect also increase the amount of video data, which increases the burden on video storage and transmission. How to solve such problems has become a research hotspot.
  • Stereo matching also called parallax estimation, is based on multi-view image data (generally binocular) acquired by the front-end camera to estimate the geometric relationship between the pixels in the corresponding image.
  • the information of the corresponding viewpoint can be obtained from the information of one viewpoint and its depth (parallax) information, thereby reducing the amount of original data and facilitating the transmission and storage of the multi-view video.
  • the stereo matching method can be roughly divided into a local stereo matching algorithm and a global stereo matching algorithm (see Scharstein D, Szeliski RA taxonomy and evaluation of dense two-frame stereo correspondence algorithms [J]. International journal of computer Vision, 2002, 47(1-3): 7-42.).
  • the local stereo matching algorithm is not accurate, but the speed is fast, which is not conducive to practical application.
  • the global stereo matching algorithm is based on the optimization of the global energy function to obtain the parallax result, which has higher accuracy but slower speed.
  • There are some improved global stereo matching algorithms that produce speeds comparable to local stereo matching algorithms such as the Fast Confidence Propagation algorithm (see Pedro F. Felzenszwalb, Daniel P. Huttenlocher. Efficient Belief Propagation for Early Vision. International Journal of Computer Vision October 2006, Volume 70, Issue 1, pp 41-54).
  • stereo matching has been widely concerned as an important link in multi-view video, and a large number of stereo matching algorithms have emerged.
  • stereo matching has been widely concerned as an important link in multi-view video, and a large number of stereo matching algorithms have emerged.
  • problems with stereo matching especially correctness and stability, which need to be further improved.
  • the present application provides a global disparity estimation method, including:
  • the first viewpoint image is an image of the target acquired from the first viewpoint
  • the second viewpoint image is an image of the target acquired from the second viewpoint
  • each third matching point is taken as the origin, and the first and second directions are searched along the first axis and the negative direction, and the pixel is searched as a search point until the search for the constraint is not satisfied.
  • the point of the condition is stopped, and all the points that satisfy the constraint are searched for as the fourth matching point; the third matching point and the fourth matching point are taken as the second matching space of the current pixel point;
  • the constraint condition includes a linear constraint condition and a spatial constraint condition based on a sample point, the linear constraint condition being a constraint of an Euclidean distance between a current pixel point and a search point, the space constraint condition being a search point and Constraining the Euclidean distance between the sampling points in color, the first axis and the second axis being perpendicular to each other;
  • the constraint is:
  • l 1 is the distance from pixel p to search point q
  • pixel p is the current pixel
  • l 2 is the distance from pixel p to sample point e i
  • O lab (p, q) is pixel p and search
  • the Euclidean distance of the point q in the color, O lab (q, e i ) is the Euclidean distance of the search point q and the sampling point e i in color
  • k 1 , k 2 , k 3 , k 4 , w 1 , w 2 is a custom parameter, and k 1 >k 2 , k 4 >k 3 , w 2 >w 1 .
  • the preset rule is such that the distance between each sampling point from four adjacent sampling points of the upper, lower, left and right is a preset distance.
  • the method further includes: performing polar line correction on the first viewpoint image and the second viewpoint image.
  • the occlusion region in the mark image is further included, specifically: taking the first view image and each block is The first reliable point L(p) from the left end in each row, the point R(pd p ) corresponding to the second viewpoint image is calculated according to the parallax d p of the point L( p ); in the second viewpoint image The point R(pd p -1) starts to find the first reliable point Rq to the left, finds its parallax d q , and calculates that the point Rq corresponds to the point L(q+d q ) in the first viewpoint image, two of the horizontal The point between the point L(p) and L(q+d q ) is the occlusion point.
  • the initial disparity is calculated using a fast confidence propagation global algorithm based on the sum of the matching costs of all points in the first matching space and the matching cost of all points in the second matching space.
  • performing image segmentation on the first view image and the second view image includes:
  • Image blocks according to color merging image blocks whose number of pixels is smaller than a preset value with image blocks closest to the color in adjacent image blocks; and/or judging that two adjacent image blocks are close in color, and two When the sum of the number of pixel points of the image block is less than the preset value, the two image blocks are merged;
  • Combining the image blocks according to the parallax merging the image blocks with the number of reliable points smaller than the preset value with the image blocks closest to the colors in the adjacent image blocks, the reliable points are obtained according to the initial parallax of each pixel in the original image. And/or, determine whether the parallax change of two adjacent image blocks is smooth, and if so, merge the two image blocks.
  • the first view image and the second view image are divided into a plurality of image blocks, specifically, the image is divided into a plurality of image blocks based on the superpixel color block.
  • determining whether the parallax change of two adjacent image blocks is smooth comprises:
  • the present application also provides a global disparity estimation system, include:
  • An image reading module configured to read in the first viewpoint image and the second viewpoint image, the first viewpoint image is an image of the target acquired from the first viewpoint, and the second viewpoint image is an image of the target acquired from the second viewpoint;
  • a matching space calculation module configured to select a sampling point on the first viewpoint image according to a preset rule, sequentially select a pixel point as a current pixel point on the first viewpoint image, and use the current pixel point as an origin, along a positive direction of the first axis And the negative direction, searching by pixel by pixel as a search point until the search for a point that does not satisfy the preset constraint is stopped, and searching for all points satisfying the constraint as the first matching point;
  • Each of the first matching points is an origin, and the pixel is searched as a search point by the pixel in the positive direction and the negative direction of the second axis until the point that does not satisfy the preset constraint is searched, and the searched satisfaction is stopped. All points of the constraint are used as a second matching point; the first matching point and the second matching point are used as the first matching space of the current pixel point;
  • each third matching point is taken as the origin, and the first and second directions are searched along the first axis and the negative direction, and the pixel is searched as a search point until the search for the constraint is not satisfied.
  • the point of the condition is stopped, and all the points that satisfy the constraint are searched for as the fourth matching point; the third matching point and the fourth matching point are taken as the second matching space of the current pixel point;
  • the constraint condition includes a linear constraint condition and a spatial constraint condition based on a sample point, the linear constraint condition being a constraint of an Euclidean distance between a current pixel point and a search point, the space constraint condition being a search point and Constraining the Euclidean distance between the sampling points in color, the first axis and the second axis being perpendicular to each other;
  • a matching cost calculation module configured to calculate a sum of matching costs of all points in the first matching space, and calculate a sum of matching costs of all points in the second matching space;
  • An initial disparity calculation module configured to calculate an initial disparity according to a sum of matching costs of all points in the first matching space and a matching cost of all points in the second matching space, and select a reliable point;
  • An image blocking module configured to perform image segmentation on the first view image and the second view image
  • a final disparity calculation module configured to perform block segmentation based on the image, and calculate a final disparity of each pixel point in the first view image and the second view image according to the initial disparity of the reliable point.
  • the constraints adopted include linear constraints and spatial beam conditions based on sample points.
  • the linear constraints are between the current pixel and the search point.
  • the constraint of the Euclidean distance on the color, the space constraint is the constraint of the Euclidean distance between the search point and the sample point. Because the above two constraints are used at the same time, the calculated matching space is closer to the image. The edge of the object, therefore, can improve the accuracy of the matching space calculation, thus ensuring the accuracy of the final disparity calculation.
  • FIG. 1 is a schematic flowchart of a global disparity estimation method according to an embodiment of the present application
  • FIG. 2 is a schematic diagram of selecting sampling points in a matching space calculation method according to an embodiment of the present application
  • FIG. 3 is a schematic diagram of calculation of a first matching space in a matching space calculation method according to an embodiment of the present application
  • FIG. 4 is a schematic block diagram of a global disparity estimation system according to an embodiment of the present application.
  • FIG. 5 is a test result of the global disparity estimation method provided by the embodiment of the present application on the Middlebury test platform.
  • this embodiment provides a global disparity estimation method, which includes the following steps:
  • the first viewpoint image and the second viewpoint image are read, the first viewpoint image is an image of a target acquired from the first viewpoint, and the second viewpoint image is an image of a target acquired from the second viewpoint.
  • the first viewpoint image is a left viewpoint image (hereinafter referred to as a left diagram)
  • the second viewpoint image is a right viewpoint image (hereinafter referred to as a right diagram) as an example.
  • the left and right images may be images in a binocular sequence taken by a binocular camera, or two images taken by a monocular camera at a certain horizontal displacement.
  • the left and right figures are both color images, and in some embodiments, may also be achromatic images.
  • the left and right images read in are the images that have been corrected by the polar line, ie the polar lines of the two images are horizontally paralleled for subsequent matching cost calculations, if two images are input. Without the over-polar line correction, you also need to perform polar line correction on the left and right images.
  • the matching space of the pixel points in the image is first calculated.
  • the matching space includes the first matching space and the second matching space, and the calculation method is as follows:
  • sampling point e is selected in the space of the left image. Specifically, the distance between each sampling point from the four adjacent sampling points of the upper, lower, left and right is a preset distance d, and all the sampling points form a grid shape. as shown in picture 2.
  • the sampling points may be selected in other manners, that is, the preset rules for selecting sampling points may be determined according to actual needs.
  • the constraint condition comprises a linear constraint condition and a spatial constraint condition based on the sample point
  • the linear constraint condition is current
  • the space constraint is the constraint of the Euclidean distance between the search point and the sample point in color.
  • a distance is extended from the two directions of the X-axis (first axis) and the Y-axis (second axis) respectively according to the color difference for the calculation of the matching space.
  • the pixel points are sequentially selected as the current pixel point p, and the point p is used as the origin.
  • the positive and negative directions along the X-axis are searched by pixel-by-pixel points as search points until the search for the constraints that do not satisfy the preset constraints are found.
  • all the points that satisfy the search condition are used as the first matching point; each of the first matching points is taken as the origin, and the positive and negative directions along the Y axis are used to search for the pixel by pixel.
  • the point p is used as the origin, and the search is performed on the pixel-by-pixel point as the search point along the positive and negative directions of the Y-axis until the point where the preset constraint is not satisfied is stopped, and the searched satisfaction constraint is obtained.
  • All the points of the condition are used as the third matching point; the third matching point is taken as the origin, and the positive and negative directions of the X-axis are used to search for the pixel-by-pixel point as the search point until the search for the constraint that does not satisfy the preset constraint is found.
  • the point stops, and all the points that satisfy the search for the constraint are taken as the fourth matching point; the third matching point and the fourth matching point are taken as the second matching space S 2 of the point p.
  • the points that meet the constraint condition are searched in the positive direction of the X-axis, the negative direction of the X-axis, the positive direction of the Y-axis, and the negative direction of the Y-axis, that is, the right arm, the left arm, the upper arm, and the lower side shown in FIG. arm.
  • constraints are:
  • l 1 is the distance from the pixel point p to the search point q
  • the pixel point p is the current pixel point
  • l 2 is the distance from the pixel point p to the sampling point e i
  • the selection is by the condition k 3 *l 1 ⁇ l 2 ⁇ k 4 *l 1 determines that O lab (p, q) is the Euclidean distance of the pixel p and the search point q in color
  • O lab (q, e i ) is the search point q and the sample point e i in color
  • the Euclidean distance, k 1 , k 2 , k 3 , k 4 , w 1 , w 2 are custom parameters, and k 1 >k 2 , k 4 >k 3 , w 2 >w 1 .
  • O lab (p, q) is the Euclidean distance of the pixel point p and the search point q on the lab color
  • O lab (q, e i ) is the search point q and the sampling point e i on the lab color. Euclidean distance. It should be noted that the value of i in the i-th sampling point e i is set to a suitable k 3 value and k 4 value, so that the value of i is unique to determine a unique sampling point.
  • condition 12 belongs to linear constraint
  • condition 3 belongs to spatial constraint based on sampling point.
  • the introduced spatial constraint is mainly used to improve the boundary region point of the object in the image, so that the calculated matching space is closer to the edge of the object in the image, and the algorithm is enhanced by referring to more reasonable color information. Stability. Therefore, under the premise of linear constraints, combined with the spatial constraints based on sampling points, the accuracy and stability of stereo matching can be better guaranteed. In other embodiments, the above constraints may be appropriately changed according to actual needs.
  • the step of calculating the matching cost of the point is also included.
  • the matching is performed within the specified range ⁇ of the right graph, and the matching cost of all points in the range and the point L p is calculated.
  • the range ⁇ is the search range, that is, the range of the disparity value.
  • the search range is on the same scanning line (polar line) as the point L p . Since the left and right images have been corrected by the polar line and the polar lines are horizontally parallel, the search range ⁇ is a horizontal direction. Line segment.
  • each point w in the first matching space S 1 of the point L p is used to match the R w+d point in the right picture, and the calculation of the matching cost of each pair of points is calculated. Obtained by the hybrid cost function, the final matching cost is the sum of the matching costs of all point pairs C 1 .
  • the sum of matching costs C 2 is calculated in the same way using the second matching space S 2 of the point L p .
  • the matching cost function of each point pair consists of three parts: a gray space census transform (center transform), a color space absolute value difference (denoted as AD), and a bidirectional gradient.
  • a gray space census transform center transform
  • AD color space absolute value difference
  • a bidirectional gradient The specific calculations of each part are as follows:
  • the use scenario of the census transformation is performed on the grayscale image.
  • the color map is converted into a grayscale image.
  • the grayscale value of the p-point in the original image is represented by GS(p), and at the same time, the calculation is performed in p.
  • the central 7x9 window remove the census value x(p,q) generated by all points q and p except p.
  • the formula is as follows:
  • x(p,q) is concatenated into a binary string B(p) according to the relative positions of p and q.
  • two corresponding bit strings can be obtained, and the difference between them is described by Hamming distance, and the cost value is as follows:
  • d represents the disparity between the corresponding pixels.
  • the absolute value difference is a more common method for measuring the similarity between two points.
  • the AD value in the color space is used, and the cost value obtained according to the AD value is as follows:
  • the RGB color of point p in the left image The RGB color of the point corresponding to the parallax d in the right picture and the left picture p, Indicates the Euclidean distance of these two colors.
  • the gradient is selected as the cost term.
  • a bidirectional gradient that is, a gradient in the horizontal and vertical directions is employed.
  • N x and N y represent the derivative (gradient) in the x and y directions, respectively
  • IL(p) is the gray value of the point to be calculated (left)
  • I R (pd) is in the other picture (right) Figure)
  • d is the parallax between two points
  • the final cost function is a weighted mixture of the above three terms, as shown in equation (6), where a, b, and g are weights to represent the contribution of each item to the final cost function value.
  • C census is the h(p,d) value of the corresponding point found in equation (3).
  • the fast belief propagation algorithm according to globally C 1 and C 2 using The initial parallax is used to improve the accuracy and stability of stereo matching.
  • the specific calculation method is as follows:
  • N(p) is a set of four points of up, down, left, and right adjacent to point p.
  • the local matching cost for point p parallax is d p is D p (d p ):
  • N(p) ⁇ q is a set of q points removed from the top, bottom, left, and right points adjacent to the point p.
  • the optimal disparity d * p (ie, the initial disparity) of point p can be obtained by minimizing the energy function E, which has the following formula:
  • ⁇ is the range of values of parallax.
  • the matching of the left and right parallax maps is used to further filter the reliable points, and d L (p) represents the left figure.
  • the parallax of the point is:
  • Match(p) 1 indicates that p is reliable, and equal to 0 indicates that p is unreliable.
  • a step of blocking the image is also included.
  • the image is first divided into several small pieces (image blocks) of a very small size.
  • the image is divided into several image blocks based on the superpixel color block, and then, here, The image blocks are combined based on color and parallax, respectively.
  • Superpixel-based color segmentation refers to taking a number of (usually large) superpixels in space, and then using spatial information and color information to determine the pixel closest to each superpixel. Each super pixel forms a block with the pixel closest to it, taking the number of super pixel points and the generated block The number is equal. Based on the color segmentation of Superpixel, the algorithm has better effect on the boundary of the object when there are enough super-pixel points, but due to the excessive number of blocks generated in this case, it will have a negative impact on the calculation.
  • the number of pixel points is p(s)
  • the number of reliable points is r(s).
  • S40 merging the image blocks according to the parallax: combining the image blocks with the number of reliable points smaller than the preset value with the image blocks closest to the color in the adjacent image blocks, and the reliable points are selected according to the initial parallax of each pixel in the original image. And/or, determine whether the parallax change of two adjacent image blocks is smooth, and if so, merge the two image blocks.
  • V Sk (i) where a and b are preset pixel widths; when max
  • ⁇ j it is determined that the current image block S and its adjacent image block
  • the parallax change of S k is smooth, wherein i ⁇ W S, Sk , W S, and Sk are subscript sets of all pairs of points at the boundary of block S and block S k , and j is a preset value.
  • the image blocking method for global disparity estimation not only the color information is used for blocking, but also the disparity information is introduced, and the accuracy of the final calculated final disparity can be further improved.
  • the left and right pictures are pictures viewed from different perspectives, some parts are not in the right picture in the left picture, and some parts are not in the left picture in the right picture. These parts are all occlusion areas. Since these regions exist only in one picture, the results calculated by the parallax calculations made by the previous method are basically wrong. These errors will affect the final estimation result. Therefore, it is necessary to use the color block to find the occlusion area. And marked as unreliable points to improve the final correct rate.
  • the left occlusion area exists in the color block, the right end of each block is adjacent to other blocks, and the left end adjacent part is unoccluded.
  • the occlusion area exists in the portion of the color block where the left end of each block is adjacent to other blocks, and the portion adjacent to the right end is unoccluded.
  • the occlusion region in the mark image is further included, specifically: taking the first reliable point of each block from the left end in each row in the left figure.
  • L(p) which is calculated according to the disparity d p of the point L(p), which corresponds to the point R(pd p ) of the right picture; in the right picture, the first reliable is found from the point R(pd p -1) to the left.
  • Point Rq find its parallax d q , calculate the point Rq corresponds to the point L(q+d q ) in the left picture, and the point between the two points L(p) and L(q+d q ) To cover the point.
  • the embodiment further includes the steps of: performing median filtering on the existing reliable points based on the color block, and removing some reliable points again. That is, when S20 is executed after S30, S20 can utilize the information in S30 when performing further screening of reliable points. It should be noted that some of the steps in FIG. 1 do not limit the strict execution order, and the order of execution may be determined according to specific needs.
  • the gradient on the X-axis and the Y-axis is first estimated.
  • the estimation method is to select some reliable points on the X-axis and p points in a color block, calculate the gradient formed by these points, and finally take the median value, that is, the gradient derivationX estimated by the p-point on the X-axis. p).
  • DerivationY(p) is obtained in the same way in the Y direction.
  • the median is sorted for all d(p i ) and the value is rounded to see if it is equal to d(p), and if not, the point is filtered out.
  • the value obtained by taking the median value for all d(p i ) and rounding the value is the final disparity d(p) of point p.
  • the final parallax can also be obtained by any of the prior art methods.
  • the present embodiment further provides a global disparity estimation system, which includes an image reading module 1000, a matching space computing module 1001, and a matching cost calculation module 1002.
  • the image reading module 1000 is configured to read in a first viewpoint image that is an image of a target acquired from a first viewpoint, and a second viewpoint image that is an image of a target acquired from a second viewpoint.
  • the matching space calculation module 1001 is configured to select a sampling point on the first viewpoint image according to a preset rule, and sequentially select a pixel point as the current pixel point on the first viewpoint image, and use the current pixel point as an origin, along the positive direction of the first axis.
  • a matching point is the origin, Exploring the pixel-by-pixel point as a search point along the positive and negative directions of the second axis until the point where the preset constraint is not satisfied is stopped, and searching for all points that satisfy the constraint as the second match Point; the first matching point and the second matching point are used as the first matching space of the current pixel point.
  • the matching space calculation module 1001 is further configured to use the current pixel point as an origin, search in the positive direction and the negative direction of the second axis, and use the pixel-by-pixel point as the search point until the point where the predetermined constraint condition is not found is stopped, And searching all the points satisfying the constraint as the third matching point; respectively searching for the third matching point as the origin, and searching along the first axis in the positive direction and the negative direction, using the pixel-by-pixel point as the search point until the search is performed.
  • the point that does not satisfy the preset constraint is stopped, all the points that satisfy the search condition are used as the fourth matching point; the third matching point and the fourth matching point are used as the second matching space of the current pixel point.
  • Constraints include linear constraints and spatial constraints based on sample points.
  • the linear constraints are the constraints of the Euclidean distance between the current pixel and the search point.
  • the spatial constraint is the color between the search point and the sample point.
  • the first axis and the second axis are perpendicular to each other.
  • the matching cost calculation module 1002 is configured to calculate a sum of matching costs of all points in the first matching space, and calculate a sum of matching costs of all points in the second matching space.
  • the initial disparity calculation module 1003 is configured to calculate an initial disparity according to a sum of matching costs of all points in the first matching space and a matching cost of all points in the second matching space, and select a reliable point.
  • the image blocking module 1004 adopts any one of the above embodiments to perform image segmentation on the original image.
  • the final disparity calculation module 1005 is configured to calculate a final disparity for each pixel within the first view image based on image segmentation.
  • the global disparity estimation system provided by this embodiment corresponds to the global disparity estimation method described above, and the working principle is not described herein again.
  • FIG. 5 is an experimental result diagram of the global disparity estimation method provided by the embodiment of the present application on the Middlebury data set.
  • the test result on the Middlebury test platform shows that the global disparity estimation method provided by the embodiment of the present application is obtained.
  • the results (line 2 results) are superior to most current methods.
  • “nonocc”, “all” and “disc” are used as evaluation indexes, and the error rate threshold is set to 1.0, that is, the difference from the true ground truth is greater than 1. Then marked as the wrong point.
  • the global disparity estimation method and system provided by the present application obtains the hybrid cost of pixel points based on a robust hybrid cost function, and aggregates the single point cost by using an improved aggregation space; then adopts a fast belief propagation global algorithm for global cost. Optimization calculations; finally, image segmentation specifically for disparity estimation and occlusion point marking are used, so the accuracy of the final disparity calculation can be greatly improved.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

一种全局视差估计方法和系统，在进行匹配空间计算时，根据预设规则在图像上选取采样点，再根据约束条件进行第一匹配空间和第二匹配空间的计算。其中，采用的约束条件包括线性约束条件和基于采样点的空间线束条件，所述线性约束条件为当前像素点与搜索点之间在颜色上的欧氏距离的约束，所述空间约束条件为搜索点与采样点之间在颜色上的欧氏距离的约束，由于同时采用了上述两个约束条件，使得计算出的匹配空间更加接近图像中物体的边缘，因此，能够提高匹配空间计算的准确性，从而保证最终视差计算的准确度。

Description

一种全局视差估计方法和系统 技术领域
本申请涉及立体匹配图像处理领域，具体涉及一种全局视差估计方法和系统。
背景技术
在传统视频系统中，用户只能被动的观看由摄像机拍摄到的画面，无法从其他视角观看不同视点的画面，而多视角视频(Multi-View Video)则允许用户从多个视点进行观看，增强了交互性及3D感官效果，在立体电视、视频会议、自动导航、虚拟现实等领域有广泛的应用前景。然而，更强的交互性及感官效果也同时增加了视频的数据量，对视频的存储及传输等增加了负担，如何解决此类问题已成为目前的研究热点。
立体匹配，也称视差估计，是根据前端摄像机获取的多目图像数据(一般为双目)，估计出对应图像中的像素点间的几何关系。利用视差估计，可以由一个视点的信息及其深度(视差)信息得到对应视点的信息，从而减少了原始数据量，为多目视频的传输及存储提供了便利。
根据具体实现细节的不同，立体匹配方法可以大致分为局部立体匹配算法和全局立体匹配算法(可参见Scharstein D，Szeliski R.A taxonomy and evaluation of dense two-frame stereo correspondence algorithms[J].International journal of computer vision，2002，47(1-3)：7-42.)。局部立体匹配算法准确性不高，但速度较快，不利于实际应用；全局立体匹配算法是基于对全局的能量函数最优化得到视差结果，其准确性较高，但速度较慢，不过，已有一些改进的全局立体匹配算法产生了与局部立体匹配算法相当的速度，如快速置信传播算法(可参见Pedro F.Felzenszwalb，Daniel P.Huttenlocher.Efficient Belief Propagation for Early Vision.International Journal of Computer Vision October 2006，Volume 70，Issue 1，pp 41-54)。
综合上述叙述可知，立体匹配作为多视角视频中的重要环节，已受到广泛关注，并有大量的立体匹配算法涌现。然而，立体匹配尚存在很多问题，特别是正确性和稳定性，需要进一步提高。
发明内容
根据本申请的第一方面，本申请提供了一种全局视差估计方法，包括：
读入第一视点图像和第二视点图像，第一视点图像为从第一视点获取的目标的图像，第二视点图像为从第二视点获取的目标的图像；
根据预设规则在第一视点图像上选取采样点；
在第一视点图上依次选取像素点作为当前像素点，以当前像素点为原点，沿第一轴线正方向和负方向，以逐个像素点作为搜索点进行搜索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第一匹配点；分别以各个第一匹配点为原点，沿第二轴线正方向和负方向，以逐个像素点作为搜索点进行探索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第二匹配点；将第一匹配点和第二匹配点作为当前像素点的第一匹配空间；
以当前像素点为原点，沿第二轴线正方向和负方向，以逐个像素点作为搜索点进行搜索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第三匹配点；分别以各个第三匹配点为原点，沿第一轴线正方向和负方向，以逐个像素点作为搜索点进行探索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第四匹配点；将第三匹配点和第四匹配点作为当前像素点的第二匹配空间；
所述约束条件包括线性约束条件和基于采样点的空间约束条件，所述线性约束条件为当前像素点与搜索点之间在颜色上的欧氏距离的约束，所述空间约束条件为搜索点与采样点之间在颜色上的欧氏距离的约束，所述第一轴线与第二轴线相互垂直；
计算第一匹配空间内所有点的匹配代价之和，计算第二匹配空间内所有点的匹配代价之和；
根据第一匹配空间内所有点的匹配代价之和和第二匹配空间内所有点的匹配代价之和计算初始视差，并筛选得到可靠点；
对第一视点图像和第二视点图像进行图像分块；
基于所述图像分块，并根据所述可靠点的初始视差分别计算第一视点图像和第二视点图像内每个像素点的最终视差。
在一实施例中，所述约束条件为：
Figure PCTCN2014089924-appb-000001
其中，l1为像素点p到搜索点q的距离，像素点p为当前像素点，l2为像素点p到采样点ei的距离，Olab(p，q)为像素点p与搜索点q在颜色上的欧氏距离，Olab(q，ei)为搜索点q与采样点ei在颜色上的欧氏距离，k1、k2、k3、k4、w1、w2为自定义参数，且k1＞k2、k4＞k3、w2＞w1。
在一实施例中，所述预设规则使得每个采样点离其上、下、左、右四个相邻采样点的距离为一预设距离。
在一实施例中，在读入第一视点图像和第二视点图像后，在选取采样点之前，还包括：对第一视点图像和第二视点图像进行极线校正。
在一实施例中，在对第一视点图像和第二视点图像进行图像分块后，计算最终视差之前，还包括标记图像中的遮挡区域，具体为：取第一视点图像每个分块在每一行中从左端开始的第一个可靠点L(p)，根据点L(p)的视差dp算得其对应于第二视点图像的点R(p-dp)；在第二视点图像中从点R(p-dp-1)开始向左边找到第一个可靠点Rq，找出其视差dq，算得点Rq对应于第一视点图像中的点L(q+dq)，水平的两个点L(p)与L(q+dq)之间的点即为遮挡点。
在一实施例中，根据第一匹配空间内所有点的匹配代价之和和第二匹配空间内所有点的匹配代价之和，采用快速置信传播全局算法计算初始视差。
在一实施例中，对第一视点图像和第二视点图像进行图像分块，包括：
将第一视点图像和第二视点图像划分为若干个图像块；
根据颜色对图像块进行合并：将像素点数量小于预设值的图像块与其相邻图像块中颜色最接近的图像块合并；和/或，判断到两个相邻图像块颜色接近，且两个图像块像素点数量之和小于预设值时，将两个图像块合并；
根据视差对图像块进行合并：将可靠点数量小于预设值的图像块与其相邻图像块中颜色最接近的图像块合并，所述可靠点为根据原始图像中各像素点的初始视差筛选得到的；和/或，判断相邻两个图像块的视差变化是否平滑，如果是，则将两个图像块合并。
在一实施例中，将第一视点图像和第二视点图像划分为若干个图像块，具体为：基于superpixel颜色分块将图像划分为若干个图像块。
在一实施例中，判断相邻两个图像块的视差变化是否平滑，包括：
找出当前图像块S与其相邻图像块Sk的边界相邻点对PS(i)、PSk(i)，PS(i)与PSk(i)为块S与块Sk的第i个相邻点对；
以PS(i)为中心搜索一个a*b的矩形方框，计算该方框中属于块S的可靠点的视差的均值VS(i)，以PSk(i)为中心搜索一个a*b的矩形方框，计算该方框中属于块Sk的可靠点的视差的均值VSk(i)，其中，a、b为预设的像素宽度；
当max|VS(i)-VSk(i)|＜j时，判断为当前图像块S与其相邻图像块Sk的视差变化平滑，其中，i∈WS，Sk，WS，Sk为块S与块Sk边界所有点对的下标集合，j为预设值。
根据本申请的第二方面，本申请还提供了一种全局视差估计系统， 包括：
图像读入模块，用于读入第一视点图像和第二视点图像，第一视点图像为从第一视点获取的目标的图像，第二视点图像为从第二视点获取的目标的图像；
匹配空间计算模块，用于根据预设规则在第一视点图像上选取采样点后，在第一视点图上依次选取像素点作为当前像素点，以当前像素点为原点，沿第一轴线正方向和负方向，以逐个像素点作为搜索点进行搜索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第一匹配点；分别以各个第一匹配点为原点，沿第二轴线正方向和负方向，以逐个像素点作为搜索点进行探索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第二匹配点；将第一匹配点和第二匹配点作为当前像素点的第一匹配空间；
以当前像素点为原点，沿第二轴线正方向和负方向，以逐个像素点作为搜索点进行搜索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第三匹配点；分别以各个第三匹配点为原点，沿第一轴线正方向和负方向，以逐个像素点作为搜索点进行探索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第四匹配点；将第三匹配点和第四匹配点作为当前像素点的第二匹配空间；
所述约束条件包括线性约束条件和基于采样点的空间约束条件，所述线性约束条件为当前像素点与搜索点之间在颜色上的欧氏距离的约束，所述空间约束条件为搜索点与采样点之间在颜色上的欧氏距离的约束，所述第一轴线与第二轴线相互垂直；
匹配代价计算模块，用于计算第一匹配空间内所有点的匹配代价之和，和计算第二匹配空间内所有点的匹配代价之和；
初始视差计算模块，用于根据第一匹配空间内所有点的匹配代价之和和第二匹配空间内所有点的匹配代价之和计算初始视差，并筛选得到可靠点；
图像分块模块，用于对第一视点图像和第二视点图像进行图像分块；
最终视差计算模块，用于基于所述图像分块，并根据所述可靠点的初始视差计算第一视点图像和第二视点图像内每个像素点的最终视差。
本申请提供的全局视差估计方法和系统中，在进行匹配空间计算时，根据预设规则在图像上选取采样点，再根据约束条件进行第一匹配空间和第二匹配空间的计算。其中，采用的约束条件包括线性约束条件和基于采样点的空间线束条件，线性约束条件为当前像素点与搜索点之间在 颜色上的欧氏距离的约束，空间约束条件为搜索点与采样点之间在颜色上的欧氏距离的约束，由于同时采用了上述两个约束条件，使得计算出的匹配空间更加接近图像中物体的边缘，因此，能够提高匹配空间计算的准确性，从而保证最终视差计算的准确度。
附图说明
图1为本申请一种实施例中全局视差估计方法的流程示意图；
图2为本申请一种实施例匹配空间计算方法中选取采样点的示意图；
图3为本申请一种实施例匹配空间计算方法中第一匹配空间的计算示意图；
图4为本申请一种实施例中全局视差估计系统的模块示意图；
图5为采用本申请实施例提供的全局视差估计方法在Middlebury测试平台上的测试结果。
具体实施方式
下面通过具体实施方式结合附图对本申请作进一步详细说明。
请参考图1，本实施列提供了一种全局视差估计方法，包括下面步骤：
S00：读入第一视点图像和第二视点图像，第一视点图像为从第一视点获取的目标的图像，第二视点图像为从第二视点获取的目标的图像。为了便于对本申请进行说明，以第一视点图像为左视点图像(下面简称左图)，第二视点图像为右视点图像(下面简称右图)为例进行说明。左图和右图可以是通过双目摄像机拍摄得到的双目序列中的图像，或者是单目摄像机在一定水平位移下拍摄得到的两幅图像。通常，左图和右图均为彩色图像，在某些实施例中，也可以是非彩色图像。
在某些实施例中，读入的左图和右图为已经过极线校正的图像，即两幅图像的极线是水平平行的，以便于后续进行匹配代价计算，若输入的两幅图像未经过极线校正，则还需要先对左图和右图进行极线校正。
S10：计算初始视差，并筛选得到可靠点。
在计算初始视差时，首先需要计算图像中像素点的匹配空间，本实施例中，匹配空间包括第一匹配空间和第二匹配空间，其计算方法如下：
根据预设规则选取采样点。首先在左图空间内选取采样点e，具体的，每个采样点离其上、下、左、右四个相邻采样点的距离为一预设距离d，所有采样点形成网格状，如图2所示。在其他实施例中，采样点的选取也可以采用其他约定方式，即用于选择采样点的预设规则可以根据实际需求制定。
根据约束条件计算第一匹配空间和第二匹配空间，其中，约束条件包括线性约束条件和基于采样点的空间约束条件，线性约束条件为当前 像素点与搜索点之间在颜色上的欧氏距离的约束，空间约束条件为搜索点与采样点之间在颜色上的欧氏距离的约束。
对于左图中的某一点p，从其分别向X轴(第一轴线)两个方向和Y轴(第二轴线)两个方向根据颜色差延生出一段距离，用于匹配空间的计算。
在左图上依次选取像素点作为当前像素点p，以点p为原点，沿X轴正方向和负方向，以逐个像素点作为搜索点进行搜索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足约束条件的所有点作为第一匹配点；分别以各个第一匹配点为原点，沿Y轴正方向和负方向，以逐个像素点作为搜索点进行探索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足约束条件的所有点作为第二匹配点；将第一匹配点和第二匹配点作为点p的第一匹配空间S1。如图3所示，为第一匹配空间S1的计算过程示意图。
之后，再以点p为原点，沿Y轴正方向和负方向，以逐个像素点作为搜索点进行搜索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足约束条件的所有点作为第三匹配点；分别以各个第三匹配点为原点，沿X轴正方向和负方向，以逐个像素点作为搜索点进行探索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足约束条件的所有点作为第四匹配点；将第三匹配点和第四匹配点作为点p的第二匹配空间S2。
以点p为原点，分别沿X轴正方向、X轴负方向、Y轴正方向、Y轴负方向搜索到符合约束条件的点，即图2所示的右臂、左臂、上臂、下臂。
在具体实施例中，约束条件为：
Figure PCTCN2014089924-appb-000002
其中，l1为像素点p到搜索点q的距离，像素点p为当前像素点，l2为像素点p到采样点ei的距离，的选取通过条件k3*l1＜l2＜k4*l1确定，Olab(p，q)为像素点p与搜索点q在颜色上的欧氏距离，Olab(q，ei)为搜索点q与采样点ei在颜色上的欧氏距离，k1、k2、k3、k4、w1、w2为自定义参数，且k1＞k2、k4＞k3、w2＞w1。例如，k1＝15、k2＝5、k3＝1.5、k4＝3、w1＝10、w2＝100。本实施例中，Olab(p，q)为像素点p与搜索点q在lab颜色上的欧氏距离，Olab(q，ei)为搜索点q与采样点ei在lab颜色上的欧氏距离。需要说明的是，第i个采样点ei中i的取值通过设置合适的k3值和k4值，使 得i的取值唯一，以确定一个唯一的采样点。
约束条件(1)中，条件①②属于线性约束，条件③属于基于采样点的空间约束。在计算匹配空间时，由于不同图片上的颜色变化快慢不一样，同一副图片的不同区域的颜色变化快慢也不一样，使得很难在单一线性约束上使得算法稳定。本实施例中，引入的空间约束主要是用于改善图像中物体的边界区域点，使得计算出的匹配空间更加接近图像中物体的边缘，由于参考了更多合理的颜色信息，也增强了算法的稳定性。因此，在线性约束的前提下，结合基于采样点的空间约束，可以更好的保证立体匹配的准确率和稳定性。在其他实施例中，可以根据实际需求适当改变上述约束条件。
在计算出点的匹配空间后，还包括计算点的匹配代价的步骤。
对于左图中的某一点Lp，在右图的指定范围Ω内进行匹配，计算该范围内所有点与点Lp的匹配代价，范围Ω是搜索范围，也即视差值的取值范围，且该搜索范围是与点Lp在同一条扫描线(极线)上的，由于左、右图已经过极线校正，且极线是水平平行的，因此搜索范围Ω为一条水平方向的线段。对于范围Ω内的每一个点的视差d，采用点Lp的第一匹配空间S1中的每个点w，去匹配右图中Rw+d点，每个点对的匹配代价的计算通过混合代价函数得到，最终匹配代价为所有点对的匹配代价之和C1。用点Lp的第二匹配空间S2以相同的方式计算匹配代价之和C2。
每个点对的匹配代价函数由三部分组成：一个灰度空间census变换(中心变换)、一个彩色空间绝对值差(记作AD)、一个双向梯度，各部分的具体计算如下所示：
(1)census变换的使用场景是在灰度图上进行的，先将彩色图转换为灰度图，原图中的p点的灰度值用GS(p)表示，同时，计算在以p为中心的7x9窗口中除去p外的所有点q与p产生的census值x(p，q)，计算公式如下：
Figure PCTCN2014089924-appb-000003
将x(p，q)根据p与q的相对位置连接成二进制串B(p)。在分别对左、右图进行计算后可得到两个对应比特串，通过汉明距离来描述它们之间的差异，得到代价值如下所示：
h(p，d)＝Ham(BL(p)，BR(p-d))............(3)
其中d代表对应像素点间的视差。
(2)AD值
绝对值差是衡量两个点相似度的较为常用的方法，本实施例中采用其在彩色空间中的AD值，根据AD值得到的代价值如下所示：
Figure PCTCN2014089924-appb-000004
其中，
Figure PCTCN2014089924-appb-000005
为左图中点p的RGB颜色，
Figure PCTCN2014089924-appb-000006
为右图中与左图p点按视差d对应的点的RGB颜色，
Figure PCTCN2014089924-appb-000007
表示这两个颜色的欧氏距离。
(3)梯度
选取梯度作为代价项，本实施例中采用双向梯度，即水平和竖直方向的梯度。其中Nx和Ny分别表示在x和y方向的导数(梯度)，IL(p)为待计算点(左图)的灰度值，IR(p-d)为其在另一张图片(右图)对应点的灰度值，d为两点间的视差，则
CGDx(p，d)＝||Nx(IL(p))-Nx(IR(p-d))||
CGDy(p，d)＝||Ny(IL(p))-Ny(IR(p-d))||............(5)
CGD＝CGDx+CGDy
(4)混合代价函数
最终的代价函数由上述三项代价项加权混合而成，如式(6)所示，其中a、b、g为各项权重，用以表示各项对最终代价函数值的贡献。
C(x，y，d)＝aCcensus+bCAD(p，d)+gCGD............(6)
其中，x、y代表坐标值，d代表点(x，y)的视差。Ccensus即为式(3)中求得的相应点的h(p，d)值。
优选的，在计算出第一匹配空间内所有点的匹配代价之和C1和第二匹配空间内所有点的匹配代价之和C2后，根据C1和C2采用快速置信传播全局算法计算初始视差，以提高立体匹配的准确率和稳定性，具体计算方式如下：
置信度B和能量函数E的关系为：
B＝e-E............(7)
此时置信度B最大化的过程就等价于能量函数E最小化的过程，则点P视差dp的能量函数可以表示为：
Figure PCTCN2014089924-appb-000008
N(p)为与点p相邻的上下左右4个点的集合，
Figure PCTCN2014089924-appb-000009
为从点r到点p经过T次迭代后传送的能量，则点p视差为dp时的局部匹配代价为Dp(dp)：
Dp(dp)＝[c1(p，dp)+c2(p，dp)]/2............(9)
且点p到点q经过t次迭代后传送的能量的计算公式可以如下：
Figure PCTCN2014089924-appb-000010
N(p)\q为与点p相邻的上下左右4个点中除去q点的集合。
点p的最佳视差d* p(即初始视差)可以通过最小化能量函数E获得，其公式如下：
Figure PCTCN2014089924-appb-000011
Ω为视差的取值范围。
S20：进一步筛选可靠点
由于算出的初始视差很多点是不可靠的，它们会影响最后的计算结果，因此，本实施例中，采用左右视差图的匹配进行可靠点的进一步筛选，dL(p)表示左图中p点的视差。则筛选公式为：
Figure PCTCN2014089924-appb-000012
match(p)等于1表示p点可靠，等于0表示p点不可靠。
在全局视差估计中，还包括对图像进行分块的步骤。在对图像进行分块时，先将图像划分为若干个尺寸极小的碎块(图像块)，优选的，本实施例基于superpixel颜色分块将图像划分为若干个图像块，之后，在此基础上分别根据颜色和视差对图像块进行合并。基于Superpixel的颜色分块是指，在空间内取若干(通常数量较大)超像素点，然后利用空间信息与颜色信息判断出与每个超像素点最接近的像素点。每个超像素点与和它最接近的像素点形成一个块，取的超像素点个数与产生的块 的数目是相等的。基于Superpixel的颜色分块，其算法在超像素点足够多的情况下对物体边界的划分效果较好，但是由于这种情况下产生的分块数目过多，将给计算带来负面的影响。
S30：根据颜色对图像块进行合并：将像素点数量小于预设值的图像块与其相邻图像块中颜色最接近的图像块合并；和/或，判断到两个相邻图像块颜色接近，且两个图像块像素点数量之和小于预设值时，将两个图像块合并。
本实施例中，假设对于图像块s，其像素点数量为p(s)，可靠点数量为r(s)。
(1)由于划分得到的碎块尺寸极小，因此其数目很大，使得后面的处理所需内存极大，所以对于像素点数目极少的块，将其与周围的块合并。当p(s)＜k1(k1为预设值)时，将该块与其颜色最接近的块进行合并，判断颜色的接近程度可以采用现有技术中的任意方式进行。
(2)如果相邻两个块的颜色足够接近，也将其合并，以提高分块的稳定性。同时要保证合并后的块不至于过大，对于块s1和s2，当p(s1)+p(s2)＜k2(k2为预设值)时，合并s1和s2。
S40：根据视差对图像块进行合并：将可靠点数量小于预设值的图像块与其相邻图像块中颜色最接近的图像块合并，可靠点为根据原始图像中各像素点的初始视差筛选得到的；和/或，判断相邻两个图像块的视差变化是否平滑，如果是，则将两个图像块合并。
由于图像分块是用于最后的视差估计(最终视差的计算)的，而前面步骤中已经计算出了初始视差。因此，根据视差对块进行合并将有助于使最后的块更适合做视差估计，提高正确率。
(1)根据前面可靠点筛选步骤，由于有些块的可靠点数量较少，以至于根据视差进行合并将会影响正确率，因此，有必要先将这些块与其他块进行合并。本实施例中，当r(s)＜k3(k3为预设值)时，将该块与其颜色最接近的块进行合并。为找出与当前块颜色最接近的块，可以采用现有技术中任意一种方式，例如，将当前块的颜色与其周围块进行对比得出。
(2)根据视差估计的特点，视差平滑变化的地方有必要将其归为一块，因此可以通过判断相邻块之间的视差变化是否平滑来决定是否将这两个块合并，如果平滑，则合并，反之，则不合并。
本实施例中，判断相邻两个图像块的视差变化是否平滑时，先找出当前图像块S与其相邻图像块Sk的边界相邻点对PS(i)、PSk(i)，PS(i)与PSk(i)为块S与块Sk的第i个相邻点对；再以PS(i)为中心搜索一个a*b的矩形方框，计算该方框中属于块S的可靠点的视差的均值VS(i)，以 PSk(i)为中心搜索一个a*b的矩形方框，计算该方框中属于块Sk的可靠点的视差的均值VSk(i)，其中，a、b为预设的像素宽度；当max|VS(i)-VSk(i)|＜j时，判断为当前图像块S与其相邻图像块Sk的视差变化平滑，其中，i∈WS，Sk，WS，Sk为块S与块Sk边界所有点对的下标集合，j为预设值。
即可定义如下公式：
Figure PCTCN2014089924-appb-000013
当th[s][sk]＜j时，块s与sk合并。
本实施例提供的用于全局视差估计的图像分块方法中，不仅利用了颜色信息进行分块，还引入了视差信息，能够进一步提高最后计算出的最终视差的准确度。
由于左右两幅图是不同视角观察到的画面，有些部分在左图中有在右图中没有，有些部分在右图中有在左图中没有，这些部分全部属于遮挡区域。由于这些区域只在一幅图中存在，按前面方法所做的视差计算算出的结果基本都是错误的，这些错误会影响最终的估算结果，因此，需要利用颜色分块将遮挡区域找出来，并标记成不可靠点，以提高最后的正确率。
以左图为例，通过人左右眼观察的方式，可以知道对于左图遮挡区域存在于颜色分块中每个块的右端与其他块相邻的部分，而左端相邻的部分是非遮挡的。对于右图遮挡区域存在于颜色分块中每个块的左端与其他块相邻的部分，而右端相邻的部分是非遮挡的。
本实施例中，在进行图像颜色分块后，计算最终视差之前，还包括标记图像中的遮挡区域，具体为：取左图每个分块在每一行中从左端开始的第一个可靠点L(p)，根据点L(p)的视差dp算得其对应于右图的点R(p-dp)；在右图中从点R(p-dp-1)开始向左边找到第一个可靠点Rq，找出其视差dq，算得点Rq对应于左图中的点L(q+dq)，水平的两个点L(p)与L(q+dq)之间的点即为遮挡点。
为了进一步提高正确率，本实施例中还包括步骤：基于颜色分块对现有可靠点做中值滤波，再次去除部分可靠点。即当S20在S30之后执行时，S20在进行进一步筛选可靠点时，可以利用S30中的信息。需要说明的是，图1中的部分步骤并不限定严格的执行顺序，其执行顺序可以根据具体需求确定。
以左图为例，对于左图中的每个可靠点p，首先估算其在X轴和Y轴上的梯度。估算方法为，在X轴上选取一些与p点在一个颜色分块内 的可靠点，计算其与这些点分别形成的梯度，最后取中值，即p点在X轴估算出来的梯度derivationX(p)。在Y方向上以相同方式得到derivationY(p)。然后对于左图中的每个点p，取其周围a*b方框内所有与点p在同一个块的可靠点qi，利用其视差d(qi)，X方向的梯度derivationX(qi)，Y方向的梯度derivationY(qi)估算点p的视差d(pi)。具体公式如下：
d(pi)＝d(qi)+(x[p]-x[qi])*derivationX[qi]
+(y[p]-y[qi])*derivationY[qi]............(14)
对所有d(pi)排序取中值并对该值进行四舍五入取整看其是否等于d(p)，如果不等则滤去该点。
S50：计算最终视差。
本实施例中，以左图为例，对于左图中的每个点p，取其周围e*f方框内所有与p在同一个分块的可靠点qi，其中e、f为预设的像素宽度，利用其视差d(qi)(即前面步骤中计算到得的初始视差)、X方向的梯度derivationX(qi)、Y方向的梯度derivationY(qi)估算点p的视差d(qi)，计算公式如下：
d(pi)＝d(qi)+(x[p]-x[qi])*derivationX[qi]
+(y[p]-y[qi])*derivationY[qi]
对所有d(pi)排序取中值并对该值进行四舍五入取整得到的值即为点p的最终视差d(p)。在其他实施例中，也可以采用现有技术中的任意一种方式来得到最终视差。
请参考图4，针对于本实施例提供的全局视差估计方法，本实施例还相应提供了一种全局视差估计系统，包括图像读入模块1000、匹配空间计算模块1001、匹配代价计算模块1002、初始视差计算模块1003、图像分块模块1004和最终视差计算模块1005。
图像读入模块1000用于读入第一视点图像和第二视点图像，第一视点图像为从第一视点获取的目标的图像，第二视点图像为从第二视点获取的目标的图像。
匹配空间计算模块1001用于根据预设规则在第一视点图像上选取采样点后，在第一视点图上依次选取像素点作为当前像素点，以当前像素点为原点，沿第一轴线正方向和负方向，以逐个像素点作为搜索点进行搜索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足约束条件的所有点作为第一匹配点；分别以各个第一匹配点为原点， 沿第二轴线正方向和负方向，以逐个像素点作为搜索点进行探索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足约束条件的所有点作为第二匹配点；将第一匹配点和第二匹配点作为当前像素点的第一匹配空间。匹配空间计算模块1001还用于以当前像素点为原点，沿第二轴线正方向和负方向，以逐个像素点作为搜索点进行搜索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足约束条件的所有点作为第三匹配点；分别以各个第三匹配点为原点，沿第一轴线正方向和负方向，以逐个像素点作为搜索点进行探索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足约束条件的所有点作为第四匹配点；将第三匹配点和第四匹配点作为当前像素点的第二匹配空间。约束条件包括线性约束条件和基于采样点的空间约束条件，线性约束条件为当前像素点与搜索点之间在颜色上的欧氏距离的约束，空间约束条件为搜索点与采样点之间在颜色上的欧氏距离的约束，第一轴线与第二轴线相互垂直。
匹配代价计算模块1002用于计算第一匹配空间内所有点的匹配代价之和，和计算第二匹配空间内所有点的匹配代价之和。
初始视差计算模块1003用于根据第一匹配空间内所有点的匹配代价之和和第二匹配空间内所有点的匹配代价之和计算初始视差，并筛选得到可靠点。
图像分块模块1004采用本实施例上述任意一种，用于对原始图像进行图像分块。
最终视差计算模块1005用于基于图像分块，计算第一视点图像内每个像素点的最终视差。
本实施例提供的全局视差估计系统与上述全局视差估计方法对应，此处不再对其工作原理进行赘述。
请参考图5，为采用本申请实施例提供的全局视差估计方法在Middlebury数据集上的实验结果图，在Middlebury测试平台上的测试结果表明，采用本申请实施例提供的全局视差估计方法得到的结果(第2行结果)优于目前大部分的方法。图5中采用“非遮挡区域(nonocc)”“所有区域(all)”“不连续区域(disc)”作为评价指标，错误率阈值被设为1.0，即与真实视差(ground truth)相差大于1则标为错误点。
本申请提供的全局视差估计方法和系统，基于一种鲁棒的混合代价函数得到像素点的混合代价，采用改进的聚合空间对单点代价进行聚合；然后采用快速置信传播全局算法进行全局代价的最优化计算；最后又采用了专门针对视差估计的图像分块以及遮挡点的标记，因此能够大大提高最终视差计算的准确度。
本领域技术人员可以理解，上述实施方式中各种方法的全部或部分步骤可以通过程序来指令相关硬件完成，该程序可以存储于一计算机可读存储介质中，存储介质可以包括：只读存储器、随机存储器、磁盘或光盘等。
以上内容是结合具体的实施方式对本申请所作的进一步详细说明，不能认定本申请的具体实施只局限于这些说明。对于本申请所属技术领域的普通技术人员来说，在不脱离本申请发明构思的前提下，还可以做出若干简单推演或替换。

Claims (10)

  1. 一种全局视差估计系统，其特征在于，包括：
    图像读入模块，用于读入第一视点图像和第二视点图像，第一视点图像为从第一视点获取的目标的图像，第二视点图像为从第二视点获取的目标的图像；
    匹配空间计算模块，用于根据预设规则在第一视点图像上选取采样点后，在第一视点图上依次选取像素点作为当前像素点，以当前像素点为原点，沿第一轴线正方向和负方向，以逐个像素点作为搜索点进行搜索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第一匹配点；分别以各个第一匹配点为原点，沿第二轴线正方向和负方向，以逐个像素点作为搜索点进行探索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第二匹配点；将第一匹配点和第二匹配点作为当前像素点的第一匹配空间；
    以当前像素点为原点，沿第二轴线正方向和负方向，以逐个像素点作为搜索点进行搜索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第三匹配点；分别以各个第三匹配点为原点，沿第一轴线正方向和负方向，以逐个像素点作为搜索点进行探索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第四匹配点；将第三匹配点和第四匹配点作为当前像素点的第二匹配空间；
    所述约束条件包括线性约束条件和基于采样点的空间约束条件，所述线性约束条件为当前像素点与搜索点之间在颜色上的欧氏距离的约束，所述空间约束条件为搜索点与采样点之间在颜色上的欧氏距离的约束，所述第一轴线与第二轴线相互垂直；
    匹配代价计算模块，用于计算第一匹配空间内所有点的匹配代价之和，和计算第二匹配空间内所有点的匹配代价之和；
    初始视差计算模块，用于根据第一匹配空间内所有点的匹配代价之和和第二匹配空间内所有点的匹配代价之和计算初始视差，并筛选得到可靠点；
    图像分块模块，用于对第一视点图像和第二视点图像进行图像分块；
    最终视差计算模块，用于基于所述图像分块，并根据所述可靠点的初始视差计算第一视点图像和第二视点图像内每个像素点的最终视差。
  2. 如权利要求1所述的系统，其特征在于，所述约束条件为：
    Figure PCTCN2014089924-appb-100001
    其中，l1为像素点p到搜索点q的距离，像素点p为当前像素点，12为像素点p到采样点ei的距离，Olab(p，q)为像素点p与搜索点q在颜色上的欧氏距离，Olab(q，ei)为搜索点q与采样点ei在颜色上的欧氏距离，k1、k2、k3、k4、w1、w2为自定义参数，且k1＞k2、k4＞k3、w2＞w1。
  3. 如权利要求1所述的系统，其特征在于，所述预设规则使得每个采样点离其上、下、左、右四个相邻采样点的距离为一预设距离。
  4. 如权利要求1所述的系统，其特征在于，图像读入模块还用于在读入第一视点图像和第二视点图像后，对第一视点图像和第二视点图像进行极线校正。
  5. 如权利要求1所述的系统，其特征在于，还包括遮挡区域标记模块，用于在所述图像分块模块对原始图像进行图像分块后，最终视差计算模块计算最终视差之前，标记图像中的遮挡区域，具体为：取第一视点图像每个分块在每一行中从左端开始的第一个可靠点L(p)，根据点L(p)的视差dp算得其对应于第二视点图像的点R(p-dp)；在第二视点图像中从点R(p-dp-1)开始向左边找到第一个可靠点Rq，找出其视差dq，算得点Rq对应于第一视点图像中的点L(q+dq)，水平的两个点L(p)与L(q+dq)之间的点即为遮挡点。
  6. 如权利要求1所述的系统，其特征在于，初始视差计算模块根据第一匹配空间内所有点的匹配代价之和和第二匹配空间内所有点的匹配代价之和，采用快速置信传播全局算法计算初始视差。
  7. 如权利要求1-6任一项所述的系统，其特征在于，图像分块模块对第一视点图像和第二视点图像进行图像分块时：
    图像分块模块将第一视点图像和第二视点图像划分为若干个图像块；
    根据颜色对图像块进行合并：将像素点数量小于预设值的图像块与其相邻图像块中颜色最接近的图像块合并；和/或，判断到两个相邻图像块颜色接近，且两个图像块像素点数量之和小于预设值时，将两个图像块合并；
    根据视差对图像块进行合并：将可靠点数量小于预设值的图像块与其相邻图像块中颜色最接近的图像块合并，所述可靠点为根据原始图像中各像素点的初始视差筛选得到的；和/或，判断相邻两个图像块的视差变化是否平滑，如果是，则将两个图像块合并。
  8. 如权利要求7所述的系统，其特征在于，图像分块模块将第一视 点图像和第二视点图像划分为若干个图像块时：图像分块模块基于superpixel颜色分块将图像划分为若干个图像块。
  9. 如权利要求7所述的系统，其特征在于，图像分块模块判断相邻两个图像块的视差变化是否平滑时：
    图像分块模块找出当前图像块S与其相邻图像块Sk的边界相邻点对PS(i)、PSk(i)，PS(i)与PSk(i)为块S与块Sk的第i个相邻点对；
    以PS(i)为中心搜索一个a*b的矩形方框，计算该方框中属于块S的可靠点的视差的均值VS(i)，以PSk(i)为中心搜索一个a*b的矩形方框，计算该方框中属于块Sk的可靠点的视差的均值VSk(i)，其中，a、b为预设的像素宽度；
    当max|VS(i)-VSk(i)|＜j时，判断为当前图像块S与其相邻图像块Sk的视差变化平滑，其中，i∈WS，Sk，WS，Sk为块S与块Sk边界所有点对的下标集合，j为预设值。
  10. 一种全局视差估计方法，其特征在于，包括：
    读入第一视点图像和第二视点图像，第一视点图像为从第一视点获取的目标的图像，第二视点图像为从第二视点获取的目标的图像；
    根据预设规则在第一视点图像上选取采样点；
    在第一视点图上依次选取像素点作为当前像素点，以当前像素点为原点，沿第一轴线正方向和负方向，以逐个像素点作为搜索点进行搜索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第一匹配点；分别以各个第一匹配点为原点，沿第二轴线正方向和负方向，以逐个像素点作为搜索点进行探索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第二匹配点；将第一匹配点和第二匹配点作为当前像素点的第一匹配空间；
    以当前像素点为原点，沿第二轴线正方向和负方向，以逐个像素点作为搜索点进行搜索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第三匹配点；分别以各个第三匹配点为原点，沿第一轴线正方向和负方向，以逐个像素点作为搜索点进行探索，直到搜索到不满足预设的约束条件的点时停止，并将搜索到的满足所述约束条件的所有点作为第四匹配点；将第三匹配点和第四匹配点作为当前像素点的第二匹配空间；
    所述约束条件包括线性约束条件和基于采样点的空间约束条件，所述线性约束条件为当前像素点与搜索点之间在颜色上的欧氏距离的约束，所述空间约束条件为搜索点与采样点之间在颜色上的欧氏距离的约束，所述第一轴线与第二轴线相互垂直；
    计算第一匹配空间内所有点的匹配代价之和，计算第二匹配空间内 所有点的匹配代价之和；
    根据第一匹配空间内所有点的匹配代价之和和第二匹配空间内所有点的匹配代价之和计算初始视差，并筛选得到可靠点；
    对第一视点图像和第二视点图像进行图像分块；
    基于所述图像分块，并根据所述可靠点的初始视差分别计算第一视点图像和第二视点图像内每个像素点的最终视差。
PCT/CN2014/089924 2014-10-30 2014-10-30 一种全局视差估计方法和系统 Ceased WO2016065579A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2014/089924 WO2016065579A1 (zh) 2014-10-30 2014-10-30 一种全局视差估计方法和系统

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2014/089924 WO2016065579A1 (zh) 2014-10-30 2014-10-30 一种全局视差估计方法和系统

Publications (1)

Publication Number Publication Date
WO2016065579A1 true WO2016065579A1 (zh) 2016-05-06

Family

ID=55856397

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2014/089924 Ceased WO2016065579A1 (zh) 2014-10-30 2014-10-30 一种全局视差估计方法和系统

Country Status (1)

Country Link
WO (1) WO2016065579A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111275730A (zh) * 2020-01-13 2020-06-12 平安国际智慧城市科技股份有限公司 地图区域的确定方法、装置、设备及存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101790103A (zh) * 2009-01-22 2010-07-28 华为技术有限公司 一种视差计算方法及装置
CN102930530A (zh) * 2012-09-26 2013-02-13 苏州工业职业技术学院 一种双视点图像的立体匹配方法
WO2013173106A1 (en) * 2012-05-18 2013-11-21 The Regents Of The University Of California Independent thread video disparity estimation method and codec

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101790103A (zh) * 2009-01-22 2010-07-28 华为技术有限公司 一种视差计算方法及装置
WO2013173106A1 (en) * 2012-05-18 2013-11-21 The Regents Of The University Of California Independent thread video disparity estimation method and codec
CN102930530A (zh) * 2012-09-26 2013-02-13 苏州工业职业技术学院 一种双视点图像的立体匹配方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
YUAN, LI ET AL.: "Initial Disparity Estimation Algorithm Based on Weighted Matching Cost", COMPUTER TECHNOLOGY AND DEVELOPMENT, vol. 21, no. 10, 31 October 2011 (2011-10-31) *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111275730A (zh) * 2020-01-13 2020-06-12 平安国际智慧城市科技股份有限公司 地图区域的确定方法、装置、设备及存储介质

Similar Documents

Publication Publication Date Title
CN113935428B (zh) 基于图像识别的三维点云聚类识别方法及系统
Li et al. PMSC: PatchMatch-based superpixel cut for accurate stereo matching
CN105517677B (zh) 深度图/视差图的后处理方法和装置
CN104680510B (zh) Radar视差图优化方法、立体匹配视差图优化方法及系统
CN102930530B (zh) 一种双视点图像的立体匹配方法
CN102903096B (zh) 一种基于单目视频的对象深度提取方法
CN108596923B (zh) 三维数据的获取方法、装置以及电子设备
CN104331890B (zh) 一种全局视差估计方法和系统
CN109255833A (zh) 基于语义先验和渐进式优化宽基线致密三维场景重建方法
CN104318576B (zh) 一种超像素级别的图像全局匹配方法
CN102750711A (zh) 一种基于图像分割和运动估计的双目视频深度图求取方法
KR101869605B1 (ko) 평면정보를 이용한 3차원 공간 모델링 및 데이터 경량화 방법
CN104899883A (zh) 一种深度图像场景的室内对象立方体检测方法
CN106408596B (zh) 基于边缘的局部立体匹配方法
CN102368826A (zh) 双视点视频到多视点视频的实时自适应生成方法
CN109887008B (zh) 基于前后向平滑和o(1)复杂度视差立体匹配方法、装置和设备
CN104182968A (zh) 宽基线多阵列光学探测系统模糊动目标分割方法
CN107016698A (zh) 基于斜平面平滑的双目立体匹配方法及装置
CN107578419B (zh) 一种基于一致性轮廓提取的立体图像分割方法
CN104408710B (zh) 一种全局视差估计方法和系统
CN109961092A (zh) 一种基于视差锚点的双目视觉立体匹配方法及系统
CN107155100B (zh) 一种基于图像的立体匹配方法及装置
CN115496790B (zh) 一种大基线光场视频深度估计方法
CN106683072B (zh) 一种基于pup图的3d图像舒适度质量评价方法及系统
WO2016065579A1 (zh) 一种全局视差估计方法和系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14904630

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14904630

Country of ref document: EP

Kind code of ref document: A1