CN104091336B - Stereoscopic image synchronous segmentation method based on dense disparity map - Google Patents

Abstract

The invention relates to a method for synchronously segmenting stereoscopic images based on dense disparity maps. First, a set of stereo images is input, and a disparity map is obtained through a stereo image matching algorithm. Then in one of the pictures, specify part of the foreground and background by drawing with a brush. According to the specified part, the prior statistical models of the color distribution of the foreground and the background, and the prior statistical models of the parallax distribution of the foreground and the background are respectively established. On this basis, under the framework of graph cut theory, constraints such as color, gradient and parallax are formalized, and energy functions are constructed. Finally, the optimization result is solved by using the maximum flow/minimum cut algorithm of the graph. If the user does not get the desired effect, he can continue to outline the wrong area in the picture until the desired result is obtained. Both the disparity distribution model and the change model adopted in the present invention are disparity statistical information, which effectively avoids the influence of disparity calculation errors. Compared with the existing method, the segmentation result obtained by the method of the present invention is more accurate.

Description

A Stereo Image Synchronous Segmentation Method Based on Dense Disparity Map

technical field

The invention belongs to the intersecting fields of computer vision, computer graphics and image processing, and relates to a method for synchronously segmenting stereoscopic images based on dense disparity maps.

Background technique

The popularity of stereoscopic images in various fields has put forward an urgent demand for intelligent processing of such data. Interactive stereo image intelligent segmentation is one of the important tasks: the user only needs to specify a small amount of foreground and background on one image in the stereo image, and the method will automatically complete the synchronous segmentation of the two images. The effectiveness of segmentation algorithms determines the accuracy of detection, recognition, classification and tracking in video surveillance applications. The segmented foreground object can be used as the input data for 3D model reconstruction to remove background interference during the reconstruction process. The segmentation algorithm and program can also help ordinary users edit the pictures of daily life captured by stereo cameras, and help film and television producers to edit stereo TVs and movies later. Examples include removing unwanted objects, compositing foreground objects into a new background, and copying and pasting foreground objects.

At present, the interactive segmentation method for a single image is relatively mature, and some have been implemented in practical applications, such as The Quick Selection tool in CS3. Compared with the segmentation of a single image, the intelligent segmentation of interactive stereo images started late. The existing basic framework for stereo image segmentation is as follows: firstly, a disparity map is obtained through a stereo matching algorithm. Each pixel value in the disparity map represents the offset of the corresponding pixel in the reference map (one of the two pre-selected images) in the matching map. That is, given a pair of stereo images and the corresponding disparity map of the left image, the corresponding pixels of the pixels of the left image in the right image can be obtained. After obtaining the disparity map, formalize the disparity clues, as well as clues such as color and gradient commonly used in single image segmentation, to form an energy function. The image segmentation problem is solved by optimizing the energy function. The quality of the disparity map has an important impact on the segmentation results. However, there are many errors in the disparity map obtained by the existing stereo matching method. The existing stereo image segmentation method based on the disparity map, such as "StereoCut: Consistent Interactive Object Selection in Stereo Image Pairs" published by Price et al. on ICCV in 2011 In ", directly formalizing the correspondence determined by the disparity map in the energy function will easily lead to segmentation errors and affect the intelligence of segmentation.

Contents of the invention

In view of the limitations of the current stereoscopic image segmentation method based on the disparity map in the use of parallax, the present invention explores a new segmentation method under the theoretical framework of the synchronous segmentation of stereoscopic images based on the disparity map, and strives to reduce the impact of matching errors on the segmentation results. Realize the purpose of improving the intelligence of the segmentation process.

To achieve this goal, the technical solution of the present invention is: after the user inputs a set of stereoscopic images, the method automatically obtains a disparity map through a stereoscopic image matching algorithm. Then, the user can designate part of the foreground and background by drawing with a brush in one of the pictures. And automatically establish the prior statistical models of the color distribution of the foreground and the background, and the prior statistical models of the parallax distribution of the foreground and the background respectively according to the specified part. On this basis, under the framework of graph cut theory, constraints such as color, gradient and parallax are formalized, and energy functions are constructed. Finally, the optimization result is solved by using the maximum flow/minimum cut algorithm of the graph. If the user does not get the desired effect, he can continue to outline the wrong area in the picture until the desired result is obtained.

Compared with the prior art, the present invention has the following advantages: based on the disparity map, the present invention establishes a statistical model of the disparity distribution of the foreground and the background, and at the same time mathematically formalizes the variation of disparity in the image, and constructs an energy function in combination with traditional constraints, and The segmentation is realized by solving the minimum value of the energy function through the graph cut algorithm. Both the disparity distribution model and the change model are disparity statistical information, which effectively avoid the influence of disparity calculation errors. The experiment proves that: compared with the existing method, under the premise of the same amount of interaction, the segmentation result obtained by the method of the present invention is more accurate.

Description of drawings

Fig. 1 is the flowchart of the method involved in the present invention;

Fig. 2 is the experimental result of the application example of the present invention: (a), (b) are the left and right images of input, (c), (d) adopt the "StereoCut: Consistent Interactive" published by Price et al. on the ICCV in 2011 The result calculated by the method in ObjectSelection in Stereo Image Pairs"; (e), (f) are the segmentation results of the present invention; the user input used by the two methods is shown in (c), (e), and the object inside Solid lines indicate the foreground, and dashed lines outside the target area indicate the background.

detailed description

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

Flow process of the present invention is as shown in Figure 1, specifically comprises the following steps:

Step 1, matching stereo images.

Read in a pair of stereo images I={I ^l , I ^r }, where I ^l and I ^r represent the left and right images respectively. Use the stereo matching algorithm to calculate the disparity maps corresponding to the left and right images, denoted by D ^l and D ^r respectively. Stereo matching can use any algorithm, such as the algorithm proposed in the paper "Efficient Belief Propagation for EarlyVision" published by Felzenszwalb et al. on CVPR04.

Step two, add foreground and background cues.

The user specifies part of the foreground and background in any one of the images through the designed interface. The embodiment of the present invention adopts a method similar to that used in "StereoCut: Consistent Interactive Object Selection in Stereo Image Pairs" published by Price et al. Draw lines of different colors to specify part of the foreground and background pixels. As shown in Figure 2(e), the pixels covered by the red line belong to the foreground, and the pixels covered by the blue line belong to the background. The subsequent steps of the present invention are not limited to the method of specifying front and background pixels used in this step, and other methods can also be used.

Step 3, establishing the color and parallax prior models of the foreground and background.

Let F represent the set of foreground pixels specified by the user, and B represent the set of background pixels specified by the user. The color of the foreground and the background, and the prior model of the disparity can be expressed in the form of GMM, histogram, and multiple clusters, and obtained by fitting or counting the colors of the corresponding pixel sets. The embodiment of the present invention adopts the K-means clustering method to cluster the color values corresponding to the pixels in F and B respectively, and obtain N _c foreground color clusters M _c background color clusters Represent the statistical model of the color distribution of the background, respectively. At the same time, use the same method to cluster the disparity values corresponding to the pixels in F and B to obtain N _d foreground disparity clusters M _d background disparity clusters Represent the disparity distribution statistical model of foreground and background respectively. The present invention proposes N _c =M _c =N _d =M _d =64.

Step 4, define the energy function.

Stereo image I={I ^l , I ^r }, including left image I ^l and right image I ^r , can be expressed as an undirected graph G=<ν,ε>. Among them, ν is the set of nodes in graph G, and ε is the set of edges. Each vertex in graph G corresponds to a pixel in stereo image I. Except for the pixels in set F and set B in the image, the remaining pixels are set to belong to set U. Interactive Stereo Image Synchronous Segmentation is to assign a label _xi to each pixel _pi in U under the constraints of the input strokes. x _i ∈ {1, 0}, denote front and background respectively. The edges in the graph G include the connection edges of adjacent pixels in the image, and the connection edges between the corresponding points of the stereo image determined by the disparity map.

The solution to the above stereo image synchronization problem is defined as the optimization problem of the following target energy function:

Among them, f _Unary (p _i , _xi ) is a unary term (Unary Term), which indicates the similarity of the color of pixel p _i , the disparity with the front and background colors and the disparity statistical model, and is also called a data term (Data Term). The higher the similarity, the larger the value of f _Unary . f _Intra (p _i , p _j ) is an Intra Plane Binary Term, which reflects the difference between all pixels in I and its neighbors (four-neighborhood or eight-neighborhood). N _Intra represents a set of adjacency relationships including all pixels in the left and right images. The larger the difference, the smaller the term. According to the principle of the graph cut algorithm, at this time, the neighboring pixels tend to take different labels. is a binary term (Inter Plane Binary Term) between images, which defines the matching result of corresponding points. The higher the matching degree, the larger the term. C _Inter represents a set containing the correspondence relationship between left and right image pixels. λ _Unary , λ _Intra , λ _Inter are the weights for adjusting the energy items.

(1) Define unary constraints

The unary constraint item includes two parts, the color unary item and the parallax unary item, which are defined as follows:

f _Unary (p _i ， _xi )＝λ _c (1-P _c ( _xi | _ci ))+λ _d (1-P _d (x|d _i )) (2)

Among them, P _c ( _xi |ci ) represents the color c _i of a given pixel p _{i , and xi takes} the probability value of the foreground or background label. Considering that the greater the probability, the smaller the energy function should be, so we take 1-P _c to represent the unary item of color. Likewise, P _c ( _xi |d _i ) represents the disparity value d _i of a given pixel p _i , where _xi takes the probability value of the foreground or background label. Take 1-P _d to denote the disparity unary term. λ _c and λ _d represent the influence weights of color and parallax respectively, λ _c +λ _d =1.

The present invention represents the color and parallax model of the foreground and background in the form of clusters (N _c foreground color clusters M _c background color clusters N _d foreground disparity clusters M _d background disparity clusters ), giving the calculation method of the unary term.

The color unary term is calculated as follows. Compare the color of each unmarked pixel with the clusters of foreground and background colors, and find the minimum distance from the centers of these clusters, which is used to describe the similarity between the pixel color and the foreground and background colors. The smaller the distance from the foreground (or background) color, the closer the color is. According to the graph cut theory, the pixel is more inclined to select the foreground (or background) label. The mathematical form of the color unary term is described as:

in, represent the minimum distances from the color c _i of the pixel p _i to the centers of various clusters of the foreground and background colors, and the expressions are respectively:

The disparity unary is calculated in the same way as the color unary.

(2) Define binary constraints in the image

The binary constraint item f _Intra (p _i , p _j ) in the image contains two items, which respectively describe the color change (ie color gradient) and parallax change (ie parallax gradient) around the pixel point, which are defined as follows:

f _Intra (p _i , p _j ) = f _c (p _i , p _j ) f _d (p _i , p _j ) (4)

Among them, f _c (p _i , p _j ) represents the similarity of colors between adjacent pixels. The closer the color is, the larger the value is. According to the principle of the graph cut algorithm, the probability of the boundary passing through the two is smaller. f _d (p _i , p _j ) represents the disparity similarity of pixel p _i with respect to adjacent pixel point p _j . The closer the parallax of the two is, the larger its value is, and according to the principle of the graph cut algorithm, the probability of the boundary passing through the two is smaller. The present invention suggests as follows for the definition form of two items:

The above two terms can also take other forms, such as the exponential form used in "StereoCut: Consistent Interactive Object Selection in Stereo Image Pairs" published by Price et al. on ICCV in 2011.

In fact, there are errors in the disparity calculation, and direct use will introduce errors into the segmentation process. The solution proposed by the present invention is to replace the parallax change between two pixels with the parallax change in a local area. Let S _j denote the area where p _i is located. The parallax change in the area S _j is represented by the variance var(S _j ). In this case, equation (6) becomes:

Wherein, S _j =A(p _i ), and the function A(p _i ) represents the area where the pixel p _i is located. The image area can be obtained by over-segmentation method, or by dividing the picture into a small set of square areas in advance.

(3) Define binary constraints between images

The inter-image binary term constrains corresponding pixels between stereo images to take the same label, defined as follows:

Among them, C represents the stereo image The likelihood of corresponding points between is an asymmetric function:

is determined based on the disparity map Between as the probability distribution function of the corresponding points. function express is the left image pixel At the corresponding points on the right image, the correspondence is determined according to the disparity map. Price et al. mentioned in "StereoCut: Consistent Interactive Object Selection in Stereo Image Pairs" published on ICCV in 2011 The five definition methods of are Uniform Distribution, Delta Function, Probability Density Distribution Function, Consistent Delta Function, and Consistent Probability Density Distribution Function. Due to the high computational complexity in the form of uniform distribution, probability density distribution function, and uniform probability density distribution function, the present invention proposes to use a Delta function or a consistent Delta function, which is defined as follows. The Delta function only uses the disparity map corresponding to a single image. Let {d ^l } be the disparity set of the left image, and the Delta function is defined as:

in, is the pixel in the left image Corresponding to the point in the figure on the right parallax. Let {d ^r } be the disparity set of the right image, and the consistent Delta function is:

in, is the pixel in the right image Corresponding to the left figure parallax.

In formula (9) express and The color similarity between, in the case of perfectly accurate parallax, However, there are errors in the current parallax calculation method, and the proposed form of the present invention is as follows:

in, is the pixel of the left image the color value, yes Corresponding point on the right color value.

Step five, find the minimum value of the energy function.

The present invention adopts a graph-cut algorithm, such as the maximum flow/ The minimum cut algorithm obtains the optimal marking result, ie, the segmentation result, by optimizing the energy function (formula (1)) defined in the present invention. If the user is not satisfied with the segmentation result, he can return to step 2 and continue to add front and background clues. Each addition will trigger a complete splitting process.

Taking the method in "StereoCut: Consistent Interactive Object Selection in Stereo Image Pairs" published by Price et al. on ICCV in 2011 as a comparison object, the effectiveness of the method of the present invention is illustrated. Both methods use a consistent Delta function (Equation (11)) as the probability distribution function between corresponding points. Figure 2 shows the effect comparison. Figure 2(a), (b) are the input left and right images. (c), (d) are the results calculated by the StereoCut method; Figure 2 (e), (f) is the segmentation result of the present invention; in the four result figures, the closed curve around the target (street lamp) marks the segmentation target The boundary of the object; the user input used by the two methods is shown in Figures (c) and (e) respectively, the solid line inside the object marks the foreground, and the dotted line outside the object marks the background. Comparing Figures (c), (d) and Figures (e), (f), it can be seen that the method of the present invention has a better segmentation effect under the premise of the same amount of interaction. If the comparison method wants to get better results, the user needs to specify more foreground and background. Therefore, the inventive method is more intelligent than the comparative method.

Claims (2)

1. A stereoscopic image synchronous segmentation method based on dense parallax map, it is characterized in that comprising the following steps: Step 1, matching stereoscopic images; Read in a pair of stereo images I={I ^l , I ^r }, I ^l and I ^r represent the left and right images respectively; use the stereo matching algorithm to calculate the disparity maps corresponding to the left and right images, and use D ^l and D ^r respectively express; Step 2, add foreground and background clues; Designate part of the foreground and background in any one of the images through the designed interface; use input devices such as mouse, touch screen or stylus to designate part of the foreground and background pixels by drawing lines of different colors on the image; Step 3, establishing the color and parallax prior models of the foreground and background; Use F to represent the foreground pixel set specified by the user, and B to represent the background pixel set specified by the user; the prior model of the color of the front and background, and the disparity is expressed in the form of multiple clusters, and is obtained by fitting or counting the colors of the corresponding pixel set : use the K-means algorithm to cluster the color values corresponding to the pixels in F and B respectively, and obtain N _c foreground color clusters M _c background color clusters Represent the statistical model of the color distribution of the background; at the same time, use the same method to cluster the disparity values corresponding to the pixels in F and B to obtain N _d foreground disparity clusters M _d background disparity clusters represent the disparity distribution statistical models of the foreground and background respectively; N _c ＝M _c ＝N _d ＝M _d ＝64; Step 4, define the energy function; Stereo image I={I ^l , I ^r }, including left graph I ^l and right graph I ^r , can be expressed as an undirected graph G=<ν,ε>; where ν is the node set in graph G, ε is a set of edges; each vertex in the graph G corresponds to a pixel in the stereo image I; except for the pixels in the set F and the set B in the image, the remaining pixels belong to the set U; the synchronous segmentation of the interactive stereo image is in Under the constraints of the input strokes, assign a label _xi to each pixel p _i in U; _xi ∈ {1,0}, representing the front and background respectively; the edges in the graph G include the connection of adjacent pixels in the image edges, and the connecting edges between the corresponding points of the stereo image determined by the disparity map; The solution to the above stereo image synchronization problem is defined as the optimization problem of the following target energy function: Among them, f _Unary (p _i , x _i ) is a unary item, which represents the similarity between the color of pixel p _i , the disparity, the front and background colors and the disparity statistical model, and is also called a data item; the higher the similarity, the higher the value of f _Unary large; f _Intra (p _i , p _j ) is a binary item in the image, reflecting the difference between all pixels in I and its four neighbors or eight neighbors, the greater the difference, the smaller the item; N _Intra means Contains a collection of adjacency relations of all pixels in the left and right images; according to the principle of the graph cut algorithm, at this time, the neighboring pixels tend to take different labels; is a binary item between images, which defines the matching result of corresponding points. The higher the matching degree, the larger the item; C _Inter represents the set containing the corresponding relationship between left and right image pixels; λ _Unary , λ _Intra , λ _Inter are Adjust the weight between each energy item; (1) Define unary constraints The unary constraint item includes two parts, the color unary item and the parallax unary item, which are defined as follows: f _Unary (p _i , _xi )＝λ _c (1-P _c ( _xi | _ci ))+λ _d (1-P _d (x|d _i )) (2) Among them, P _c ( _xi |ci ) represents the color c _i of a given pixel p _{i , and xi takes} the probability value of the foreground or background label; considering that the greater the probability, the smaller the energy function should be, so take 1-P _c represents the color unary item; similarly, P _c ( _xi |d _i ) represents the disparity value d _i of a given pixel p _i , and _xi takes the probability value of the foreground or background label; taking 1-P _d represents the disparity unary item; λ _c , λ _d represent the influence weights of color and parallax respectively, λ _c +λ _d ＝1; Represent the color and disparity model of foreground and background in the form of clusters, including N _c foreground color clusters M _c background color clusters N _d foreground disparity clusters M _d background disparity clusters Give the calculation method of the unary term; The color unary term is calculated as follows: compare the color of each unlabeled pixel with the clusters of foreground and background colors, and find the minimum distance from the center of these clusters, which is used to describe the pixel color and the foreground and background colors. The similarity; the smaller the distance from the foreground or background color, the closer the color is, according to the graph cut theory, the pixel is more inclined to select the foreground or background label; the mathematical form of the color unary term is described as: in, represent the minimum distances from the color c _i of the pixel p _i to the centers of various clusters of the foreground and background colors, and the expressions are respectively: The calculation process of the disparity unary item is the same as that of the color unary item; (2) Define binary constraints in the image The binary constraint term f _Intra (p _i ,p _j ) in the image contains two items, which respectively describe the color change and parallax change around the pixel point, that is, the color gradient and the parallax gradient, which are defined as follows: f _Intra (p _i ,p _j )＝f _c (p _i ,p _j )f _d (p _i ,p _j ) (4) Among them, f _c (p _i , p _j ) represents the similarity of colors between adjacent pixels. The closer the color is, the larger the value is. According to the principle of the graph cut algorithm, the probability of the boundary passing through the two is smaller; f _d ( p _i , p _j ) represents the similarity of disparity between pixel p _i and adjacent pixel point p _j ; the closer the two disparities are, the larger the value is, and according to the principle of the graph cut algorithm, the probability of the boundary passing through the two is smaller ; The definitions of the two terms are as follows: In fact, there is an error in the disparity calculation, and direct use will introduce the error into the segmentation process; the solution is to replace the parallax change between two pixels with the parallax change of the local area; let S _j represent the area _where p _i is located; The parallax change is represented by the variance var(S _j ); in this case, formula (6) becomes: Wherein, S _j =A(p _i ), the function A(p _i ) represents the area where the pixel p _i is located; the image area can be obtained by over-segmentation, or by dividing the picture into small square area sets in advance; (3) Define binary constraints between images The inter-image binary term constrains corresponding pixels between stereo images to take the same label, defined as follows: Among them, C represents the stereo image The likelihood of corresponding points between is an asymmetric function: is determined based on the disparity map between as the probability distribution function of corresponding points; the function express is the left image pixel At the corresponding point on the right image, the corresponding relationship is determined according to the disparity map; Use the Delta function or a consistent Delta function, defined as follows; the Delta function only uses the disparity map corresponding to a single image; let {d ^l } be the disparity set of the left image, and the Delta function is defined as: in, is the pixel in the left image Corresponding to the point in the figure on the right disparity; let {d ^r } be the disparity set of the right image, and the consistent Delta function is: in, is the pixel in the right image Corresponding to the left figure parallax; In formula (9) express and The color similarity between, in the case of perfectly accurate parallax, However, there is an error in the current disparity calculation method, and the following form is adopted to eliminate the error: in, is the pixel of the left image the color value, yes Corresponding point on the right the color value; Step 5, solving the minimum value of the energy function; Adopt the graph cut algorithm, by optimizing the energy function defined in the present invention, i.e. formula (1), obtain optimal labeling result, i.e. segmentation result; As unsatisfactory to segmentation result, return to step 2, continue to add front, background clue ; Each time an item is added, a complete segmentation process will be triggered. 2. a kind of stereoscopic image synchronous segmentation method based on dense disparity map according to claim 1, it is characterized in that, the color of front and background described in step 3, disparity prior model can also adopt GMM, histogram form expression .