SG193942A1

SG193942A1 - A method and system for interactive image segmentation

Info

Publication number: SG193942A1
Application number: SG2013071782A
Authority: SG
Inventors: T N Ahn Nguyen; Jianfei Cai; Juyong Zhang; Jianmin Zheng
Original assignee: Univ Nanyang Tech
Priority date: 2011-04-21
Filing date: 2012-04-23
Publication date: 2013-11-29
Also published as: WO2012144957A1

Abstract

The present invention proposes using a convex active contour model for interactive segmentation of an input image. A user specifies that at least one foreground seed of the image is part of the foreground, and at least one background seed is part of the background. One or more probabilistic functions are derived, indicating the likelihood of other pixels being part of the foreground or background. The probabilistic functions are incorporated into the definition of a regional term of the active contour model, and optionally also a boundary term of the active contour model.

Description

A Method and System for interactive Image Segmentation

Field of the invention = The present invention relates to a method and system for performing interactive . image segmentation. }

Background of the Invention

Interactive image segmentation, which incorporates user interaction to define : the desired content to be extracted, has received much attention in recent : oo years. Note that in this document, “interactive image segmentation” or “semi- automatic image segmentation” means image segmentation using user inputs whereas “automatic image segmentation” means image segmentation without using any user input. :

Many interactive image segmentation algorithms have been proposed. in general, interactive image segmentation algorithms can be classified into two categories: boundary-based approaches and region-based approaches.

In boundary-based approaches, the user is often asked to specify at least one initial contour that is close to a desired boundary. The active contours or Snake method [14] then evolves the initial contour to the desired boundary without : using ‘additional user inputs. Alternatively, the user may be asked to specify boundary. points on the desired boundary. The methods of intelligent scissors ~ [15, 11] apply Dijkstra’s shortest path algorithm to find the cutting contour between neighboring boundary points that are specified by the user. A problem with boundary-based approaches is that they require great care to specify the initial contours or the boundary points, especially for complex shapes. | | oo

Considering the above-mentioned problem with boundary-based approaches, several recent interactive image segmentation algorithms have instead taken regional information as the input. In particular, in “these region-based : approaches, the user is often asked to draw two types of strokes to label some : pixels in the input image as foreground or background pixels (whereby the foreground pixels form the desired object to be segmented and the background pixels form the rest of the image), after which the algorithm completes the labeling for the remaining pixels in the input image. oo | Most existing region-based interactive segmentation algorithms work by treating ~ the input image as a weighted graph with nodes corresponding to pixels in the image and with edges placed between neighboring pixels. They further work by minimizing a certain energy. function on this weighted graph to produce segmentation of the input image. Examples of existing region-based interactive segmentation algorithms include the Graph Cut based methods [3, 16], the

Random Walks based methods [13, 21, 20] and the Geodesic methods [1, 9].

Although the existing region-based algorithms require much less user effort for the initialization step as compared to the boundary-based approaches, they suffer from problems that undermine their segmentation performance.

Therefore, existing region-based algorithms often do not achieve robust and accurate segmentation of the input image.

For example, one problem of existing region-based algorithms is that they are sensitive to user inputs. As pointed out in [17], the Graph Cut algorithm is "sensitive to the number of seeds input by the user (i.e. the number of pixels which the user labels as foreground or background pixels), while the Random

Walks and Geodesic algorithms are sensitive to locations of these seeds. This is mainly due to the different behaviors of the energy functions used in the algorithms. For example, the Graph Cut algorithm operates by minimizing the total edge weights along the cut (or boundary of the segmentation). Thus, it may return very small segmentations in the case where a small number of seeds is provided by the user. This is known as the “small cut” problem. The Random

Walks algorithm operates by minimizing the Dirichlet energy functional with respect to the boundary conditions (locations of the seeds input by the user) and different boundary conditions tend to result in different harmonic functions.

Figs. 1(a) — (d) and 2(a) — (d) .illustrate the above-mentioned user input sensitivity problem by showing the segmentation results of different existing region-based algorithms including the Random Walks algorithm [13], the

Co Geodesic algorithm [1] and the Grab Cut algorithm [16]. Note that the Grab Cut algorithm is an advanced version of the Graph Cut algorithm. In particular, Fig. oo 1(a) shows an input image with a first set of user input strokes. Two of the input © 10 strokes 102 (the light, thick lines) label a first set of pixels as foreground pixels.

The other two of the input strokes 104 (the dark, thick lines) label a second set of pixels as background pixels. Fig. 2(a) shows the same input image with a second set of user input strokes 202, 204 labeling the foreground and background pixels respectively. Figs. 1(b) and 2(b) respectively show segmented images obtained by applying the Random Walks algorithm on the . input image with the first and second sets of user input strokes 102, 104, 202, 204, Figs. 1(c) and 2(c) respectively show segmented images obtained by applying the Geodesic algorithm on the input image with the first and second sets of user input strokes 102, 104, 202, 204, and Figs. 1(d) and 2(d) respectively show segmented images obtained by applying the Grab Cut algorithm on the input image with the first and second sets of user input strokes 102, 104, 202, 204. As shown in Figs. 1(a) — (d) and 2(a) — (d), the segmentation results for all three region-based algorithms i.e. the Random

Walks, Geodesic and Grab Cut algorithms with the first set of user input strokes 102, 104 differ from those with the second set of user input strokes 202, 204.

This indicates that the algorithms are sensitive to the user inputs.

Another problem of existing region-based algorithms is that the cutting contours generated by the region-based approaches, especially by the Random Walks } 30 and Geodesic algorithms, are usually jaggy and do not snap to geometry features. This can be seen from Figs. 1(a) — (d) and 2(a) — (d).

Thus, additional refinement steps are often needed to improve the segmentation performance of the existing region-based methods. In fact, most of the state-of-the-art interactive image segmentation methods [16, 13, 1, 20] rely on additional user inputs to either globally or locally refine the cutting contour(s). However, when dealing with images comprising complex shapes, : the user is often required to provide a lot of additional strokes or boundary points, and is thus often burdened with laborious refinement or editing. It has been proposed to use the active contours model or Snakes model [14] to refine

A initial boundary contour(s) produced by a region-based segmentation approach

[6] without incorporating any additional user input. However, the refinement based on the Snakes model is only able to change the contour(s) locally for

So smoothness but is incapable of evolving the entire contour(s) to snap to geometry features or edges. It is also incapable of handling topology changes of the evolving contour(s).

Summary of the invention

The present invention aims to provide a new and useful method and system for refining one or more initial contours in an input image.

In general terms, the present invention proposes using an active contour model for interactive segmentation of an input image, using user input which specifies that at least one background seed of the image is part of the background, and : at least one foreground seed is part of the foreground. Active contour models are known from techniques for automatic segmentation, but have not been successfully applied in interactive segmentation. ~The active contour model is solved by iteratively refining a candidate function u to find a stationary point (maximum or minimum) of the active contour model, ~~ 30 and the final iteration of the refinement of the candidate function is used to : provide the segmentation result. The initial value of the candidate function is preferably chosen to be consistent with one or more initial estimates of the correct segmentation outline, referred to here as “initial contours” in the input image. While it is true that active contour models have been proposed for : refining initial contours in input images, it has not been previously known how to incorporate data from user inputs into the active contour models. This is a non- 5 trivial problem. For example, directly incorporating user inputs as hard . constraints into the active contour model does not lead to better performance. ) The present invention proposes several different ways for doing this. Most of oo ~~ these use one or more probabilistic functions derived using the user input, and indicative of the likelihood of given pixels being part of the foreground or background. One or more of the probabilistic functions are incorporated into the : definition of a regional term of the active contour model, and preferably also a boundary term of the active contour model.

One possible probabilistic function is a function P(x) consistent with initial contours derived from another segmentation algorithm. It is consistent because itis derived from the initial contours (which were obtained using user inputs), or because it was from the function that the initial contours were derived.

Other possible probabilistic functions may be derived from the colors of the input image, compared to the colors of the seeds specified by the user.

Additionally, the regional function may be chosen such that the resultant

Co segmentation will agree with the regions specified by the user.

The invention may be expressed as a method, or as a computer system for ~ performing such a method. This computer system may be integrated with a device for capturing images. The invention may also be expressed as a computer program product, such as one recorded on a tangible computer medium, containing program instructions operable by a computer system to : perform the steps of the method. :

Brief Description of the Figures

Embodiments of the invention will now be illustrated for the sake of example only with reference to the following drawings, in which: | :

Fig. 1(a) shows an input image with a first set of user input strokes and "Figs. 1(b) — (d) show segmentation results obtained by applying different existing interactive image segmentation methods on the input image of Fig. 1(a) . | with the first set of user input strokes; oo Fig. 2(a) shows the input image of Fig. 1(a) with a second set of user input strokes and Figs. 2(b) — (d) show segmentation results obtained by applying different existing interactive image segmentation methods on the input + image of Fig. 2(a) with the second set of user input strokes;

Fig. 3 shows a flow diagram of a method for segmenting an input image according to an embodiment of the present invention;

Fig. 4 shows a flow diagram of a step of the method of Fig. 3 for refining one or more initial contours in the input image;

Fig. 5(a) shows an input image and Figs. 5(b) — (g) show results obtained when the method of Fig. 3 is applied on the input image of Fig. 5(a) with and without incorporating edge detection of a GMM probability map of the input image of Fig. 5(a);

Figs. 6(a) — (b), (e) show an input image, a trimap associated with the input image and the input image with user input strokes, and Figs. 6(c) — (d), (f) — (g) show results obtained by applying the Random Walks with AT method and the method of Fig. 3 on the input image using the provided trimap and the user input strokes;

Fig. 7(a) shows an input image and Figs. 7(b) — (d) show results obtained ‘by applying the method of Fig. 3 on the input image of Fig. 7(a) with initial contours in the input image of Fig. 7(a) obtained by different existing interactive image segmentation methods; :

Fig. 8(a) shows an input image and Figs. 8(b) — (d) show results obtained by applying the method of Fig. 3 on the input image of Fig. 8(a) with initial contours in the input image of Fig. 8(a) obtained by different existing interactive ” image segmentation methods;

Figs. 9(a) and (b) show segmentation results obtained by applying the method of Fig. 3 on the input image of Fig. 1(a) with the first and second sets of = user input strokes respectively;

Fig. 10(a) shows an input image with a first set of user input strokes and - a second set of user input strokes, and Figs. 10(b) — (e) show segmentation results obtained by applying different existing interactive segmentation methods and the method of Fig. 3 on the input image of Fig. 10(a). with the first and second sets of user input strokes; | :

Figs. 11(a) — (h) show an input image with different sets of user input strokes, and segmentation results from applying the Grab Cut method and the . method of Fig. 3 on the input image with the different sets of user input strokes;

Fig. 12(a) shows four input images and Figs. 12(b) — (e) show segmentation results of different existing interactive segmentation methods and . the method of Fig. 3 on the four input images of Fig. 12(a); : Fig. 13(a) shows an input image and Figs. 13(b) — (g) show segmentation results obtained by applying different existing interactive segmentation methods and the method of Fig. 3 on the input image of Fig. 13(a);

Fig. 14(a) shows an input image and Figs. 14(b) — (e) show segmentation results obtained by applying different existing interactive segmentation methods and the method of Fig. 3 on the input image of Fig. 14(a); : Fig. 15(a) shows an input image comprising an object of interest and the segmentation of the object of interest by the method of Fig. 3, and Figs. 15(b) — (d) each shows an image comprising the same object of interest but captured at a different view from that in the input image and the segmentation of the object of interest; :

Figs. 16(a) — (d) show two input images which the method of Fig. 3 is unable to handle well and segmentation results of applying the method of Fig. 3 on these two input images; and , Figs. 17(a) — (c) show segmentation results obtained by applying the method of Fig. 3 on three medical images.

Detailed Description of the Embodiments

Referring to Fig. 3, the steps are illustrated of a method 300 which is an embodiment of the present invention, and which segments an input image.

Method 300 may be referred to as an interactive image segmentation method.

The input image to method 300 may be a two-dimensional (2D) image

Co comprising a plurality of pixels with respective intensity values. Preferably, there may be color data describing the image, such as a plurality of color intensity values for each pixel, corresponding to the intensities of respective colors. If so, ~~ the intensity value for each pixel may be a combination of the color intensity values.

The plurality of pixels in the input image consists of foreground pixels forming a . foreground region (comprising an object of interest to be segmented) and background pixels forming a background region.

The user inputs data which specifies one or more foreground seeds, and one or more background seeds. These seeds are pixels which the user labeled as foreground or background pixels by for example, drawing foreground strokes in i. the foreground region (with the foreground seeds comprised in the foreground ~ strokes) and background strokes in the background region (with the background seeds comprised in the background strokes) of the input image. In other words, these foreground and background seeds respectively indicate a subset of the foreground pixels and a subset of the background pixels of the input image. The : subset of the foreground pixels forms at least one foreground seed region whereas the subset of the background pixels forms at least one background seed region.

Method 300 comprises two steps: step 302 and step 304. In step 302, one or " more initial contours in the input image are obtained. Then, in step 304, the one or more initial contours are refined.

These steps will now be described in more detail. . oo Step 302: Obtain one or more initial contours in the input image oo In step 302, one or more initial contours in the input image are obtained. These one or more initial contours represent an initial segmentation of the plurality of pixels in the input image into an initial estimated set of foreground pixels and an initial estimated set of background pixels. : : A

An image segmentation algorithm is performed to obtain the one or more initial contours. In the embodiment, an interactive image segmentation algorithm is used to obtain the initial contour(s) based on the seeds. However, in principle, oo the initial contour could be obtained in another way. Both region-based interactive image segmentation algorithms and boundary-based interactive a image segmentation algorithms may be used, although it is preferable to use the region-based interactive image segmentation algorithms. Automatic image segmentation algorithms may also be used to obtain the initial contour(s) in step 302. Alternatively, the initial contour(s) may simply be contour(s) input by the user.

In one example, the Geodesic algorithm which is a region-based interactive image segmentation algorithm is used to obtain the initial contour(s) in step 302. ~ Using the Geodesic algorithm to obtain the initial contour(s) is advantageous due to its high processing speed and its ability to avoid the “small-cut” problem.

Other region-based interactive image segmentation algorithms such as the

Random Walks algorithm [13] may also be used for obtaining the initial contour(s).

As part of the Geodesic algorithm or Random walks algorithm, a probabilistic function in the form of a probability map P(x) is obtained. It is from this that the one or ‘more initial contours ‘may be derived. The probability map P(x) comprises a plurality of probability points corresponding to respective pixels xin the input image. Each probability point in the probability map P(x) takes a value within the range of 0 and 1 i.e. [0,1], with this value indicating a probability

Co that its corresponding pixel x belongs to the foreground region in the input . image. The probability map P(x) may also be referred to as an alpha map. 10° In one example, the Geodesic algorithm was used to obtain the initial contours. fn this example, for each pixel x in the input image, a first geodesic distance : © between the pixel x and the closest foreground seed region, and a second geodesic distance between the pixel x and the closest background seed region are computed. The value of the probability point corresponding to the pixel x is : 15 then calculated based on these geodesic distances. For example, the value of . the probability point may be calculated according to Equation (1). In Equation (1), P(x) represents the value of the probability point corresponding to the pixel x, Dg(x) represents the first geodesic distance of the pixel x to the closest foreground seed region and Dg (x) represents the second geodesic distance of the pixel x to the closest background seed region.

P= (1)

Dr (x) + Dg(x)

Once the probability map is available, we initialize the candidate function u(x) in the algorithm described below by assigning it to be equal to P(x).

In some other segmentation algorithms such as the Grab Cut algorithm, only a hard segmentation map is produced. That is, no probability map is obtained as : : part of the process to obtain the initial contours. The hard segmentation map also comprises a plurality of points corresponding to respective pixels in the input image. However, each point in the hard segmentation map merely v indicates whether the corresponding pixel belongs to the initial estimated set of foreground pixels or the initial estimated set of background pixels. Generally, a point -in the hard segmentation map with a value of 0 indicates that the Co corresponding pixel belongs to the initial estimated set of background pixels whereas a point in the hard segmentation map with a value of 1 indicates that the corresponding pixel belongs to the initial estimated set of foreground pixels. oo If such segmentation algorithms were used to obtain the initial contour(s), the oo probability map P(x) may be calculated using the hard segmentation map. This is further elaborated as follows. : ~~ Boundary pixels comprising pixels forming the one or more initial contours are ) first identified using the hard segmentation map and for each identified boundary pixel, the value of its corresponding probability point is set to a predetermined probability value for example, 0.5. Next, near-border pixels comprising pixels within a predetermined Euclidean distance D from their respective nearest boundary pixels (i.e. within a band of width D around the boundary pixels) are located. D may range from 8 to 50 pixels. For each identified near-border pixel, the value of the corresponding probability point is then calculated based on a Euclidean distance between the near-border pixel : and its nearest boundary pixel. For example, for each near-border pixel which belongs to the initial estimated set of foreground pixels (as indicated by the hard segmentation map), the value of the corresponding probability point is set as a value between 0.5 to 1 (not inclusive of 0.5 but inclusive of 1), whereby the larger the Euclidean distance between the near-border pixel and its nearest boundary pixel, the larger the probability value assigned to the probability point. ~ On the other hand, for each near-border pixel which belongs to the initial estimated set of background pixels (as indicated by the hard segmentation map), the value of the corresponding probability point is set as a value between ~~ 30 0 to 0.5 (inclusive of 0 but not inclusive of 0.5), whereby the larger the

Euclidean distance between the near-border pixel and its nearest boundary

: pixel, the smaller the probability value assigned to the probability point.

Furthermore, it is assumed that the initial segmentation of the pixels far away from the boundary pixels is likely to be correct. Therefore, for each of the : remaining pixels which are not part of the boundary pixels or the near-border pixels, the value of the corresponding probability point is set to 1 if the pixel belongs to the initial estimated set of foreground pixels and is set to 0 if the pixel oo belongs to the initial estimated set of background pixels.

In particular, denoting the hard segmentation map as H(x), whereby a point H(x) with a value of 1 indicates that the corresponding pixel x belongs to the initial estimated set of foreground pixels and a point H(x) with a value of zero indicates that the corresponding pixel x belongs to the initial estimated set of - background pixels, the probability map P(x) may be derived based on the ~ following equation in which d(x) denotes the Euclidean distance between the pixel x and its nearest boundary pixel, and D is the threshold value on d(x). As mentioned above, D ranges from 8 to 50 pixels. 1 if dx)=2D and H(x)=1

P(x) fa i d(x)<D and H(x)=1 ~ [0.5-0.5(d(x)/D)* if d(x)<D and H(x)=0 0 if dx)2D and H(x)=0

In some examples, a further step for checking the hard segmentation map against color distributions of the foreground and background regions in the input image for consistency may be implemented. Implementing this further step is preferable. In other words, the probability map P(x) may be calculated using both the hard segmentation map, and foreground and background color distributions of the input image. These color distributions may be estimated from the user inputs. Co in one example, the color distributions of the foreground and background regions are represented by foreground and background Gaussian Mixture

Models (GMMs) which are respectively estimated from foreground and > background seeds input by the user. From the foreground and background

GMMs, a foreground probability that a pixel x belongs to the foreground region i.e. P(x) is calculated. This foreground probability’ may be a normalized log . likelihood. expressed in the form P(x)=——09FxIF ‘where : —-logP.(x| F)—logP.(x| B) oo

Co Pr(x| F)and Pr(x | B) respectively denote the probabilities that the pixel x fits the foreground and background GMMs. The foreground probabilities are then compared against predetermined thresholds. In particular, for each pixel x originally classified as a foreground pixel (i.e. belonging to the initial estimated set of foreground pixels), if the pixel xhas a foreground probability less than a first threshold (for example, 0.4), it is reclassified as an ambiguous pixel. RE

Similarly, for each pixel x originally classified as a background pixel (i.e. . belonging to the initial estimated set of background pixels), if the pixel x has a foreground probability greater than a second threshold (for example, 0.6), it is re-classified as an ambiguous pixel. The values of the probability points corresponding to these ambiguous pixels are then set to the predetermined probability value used for the probability points corresponding to the boundary pixels (for example, 0.5).

Note that if the initial contours are simply contours input by the user, a hard : segmentation map may be formed from the initial contours and in this case, the above steps of obtaining the probability map P(x) using the hard segmentation map or both the hard segmentation map and the color distributions may be used.

Step 304: Refine the one or more initial contours :

In step 304, the one or more initial contours obtained in step 302 are refined. oo Step 304 employs an active contour model which is a non-obvious improvement from the existing convex active contour model. This active contour model employed in step 304 retains the property of being convexand may be referred to as a constrained active contour model.

Different types of active contour models are currently available. The classical active contour model [14] is mostly used for local adjustment of initial contour(s) in an input image to enhance smoothness of the contour(s). The Geodesic ~~. active contours model proposed in [5] is capable of evolving entire initial oo contour(s) (or boundary contour(s)) to snap to geometry edges, but it (especially . its edge detection function) heavily depends on boundary information. The active contour without edges model [8] adds regional information and removes the dependency on boundary information, but is often trapped in a local ~ minimum due to its non-convex modeling. The convex active contour model [4] EE has mainly been used for automatic image segmentation and is able to find a global minimum solution for the segmentation problem. The convex active contour model [4] is discussed in greater detail below.

Convex Active Contour Model

The convex active contour model makes use of both boundary and regional information in an input image to find a global “optimal” solution for segmenting the input image. The convex active contour model [4] can generally be expressed as oo min([ g, Vea + A h,uax) | (2) | | . with the following symbol definitions: ) e u is a candidate function on the image domain Q of the input image.

Throughout the image domain Q, the function u takes on a value between 0 and 1 at each pixel x. The input image is segmented into the foreground and background regions by thresholding the function vu.

e Function g, is a boundary function and may be an edge detection function such as | : oo ae | ®

Co 1+|Vi(x)| : where I(x) is the intensity value of the pixel x. : . Function h, is a regional function that gives a measure of the intensity values : of the pixels x belonging to the foreground and background regions of the input image. Particularly, h, may be formulated according to Equation (4). h, = hr — he @ h” and h* are the foreground and background regional functions : : respectively and may be defined as oo [ (ax

Con X) = tt HA t= Be . Cin - oo [Max

IP (Cours X) = (Mowe = HX)? Hoy = = (5) fax

Cou where Hand pu, are the mean intensity values of the pixels belonging to the foreground and background regions respectively.

As shown in Equation (2), the convex active contour model further comprises a tradeoff factor 4 which balances the boundary term [ g,[Vulax comprising the oo boundary function g, and the regional term [ hr uax comprising the regional function h, . The boundary term [ go|vujax favors segmentation of the input image along curves that the boundary function g, (which may be an edge : : detection function) reaches a minimum and along areas where the gradient is small. It also favors segmentation of the input image at locations with smooth boundary curves. The regional term [ hudx ensures that the segmentation of the input image complies with some regional coherence criteria defined in the regional function h,. The tradeoff factor 4 allows the user to control the scale oC | and the smoothness of the segmentation. A smaller 4 tends to result in a ~~ smoother contour.

After solving the optimization problem of Equation (2) to obtain function u, the segmentation of the input image into foreground and background regions may : be derived by thresholding the function uv according to Equation (6) as shown oo below where T is the cutoff parameter, C,, is the segmented foreground region . and Co. is the segmented background region. The segmentation is usually : 15 quite robust to different values of T because the converged u values are : typically close to either 0 or 1. In one example, T is set as 0.5.

Co ={xIux)>T} C, =Q\C, (6) 20 . oo oo

The segmentation problem based on the convex active contour model of ~ Equation (2) is usually solved by a three-step alternate iterative approach using - steps 1 — 3 as follows.

Step 1: Fix the segmentation, i.e. C,;and C,,, and update h, .

Step 2: Fix h, to find the function u from Equation (2). ~ Step 3: Update C,; and C,, according to Equation (6).

The above steps 1 — 3 are repeated until the functionu converges (i.e. when .

C, and C,, no longer change by much). It can be seen that the computation bottleneck of this three-step alternate iterative approach lies in step 2, i.e. solving the optimization problem of Equation (2). oo

Split Bregman Solver ’ . Several methods have been proposed to solve Equation (2) for a given h,. For » oo example, Chan et al. [7] proposed to either enforce the inequality constraint of Co (0<u<1) using an exact penalty function, which is non-differentiable, or regularize the penalty function, which does not exactly enforce the inequality ~~ © constraint. Bresson et al. [4] proposed using a splitting/regularization approach to minimize Equation (2). However, the method in Bresson et al. [4] “smears” the values of u near the boundaries of the object of interest, and makes the segmentation results more dependent on the cutoff parameter T. This may : result in the elimination of object details.

Recently, Goldstein et al. [12] proposed using the Split Bregman method to : solve Equation (2). The Split Bregman method is not only able to solve the convex active contour model but is also a much more efficient solver. The use of this Split Bregman solver as introduced in [12] is described below.

In particular, instead of solving Equation (2) directly, Bregman introduced a vectorial function d into the convex active contour model as min [ gla] + 4h, uax 7) with the constraint of d=Vu. This constraint is enforced using the efficient

Bregman iteration approach defined as

N (ut, d") =arg min [ gyld] + ah, + Slo ~ Vu ox (8)

-. b*! — b* + yuk dr Co (9) where k= 0 is the iteration index. The computations of Equation (8) and (9) are repeated until u converges. : oo

Since Equation (8) is differentiable, it can be solved using a simple alternating : method. Specifically, Equation (8) is first differentiated with respect to u using the Euler-Lagrange Differential Equation, which results in the following optimality condition for uv: : pAU = Ah, + pdiv(d = b*), ue [0] (10)

In Equation (10), Au is the Laplacian of u and div(d* — b*)is the divergence of (d* ~ bY). Based on Equation (10), u**'can be approximated by a Gauss-Seidel iterative method [12] and Equation (8) may then be solved with respect to d. It ~ has been shown in [18] that the minimizing solution d**' may be given by soft- : thresholding as shown in Equation (11).

Vuk + br . ad" = —————max(vVu*™ + b*|- u7'g,,0) 11 vu +b" | HG ) ( ) | oo

Sub-steps of step 304

As mentioned above, step 304 refines the one or more initial contours from step 302 by employing an active contour model which is a non-obvious improvement from the known convex active contour model. This improved active contour : model may be referred to as a constrained active contour model and can also be expressed in the form of Equation (2). in other words, like the known active . contour model, the constrained active contour model has. a boundary term :

[ 9:|vujox which comprises a boundary function g,. and a regional term : { huax which comprises a regional function h, .

Referring to Fig. 4, sub-steps of step 304 are shown. In particular, step 304 comprises sub-steps 304a, 304b and 304c. In sub-step 304a, the regional ~~ function h, of the constrained active contour model is calculated and in sub- . oo step 304b, the boundary function g, of the constrained active contour model is oo ~ calculated. Next, in sub-step 304c, the constrained active contour model with : the calculated regional and boundary functions is solved. | | N ~ Sub-step 304a: Calculate the regional function of the constrained active contour | : model © In sub-step 304a, the regional function h, of the constrained active contour model! is calculated. Similar to the regional function h, of the convex active contour model, the regional function h, of the constrained active contour model comprises information relating to the foreground and background regions of the input image. However, the regional function h, of the constrained active contour model is calculated based on the user inputs as follows.

First, color distributions of the foreground and background regions in the input

BE : image are estimated from the user inputs and the color data describing the input image if they are not already available from for example, the calculation of the probability map P(x) using the hard segmentation map as described above.

Foreground and background seeds input by the user usually give an excellent description about the color distributions of the foreground and background :

Lo regions. Thus, in one example, in sub-step 304a, foreground and background : ~ GMMs [19] representing the color distributions of the foreground and background regions are first respectively estimated from the foreground and background seeds. ©

Next, for each pixel x of the input image, a foreground probability that the pixel xbelongs to the foreground region and a background probability that the pixel : x belongs to the background region are calculated using the estimated color ~~ distributions of the foreground and background regions. oo In one example, the foreground and background probabilities are calculated © 10 using the foreground and background GMMs estimated from the foreground and background seeds as follows. Co

Let Pr(x| F)and Pr(x| B) respectively denote the probabilities that pixel x in the input image fits the foreground and background GMMs. The foreground and background probabilities of the pixel x are then respectively calculated as oo probabilistic functions in the form of normalized log likelihoods P(x) and P(x) using Equations (12) and (13). Alternatively, the foreground and background probabilities of the pixel x may also be calculated as normalized likelihoods

P-(x) and Pg(x) using equations similar to Equations (12) and (13) but without the log functions. _logP,(x| F

P(x) ——00F(XIF) (12) . —logP.(x|F)-logP.(x |B) oo —-logP.(x|B | :

Pon) = ——— 0918 (19) —logP,.(x| F)=logP.(x| B) : oo

Next, the regional function h, at each pixel x i.e. h (x) is calculated based on the foreground and background probabilities of the pixel x. -

In one example, the regional function h, at each pixel x is calculated by calculating a difference between the foreground and background probabilities of the pixel x. As shown in Equation (14), this may be performed using the : normalized log likelihoods P:(x) and Ps(x) from Equations (12) and (13) ie. using regional information derived from the foreground and background seeds ~ input by the user. In this case, the regional function h, is defined using both the oo candidate function u, and the probabilistic functions P-(x) and P(x). oo AX)=RX)-Px) | oo 0 ommenEm e

Calculating the regional function h, based on the foreground and background probabilities ensures that the one or more initial contours evolve towards one or ) more contours complying with the known color distributions or GMM models. - For instance, if the regional function h, is calculated using Equation (14), for a pixel x, if P(x) > P:(x), Pg(x)— P(x) is positive and u(x) tends to decrease during the contour evolution to. minimize Equation (2). Therefore, in this case, u(x) is more likely going to be less than or equal to the cutoff parameter T in

Equation (6) i.e. u(x) < T and the pixel x is more likely going to be classified as ) a background pixel. on the other hand, if P(x) < P(x), P(x) = Pe (x) is ~~ negative and u(x) tends to increase during the contour evolution to minimize :

Equation (2). Therefore, in this case, u(x) is more likely going to be greater than the cutoff parameter T i.e. u(x) > T and pixel x is more likely going to be ~ classified as a foreground pixel. : . 25 ~~ Calculating the regional function A, in the above-described manner may fail in cases whereby the foreground and background color distributions are not well ~~ separated. Therefore, to avoid this problem and also to make use of the initial segmentation obtained in step 302, the regional function h; may be calculated further based on the one or more initial contours. The regional function h, may. also be calculated by weighting its calculation based on the user inputs and its | : calculation based on the initial contours with a tradeoff factor. | oo

In one example, h, is calculated according to Equation (15).

Co oo h, (x) = a Py (x) — Pe(x)) + (1-@)(1- 2P()) (15)

As shown in Equation (15), the regional function h, in this example comprises three components, | | Co :

The first component of h, i.e. (Ps(x)~ P-(x)) is calculated based on the user inputs. In particular, it reflects the foreground and background color distributions . 15 of the input image estimated from the foreground and background seeds, and co modeled by the foreground and background GMMs. This component corresponds to the calculation of h, using Equation (14). In particular, in

Equation (15), the foreground and background probabilities of each pixel x are obtained and the normalized difference of the two probabilities is calculated to give the value of this first component of h, at pixel x.

The second component of h.i.e. (1-2P(x))is calculated based on the one or more initial contours i.e. it incorporates the initial segmentation results. In particular, the probability map P(x) representing the one or more initial contours obtained from step 302 is included in this second component and the regional function h, at each pixel x of the input image is calculated based on the value of the corresponding probability point on the probability map P(x).

The second component of h, in Equation (15) prevents the refined contour(s) Co from drifting too far away from the initial contour(s) (i.e. initial segmentation).

Specifically, when P(x)>0.5, (1-2P(x)) is negative and u(x) tends to increase to minimize Equation (2). This tendency of u(x) to be larger favors classifying the pixel x as a foreground pixel. The reverse is true when

P(x) < 0.5. More specifically, when P(x) < 0.5, (1-2P(x))is positive and u(x) : 5 tends to decrease to minimize Equation (2) which favors classifying the pixel | : x as a background pixel. aN oo . The third component of h, in Equation (15) is a, < [0,1] which is the tradeoff factor as mentioned above. More specifically, the regional function h, in this example is calculated by weighting its calculation based on the user inputs and its calculation based on the one or more initial contours with the tradeoff factor oo a . Therefore, h, in Equation (15) may be described as a weighted average of the first and second components.

When the foreground and background colors in the input image are well separable, it is preferable that the first term. (P;(x)— P-(x)) in Equation (15) dominates. On the other hand, when the foreground and background colors in the input image are not well separable, the second term in Equation (15) preferably dominates. Therefore, in one example, the tradeoff factor a is set to be a distance between the estimated color distributions of the foreground and background regions i.e. if Equation (15) is used, a distance between the foreground and background GMMs estimated from the foreground and oo background seeds [20]. The distance between the foreground and the - background GMMs is measured according to Equation (16) where n represents ‘the total number of pixels in the input image. | . «Loan | F)—logP,.(x; | J (16) - niF|logP.(x; | F)+logP.(x; | B)

It can be observed that minimizing Equation (2) with the regional function h(x) tending to positive infinity (i.e. h(x) >) forces u(x) to 0 whereas minimizing Equation (2) with the regional function h(x) tending to negative infinity (i.e. h(x) > —o) forces u(x) to 1. Therefore, it may be possible - to enforce some hard constraints in the contour evolution process. ~

In one example, pixels with little or no ambiguity in how they should be oo classified are used as the hard constraints. In this example, at each pixel x with oo little or no ambiguity that it is a foreground pixel, the regional function h, is set to a. predetermined negative value with an extremely large magnitude. This extremely large magnitude may be greater than 1000. Such pixels x may | - include one or more of: the foreground seeds input by the user and the pixels with corresponding probability points having very large values above a first cutoff value (for example, P(x) > 0.9). On the other hand, at each pixel x with - 15 little or no ambiguity that it is a background pixel, the regional function A, is set toa predetermined positive value with an extremely large magnitude. Similarly, this extremely large magnitude may be greater than 1000. Such pixels x may include one or more of: the background seeds input by the user and the pixels with corresponding probability points having very small values below a second cutoff value (for example, P(x) <0.1) By using the hard constraints, the - refined contour(s) can more closely comply with the user inputs such as the : foreground and background seeds. In addition, more information from the initial segmentation in step 302 (for example, in the form of the probability map P(x) ) can be exploited.

Sub-step 304b: Calculate the boundary function of the constrained active : contour model

In sub-step 304b, the boundary function g, of the constrained active contour © 30 model is calculated. Similar to the boundary function g, of the convex active contour model, the boundary function g, of the constrained active contour "model comprises information relating to edges of the foreground and background regions in the input image. ~The boundary term [ g,|vulax in Equation (2) is essentially a weighted total = variation of the function u. The boundary function g, calculated using Equation . ~ (3) encourages the segmentation of the input image along curves where the boundary function g, (which may be an edge detection function) is minimal. oo oo Although it is possible to use Equation (3) for the calculation of g,in sub-step 304b, at locations in the input image with weak edges, the initial contours (or boundaries) are likely to be smoothed out using this calculation of g, and object = : details may be lost. oo :

Therefore, in one example of sub-step 304b, the boundary function g, is : calculated based on the user inputs. Such a calculation may comprise

B 15 “incorporating a probabilistic function in the form of a color probability map reflecting the user inputs (for example, a GMM probability map P(x) as shown in Equation (12) estimated from foreground and background seeds) into the calculation of g,. This can enhance the edge detection in the input image.

In a more specific example, a color probability map is first formed from the foreground probabilities calculated for the pixels of the input image. This color probability map comprises information from the user inputs and information relating to variations in intensity values in the input image. Edges in the color : probability map and in the input image are then detected and the boundary function g, is subsequently calculated based on the detected edges in the color probability map and in the input image. Thus, the boundary function g,in this case is defined using the candidate function u(x) and information relating to variations in intensity values in the input image. So

In particular, the boundary function g, may be calculated as 9,=89.+00-6)9. | (17) where g, represents the results of (in other words, the detected edges from) applying edge detection to a color probability map in the form of a GMM oo probability map P-(x), g,represents the results of (in other words, the detected oo edges from) applying edge detection to the input image and pS, Be [0] is a tradeoff factor. As shown in Equation (17), the boundary function g, is calculated by weighting the detected edges in the color probability map and the ~~ detected edges in the input image with the tradeoff factor Ji In other words, the edges detected by this step of the constrained active contour model may be viewed as a weighted average of the results of applying edge detection to the color probability map (GMM probability map F:(x)) and the results of applying edge detection to the input image. ~ f may be computed in a similar manner as « given in Equation (16). More specifically, when the foreground and background colors are well separable, it is preferable that g. dominates. On the other hand, when the foreground and background colors are not well separable, g, should preferably dominate.

Therefore, similar to the tradeoff factor « , the tradeoff factor § may be set to a distance between the estimated color distributions of the foreground and background regions (for example, a distance between the foreground and background GMMs respectively estimated from the foreground and background seeds). The distance between the foreground and background GMMs is measured according to a Note that the boundary function g, which is also an edge detection function returns values : between 0 to 1 and a small value of g, corresponds to a likely edge.

Figs. 5(a) — (g) show segmentation results when method 300 is applied on an : input image with and without incorporating edge detection of the GMM probability map of the input image in sub-step 304b. In particular, Fig. 5(a) oo 5 shows the input image with the foreground and background strokes input by the user and Fig. 5(b) shows the GMM probability map P-(x) of the input image. ~~ Figs. 5(c) and (d) respectively show the segmentation results obtained with the : = boundary function g, derived using Equation (3) and the boundary function g, derived using Equation (17). Figs. 5(e) and (f) respectively show the results of applying edge detection to the input image and to the GMM probability map

P.(x) i.e. g, and g, whereas Fig. 5(g) illustrates the boundary function ag, l : | derived using Equation (17). From Fig. 5(c), it can be seen that some boundary problems 502 arise when g, is calculated from Equation (3). Furthermore, from

Figs. 5(c) and (d), it can be seen that neorporaing g. which comprises . 15 information from the user inputs enhances the conventional edge detection . - result g,, especially at the weak edges, leading to a more accurate boundary contour.

Sub-step 304c: Solve the constrained active contour model with the calculated regional and boundary functions to obtain one or more refined contours :

In sub-step 304c, the constrained active contour model with the regional and boundary functions calculated from sub-steps 304a and 304b is solved. This obtains one or more refined contours representing a refined segmentation of the plurality of pixels in the input image into a refined estimated set of foreground ~ pixels and a refined estimated set of background pixels. In particular, stationary ‘point values of the candidate function u(x) corresponding to a stationary point : of the constrained active contour model are obtained and are then used to obtain the one or more refined contours.

To initialize the solving of the constrained active contour’ model for contour refinement (or evolution), the function u(x) in Equation (2) is initially set to be the probability map P(x) from step 302. oo

E 5 Note that unlike the regional function h, calculated using Equation (4) in the convex active contour model, the regional function h, calculated in the

Co | constrained active contour model using for example Equation (14) or (15) is oo fixed given the user inputs and the initial segmentation results from step 302.

Therefore, there is no need to use all three steps in the alternate iterative approach described above to solve the constrained active contour model.

Instead, only step 2 of this three-step alternate iterative approach is required i.e. it is only required to find the function u(x) for Equation (2) based on the fixed h,. This allows the constrained active contour model to be solved more quickly than the convex active contour model. 1 : In one -example, the constrained active contour model is solved (i.e. the stationary point values of the candidate function u(x) are obtained) using the

Split Bregman method. As discussed earlier, using the Split Bregman method is ‘advantageous as it can solve the constrained active contour model in a fast manner [12]. Co

Note that although step 304 is described as part of an interactive segmentation

Co algorithm (method 300) above, it may be implemented as a stand-alone method which takes in an input image with one or more initial contours and may be used for refining the one or more initial contours in the input image.

Furthermore, step 304 employing the constrained active contour model is able to refine the initial contour(s) without additional user inputs (unless no user inputs were obtained for step 302, which in this case, the additional user inputs ~~ may simply comprise user inputs required for obtaining initial contour(s) with a region-based interactive image segmentation algorithm). However, note that step 304 is also capable of incorporating additional user inputs beyond those : required for obtaining the initial contour(s) for further user guided contour evolving. This may be useful when the one or more initial contours segment the input image in a very poor manner or when a highly accurate segmentation of the input image is required. oo Experimental Results

Co oo Experiments were conducted to evaluate the performance of method 300. In © 10 these | experiments, the tradeoff factor A in Equation (2) describing the constrained active contour model was set as 1000 and the Split Bregman

Solver was used to solve the constrained active contour model with yin EE

Equation (8) set as 10000. The parameter Tin Equation (6) was set as 0.5. The other parameters such as a and fwere set according to the description above and all the parameters were set in the same way for all the input images. The experimental results are discussed below.

Images in the commonly used MSRC ground truth data set [16] were used as [input images to method 300 for evaluating the performance of method 300. The

MSRC data set contains 50 images, each of which is associated with a trimap and the segmentation ground truth (i.e. the desired segmentation of the : foreground and background regions in the image). Several versions of method 300 using different algorithms to obtain the one or more initial contours in step : 302 were tested. The different algorithms include the Geodesic algorithm, the

Random Walks algorithm, the Grab Cut algorithm and the Yang's algorithm [20].

Furthermore, the input images were also segmented using the Geodesic algorithm alone so as to compare the performance of method 300 against the : ) performance of the Geodesic algorithm on its own. For a fair comparison, the same trimaps provided by the MSRC data set were used as the user inputs for the Geodesic algorithm and for method 300. BN

Table 1 summarizes error rates (percentage of mislabeled: pixels) achieved by different state-of-the-art interactive segmentation algorithms and by method 300 with the images in the MRSC ground truth data set used as the input images.

The error rates for the different state-of-the-art segmentation algorithms other = than the Geodesic algorithm are quoted from the best results reported in literature. Note that these error rates were also achieved using the same ~ trimaps provided by the MSRC data set as mentioned above.

Geodesic [1] . 5.21 (from the implementation of

Lo | oo the Geodesic method on the B mm

Method 300 (using the Geodesic method | 3.768 fr UE ‘Method 300 (using the Random walks | 3.77

Method 300 (using the Grab Cut method in | 4.907 zg * [ Method 300 (using the Yang et. al. method | 3.765 oo

Table 1 :

SE From Table 1, it can be seen that method 300 is capable of achieving very low error rates, significantly outperforming state-of-the-art interactive image segmentation algorithms such as the Gaussian Mixture Markov Random Field - (GMMRF) method [2], the Geodesic method [1], the Random Walks method

[13], the Segmentation by transduction method [10], the Grab Cut method [16] and the Yang's method [20]. In particular, method 300 is able to significantly lower the error rates by refining the initial contours derived from the state-of-the- art interactive image segmentation algorithms.

In addition, it can be seen that the constrained active contour model employed oo in step 304 is insensitive to the initial contours derived in step 302 since initializing the contours using different methods lead to almost the same error rate. Moreover, the speed of convergence of method 300 is very high, taking . less than half a second to optimize the segmentation results of (i.e. refine the initial contours in) images with a resolution of 640 x 480.

Note that the error rate obtained from method 300 using the Grab Cut method in step 302 is not as low as the other error rates associated with method 300. This - is because the probability map used in this version of method 300 is not as accurate. This is in turn because only a hard segmentation map is produced in the Grab Cut method and the probability map has to be calculated from this hard segmentation map.

On the other hand, method 300 works ideally with the Geodesic method in step 302 to create a comprehensive segmentation framework. Among the various versions of method 300, the version using the Geodesic method in step 302 performs the best, achieving not only a very low error rate but also a fast running speed, one that is even faster than the speed of running the Random ~ Walks method alone or the Grab Cut method alone. In particular, method 300 using the Geodesic method for pre-segmentation in step 302 takes less than three seconds in total to segment an image with a resolution of 640 x 480 when implemented in C++ on a PC with Intel 2.67 GHz CPU and a 2 GB RAM. The Co speed of convergence of the constrained active contour optimization in such an implementation takes less than half a second. :

As shown in Table 1, applying the segmentation by transduction (SBT) method : ‘reported in [10] on the MSRC data set resulted in a high error rate of 5.4%, same as that obtained when the Random Walks method was applied to the data - 5 set. However, a lower error rate of 3.3% was achieved [10] when the SBT method was applied together with an adaptive threshold (AT) method. In

Co particular, the authors of [10] proposed applying the AT method in a post- processing step for the Random Walks method and the SBT method. This

Co subsequently reduced the error rates of the Random Walks method and the

SBT method from 5.4% t0.3.3%. As explained in [10], this large reduction in the error rates after applying the AT method is due to the particular form of seeds in the MSRC data set. More specifically, as the unlabelled pixels in each image of the MSRC data set only cover a small band along the object boundary in the image, it is not surprising that a naive segmentation approach such as the AT method that tracks the skeleton of the unlabelled pixels can perform well. . However, the AT method is very specific to the MSRC data set or any other - boundary brush tool where the unseeded region only covers a small band along the object boundary. The AT method does not work well for images with large and unregulated unknown regions, as shown in Figs. 6(a) — (9). | oo

In particular, Fig. 6(a) shows an input image and Fig. 6(b) shows a trimap associated with the input image. Figs. 6(c) and (d) respectively show the segmentation results from applying the Random Walks (RW) with AT method and method 300 (with the Geodesic method in step 302) on the input image using the provided trimap. Fig. 6(e) shows the input image with user input : strokes, and Figs. 6(f) and (g) respectively show the segmentation results from applying the RW with AT method and method 300 (with the Geodesic method in step 302) on the input image using the user input strokes. As shown in Figs. 6(c) and (f), the RW with AT method works well using the provided trimap but fails using the sparse user input strokes. In contrast, method 300 is more - general and works well even on images with large unknown regions. This is not only evident from Fig. 6(g) but also from the other figures showing sparsely drawn user input strokes which leave large unknown areas in the input images ; and further showing the good segmentation results achieved by method 300 despite these large unknown regions. The good visual results shown in several of the figures indicate that the performance of method 300 does not rely on . oo 5 carefully designed trimaps, as in those in the MSRC data set. - Figs. 7(a) — (d) shows results obtained using method 300 with the initial

N contours in step 302 derived using the Geodesic method, the Random Walks oo method and the Grab Cut method. In particular, Fig. 7(a) shows the input image with user input foreground stroke 702 and background stroke 704, Fig. 7(b) shows the probability maps P(x) obtained with the Geodesic method used in : step 302 (left), with the Random Walks method used in step 302 (middle) and with the Grab Cut method used in step 302 (right), Fig. 7(c) shows the initial ~ contours derived in step 302 using the Geodesic method (left), the Random

Walks method (middle) and the Grab Cut method (right) and Fig. 7(d) shows the : final segmentation results i.e. the refined contours with the Geodesic method used in step 302 (left), with the Random Walks method used in step 302 (middle) and with the Grab Cut method used in step 302 (right). It can be seen from Figs. 7(a) — (d) that the constrained active contour model employed in step 304 is able to successfully refine initial contours obtained from existing region- based methods. Figs. 7(a) — (d) further show that the constrained active contour model is insensitive to the initial contours derived in step 302 since the final segmentation results as shown in Fig. 7(d) are very similar regardless of the method used in step 302. In other words, method 300 is insensitive to the algorithm chosen to derive the initial contours in step 302.

The same is illustrated in Figs. 8(a) — (d). In particular, Fig. 8(a) shows an input image with user input foreground strokes 802 and background strokes 804.

Figs. 8(b) — (d) show resuits obtained by applying method 300 on the input image with the Geodesic method, the Random Walks method and the Grab Cut method used in step 302 respectively. In particular, the probability maps are shown as the left images of Figs. 8(b) — (d), the initial contours are shown as the middle images of Figs. 8(b) — (d) and the refined contours are shown as the right images of Figs. 8(b) — (d). As shown in Figs. 8(a) — (d), the constrained active contour model is able to successfully refine initial contours obtained from existing region-based methods. Similar to Figs. 7(a) — (d), Figs. 8(a) — (d) also : show that the constrained active contour model is insensitive to the initial ~~ contours derived in step 302 since the refined contours are very similar regardless of the method used in step 302.

Figs. 9(a) — (b) show segmentation results obtained by applying method 300 (using the Geodesic method in step 302) on the input image of Figs. 1(a) and : 2(a).- In particular, Fig. 9(a) shows the results with the first set of user input strokes 102, 104 as shown in Fig. 1(a) whereas Fig. 9(b) shows the results with the second set of user input strokes 202, 204 as shown in Fig. 2(a). In contrast to the existing region-based algorithms (see Figs. 1(b) — (d) and 2(b) — (d)), it . can be seen from the similarity between Figs. 9(a) and 9(b) that method 300 is robust and relatively insensitive to the user inputs. Furthermore, method 300 is able to smooth the initial contour from the Geodesic method and produce a contour which snaps to geometric edges. oo ~The same is illustrated in Figs. 10(a) — (d) which compare the segmentation results obtained with different user inputs. oo

In particular, Fig. 10(a) shows an input image with a first set of user input strokes (left) and a second set of user input strokes (right). Fig. 10(b) shows the segmentation results obtained by applying the Random Walks method on the input image with the first set of user input strokes (left) and the second set of ~ user input strokes (right). Fig. 10(c) shows the segmentation results obtained by applying the Geodesic method on the input image with the first set of user input strokes (left) and the second set of user input strokes (right). Fig. 10(d) shows the segmentation results obtained by applying method 300 (using the Random

Walks method in step 302) on the input image with the first set of user input strokes (left) and the second set. of user input strokes (right). Fig. 10(e) shows ’ the segmentation results obtained by applying method 300 (using the Geodesic ~~ method in step 302) on the input image with the first set of user input strokes (left) and the second set of user input strokes (right). | CL : As aforementioned, the Random Walks and Geodesic methods tend to be sensitive to locations of the seeds input by the user. It can be seen from Figs. 10(b) and (c) that, for the Random Walks and Geodesic methods, different user

Co inputs result in different segmentation results. In contrast, method 300 is able to . generate stable results insensitive to the user inputs, as shown in Figs. 10(d) and (e). oo Lo Co :

Note that although the Grab Cut method is less sensitive to the seed locations as compared to the Random Walks and Geodesic methods, it is sensitive to the number of seeds and has the “small cut” problem as illustrated in Figs. 1(d), © 2(d) and further illustrated in Figs. 11(a) — (h).

In particular, Figs. 11(a), (c), (e), (g) show an input image with different sets of } user input strokes. The object of interest to be segmented in the input image is . the fireman on the left. Figs. 11(b), (d), (f) and (h) respectively show the segmentation results by applying the Grab Cut method on the input image with the user input strokes of Figs. 11(a), (c), (e) and (g). Fig. 11(i) shows the input oo image with two user input strokes and Fig. 11(j) shows the segmentation result from applying method 300 (using the Geodesic method in step 302) on the input image with the two user input strokes in Fig. 11(i).

As shown in Figs. 11(a) — (h), due to the similarity in appearance of the two ~~ firemen, the segmentation results from the Grab Cut method tend to include not oo only the object of interest but also the other fireman (on the right). Even when : ~~ 30 more background strokes are drawn over the other fireman on the right as shown in Fig. 11(c), the Grab Cut method still chooses the minimum cut around the added strokes (see Fig. 11(d)). The Grab Cut method is only able to produce relatively good segmentation results when a large number of user input strokes is available (see Figs. 11(e) — (h)). Moreover, the Grab Cut method relies heavily on the global color model and this often results in disjointed parts or noisy parts in the segmentation results (see Figs. 11(a) — (h), and also Figs. = hE 5 1(d), 2(d) and 12(d) below). This is especially so when the foreground and background regions in the input image have similar colors.

In contrast, method 300 (using the Geodesic method in step 302) is able to produce good segmentation results (see Fig. 11(j)) with only two user input strokes. This is mainly because in this version of method 300, the Geodesic method helps propagate the seeds and introduces spatial or location information into the constrained active contour model. This significantly mitigates the “small cut” problem. The same advantage can be achieved when the Random Walks method is applied in step 302. i} Note that the MSRC data set is the only publicly available data set with trimaps oo provided but is somewhat biased because with the provided trimaps, the input images only contain small unknown regions (in other words, for a majority of the pixels in the input image, it is known whether these pixels belong to the foreground or background regions of the image). In these cases, the Geodesic and Random Walks methods tend to perform relatively well. Thus, method 300 “and the different existing segmentation algorithms were also tested on images : with large unknown regions (in other words, for a majority of the pixels in the input image, it is unknown whether these pixels belong to the foreground or background regions of the image). oo ~ Figs. 12(a) — (e) show segmentation results of different existing segmentation algorithms and method 300 when four different images with large unknown . regions were used as the input images. oo :

In particular, Fig. 12(a) shows the four input images, and Figs. 12(b) — (e) respectively show the segmentation results when the Random Walks method, :

the Geodesic method, the Grab Cut method and method 300 (using the co

Geodesic method in step 302) were performed on the respective images.

It can be seen from Figs. 12(b) — (c) that with input images having large = unknown regions, the Geodesic and Random Walks methods perform poorly, producing inaccurate and jaggy boundary contours. Although the performance of the Grab Cut method (see Fig. 12(d)) is much better as compared to that of the Geodesic and Random Walks methods, its results still contain clearly visible oo artifacts, e.g. around the neck of the man (see leftmost image of Fig. 12(d)), the . right elbow of the boy (see second image from the left of Fig. 12(d)), the horse’s head and leg regions (see third image from the left of Fig. 12(d)) and : the bottom of the boat (see rightmost image of Fig. 12(d)). On the contrary, - method 300 is capable of producing accurate and smooth contours which snap to geometry edges as shown in Fig. 12(e). - oo Figs. 13(a) ~ (q) show segmentation results of different existing interactive image segmentation algorithms and method 300. In particular, Fig. 13(a) shows an input image with user input foreground and background strokes. Figs. 13(b) — (d) respectively show the segmentation results obtained by applying the ‘Geodesic method, the Random Walks method and the Grab Cut method on the input image. Figs. 13(e) — (9) respectively show the segmentation results a. obtained by further applying step 304 of method 300 on the segmentation : results in Figs. 13(b) — (d). As shown in Figs. 13(b) ~ (@), the three state-of-the-

Co art interactive segmentation algorithms, especially the Geodesic method and a 25 the Random Walks method, produce inaccurate and jaggy boundary contours. -

On the other hand, step 304 of method 300 is able to reduce the errors in the ~ boundary contours by smoothing the contours and making them snap to geometry edges.

Furthermore, method 300 displays the ability to handle topology changes of the : boundary contour around the object of interest, which is not achievable by the : | classical Snakes model. Fig. 14(a) shows an input image whereas Figs. 14(b)-

BE (e) respectively show segmentation results obtained by applying the Random

Walks ‘method, method 300 (using the Random Walks method in step 302), the -

Geodesic method and method 300 (using the Geodesic method in step 302) on the input image. =

As shown in Figs. 14(b) — (e), the Random Walks method produced two initial oo boundary contours that separate the object of interest into two halves and the oo Geodesic method produced only one initial closed boundary contour around the a object of interest. In contrast, method 300 is capable of evolving (without oo additional user inputs) these initial boundary contours to one outer refined contour outlining the object of interest and one inner refined contour outlining the gap 1402 between the object's left arm and body. Therefore, it can be seen ~ + that method 300 is capable of handling topology changes of the boundary contour.

A multi-view image cutting tool employing method 300 may be developed for segmenting multi-view images. Multi-view images comprise a plurality of images which in turn. comprise the same object of interest but captured at different views. in other words, each image of the plurality of images comprises the object of interest captured at a view different from the other images. 0

Figs. 15(a) — (d) show an example of applying such a multi-view image cutting tool based on method 300 (using the Geodesic method in step 302). More specifically, Fig. 15(a) shows an input image to method 300 and the © segmentation of the input image by method 300, whereby the input image is one of the multi-view images. Figs. 15(b) — (d) shows the segmentation of other © multiview images. oo in the above example, user input foreground strokes 1502 and background strokes 1504 are first manually input into a subset of the multi-view images : (including the input image shown in Fig. 15(a)) and method 300 as employed in ~~. : the image. cutting tool is run to segment the subset of the mulii-view images.

Then, the multi-view image cutting tool generates the segmentation results for Le the remaining multi-view images ‘based on the segmentation of the subset of y multi-view images. The object contours 1506 are thus obtained. ~The multi-view image cutting tool may employ a 5-step process as follows:

Step 1: Segment a subset l,, of the multi-view images using method 300 with - oo the user's input strokes to obtain silhouettes (i.e. segmented foreground regions ’ | containing the object of interest) in these images.

Step 2: Generate an initial silhouette for each of the remaining multi-view oo images I, I. ¢ 1,. This may be done by first generating a visual hull containing the object of interest from the silhouettes obtained in step 1 (using for example, a shape-from-silhouette method [24]), and then projecting the visual hull on each of the remaining images [to generate the initial silhouette for the image. : 15 -

Step 3: Evolve the initial silhouette generated in step 2 for each of the remaining multi-view images Ito snap to nearby geometry features in the image using for example, the known convex active contour model [4] or the constrained active contour model. | -

Step 4: To ensure segmentation coherence across the different views, perform a three-dimensional (3D) object segmentation via a 3D graph-cut. A method similar to the method described in reference [25] may be used in step 4.

Furthermore, the method may employ not only the color cue used in reference

[25], but also one or more of: a silhouette cue, hard constraints from initial user ~ interaction and a stereo-matching cue. : : Steps 3 and 4 above are repeatedly performed in an iterative manner until the

Co silhouettes in the images converge. In each iteration, the refined silhouette from: : ~ 30 the active contour model provides better color and silhouette cues for the 3D graph-cut in step 4 and in retum, the graph-cut produces better initial silhouettes for the next silhouette refinement in step 3.

Step 5: If necessary, refine the segmentation results of one or more multi-view images with extra foreground or background strokes, and perform the above- mentioned iterative process again. This step is optional and is usually ~~ performed when highly accurate segmentation results are required. :

Co As shown from the experimental results above, method 300 employing the constrained active contour model in step 304 is a robust and accurate interactive image segmentation method. Method 300 significantly outperforms + state-of-the-art interactive segmentation methods. The state-of-the-art ~ interactive image segmentation algorithms are often sensitive to user inputs and are unable to produce accurate cutting contours with one-shot user inputs i.e. they are unable to produce accurate cutting contours with just the user inputs for producing the one or more initial contours for example, the user input foreground and background ‘strokes. Furthermore, the state-of-the-art interactive image segmentation algorithms frequently rely on laborious user editing to refine the segmentation boundary or the initial contour(s). In contrast, method 300 exhibits many desired properties for a good segmentation tool, including the robustness to user inputs. In addition, the constrained active - contour model employed in step 304 of method 300 exhibits robustness to different contour initializations. Hence, method 300 is capable of producing good results reflecting user intentions and geometric features of the input image. Method 300 also has the ability to produce smooth and accurate boundary contours and handle topology changes. In addition, method 300 has a ~ high processing speed and is able to operate using simple inputs of foreground and background strokes from the user which does not require much effort on the user's part. To further evaluate the performance of the different interactive image segmentation methods and method 300, advanced evaluation methods = - such as the user-simulation based approach proposed in [23] may be adopted.

The advantages of method 300 may be partly attributed to its employment of the constrained active contour model in step 304 which improves the powerful continuous-domain convex active. contour. model. The constrained active contour model is able to refine initial contours obtained from state-of-the-art region-based methods, including the Geodesic method and the Random Walks ~~ method. As mentioned above, in method 300 for interactive segmentation of the oo input image, some of these region-based methods (preferably, the Geodesic oo method due to its fast processing speed) may be used in step 302 to generate © 10 one or more initial contours and the constrained active contour model is then ) applied in step 304 to refine and optimize these initial contour(s). Employing oo state-of-the-art region-based. methods with the constrained active contour = model in method 300 allows the use of the seed propagation and location features introduced by for example the Geodesic method or the Random Walks method to reduce the possible “small cut” problem in the known convex active contour model. It further allows the use of the powerful contour evolving ~~. capability provided by a convex active contour model to absorb the non- robustness of the region-based approaches.

Using continuous-domain convex methods allows one to avoid the inherent grid bias in all discrete graph-based methods. Furthermore, the processing speed of these methods is high because fast and global numerical solvers for these “methods through convex optimization [4], [22] exist. For example, the convex

BN active contour model when applied for automatic image segmentation, can give a global “optimal” result (with respect to its energy function) without involving ~ user interaction. However, when segmenting complex images, the automatic image segmentation based on the convex active contour model often results in over segmentation with small/trivial solutions [4, 7]. Furthermore, the convex active contour model so far has mainly been applied for automatic image segmentation without incorporating user inputs. To date, it is not clear how to : apply the convex active contour model for interactive image segmentation incorporating user inputs. This is also a non-trivial problem. For example,

directly incorporating user inputs as hard constraints into the convex active contour model does not lead to better performance. It is also non-trivial to apply the convex active contour model for refinement of initial contours in input images. For example, directly applying the convex active contour model on the =~ initial contours usually results in the refined contours drifting away from not only the initial contours but also, the user input seeds. This is because the convex : ~~ active contour model will tend to produce a global “optimal” solution that is oo independent of the initial contours. © 10 On the other hand, the constrained active: contour model (as employed in step 304 of method 300) is a strong boundary refinement method which improves . . the existing convex active contour model. The constrained active contour model makes use of both the boundary and regional information in the input image to find a global “optimal” solution for the segmentation problem. Step 304 may be implemented as a stand-alone method for refining one or more initial contours in : an input image and can be applied with existing interactive segmentation - methods to improve their segmentation results.’

The constrained active contour model (as employed in step 304 of method 300) is advantageous as its regional function h, incorporates information from the : user input. Thus, by using the constrained active contour model for contour : refinement, the evolving contour(s) complies with ‘user inputs and reflects the user intention. In addition, the regional function h, of the constrained active contour model may further incorporate information from the initial segmentation results comprising the one or more initial contours. This gives a further advantage in that the evolving contour(s) do not drift too far away from the initial contour(s) (generated by for example, an interactive region-based method).

As mentioned above, information about the initial segmentation results may be incorporated into the regional function h, of the constrained active contour ~ model by representing this information in the form of a probability map that comprises comprehensive information about the results. This probability map Co represents the one or more initial contours in the input image and may be used in the constrained active contour model to refine the contour(s).

Furthermore, the boundary function g, of the constrained active contour model : (as employed in step 304 of method 300) may incorporate the user inputs and : this gives a further advantage to the constrained active contour model. The boundary function g, of the existing convex active contour model favors segmentation along curves in the input image where the boundary function a, (which is usually an edge detection function) reaches a minimum. The boundary function g, in the existing convex active contour model also favors : segmentation at locations in the input image with smooth boundary curves.

Conventional edge detection functions (such as the function shown in Equation (3) are usually used as the boundary functions g, in the existing convex active contour model and this tends to smooth out the initial contour(s) at locations in the input image with weak edges. On the other hand, by incorporating information from the user inputs into the boundary function g, of the constrained -active contour model, edge detection can be effectively enhanced. Information from the user inputs incorporated into the boundary function g, of the © 20 constrained active contour model may be in the form of color probability maps for example, GMM probability maps or GMM color models.

Furthermore, the constrained active contour model retains a strong contour- evolving ability similar to that of the existing convex active contour model. This allows the constrained active contour model to evolve the one or more initial contours entirely to absorb the non-robustness of region-based approaches and at the same time, keep the contour(s) smooth and snapping to geometry features in the input image. :

The operating speed of method 300 and of the constrained active contour model is also high due to the fact that the constrained active contour model (employed in step 304 of method 300) can be solved quickly by a fast and global numerical solver such as the Split Bregman Method. In addition, the convergence speed when solving the constrained active contour model is typically higher than that when solving the existing convex active contour model.

The constrained active contour model can also be solved more quickly than the

Co oo existing convex active contour model because only step 2 of the three-step oo - alternate iterative approach is required to solve the constrained active contour model whereas all three steps of this approach are required to solve the existing convex active contour model. If the Geodesic algorithm is used for contour : initialization in step 302 of method 300, the processing speed of method 300 -can be further increased. :

As method 300 is essentially a hard segmentation method, it may not handle transparent or semitransparent boundaries such as semi-lucent hair well.

Furthermore, method 300 assumes that the shape of the object of interest is ~ smooth and can be well described by the weighted shortest boundary length.

Therefore, it may not handle very sophisticated shapes such as bush branches or hair well. Figs. 16(a) and (c) shows two example input images which method 300 is unable to segment well. In particular, Figs. 16(b) and (d) show the segmentation results from applying method 300 (with the Geodesic method in step 302) on the input images in Figs. 16(a) and (c) respectively. It can be seen

Co that method 300 is unable to segment the deer antlers and the tree branches in a clean manner. Nevertheless, despite these limitations, method 300 is still a robust and accurate interactive segmentation method that outperforms state-of- the-art interactive segmentation methods.

As mentioned above, step 304 empioying the constrained active contour model may be implemented as a stand-alone method for refining one or more initial contours in an input image. Such a contour refinement method based on step 304 may be integrated with a commercial image editing software as a plug-in.

For example, it may be integrated with Adobe Photoshop or Gimp. Such a contour refinement method based on step 304 may also be developed as a Co stand-alone image editing software, or as a gadget for consumer digital : apparatus comprising for example, one or more of: a digital camera, a mobile Co ~ phone and a PDA to cater to the increasing demand from digital media consumers. Furthermore, the constrained active contour model employed in oo step 304 may be applied for other segmentation problems such as ‘image ~~ matting or video segmentation. oo Furthermore, method 300 may be applied on medical images. For example, the : input image to method 300 may be a CT image or an MRI image. As CT and

MRI images do not comprise color data, it is usually more difficult to segment these images and the user has to be more careful in defining the foreground and background seeds. When applying method 300 to these images, the tradeoff factors «in Equation (15) and gin Equation (17) are set as zero. B 3D volume segmentation of organs and/or tumors from CT or MRI images is extremely important in the practice of clinical medicine. Powerful segmentation techniques are thus of substantial importance in CT and MRI volume assessment. A good segmentation technique enables a user (such as a clinical radiologist) to accurately and efficiently obtain 3D volume assessments of an : organ (e.g. liver) and/or a tumor from 2D images that are obtained in routine ~ clinical practice. This allows for tumor treatment assessment and for cases in which -an entire organ such as the liver is being measured, accurate determination of feasibility of organ resection and transplantation. The 3D data images will also be useful to the surgeons in preoperative surgical planning for deciding the plane of resection.

One particular example is the surgical management of liver cancer. Primary liver cancer, where viral hepatitis B is endemic and a known causative factor, is a leading cause of several cancer-induced deaths. It has been shown that a liver remnant of 20-40% its original (“ideal”) volume is sufficient to sustain normal metabolic function, depending on the function of the native liver. Thus,

for some cases of primary liver cancer, surgical resection allows for cure, although this is often limited by the presence of concomitant liver cirrhosis. In : general, surgical management for surgical resection involves determining the : feasibility of liver resection and transplantation, tumor treatment assessment . : and surgical planning. It is extremely difficult to perform these steps of surgical } management without image segmentation.

Co Most of the existing methods for 3D volume segmentation perform automatic on oo image segmentation. It is difficult to obtain satisfactory segmentation results using automatic image segmentation when the densities of (or the signal intensities of the pixels or voxels in the CT or MRI images belonging to) adjacent normal ‘structures are identical or similar. For example, in CT or MRI - images comprising a liver, the diaphragm and organs such as the stomach and the colon adjacent to one another and to the liver may not be accurately segmented from the liver by the automatic image segmentation methods. } To date, most methods used in hospitals (for example, the Siemens Leonardo - method) are still based on manual operations, requiring freehand ROI (region of interest) determination, which is very tedious and time-consuming. According to a doctor practicing in a hospital in Singapore (more specifically, Tan Tock Seng

Hospital), to perform 3D volume assessment of CT or MRI images, the doctors and/or medical researchers in the hospital usually have to segment the images manually. Because such manual methods are time-consuming and labor- . intensive, this deters a more regular use of CT or MRI volumetry, thus limiting

Co the potential of CT and MRI volumetry.

Some interactive image segmentation methods (or semi-automatic image ~ segmentation methods) have been recently introduced in medical image oo segmentation. In these interactive image segmentation methods, the users (e.g. doctors) may be asked to provide certain kinds of initializations such as initial © 30 contours or seeds indicating target objects to be segmented, and may also be asked to set certain parameters for the computerized segmentation. ‘Manual refinement may also be used after the computerized segmentation. It has been shown that in general, interactive image segmentation methods outperform the automatic image segmentation methods.

However, as mentioned earlier, there are several problems with existing interactive image segmentation methods. For example, they are sensitive to user inputs and often require a lot of additional user inputs to. refine the initial : oo contour(s) especially when segmenting images comprising objects with complex shapes. Thus, the user is often burdened with laborious editing when existing oo interactive image segmentation methods are used. | oo ~ Method 300 as described above can mitigate these problems by providing a - more accurate interactive image segmentation method and by combining multiple types of intuitive inputs. This can relieve users such as the doctors and medical researchers from tedious manual labeling, and at the same time, : | provide flexibilities for them to easily guide and refine the segmentation of the target objects in CT or MRl images.

Figs. 17(a) — (c) show segmentation results obtained by applying method 300 on medical images. In particular, Fig. 17(a) shows a first medical image with an user input foreground stroke 1702a, user input background strokes 1704a and = the generated cutting contour (i.e. refined contour) 1706a obtained from method 300, Fig. 17(b) shows a second medical image with user input foreground : strokes 1702b, user input background strokes 1704b and the generated cutting oo contour 1706b obtained from method 300, Fig. 17(c) shows a third medical image with user input foreground strokes 1702c, user input background strokes 1704c and the generated cutting contour 1706c obtained from method 300. It can be seen from Figs. 17(a) — (c) that method 300 can be effectively used in : the segmentation of CT or MRI images for medical purposes.

REFERENCES Co

[1] X. Bai and G. Sapiro. A geodesic framework for fast interactive image and video segmentation and matting. In IEEEICCV, pages 1-8, 2007.1,2,4,5,7

[2] Blake, A. et al. Interactive image segmentation using an adaptive GMMRF ~~ model. ECCV, pages 428-441, 2004. 5 :

[3]. Boykov and M. Jolly. Interactive graph cuts for optimal boundary and :

N region segmentation of objects in ND images. In IEEE ICCV, pages 105-112, oo 2001. 1 | LL

[4] Bresson, X. et al. Fast global minimization of the active contour/snake model. Journal of Mathematical Imaging and Vision, 28(2):151-167, 2007. 2,3

[5] V. Caselles, R. Kimmel, and G. Sapiro. Geodesic active contours.

International journal of computer vision, 22(1 ):61-79, 1997. 2

[6] F. Chan, F. Lam, P. Poon, H. Zhu, and K. Chan. Object boundary location by region and contour deformation. Vision, Image and Signal Processing, IEE

Proceedings, 143(6):353-360, 2002. 2

[7] T. Chan, - S. Esedoglu, and M. Nikolova. Algorithms for finding global minimizers of image segmentation and denoising models. SIAM J. Appl. Math., © 66, no. 5:1632-1648, 2006. 2, 3

[8] T. Chan and L. Vese. Active contours without edges. IEEE Transactions on

Image Processing, 10(2):266-277, 2001. 3

[9] A. Criminisi, T. Sharp, and A. Blake. Geos: Geodesic image segmentation.

To ECCV, pages 99-112, 2008. 1 :

[10] Duchenne, O. et al. Segmentation by transduction. In IEEE CVPR, pages 1-8,2008. 5

[11] A. Falcao, J. Udupa, and F. Miyazawa. An ultra-fast user steered image segmentation paradigm: live wire on the fiy. IEEE Transactions on Medical

Imaging, 19(1):55-62, 2002. 1

[12] T. Goldstein, X. Bresson, and S. Osher. Geometric applications of the split

Bregman method: Segmentation and surface reconstruction. Journal of

Scientific Computing, 45, no. 1-3:272~293, 2010. 2, 3, 4

[13] L. Grady. Random walks for image segmentation. IEEE Transactions on : ~~ Pattern Analysis and Machine Intelligence, 28(11):1768-1783, 2006. 1, 2, 4, 5, 7 Co

[14] M. Kass, A. Witkin, and D. Terzopoulos. Snakes: Active contour models. 1JoV, 1(4):321-331, 1988. 1, 2

[15] E. Mortensen and W. Barrett. Interactive segmentation with intelligent . scissors. Graphical Models and Image Processing, 60(5):349-384, 1998. 1 ~ [16] C. Rother, V. Kolmogorov, and A. Blake. Grabcut: Interactive foreground oo Co extraction using iterated graph cuts. In ACM SIGGRAPH, 2004. 1, 2, 5, 7

[17] A. Sinop and L. Grady. A seeded image segmentation framework unifying graph cuts and random walker which yields a new algorithm. In IEEE ICCV, pages 1-8, 2007: 1 | oo

[18] Y. Wang, W. Yin, and Y. Zhang. A fast algorithm for image deblurring with total variation regularization. Rice University CAAM Technical Report TR07-10, 2007.4 . [19] C. Yang, R. Duraiswami, N. Gumerov, and L. Davis. Improved fast Gauss transform and efficient kernel density estimation. In IEEE ICCV, pages 664— 671,2003.4

[20] W. Yang, J. Cai, J. Zheng, and J. Luo. User-friendly interactive image ‘segmentation through unified combinatorial user inputs. IEEE Transactions on

Image Processing, 19(9):2470-2479, Sept. 2010. 1, 5

[21] J. Zhang, J. Zheng, and J. Cai. A diffusion approach to seeded image segmentation. In IEEE CVPR, 2010. 1

[22] M. Unger, T. Pock, W. Trobin, D. Cremers, and H. Bischof, “Tvseg - interactive total variation based image segmentation,” in British Machine Vision

Conference, 2008. : [23] K. McGuinness and N. E. O'Connor, “Toward automated evaluation of interactive segmentation,” Computer Vision and Image Understanding, vol. 115, no. 6, pp. 868-884, 2011.

[24] A. Laurentini. The visual hull concept for silhouette-based image oo understanding. IEEE Trans. PAMI, pages 150-162, 1994. ~

[25] N. Campbell, G. Vogiatzis, C. Hernandez, and R. Cipolla. Automatic 3D object segmentation in multiple views using volumetric graph-cuts. Image and

Vision Computing, 28(1):14-25, 2010.

Claims

1. A method for segmenting an input image, wherein the input image comprises at least one intensity value for each of a plurality of pixels, the pixels } : comprising foreground pixels forming a foreground region and background pixels forming a background region, the method comprising: oo receiving user inputs specifying that at least one foreground seed of the input image is part of the foreground region and at least one background seed oo of the input image is part of the background region; CL oo using the user inputs to generate one or more probabilistic functions, each probabilistic function having a value at each pixel of the input image indicative of a likelihood of that pixel being a foreground pixel; - obtaining a regional function of an active contour model, the regional function being defined using at least one of said probabilistic functions and a candidate function which takes a value at each pixel of the input image; . Co obtaining a boundary function of the active contour model, wherein the = ) boundary function is defined using said candidate function and information relating to variations in intensity values in the input image; obtaining stationary point values of the candidate function corresponding to a stationary point of the active contour model; and oo So using the stationary point values of the candidate function to obtain one - or more refined contours representing a segmentation of the plurality of pixels into an estimated set of foreground pixels and an estimated set of background oo pixels.

2. A method according to claim 1, in which said step of generating one or more probabilistic functions comprises: oo estimating color distributions of the foreground and background regions from the user inputs and color data describing the input image; and for each pixel of the input image, calculating a foreground probability that the pixel belongs to the foreground region and a background probability that the pixel belongs to the background region using the estimated color distributions of : the foreground and background regions. oo

3. A method according to claim 2, wherein said step of obtaining the regional function further comprises calculating the regional function at each-pixel of the = input image by calculating a difference between the foreground and background probabilities of the pixel. oo

4. A method according to any one of the preceding claims, wherein said step of ~ 10 generating one or more probabilistic functions comprises using the user inputs to perform a segmentation of the input image to obtain a segmentation

~~. according to one or more initial contours, one said probabilistic function being a : probability map consistent with the one or more initial contours.

5. A method according to claim 4 when dependent on claim 3, wherein the - regional function is calculated using a sum of said difference between the ~ foreground and background probabilities and said probability map, weighted by a first tradeoff factor. :

6. A method according to claim 5, wherein the first tradeoff factor is calculated based on a distance between the estimated color distributions of the foreground and background regions.

: 7. A method according to any one of claims 4 — 6, wherein said one or more initial contours are obtained from said probability map. )

8. A method according to claim 7, wherein the probability map comprises probability points corresponding to respective pixels, and the at least one : foreground seed and at least one background seed respectively form at least ~~ 30 one foreground seed region and at least one background seed region, and wherein calculating the probability map further comprises:

calculating the value ofeach probability point based on a first geodesic distance between the corresponding pixel and the closest foreground seed region, and a second geodesic distance between the corresponding pixel and the closest background seed region.

:

9. A method according to claim 4, wherein the probability map comprises probability points corresponding to respective pixels and is calculated by: : identifying boundary pixels comprising pixels forming the one or more oo initial contours; = © 10 identifying near-border pixels comprising pixels within a predetermined Euclidean distance from respective nearest boundary pixels; oo for each identified boundary pixel, setting the value of the corresponding probability point to a predetermined probability value; and : for each identified near-border pixel, calculating the value of the corresponding probability point based on a Euclidean distance between the - near-border pixel and a nearest boundary pixel.

10. A method according to claim 9, wherein obtaining the regional function further comprises: ’ setting the regional function at the pixels with corresponding probability points having values above a first cutoff value t0 a predetermined negative ~~ value; and setting the regional function at the pixels with corresponding probability points having values below a second cutoff value to a predetermined positive : value. :

11. A method according to any preceding claim, wherein obtaining the regional function further comprises one or more of: setting the regional function at the at least one foreground seed to a predetermined negative value; and : setting the regional function at the at least one background seed to a predetermined positive value.

12. A method according to any one of the preceding claims, wherein the oo boundary function is calculated based on at least one of said probabilistic functions. | | 0 S

13. A method" according to claim 2, wherein obtaining the boundary function further comprises: forming a color probability map from the foreground probabilities of the oo pixels of the input image; detecting edges in the color probability map and in the input image; and obtaining the boundary function based on the detected edges in the color

~~. probability map and in the input image. . Co

14. A method according to claim 13, wherein obtaining the boundary function further comprises weighting the detected edges in the color probability map and the detected edges in the input image with a second tradeoff factor. :

15. A method according to claim 14, wherein the second tradeoff factor is calculated based on a distance between the estimated color distributions of the foreground and background regions. : ~~

16. A method according to any one of the preceding claims, wherein the stationary point values of the candidate function are obtained using a Split : Bregman method.

17. A method according to any one of the preceding claims, wherein the - input image is a MRI image ora CT image. ~~

18. A method for segmenting a plurality of images wherein each of the ~~ 30 plurality of images comprises an object of interest captured at a view different from others of the plurality of images and wherein the method comprises:

segmenting a subset of the plurality of images by a method according to any preceding claim; and segmenting the rest of the plurality of images based on the segmentation of the subset of the plurality of images.

.

19. A computer system having a processor arranged to perform a method according to any one of claims 1 — 18. Co ~

20. A computer program product such as a tangible data storage device, readable by a computer and containing instructions operable by a processor of a computer system to cause the processor to perform a method according to

~~. anyone of claims 1-18. - oo SE