CN104050682B - Image segmentation method fusing color and depth information - Google Patents

Abstract

The invention discloses an image segmentation method fusing color and depth information. According to the method, firstly, a meanshift algorithm is used for segmenting an input color image to obtain an over-segmentation region set, and then similarities among all the regions are calculated and include color similarities, depth similarities and fusions of the color similarities and the depth similarities; then according to a depth image, seed regions of a target and seed regions of a background are automatically selected; finally, an MSRM algorithm is used for merging the regions, so that a final segmentation result is obtained. In the process of calculating the similarities among the regions, the color information is used, besides, the depth information is dynamically fused, and the problem that when the target color and the background color are similar, and namely a long-scale contrast edge exists between objects, correct segmentation can not be achieved is solved. The seed regions are automatically selected by the utilization of the depth information of the image, the seed regions of the target and the seed regions of the background do not need to be marked manually and alternately, region characteristics of the depth image are directly used for determining the seed regions instead of edge characteristics, and therefore high robustness is achieved.

Description

A kind of Fusion of Color and the image partition method of depth information

Technical field

The invention belongs to computer vision field is and in particular to the image segmentation side of a kind of Fusion of Color and depth information Method.

Background technology

Image segmentation is to divide the image into several regions specific, with unique properties and propose interesting target Technology and process.It is the committed step that image procossing arrives graphical analyses.The target extracting after image segmentation can be used for figure As semantics recognition, the field such as image retrieval.Traditional images dividing method is generally basede on the external appearance characteristic of image, such as color, brightness, Texture, shape, structure etc..And really object exists in three-dimensional world, object should be defined by connectedness physically, knot The depth information closing image carrys out segmentation figure picture, can obtain and have certain semanteme efficiently against over-segmentation and less divided problem Image segmentation result.

At present, image partition method is countless, the region merging technique based on maximum similarity that wherein [document 1] proposes Method (hereinafter referred to as msrm algorithm) is fairly simple, to picture material self adaptation, need not arrange similarity threshold, Neng Goucong in advance Objective contour is extracted in complex background.The advantage of msrm algorithm is: cuts (graph cut) with the classic algorithm figure of man-machine interactively Compare, under the conditions of identical man-machine interactively, segmentation effect is more preferable.It is limited in that: need labelling to cover main characteristic area Domain, when shade, low contrast edge and fuzzy region, segmentation is failure.

Have a lot of documents that depth image is introduced image segmentation field in recent years, [document 2] propose a kind of based on depth and The image object partitioning algorithm of colouring information, carries out over-segmentation first with meanshift partitioning algorithm to target image, simultaneously Obtain the dense depth map of axonometric chart pair by binocular stereo vision algorithm, select from over-segmentation result according to depth discontinuity Take the seed point set proceeding that " exquisite " is split, algorithm distribution label is cut to the region figure of unallocated Seed label, and right The adjacent area not having depth noncoherent boundary each other but having different labels is merged.The limitation of this algorithm exists In: when (1) is cut algorithm (graph cut) and carried out global optimization with figure, only use colouring information；(2) image two obtaining Value depth discontinuous edge is strongly depend on experimental threshold values, and edge line in itself be interruption, discontinuous, this will affect seed The reliability that point is chosen.[document 3] proposes a kind of multimode semantic segmentation method based on color and depth information, and this algorithm is by stricture of vagina Reason, color description and 3d description are fused together by Markov random field model, are super-pixel distribution label.The party Method needs to train, computationally intensive.What [document 4] proposed is positioned using colored and depth image in robot visual guidance system Object, when being only applicable to that in image, several object colors are single and completely the same, the difference using depth is distinguished.

Document 1:ning j., zhang l., zhang d., et al.interactive image segmentation by maximal similarity based region merging.pattern recognition,2010,43(2): 445-456；

Document 2: Pi Zhiming, Wang Zengfu. merge the image object partitioning algorithm of depth and colouring information. pattern recognition and people Work intelligence, 2013,26 (2): 151-158；

Document 3:islem jebari, david filliat.color and depth-based superpixels For background and object segmentation.procedia engineering, 2012,41:1307- 1315；

Document 4:jos é-juan, hern á ndez-l ó pez, ana-linnet.et al.detecting objects using color and depth segmentation with kinect sensor.procedia technology, 2012,3:196-204.

Content of the invention

It is an object of the invention to provide one kind can more accurately in addition when target and background color is complicated and close Distinguish and can utilize the image partition method in the region characteristic automatic selected seed region of depth image.

The technical solution adopted in the present invention is: the image partition method of a kind of Fusion of Color and depth information, its feature It is, comprise the following steps:

Step 1: using meanshift algorithm, input color image is made to split, obtain overdivided region set g= {gⁱ}_{I=1 ..., rn}, wherein, subscript i represents region sequence number, and rn is region total number；

Step 2: calculate the similarity between each region in g, including color similarity s_cWith depth similarity s_d, Yi Jiyan Color similarity s_cWith depth similarity s_dFusion；

Step 3: target and background seed region is automatically chosen according to depth image；

Step 4: carry out region merging technique using msrm algorithm, obtain final segmentation result.

Preferably, the color similarity between each region in the calculating g described in step 2, it implements process is Define the color similarity s of any two region r and q in g using bhattacharyya coefficient_c:

$s_c=\underset{u=1}{\overset{u}{\σ}}\sqrt{{\mathrm{hist}}_r^u\·{\mathrm{hist}}_q^u}$

Wherein, hist_rAnd hist_qIt is respectively the normalization color histogram of region r and q, subscript u represents histogrammic u Individual element, u is histogrammic footstalk number.

Preferably, the depth similarity between each region in the calculating g described in step 2, it implements process is The depth value of the pixel in each region in g is taken arithmetic average as the depth value in this region, constitute regional depth set d= {dⁱ}_{I=1 ..., rn}, subscript i represents region sequence number, defines depth similarity s of any two region r and q in g_d:

$s_d=\frac{-\left|d^r-d^q\right|}{max{\left\{d^i\right\}}_{i=1,\mathrm{...},rn}-\min {\left\{d^i\right\}}_{i=1,...,rn}}$

Wherein, max { dⁱ}_{I=1 ..., rn}Represent that all regional depths take maximum, min { dⁱ}_{I=1 ..., rn}For all regional depths Minima in addition to 0.

Preferably, the described depth value using the pixel in each region in g takes arithmetic average as the depth in this region Value, the depth for partial pixel in image not can determine that due to the reason such as blocking, and in the depth image being given be using 0 as The situation that depth value is filled up, concrete processing method is: overdivided region set g is mapped in depth image, if too Cut region gⁱThe depth value of middle element is all 0, show this region object depth information do not know, then only consider this region with The color similarity of adjacent area；If overdivided region gⁱIn have Partial Elements depth value be 0, then calculate this regional depth Value dⁱWhen, only to overdivided region gⁱThose pixels that middle depth is not zero take arithmetic average.

Preferably, the color similarity s described in step 2_cWith depth similarity s_dFusion, s_cAnd s_dTotal after fusion Similarity be:

S=s_c+w·s_d

Wherein, s_cAnd s_dThe weight w merging to be described using nonlinear sigmoid curve:

$w=\frac{a}{0.1+\exp \left(\frac{-(s_c-b)}{c}\right)}$

The maximum that wherein a determines sigmoid curve approaches value, b and c determine respectively sigmoid curve displacement and Steep.

Preferably, automatically choosing target and background seed region according to depth image described in step 3, it implements Process is first with k-means algorithm, the element in regional depth set d to be clustered, and classification number takes k=2, automatically Be polymerized to two big class, i.e. target and background, then respectively from this some region of two big apoplexy due to endogenous wind random chooses as target and background Seed region.This clustering method is simple, solves the problems, such as to need in Ning Ji peak method manually to participate in marking.

Present invention advantage compared with prior art is:

(1) between zoning during similarity, not only utilize colouring information, go back dynamic fusion depth information, then When in image, target and background color is close, when low contrast edge occurring between object and object, can by depth not With distinguishing；

(2) utilize image depth information automatic selected seed region, mark the kind of target and background without man-machine interactively Subregion, directly utilizes the region characteristic of depth image, rather than local edge, to determine seed region, has preferable robust Property.

Brief description

Fig. 1: be flow chart of the present invention；

Fig. 2: be the sigmoid curve used in the embodiment of the present invention；

Fig. 3-1: be the coloured image of input in the embodiment of the present invention；

Fig. 3-2: be the depth image of input in the embodiment of the present invention；

Fig. 4: be the result that in the embodiment of the present invention, coloured image is made with meanshift segmentation；

Fig. 5-1: be that in the embodiment of the present invention, regional depth is made with k-means cluster result；

Fig. 5-2: be the seed region chosen in the embodiment of the present invention；

Fig. 6-1: be the final segmentation result being obtained according to the inventive method in the embodiment of the present invention；

Fig. 6-2: be the final segmentation result being obtained according to Ning Ji peak method in the embodiment of the present invention.

Specific embodiment

Understand for the ease of those of ordinary skill in the art and implement the present invention, below in conjunction with the accompanying drawings and embodiment is to this Bright be described in further detail it will be appreciated that described herein enforcement example be merely to illustrate and explain the present invention, not For limiting the present invention.

Ask for an interview Fig. 1, Fig. 2, Fig. 3-1, Fig. 3-2 and Fig. 4, as a example the present invention adopts the segmentation of Aloe potted landscape shown in Fig. 3-1, Fig. 3-2 is the depth image of Aloe potted landscape, make use of the information of depth image that coloured image is made more accurately to split.The present invention Be the technical scheme is that the image partition method of a kind of Fusion of Color and depth information, comprised the following steps:

Step 1: using the framework of Ning Jifeng, using meanshift segmentation software edison to the Aloe basin shown in Fig. 3-1 Scape is split, and asks for an interview Fig. 4, is over-segmentation result, obtains overdivided region set g={ gⁱ}_{I=1 ..., rn}, wherein, subscript i generation Table section sequence number, rn is region total number；Software relevant parameter is all using default setting.It can be seen that the background based on texture is divided It is slit into a large amount of discreet region, and Aloe potted landscape is divided into larger region unit.

Step 2: 3 Color Channels of the rgb coloured image of input are divided into 16 deciles by codomain [0,255] respectively, The color histogram in so each region just calculates in the feature space for 16 × 16 × 16=4096 for the dimension.Then right Color histogram normalization, the color similarity of two adjacent areas of calculating:

hist_rAnd hist_qIt is respectively the normalization color histogram of any two region r and q in g, subscript u represents Nogata U-th element of figure.Create two-dimentional similarity matrix sm, dimension is rn × rn.If two regions non-conterminous it is impossible to be closed And, make sm_ij=0, each region oneself is maximum with the similarity of oneself, is set to 1, i.e. diagonal entry sm_ii=1, adjacent area According to formula one calculate color similarity, span be [0,1).

The depth value of the pixel in each region in g is taken arithmetic average as the depth value in this region, constitute set d= {dⁱ}_{I=1 ..., rn}, subscript i represents region sequence number.Assume that ith zone has n pixel, the depth value set of each pixel is {x₁,…,x_n, then:

Define depth similarity s of two region r and q_d:

In formula three, max { dⁱ}_{I=1 ..., rn}Represent that all regional depths take maximum, min { dⁱ}_{I=1 ..., rn}Deep for all regions Minima in addition to 0 for the degree.If overdivided region gⁱThe depth value of middle element is all 0, then it is believed that this region no depth Information, only considers the color similarity in this region and adjacent area；If overdivided region gⁱIn have Partial Elements depth value be 0, then calculate this regional depth value dⁱWhen, only arithmetic average is taken to the pixel that depth is not zero.From formula three, depth phase Like degree s_dSpan be [- 1,0], the depth difference in two regions is bigger, then depth similarity is less.

Step 3: color similarity s_cWith depth similarity s_dFusion, s_cAnd s_dAfter fusion, total similarity is:

S=s_c+w·s_d(formula is wantonly)

Ask for an interview Fig. 2, s_cAnd s_dThe weight w merging to be described using nonlinear sigmoid curve:

The maximum that wherein a determines sigmoid curve approaches value, b and c determine respectively sigmoid curve displacement and Steep.

It is demonstrated experimentally that a=1, the objective contour effect extracted when b=0.2, c=0.5 is preferable.Due to color similarity s_c ∈ [0,1], Fig. 2 only show and work as a=1, a part for sigmoid curve when b=0.2, c=0.5.

Step 4: with k-means algorithm to d={ dⁱ}_{I=1 ..., rn}Middle element is clustered, and takes k=2, is polymerized to two big class, that is, Target class r_oWith background classes r_b.Assume r_oIncluding m region, r_bComprise n region, then m+n=rn.Respectively from this two big apoplexy due to endogenous wind M1 and n1 region of random choose is as the seed region of target and background, wherein m1 < m, n1 < n.M1 and n1 specific number is permissible By user from primary input program, find in practice, 1/40th effects that m1 and n1 is about taken as m and n are just fine, no With taking too many seed region.If Fig. 5-1 is this example region depth cluster result, white represents target class, and black represents the back of the body Scape class, then randomly selects 5 seed regions from target class (containing 191 overdivided regions), background classes are (containing 1267 over-segmentations Region) 27 seed regions of random choose, because in background classes, each region area is less, region quantity is larger, therefore can fit Choose some seed regions when, as shown in Fig. 5-2, green area represents target seed region, and blue region represents background kind more Subregion.

Step 5: region merging technique is carried out using msrm algorithm, obtains final segmentation result.

If m_bFor the background area set of labelling, m_oFor the target area set of labelling, n is unmarked regional ensemble, then The basic procedure of msrm algorithm is as follows:

Step 5.1: merge the background area of unmarked region and labelling.For each region b ∈ m_b, find the adjacent of it Set of regionsFor each a_iAndFind its Neighbourhood setObviouslyCalculateIfThen by region a_iIt is merged into b.Update set m_bAnd n.Until m_bIn region can not find new combined region till；

Step 5.2: the region in self-adopt combination n.In the same manner, for each region p ∈ n, find its neighborhood collectionFor each h_iMeetAndFind its neighborhood collectionObviouslyCalculateIfThen by region h_iIt is merged into p.Update collection Close n.Till the region in n can not find new combined region.

Circulation carries out step 5.1 and step 5.2 until unmarked region can neither be merged into m_bIn, nor close in n And.This algorithm progressively by unmarked region merging technique in background area, remaining unmarked region is automatically incorporated into target area In.

Under conditions of seed region is chosen unanimously, Fig. 6-1 is the final segmentation result being obtained according to the inventive method, figure 6-2 is the final segmentation result being obtained according to Ning Jifeng et al. proposition method.It is seen that, the color due to curtain in background is Absinthe-green decorative pattern, and Aloe leave color is close, the target area therefore do not laid down hard and fast rule as seed all can be by mistakenly It is merged in background classes.And background and target can be precisely separating by the present invention using the difference of depth.

It should be appreciated that the part that this specification does not elaborate belongs to prior art.

It should be appreciated that the above-mentioned description for preferred embodiment is more detailed, can not therefore be considered to this The restriction of invention patent protection scope, those of ordinary skill in the art, under the enlightenment of the present invention, is weighing without departing from the present invention Profit requires under protected ambit, can also make replacement or deform, each fall within protection scope of the present invention, this Bright scope is claimed should be defined by claims.

Claims (3)

1. the image partition method of a kind of Fusion of Color and depth information is it is characterised in that comprise the following steps:

Step 1: using meanshift algorithm, input color image is made to split, obtain overdivided region set g= {gⁱ}_{I=1 ..., rn}, wherein, subscript i represents region sequence number, and rn is region total number；

Step 2: calculate the similarity between each region in g, including color similarity s_cWith depth similarity s_d, and color phase Like degree s_cWith depth similarity s_dFusion；

Wherein calculate the depth similarity between each region in g, it implements process is by the depth of the pixel in each region in g Angle value takes arithmetic average as the depth value in this region, constitutes regional depth set d={ dⁱ}_{I=1 ..., rn}, subscript i represents region Sequence number, defines depth similarity s of any two region r and q in g_d:

$s_d=\frac{-|d^r-d^q|}{max{\left\{d^i\right\}}_{i=1,...,rn}-\min {\left\{d^i\right\}}_{i=1,...,rn}}$

Wherein, max { dⁱ}_{I=1 ..., rn}Represent that all regional depths take maximum, min { dⁱ}_{I=1 ..., rn}For all regional depths except Minima beyond 0；

Wherein the depth value of the pixel in each region in g is taken arithmetic average as the depth value in this region, in the middle part of image The depth of point pixel not can determine that, and is using 0 situation about being filled up as depth value in the depth image being given, and concrete processes Method is: overdivided region set g is mapped in depth image, if overdivided region gⁱThe depth value of middle element is all 0, table The depth information of this region object bright does not know, then only consider the color similarity in this region and adjacent area；If over-segmentation Region gⁱIn have Partial Elements depth value be 0, then calculate this regional depth value dⁱWhen, only to overdivided region gⁱMiddle depth Those pixels being not zero take arithmetic average；

Step 3: target and background seed region is automatically chosen according to depth image；

Step 4: the method using the region merging technique based on maximum similarity carries out region merging technique, obtains final segmentation result.

2. Fusion of Color according to claim 1 and depth information image partition method it is characterised in that: in step 2 Described color similarity s_cWith depth similarity s_dFusion, s_cAnd s_dAfter fusion, total similarity is:

S=s_c+w·s_d

Wherein, s_cAnd s_dThe weight w merging to be described using nonlinear sigmoid curve:

$w=\frac{a}{0.1+\exp \left(\frac{-(s_c-b)}{c}\right)}$

The maximum that wherein a determines sigmoid curve approaches value, b and c determines the displacement of sigmoid curve and precipitous respectively Degree.

3. Fusion of Color according to claim 1 and depth information image partition method it is characterised in that: step 3 institute That states chooses target and background seed region automatically according to depth image, and it implements process is to calculate first with k-means Method clusters to the element in regional depth set d, and classification number takes k=2, is automatically polymerized to two big class, i.e. target and background, Then respectively from this some region of two big apoplexy due to endogenous wind random chooses as target and background seed region.