CN110570436A - Image segmentation method based on depth perception - Google Patents

Image segmentation method based on depth perception Download PDF

Info

Publication number
CN110570436A
CN110570436A CN201910661047.5A CN201910661047A CN110570436A CN 110570436 A CN110570436 A CN 110570436A CN 201910661047 A CN201910661047 A CN 201910661047A CN 110570436 A CN110570436 A CN 110570436A
Authority
CN
China
Prior art keywords
pixel
target
image
gaussian
depth
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910661047.5A
Other languages
Chinese (zh)
Inventor
卢书芳
丁雪峰
寿旭峰
朱翔
高飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University of Technology ZJUT
Original Assignee
Zhejiang University of Technology ZJUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University of Technology ZJUT filed Critical Zhejiang University of Technology ZJUT
Priority to CN201910661047.5A priority Critical patent/CN110570436A/en
Publication of CN110570436A publication Critical patent/CN110570436A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/136—Segmentation; Edge detection involving thresholding
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G06T5/70—

Abstract

The invention discloses an image segmentation method based on depth perception, which comprises the following steps: (1) acquiring a depth map and a color map of a target and preprocessing the depth map and the color map; (2) extracting a minimum depth range containing a target from the depth map, and deleting pixel points at positions outside a depth threshold value in the corresponding color map; (3) and (3) performing picture frame twice on the color image processed in the step (2), wherein the first picture frame is divided into a minimum area containing a complete target, the second picture frame is divided into a positive foreground area in the minimum area, and all pixels outside the first picture frame are used as background pixels TBThe second picture frame has all pixelsPart as a target pixel TFThe pixel between two frames is used as a possible target pixel TU(ii) a (4) And estimating a Gaussian mixture model of the target and the background according to pixels possibly belonging to the target, and performing iterative segmentation by using the Gaussian mixture model. The invention has better segmentation effect and higher segmentation accuracy.

Description

Image segmentation method based on depth perception
Technical Field
The invention belongs to the technical field of digital image processing, and particularly relates to an image segmentation method based on depth perception.
Background
Image segmentation is a classic image problem, is the basis of high-level work of digital image processing, and is widely researched. Common image segmentation methods include region growing, watershed, cluster segmentation, edge detection, threshold segmentation, and the like.
The energy minimization framework based on a Markov Random Field (MRF) is a research hotspot of image segmentation, and the theory is novel in that the energy minimization framework can be combined with various theoretical knowledge to carry out global optimal solution.A GrabCut algorithm is provided based on Graphcut in Rother and the like in 2004.
chinese patent publication No. CN108596919A discloses an automatic image segmentation method based on a depth map, which fuses depth information and improves segmentation accuracy of the algorithm on the basis of achieving automatic segmentation of GrabCut with saliency.
the maximum stream minimum cut algorithm in the GrabCut algorithm has the defects of sensitivity to local noise, poor edge effect of target extraction and the like. Depth perception is introduced, the depth range of the image can be normalized to be in a range of 0-255 by using a depth image, and the edge segmentation effect is better for scenes with complex target backgrounds and small edge gradients.
disclosure of Invention
in order to overcome the defects in the prior art, the invention provides an image segmentation method based on depth perception, which has a better segmentation effect and more accurate image edge segmentation.
An image segmentation method based on depth perception comprises the following steps:
(1) Acquiring a depth map and a color map of a target and preprocessing the depth map and the color map;
(2) extracting a minimum depth range containing a target from the depth map, and deleting pixel points at positions outside a depth threshold value in the corresponding color map;
(3) And (3) performing picture frame twice on the color image processed in the step (2), wherein the first picture frame is divided into a minimum area containing a complete target, the second picture frame is divided into a positive foreground area in the minimum area, and all pixels outside the first picture frame are used as background pixels TBAll pixels in the second picture frame are used as target pixels TFThe pixel between two frames is used as a possible target pixel TU；
(4) And estimating a Gaussian mixture model of the target and the background according to pixels possibly belonging to the target, and performing iterative segmentation by using the Gaussian mixture model.
according to the method, the depth image target threshold value is segmented, the complete target minimum depth threshold value range is extracted, the image frames are processed twice, the complete target minimum area is segmented from the depth image in the first time, the foreground area is marked in the second time, the target clear edge is obtained from the complex background, the calculated amount of subsequent processing is reduced, and finally iterative segmentation is carried out according to a Gaussian mixture model.
In the step (1), the preprocessing includes aligning the depth map and the color map, and adjusting contrast and brightness.
in step (3), for TBInitializing the label alpha of the pixel n for each pixeln0; for TFInside ofinitializing the label alpha of the pixel n for each pixel nn2; for TUfor each pixel n, initializing the label alpha of the pixel nn＝1。
In the step (4), the specific steps of performing iterative segmentation according to the gaussian mixture model are as follows:
(4-1) clustering pixels belonging to a target and a background into K classes respectively through a K-mean algorithm, namely K Gaussian models in the GMM, wherein each Gaussian model in the GMM obtains a plurality of pixel sample sets;
(4-2) assigning a gaussian component in the GMM to each pixel;
the purpose of the step is to find the Gaussian component corresponding to each pixel with the highest probability and distribute the Gaussian component to the corresponding Gaussian component; if pixel n is the target pixel, then the RGB value of pixel n is substituted into each Gaussian component in the target GMM with the highest probability, i.e., the kth of pixel nnIndividual gaussian components:
Wherein alpha isnIs a pixel label, θ, knIs the GMM parameter, znAre pixels in an RGB spatial image.
(4-3) obtaining a pixel sample set of a Gaussian model according to the classification of the Gaussian component to which each pixel belongs in the step (4-1), and meanwhile, obtaining parameter mean and covariance through RGB value estimation of the pixel samples, wherein the weight of the Gaussian component can be determined through the ratio of the number of pixels belonging to the Gaussian component to the total number of pixels:
Wherein D isn(. h) is a Gaussian mixture model, theta, k are GMM parameters, alpha is a label of a pixel n, and z is a pixel point of an RGB space image;
(4-4) establishing a graph according to a Gibbs energy item, solving weights t-link and n-link, then segmenting by a max-flow/min-cut algorithm, and segmenting the maximum flow according to the mean value, covariance and weight of the maximum flow, wherein the formula is as follows:
Wherein α is a pixel label, α ═ 0 is a background, α ═ 1 is a possible region, α ═ 2 is a target region, θ, and k is a GMM parameter;
(4-5) repeating the steps (4-2) to (4-4) in the iterative minimization process until convergence, and obtaining a segmentation template for separation.
In the step (4), after the segmentation is finished, the method further comprises the step of smoothing the segmented boundary by using a border-matching. For the picture which can not be subjected to edge enhancement, the target edge segmented by adopting the depth perception method is smoother and clearer. The smoothing process specifically includes the following steps:
According to the boundary information of the image, selecting a mask of the image to use a Canny operator to carry out edge detection, parameterizing the image contour through the obtained edge information, finding out pixel points which belong to the same continuous contour, storing the information of the pixel points of all contours, and obtaining delta and sigma of each pixel point according to the principle of energy function minimization, wherein the energy function is as follows:
Wherein D isnis a data item, V is a smoothing item; α, anIs a pixel label, Δt,σtDelta and sigma for a pixel point.
And calculating through a Sigmoid function, and fusing the alphaMask of the whole image with the original image to obtain the image after the border matching.
compared with the prior art, the invention has the following beneficial effects:
The invention effectively overcomes the defects of inaccurate target edge and incomplete target of color image segmentation by combining the depth map and the color map to segment the target. The method provided by the invention has better segmentation effect and higher segmentation accuracy.
Drawings
FIG. 1 is a schematic flow chart of an image segmentation method based on depth perception according to the present invention;
FIG. 2 is a color image and a depth image of an image to be segmented according to an embodiment of the present invention, in which (a) is the color image and (b) is the depth image;
FIG. 3 is a color drawing of an embodiment of the present invention with areas outside the depth threshold removed;
FIG. 4 is a diagram illustrating a first time a region containing a complete target according to an embodiment of the present invention;
FIG. 5 illustrates a second outlined region of positive foreground in an embodiment of the present invention;
FIG. 6 is a diagram illustrating the effect of the final segmentation in the embodiment of the present invention;
fig. 7 shows other image segmentation results according to the embodiment of the present invention, wherein (a) is before segmentation and (b) is after segmentation.
Detailed Description
the invention will be described in further detail below with reference to the drawings and examples, which are intended to facilitate the understanding of the invention without limiting it in any way.
as shown in fig. 1, an image segmentation method based on depth perception is characterized by comprising the following steps:
And S01, acquiring a depth map and a color map of the target and preprocessing the depth map and the color map.
Referring to fig. 2, the depth map D1 and the color map C1 are aligned, and the contrast and brightness of the color map C1 are adjusted.
S02, extracting the minimum depth range including the complete target from the depth map D1, and deleting the corresponding pixel points outside the depth threshold range in the color map C1 to obtain a map C2 to reduce the amount of computation, as shown in fig. 3.
s03, the color image C2 processed in step (2) is framed twice to mark different image areas.
(3-1) first framing an area containing a complete target, and taking all pixels outside the frame as background imagesPlain TBAs shown in fig. 4.
(3-2) framing the area determined to be the target in the first frame, and framing T in the second frameFall as foreground pixels, pixel T between two blocksUAll as "possible target" pixels, as shown in fig. 5.
(3-3) to TBFor each pixel n in the array, initializing the label alpha of the pixel nn0, namely the background pixel; for TFfor each pixel n in the array, initializing the label alpha of the pixel nn2; to TUEach pixel n in the array, initializing the label alpha of the pixel nn1, i.e. a pixel that is "likely to be a target".
(3-4) obtaining a target (alpha) according to the probabilityn1), the rest being the pixels belonging to the background (α)n0) and certainly belong to the target (α)n2) from pixels that may belong to the object, the GMMs of the object and the background are estimated.
S04, iterating the energy minimization segmentation.
(4-1) clustering pixels belonging to a target and a background into K classes respectively through a K-mean algorithm, namely K Gaussian models in GMM (Gaussian mixture model), wherein each Gaussian model in GMM obtains a plurality of pixel sample sets;
(4-2) assigning a Gaussian component in GMM to each pixel, if pixel n is the target pixel, then substituting the RGB value of pixel n into each Gaussian component in the target GMM with the highest probability, i.e., the kth of pixel nnIndividual gaussian components:
Wherein alpha isnIs a pixel label, θ, knIs the GMM parameter, znare pixels in an RGB spatial image.
And (4-3) obtaining a pixel sample set of a Gaussian model according to the classification of the Gaussian component to which each pixel belongs in the step (4-1), and simultaneously obtaining parameter mean and covariance through RGB value estimation of the pixel samples, wherein the weight of the Gaussian component can be determined through the ratio of the number of the pixels belonging to the Gaussian component to the total number of the pixels.
Wherein D isn(. cndot.) is a Gaussian mixture model, theta, k are GMM parameters, alpha is a label of a pixel n, and z is a pixel point of an RGB space image.
(4-4) establishing a graph according to a Gibbs energy item, solving weights t-link and n-link, then segmenting by a max-flow/min-cut algorithm, and segmenting the maximum flow according to the mean value, covariance and weight of the maximum flow, wherein the formula is as follows:
where α is a pixel label, α ═ 0 is a background, α ═ 1 is a possible region, α ═ 2 is a target region, and θ, k is a GMM parameter.
(4-5) repeating the steps (4-2) to (4-4) in the iterative minimization process until convergence.
(4-6) smoothing the segmented boundary by using a clipper-matching, wherein the specific process is as follows:
According to the boundary information of the image, selecting a mask of the image to use a Canny operator to carry out edge detection, parameterizing the image contour through the obtained edge information, finding out pixel points which belong to the same continuous contour, storing the information of the pixel points of all contours, and obtaining delta and sigma of each pixel point according to the principle of energy function minimization, wherein the energy function is as follows:
Wherein D isnis a data item, V is a smoothing item;
Calculating through a Sigmoid function, fusing the alphaMask obtained from the whole image with the original image to obtain a boundary-matched image, and finally obtaining a segmentation result as shown in fig. 6.
For the picture which can not be subjected to edge enhancement, the target edge segmented by adopting the depth perception method is smoother and clearer, and fig. 7 shows the segmentation results of other images obtained by utilizing the segmentation method of the invention, wherein (a) is a picture before segmentation and (b) is a picture after segmentation.
compared with the existing segmentation method, the method provided by the invention has better segmentation effect and higher segmentation accuracy.
The embodiments described above are intended to illustrate the technical solutions and advantages of the present invention, and it should be understood that the above-mentioned embodiments are only specific embodiments of the present invention, and are not intended to limit the present invention, and any modifications, additions and equivalents made within the scope of the principles of the present invention should be included in the scope of the present invention.

Claims (6)

1. An image segmentation method based on depth perception is characterized by comprising the following steps:
(1) Acquiring a depth map and a color map of a target and preprocessing the depth map and the color map;
(2) extracting a minimum depth range containing a target from the depth map, and deleting pixel points at positions outside a depth threshold value in the corresponding color map;
(3) And (3) performing picture frame twice on the color image processed in the step (2), wherein the first picture frame is divided into a minimum area containing a complete target, the second picture frame is divided into a positive foreground area in the minimum area, and all pixels outside the first picture frame are used as background pixels TBAll pixels in the second picture frame are used as target pixels TFthe pixel between two frames is used as a possible target pixel TU；
(4) And estimating a Gaussian mixture model of the target and the background according to pixels possibly belonging to the target, and performing iterative segmentation by using the Gaussian mixture model.
2. the depth perception-based image segmentation method as claimed in claim 1, wherein in step (1), the preprocessing includes aligning a depth map and a color map, and adjusting contrast and brightness.
3. The depth perception-based image segmentation method according to claim 1, wherein in the step (3), T is measuredBinitializing the label alpha of the pixel n for each pixeln0; for TFfor each pixel n, initializing the label alpha of the pixel nn2; for TUFor each pixel n, initializing the label alpha of the pixel nn＝1。
4. the image segmentation method based on depth perception according to claim 1, wherein in the step (4), the iterative segmentation according to the gaussian mixture model specifically comprises the steps of:
(4-1) clustering pixels belonging to a target and a background into K classes respectively through a K-mean algorithm, namely K Gaussian models in the GMM, wherein each Gaussian model in the GMM obtains a plurality of pixel sample sets;
(4-2) assigning a gaussian component in the GMM to each pixel;
(4-3) obtaining a pixel sample set of a Gaussian model according to the classification of the Gaussian component to which each pixel belongs in the step (4-1), and meanwhile, obtaining parameter mean and covariance through RGB value estimation of the pixel samples, wherein the weight of the Gaussian component can be determined through the ratio of the number of pixels belonging to the Gaussian component to the total number of pixels:
Wherein D isn(. h) is a Gaussian mixture model, theta, k are GMM parameters, alpha is a label of a pixel n, and z is a pixel point of an RGB space image;
(4-4) establishing a graph according to a Gibbs energy item, solving weights t-link and n-link, then segmenting by a max-flow/min-cut algorithm, and segmenting the maximum flow according to the mean value, covariance and weight of the maximum flow, wherein the formula is as follows:
wherein α is a pixel label, α ═ 0 is a background, α ═ 1 is a possible region, α ═ 2 is a target region, θ, and k is a GMM parameter;
(4-5) repeating the steps (4-2) to (4-4) in the iterative minimization process until convergence, and obtaining a segmentation template for separation.
5. The image segmentation method based on depth perception according to claim 4, wherein in the step (4-2), Gaussian components in GMM are allocated to each pixel, the probability of the Gaussian component corresponding to each pixel is found to be the highest, and the Gaussian component is allocated to the corresponding Gaussian component; if pixel n is the target pixel, then the RGB value of pixel n is substituted into each Gaussian component in the target GMM with the highest probability, i.e., the kth of pixel nnIndividual gaussian components:
Wherein alpha isnIs a pixel label, θ, knIs the GMM parameter, znare pixels in an RGB spatial image.
6. The method of image segmentation based on depth perception according to claim 1, wherein in the step (4), after the segmentation, the method further includes smoothing the segmented boundary by using a border-matching, and the specific process is as follows:
According to the boundary information of the image, selecting a mask of the image to use a Canny operator to carry out edge detection, parameterizing the image contour through the obtained edge information, finding out pixel points which belong to the same continuous contour, storing the information of the pixel points of all contours, and obtaining delta and sigma of each pixel point according to the principle of energy function minimization, wherein the energy function is as follows:
Wherein D isnfor data items, V for smoothing items, alphanIs a pixel label, Δt,σtDelta and sigma of a pixel point;
and calculating through a Sigmoid function, and fusing the alphaMask of the whole image with the original image to obtain the image after the border matching.
CN201910661047.5A 2019-07-22 2019-07-22 Image segmentation method based on depth perception Pending CN110570436A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910661047.5A CN110570436A (en) 2019-07-22 2019-07-22 Image segmentation method based on depth perception

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910661047.5A CN110570436A (en) 2019-07-22 2019-07-22 Image segmentation method based on depth perception

Publications (1)

Publication Number Publication Date
CN110570436A true CN110570436A (en) 2019-12-13

Family

ID=68773815

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910661047.5A Pending CN110570436A (en) 2019-07-22 2019-07-22 Image segmentation method based on depth perception

Country Status (1)

Country Link
CN (1) CN110570436A (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105989604A (en) * 2016-02-18 2016-10-05 合肥工业大学 Target object three-dimensional color point cloud generation method based on KINECT
CN107463622A (en) * 2017-07-06 2017-12-12 西南交通大学 A kind of automatic Symbolic method for keeping landmark shape facility
CN108447068A (en) * 2017-12-22 2018-08-24 杭州美间科技有限公司 Ternary diagram automatic generation method and the foreground extracting method for utilizing the ternary diagram
CN108596919A (en) * 2018-04-24 2018-09-28 重庆邮电大学 A kind of Automatic image segmentation method based on depth map

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105989604A (en) * 2016-02-18 2016-10-05 合肥工业大学 Target object three-dimensional color point cloud generation method based on KINECT
CN107463622A (en) * 2017-07-06 2017-12-12 西南交通大学 A kind of automatic Symbolic method for keeping landmark shape facility
CN108447068A (en) * 2017-12-22 2018-08-24 杭州美间科技有限公司 Ternary diagram automatic generation method and the foreground extracting method for utilizing the ternary diagram
CN108596919A (en) * 2018-04-24 2018-09-28 重庆邮电大学 A kind of Automatic image segmentation method based on depth map

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
CARSTEN ROTHER 等: ""GrabCut" — Interactive Foreground Extraction using Iterated Graph Cuts", 《ACM》 *
SHUFANG LU 等: "Depth-aware image vectorization and editing", 《THE VISUAL COMPUTER》 *
王飞 等: "一种基于图的交互式目标分割算法", 《计算机工程与应用》 *

Similar Documents

Publication Publication Date Title
CN109961049B (en) Cigarette brand identification method under complex scene
CN107452010B (en) Automatic cutout algorithm and device
CN109859171B (en) Automatic floor defect detection method based on computer vision and deep learning
CN108537239B (en) Method for detecting image saliency target
EP2811423B1 (en) Method and apparatus for detecting target
CN111415363B (en) Image edge identification method
CN108447068B (en) Ternary diagram automatic generation method and foreground extraction method using ternary diagram
CN108846844B (en) Sea surface target detection method based on sea antenna
CN109975196B (en) Reticulocyte detection method and system
CN113298809B (en) Composite material ultrasonic image defect detection method based on deep learning and superpixel segmentation
CN111275727A (en) Cell segmentation method, cell segmentation apparatus, and computer-readable storage medium
CN108038458B (en) Method for automatically acquiring outdoor scene text in video based on characteristic abstract diagram
CN111868783B (en) Region merging image segmentation algorithm based on boundary extraction
CN108335268A (en) A method of the coloured image deblurring based on blind deconvolution
CN107704864B (en) Salient object detection method based on image object semantic detection
CN105913434B (en) A kind of positioning of leucocyte and iterative splitting approach
CN116433978A (en) Automatic generation and automatic labeling method and device for high-quality flaw image
CN116596899A (en) Method, device, terminal and medium for identifying circulating tumor cells based on fluorescence image
CN110570436A (en) Image segmentation method based on depth perception
CN112270683B (en) IHC digital preview image identification and organization foreground segmentation method and system
CN116563659A (en) Optical smoke detection method combining priori knowledge and feature classification
CN112581487A (en) Method for automatically extracting detection area and positioning kernel
CN115601358B (en) Tongue picture image segmentation method under natural light environment
CN112819841B (en) Face region segmentation method and device, computer equipment and storage medium
Futagami et al. Fast and robust building extraction based on HSV color analysis using color segmentation and GrabCut

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20191213