CN110610505A - Image segmentation method fusing depth and color information - Google Patents
Image segmentation method fusing depth and color information Download PDFInfo
- Publication number
- CN110610505A CN110610505A CN201910909933.5A CN201910909933A CN110610505A CN 110610505 A CN110610505 A CN 110610505A CN 201910909933 A CN201910909933 A CN 201910909933A CN 110610505 A CN110610505 A CN 110610505A
- Authority
- CN
- China
- Prior art keywords
- image
- color
- pixel
- depth
- edge
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000003709 image segmentation Methods 0.000 title claims abstract description 33
- 238000001914 filtration Methods 0.000 claims abstract description 12
- 230000011218 segmentation Effects 0.000 claims abstract description 12
- 230000003595 spectral effect Effects 0.000 claims abstract description 7
- 238000007781 pre-processing Methods 0.000 claims abstract description 4
- 239000011159 matrix material Substances 0.000 claims description 9
- 238000010276 construction Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 238000007499 fusion processing Methods 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 238000002372 labelling Methods 0.000 claims description 2
- 238000005259 measurement Methods 0.000 claims description 2
- 238000003384 imaging method Methods 0.000 abstract description 4
- 238000004364 calculation method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000003086 colorant Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
- G06T2207/20032—Median filtering
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Probability & Statistics with Applications (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an image segmentation method fusing depth and color information, which mainly comprises the following steps: preprocessing RGB-D data: carrying out median filtering by taking the image edge information as a guide to enhance the quality of the depth image; super-pixel segmentation of RGB-D images: fusing color and depth information, and performing over-segmentation on the image; super-pixel merging: similar superpixels are merged by adopting a spectral clustering method based on graph theory, clustering is converted into the problem of graph division, and the image segmentation is completed. According to the invention, the depth map is converted into the three-dimensional point cloud according to the imaging geometric principle, so that the information of depth and color is integrated, the image is segmented, and the quality and the precision of image segmentation are improved.
Description
Technical Field
The invention relates to the field of image processing and computer vision, in particular to an image segmentation method fusing depth and color information.
Background
Image segmentation is a popular topic in the field of computer vision, and plays an important role in a variety of applications such as object recognition, target positioning and tracking, image retrieval, three-dimensional reconstruction, robot navigation and positioning, and the like. The traditional RGB image segmentation method divides an image into non-overlapping connected regions by using low-level features such as color space, texture, color distribution histogram and the like, so that the same region has high similarity, and different regions have larger difference. These methods are difficult to distinguish when adjacent different objects in the image are similar in color, or when the contrast of the edge features is low.
In recent years, with the rapid development of sensor technology, a large number of consumer-grade deep acquisition devices gradually come to the market and are widely applied, such as microsoft Kinect and Intel's real sense series. These sensors usually carry a color camera and can acquire the registered depth and color images synchronously, which provides more information and possibility for the traditional image segmentation technology based on two-dimensional color space.
Disclosure of Invention
The invention aims to provide an image segmentation method for acquiring RGB-D data, fusion depth and color information by using a depth sensor, which is mainly applied to object recognition of indoor scenes.
The registered depth and color images are synchronously acquired, so that not only can two-dimensional color information of a scene be acquired, but also three-dimensional space information of the scene can be acquired, and therefore, objects which are difficult to distinguish in the two-dimensional color space can be possibly distinguished through position information in the three-dimensional space, and targets can be identified from the background. Based on the principle, the invention provides an image segmentation method fusing depth and color information, which mainly comprises the following steps:
step 1, RGB-D data preprocessing: carrying out median filtering by taking the image edge information as a guide to enhance the quality of the depth image;
step 2, RGB-D image superpixel segmentation: fusing color and depth information, and performing over-segmentation on the image;
and 3, super-pixel combination: similar superpixels are merged by adopting a spectral clustering method based on graph theory, clustering is converted into the problem of graph division, and the image segmentation is completed.
Further, the method of step 1 comprises:
respectively carrying out edge extraction on the color image and the depth image to generate respective edge images;
and (3) carrying out fusion processing on the color edge image and the depth edge image: on the depth edge image, calculating the gradient direction of each pixel, if the edge direction of the corresponding color edge image is similar to the direction, using the color image edge, otherwise, using the depth image edge, thus obtaining the fused edge image information;
on the depth image, performing median filtering on each effective pixel by adopting a template with the size of 3 multiplied by 3;
color space transformation: and converting the color image acquired by the camera from an RGB color space to a CIELab color space.
Further, the step 2 comprises the following steps:
2.1 three-dimensional point cloud reconstruction based on depth information: converting pixel coordinates (X, Y) in the depth map into three-dimensional space coordinates (X, Y, Z) based on depth information in the depth map and intrinsic parameters of the camera;
2.2 superpixel Pre-segmentation and initialization of clustering centers: assuming that the image has N pixels, the image is divided into K superpixel blocks, each superpixel block has N/K pixels, and the space between each superpixel block isSetting an initial clustering center at the center of the superpixel block;
2.3 region clustering labeling: in the 2S multiplied by 2S neighborhood range of each super pixel clustering center, calculating the distance between each effective pixel and the clustering center in the 8-dimensional feature space [ L, a, b, X, Y, X, Y, Z ]:
D＝DLab+αDxy+βDXYZ
where subscript c denotes the cluster center of each superpixel, subscript i denotes each valid pixel within the search range, DLab,Dxy,DXYZRespectively representing Euclidean distances between an effective pixel i and a clustering central point on a color feature Lab, an image two-dimensional coordinate xy and a depth-based three-dimensional space coordinate XYZ, wherein alpha and beta are weights used for balancing the importance degree of each item, and D is a final feature distance set according to specific scene data;
dividing each pixel into super pixels to which the cluster center with the minimum characteristic distance D belongs, namely marking the super pixels as the same class as the cluster center;
2.4 updating the clustering center: after all the effective pixels are marked, updating the clustering center according to the pixel classification result, namely counting the number of pixels contained in each super-pixel block and calculating the average value of the number of pixels to obtain a new clustering center;
2.5 iterative clustering: and repeating the steps of 2.3 and 2.4 until the clustering is stable, namely the clustering center and the pixel mark are not changed any more, so that the super-pixel segmentation is completed.
Further, the step 3 comprises the following steps:
3.1 similarity matrix construction: and taking each super pixel as a node of the graph, and taking the similarity between the super pixels as a weight of an edge to construct an undirected graph of all the super pixels. Assuming that the number of superpixels is K, the similarity matrix W of the graph belongs to RK×KIs aA symmetric array of each element wpqThe degree of similarity between a superpixel p and a superpixel q (p is 1, …, K; q is 1, …, K) is measured by color similarity, texture similarity and three-dimensional spatial connectivity:
wpq＝Dcolor(p,q)+Dtexture(p,q)+γDspace
wherein D iscolor(p, q) represents color similarity, Dtexture(p, q) represents texture similarity, DspaceRepresenting three-dimensional space connectivity, wherein gamma is a weight factor, and adjusting the weight of the three-dimensional space connectivity according to the quality of depth data, and the three-dimensional space refers to the three-dimensional space reconstructed based on the depth information;
and 3.2, performing spectral clustering on the superpixels according to the constructed similarity matrix, and combining the superpixels with high similarity to finish image segmentation.
Further, the median filtering method is as follows:
let D (x, y) be the current valid pixel depth value, ND(x, y) is its 8 neighbors, then:
D(x,y)＝median(ND(x,y))
if no edge pixel exists in the neighborhood of the current pixel 8, calculating according to the formula; if the edge pixel exists, median filtering is carried out according to the neighborhood pixel on one side of the edge where the current pixel is located.
Further, the specific method for converting the single-dimensional information of the depth map into the three-dimensional space coordinate is
Wherein (x, y) is the image pixel coordinate of a certain point in the depth image, d is the depth value of the point in the depth image, fx、fyFocal lengths of the depth camera in the x and y directions, respectively, (c)x,cy) As principal point coordinates, fx,fy,cx,cyGiven by the camera manufacturer, (X, Y, Z) is the three-dimensional spatial coordinates of the point.
Further, the color similarity calculation method comprises the following steps:
on the RGB color space, for the super pixel p, respectively counting the normalized color histogram H on the three channels of RGBp,R,Hp,G,Hp,BThe same applies to superpixel q; computing Bhattacharyya coefficients between the superpixel p and q three channel color histograms:
where i is the normalized pixel value;
then the sum of the Bhattacharyya coefficients between the three channel histograms of RGB is the color similarity:
Dcolor(p,q)＝BhR(p,q)+BhG(p,q)+BhB(p,q)。
further, the texture similarity calculation method comprises the following steps:
in Lab color space, an L channel value is selected to describe a Local Binary Pattern (LBP) histogram of a super pixel, and a Bhattacharyya coefficient is adopted to measure the texture similarity D between the super pixels p and qtexture(p,q)。
Further, the method for calculating the connectivity of the three-dimensional space comprises the following steps:
adopting a Gaussian kernel function, wherein sigma is a scale factor, setting according to an actual scene, and XYZ is a three-dimensional space coordinate of a superpixel clustering center, so that the three-dimensional space connectivity between the points p and q
According to the invention, a depth map is converted into three-dimensional point cloud according to the imaging geometric principle, so that the information of depth and color is integrated, and the image is subjected to superpixel segmentation and clustering, thereby completing image segmentation. The method makes full use of spatial position information contained in the depth image registered with the color image, so that objects which are difficult to distinguish in a two-dimensional color space are distinguished through the position information in a three-dimensional space, targets are identified from the background, and the quality and the precision of image segmentation are improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a flow diagram of a method for image segmentation fusing depth and color information according to an exemplary embodiment;
fig. 2 is an example of a segmentation effect of an image segmentation method applying fused depth and color information according to an exemplary embodiment.
Detailed Description
Reference will now be made in detail to various exemplary embodiments of the invention, the detailed description should not be construed as limiting the invention but as a more detailed description of certain aspects, features and embodiments of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made in the specific embodiments of the present disclosure without departing from the scope or spirit of the disclosure. Other embodiments will be apparent to those skilled in the art from consideration of the specification. The specification and examples are exemplary only.
A flow chart of an exemplary embodiment of the present invention is given in fig. 1. As shown in the figure, the specific steps of this embodiment include:
step 1: and preprocessing RGB-D data. Due to the influences of working distance, shielding, noise and the like, the depth image needs to be preprocessed, median filtering is carried out by taking image edge information as guidance, and the quality of the depth image is enhanced. The method specifically comprises the following steps:
1.1, carrying out edge extraction on the color image and the depth image to generate respective edge images. Preferably, the present embodiment employs a Canny edge detection operator.
1.2 in the depth edge image, 8 neighborhoods of each edge pixel are detected, and if no invalid pixel exists, the pixel is a real edge pixel. And then, carrying out fusion processing on the color edge image and the depth edge image: on the depth edge image, the gradient direction of each pixel is calculated, if the direction of the corresponding color edge image edge is approximate, the color image edge is used, otherwise, the depth image edge is used. Thus, the fused edge image information is obtained.
1.3 on the depth image, median filtering is performed for each effective pixel. Preferably, a 3 × 3 size template is used: let D (x, y) be the current valid pixel depth value, ND(x, y) is its 8 neighbors, then:
D(x,y)＝median(ND(x,y))
if no edge pixel exists in the neighborhood of the current pixel 8, calculating according to the formula; if the edge pixel exists, median filtering is carried out according to the neighborhood pixel on one side of the edge where the current pixel is located.
1.4 color space transformation: and converting the color image acquired by the camera from an RGB color space to a CIELab color space. Because it has wider color gamut and can make up the defect of uneven distribution of the RGB color model.
Step 2: and (5) carrying out super-pixel segmentation on the RGB-D image. The complexity of the scene makes it difficult to directly and accurately segment the image, so that the whole image is segmented into a large number of small blocks, and adjacent similar pixels are divided into the same block, which is called super-pixel. Fusing color and depth information, and segmenting an image, specifically comprising the following steps:
2.1 three-dimensional point cloud reconstruction. The depth map contains three-dimensional space information, but the x and y coordinates of the depth map are pixel coordinates, so that the information of an object in one direction in the three-dimensional space can be reflected, the information of the other two dimensions can be lost when the depth map is directly used, and therefore the single-dimensional information of the depth map needs to be converted into the three-dimensional space coordinates.
Let (X, Y, Z) be the three-dimensional spatial coordinates of a certain point, (X, Y) be the corresponding image pixel coordinates in the depth image, and d be the corresponding depth value in the depth image. f. ofx,fyFocal lengths of the depth camera in the x, y directions, respectively, (c)x,cy) As principal point coordinates, fx,fy,cx,cyGiven by the camera manufacturer, commonly referred to as intrinsic parameters. According to the imaging geometry principle, three-dimensional space coordinates can be calculated from the depth values:
at this time, each effective pixel of the RGB-D image can be described by the color feature L ab, the image two-dimensional coordinates x y, and the three-dimensional space coordinates XYZ to measure the color similarity and the continuity of the two-dimensional space and the three-dimensional space.
2.2 initializing the cluster centers. Assuming that the image has N pixels, the image is divided into K superpixel blocks, each superpixel block has N/K pixels, and the space between each superpixel block isThe initial cluster center is set at the center of the superpixel block.
2.3 region clustering labels. In the 2S multiplied by 2S neighborhood range of each super pixel clustering center, calculating the distance between each effective pixel and the clustering center in the 8-dimensional feature space [ L, a, b, X, Y, X, Y, Z ]:
D＝DLab+αDxy+βDXYZ
where subscript c denotes the cluster center of each superpixel, subscript i denotes each valid pixel within the search range, DLab,Dxy,DXYZAnd respectively representing Euclidean distances between the effective pixel i and a clustering central point on a color feature L ab, an image two-dimensional coordinate xy and a three-dimensional space coordinate XYZ, wherein alpha and beta are weights, are used for balancing the importance degree of each item, and are set according to specific scene data. D is the final feature distance. Each pixel is classified into the super-pixel to which the cluster center with the smallest characteristic distance belongs, namely, the super-pixel is marked as the same class as the cluster center.
2.4 updating the cluster center. And after all the effective pixels are marked, updating the clustering center according to the pixel classification result. Namely, the number of pixels contained in each superpixel block is counted, and the average value of the number of pixels is calculated to obtain a new clustering center.
And 2.5, carrying out iterative clustering, and repeating the steps of 2.3 and 2.4 until the clustering is stable, namely the clustering center and the pixel mark are not changed any more. Generally, 5-6 times of iteration can obtain stable clustering, and the super-pixel segmentation is completed.
And step 3: and (4) super-pixel combination. The method comprises the following steps of merging similar superpixels by adopting a spectral clustering method based on graph theory, converting clustering into a graph partitioning problem, and completing image segmentation:
3.1 similarity matrix construction. And taking each super pixel as a node of the graph, and taking the similarity between the super pixels as a weight of an edge to construct an undirected graph of all the super pixels. Assuming that the number of superpixels is K, the similarity matrix W of the graph belongs to RK×KIs a symmetric array of each element wpqFor the similarity between superpixel p and superpixel q, the color similarity, texture similarity and three-dimensional spatial connectivity are adopted for measurement:
wpq＝Dcolor(p,q)+Dtexture(p,q)+γDspace
wherein gamma is a weight factor, and the weight of the three-dimensional space connectivity is adjusted according to the quality of the depth data.
Color similarity: on the RGB color space, for the super pixel p, respectively counting the normalized color histogram H on the three channels of RGBp,R,Hp,G,Hp,BThe same applies to superpixel q. Computing Bhattacharyya coefficients between the superpixel p and q three channel color histograms:
where i is the normalized pixel value. The sum of the Bhattacharyya coefficients among the three channel histograms of RGB is then the color similarity:
Dcolor(p,q)＝BhR(p,q)+BhG(p,q)+BhB(p,q)
texture similarity: in Lab color space, an L channel value is selected to describe the super-pixels by a local binary pattern LBP histogram, similar to color similarity, and Bhattacharyya coefficients are adopted to measure texture similarity D between the super-pixelstexture(p,q)。
Three-dimensional space connectivity: adopting a Gaussian kernel function, wherein sigma is a scale factor, setting according to an actual scene, and XYZ is a three-dimensional space coordinate of a superpixel clustering center:
and 3.2, performing spectral clustering on the superpixels according to the constructed similarity matrix, and combining the superpixels with high similarity to finish image segmentation.
Therefore, according to the method, the depth map is converted into the three-dimensional point cloud according to the imaging geometric principle, information of depth and color is further integrated, and the image is subjected to superpixel segmentation and clustering, so that image segmentation is completed. The method makes full use of spatial position information contained in the depth image registered with the color image, so that objects which are difficult to distinguish in a two-dimensional color space are distinguished through the position information in a three-dimensional space, targets are identified from the background, and the quality and the precision of image segmentation are improved.
Application effect example:
as shown in fig. 2, the colors of the scene and the background are similar in the figure (as shown in fig. 2(a)), the contrast of the edge features is also low, and the segmentation is difficult by using the traditional RGB image segmentation method, and by using the method of the present invention, the segmentation is performed by comprehensively using the information of the color and the depth (as shown in fig. 2(b)), and the result is shown in fig. 2(c), wherein each part is clearly segmented.
The foregoing is merely an illustrative embodiment of the present invention, and any equivalent changes and modifications made by those skilled in the art without departing from the spirit and principle of the present invention should fall within the protection scope of the present invention.
Claims (9)
1. An image segmentation method for fusing depth and color information mainly comprises the following steps:
step 1, RGB-D data preprocessing: carrying out median filtering by taking the image edge information as a guide to enhance the quality of the depth image;
step 2, RGB-D image superpixel segmentation: fusing color and depth information, and performing over-segmentation on the image;
and 3, super-pixel combination: similar superpixels are merged by adopting a spectral clustering method based on graph theory, clustering is converted into the problem of graph division, and the image segmentation is completed.
2. The image segmentation method according to claim 1, wherein the method of step 1 comprises:
respectively carrying out edge extraction on the color image and the depth image to generate respective edge images;
and (3) carrying out fusion processing on the color edge image and the depth edge image: on the depth edge image, calculating the gradient direction of each pixel, if the edge direction of the corresponding color edge image is similar to the direction, using the color image edge, otherwise, using the depth image edge, thus obtaining the fused edge image information;
on the depth image, performing median filtering on each effective pixel by adopting a template with the size of 3 multiplied by 3;
color space transformation: and converting the color image acquired by the camera from an RGB color space to a CIELab color space.
3. The image segmentation method according to claim 1, wherein the step 2 comprises the steps of:
2.1 three-dimensional point cloud reconstruction based on depth information: converting pixel coordinates (X, Y) in the depth map into three-dimensional space coordinates (X, Y, Z) based on depth information in the depth map and intrinsic parameters of the camera;
2.2 superpixel Pre-segmentation and initialization of clustering centers: assuming that the image has N pixels, the image is divided into K superpixel blocks, each superpixel block has N/K pixels, and the space between each superpixel block isSetting an initial clustering center at the center of the superpixel block;
2.3 region clustering labeling: in the 2S multiplied by 2S neighborhood range of each super pixel clustering center, calculating the distance between each effective pixel and the clustering center in the 8-dimensional feature space [ L, a, b, X, Y, X, Y, Z ]:
D＝DLab+αDxy+βDXYZ
where subscript c denotes the cluster center of each superpixel, subscript i denotes each valid pixel within the search range, DLab，Dxy，DXYZRespectively representing Euclidean distances between an effective pixel i and a clustering central point on a color feature Lab, an image two-dimensional coordinate xy and a depth-based three-dimensional space coordinate XYZ, wherein alpha and beta are weights used for balancing the importance degree of each item, and D is a final feature distance set according to specific scene data;
dividing each pixel into super pixels to which the cluster center with the minimum characteristic distance D belongs, namely marking the super pixels as the same class as the cluster center;
2.4 updating the clustering center: after all the effective pixels are marked, updating the clustering center according to the pixel classification result, namely counting the number of pixels contained in each super-pixel block and calculating the average value of the number of pixels to obtain a new clustering center;
2.5 iterative clustering: and repeating the steps of 2.3 and 2.4 until the clustering is stable, namely the clustering center and the pixel mark are not changed any more, so that the super-pixel segmentation is completed.
4. The image segmentation method according to claim 3, wherein the step 3 comprises the steps of:
3.1 similarity matrix construction: and taking each super pixel as a node of the graph, and taking the similarity between the super pixels as a weight of an edge to construct an undirected graph of all the super pixels. Assuming that the number of superpixels is K, the similarity matrix W of the graph belongs to RK×KIs a symmetric array of each element wpqFor the degree of similarity between a superpixel p and a superpixel q (p 1., K, q 1., K), the color similarity, texture similarity, and three-dimensional spatial connectivity are used for measurement:
wpq＝Dcolor(p，q)+Dtexture(p，q)+γDspace
wherein D iscolor(p, q) represents color similarity, Dtexture(p, q) represents texture similarity, DspaceRepresenting three-dimensional space connectivity, wherein gamma is a weight factor, and adjusting the weight of the three-dimensional space connectivity according to the quality of depth data, and the three-dimensional space refers to the three-dimensional space reconstructed based on the depth information;
and 3.2, performing spectral clustering on the superpixels according to the constructed similarity matrix, and combining the superpixels with high similarity to finish image segmentation.
5. The image segmentation method according to claim 2, wherein the median filtering method is:
let D (x, y) be the current valid pixel depth value, ND(x, y) is its 8 neighbors, then:
D(x,y)＝median(ND(x,y))
if no edge pixel exists in the neighborhood of the current pixel 8, calculating according to the formula; if the edge pixel exists, median filtering is carried out according to the neighborhood pixel on one side of the edge where the current pixel is located.
6. The image segmentation method according to claim 3, wherein the specific method for converting the one-dimensional information of the depth map into three-dimensional space coordinates is
Wherein (x, y) is the image pixel coordinate of a certain point in the depth image, d is the depth value of the point in the depth image, fx、fyFocal lengths of the depth camera in the x and y directions, respectively, (c)x，cy) As principal point coordinates, fx，fy，cx,cyGiven by the camera manufacturer, (X, Y, Z) is the three-dimensional spatial coordinates of the point.
7. The image segmentation method according to claim 4, wherein the color similarity is calculated by:
on the RGB color space, for the super pixel p, respectively counting the normalized color histogram H on the three channels of RGBp,R,Hp,G,Hp,BThe same applies to superpixel q; computing Bhattacharyya coefficients between the superpixel p and q three channel color histograms:
where i is the normalized pixel value;
then the sum of the Bhattacharyya coefficients between the three channel histograms of RGB is the color similarity:
Dcolor(p，q)＝BhR(p，q)+BhG(p，q)+BhB(p，q)。
8. the image segmentation method according to claim 4, wherein the texture similarity is calculated by:
in Lab color space, an L channel value is selected to describe a Local Binary Pattern (LBP) histogram of a super pixel, and a Bhattacharyya coefficient is adopted to measure the texture similarity D between the super pixels p and qtexture(p，q)。
9. The image segmentation method according to claim 4, wherein the three-dimensional space connectivity is calculated by:
adopting a Gaussian kernel function, wherein sigma is a scale factor, setting according to an actual scene, and XYZ is a three-dimensional space coordinate of a superpixel clustering center, so that the three-dimensional space connectivity between the points p and q
。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910909933.5A CN110610505A (en) | 2019-09-25 | 2019-09-25 | Image segmentation method fusing depth and color information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910909933.5A CN110610505A (en) | 2019-09-25 | 2019-09-25 | Image segmentation method fusing depth and color information |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110610505A true CN110610505A (en) | 2019-12-24 |
Family
ID=68893433
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910909933.5A Pending CN110610505A (en) | 2019-09-25 | 2019-09-25 | Image segmentation method fusing depth and color information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110610505A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111079713A (en) * | 2019-12-31 | 2020-04-28 | 帷幄匠心科技（杭州）有限公司 | Method for extracting pedestrian color features and terminal equipment |
| CN111190981A (en) * | 2019-12-25 | 2020-05-22 | 中国科学院上海微系统与信息技术研究所 | Method and device for constructing three-dimensional semantic map, electronic equipment and storage medium |
| CN111709483A (en) * | 2020-06-18 | 2020-09-25 | 山东财经大学 | Multi-feature-based super-pixel clustering method and equipment |
| CN112183378A (en) * | 2020-09-29 | 2021-01-05 | 北京深睿博联科技有限责任公司 | Road slope estimation method and device based on color and depth image |
| CN112364730A (en) * | 2020-10-29 | 2021-02-12 | 济南大学 | Hyperspectral ground object automatic classification method and system based on sparse subspace clustering |
| CN112785608A (en) * | 2021-02-09 | 2021-05-11 | 哈尔滨理工大学 | Medical image segmentation method for improving SNIC (single noise integrated circuit) based on adaptive parameters |
| CN112880563A (en) * | 2021-01-22 | 2021-06-01 | 北京航空航天大学 | Single-dimensional pixel combination mode equivalent narrow-area-array camera spatial position measuring method |
| CN112991238A (en) * | 2021-02-22 | 2021-06-18 | 上海市第四人民医院 | Texture and color mixing type food image segmentation method, system, medium and terminal |
| CN113199479A (en) * | 2021-05-11 | 2021-08-03 | 梅卡曼德（北京）机器人科技有限公司 | Trajectory generation method and apparatus, electronic device, storage medium, and 3D camera |
| CN113542142A (en) * | 2020-04-14 | 2021-10-22 | 中国移动通信集团浙江有限公司 | Portrait anti-counterfeiting detection method and device and computing equipment |
| WO2021228194A1 (en) * | 2020-05-15 | 2021-11-18 | 上海非夕机器人科技有限公司 | Cable detection method, robot and storage device |
| CN113689549A (en) * | 2021-08-03 | 2021-11-23 | 长沙宏达威爱信息科技有限公司 | Modeling method and digital design system |
| CN116778095A (en) * | 2023-08-22 | 2023-09-19 | 苏州海赛人工智能有限公司 | Three-dimensional reconstruction method based on artificial intelligence |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103455984A (en) * | 2013-09-02 | 2013-12-18 | 清华大学深圳研究生院 | Method and device for acquiring Kinect depth image |
| CN104574375A (en) * | 2014-12-23 | 2015-04-29 | 浙江大学 | Image significance detection method combining color and depth information |
| CN105469369A (en) * | 2015-11-27 | 2016-04-06 | 中国科学院自动化研究所 | Digital image filtering method and system based on segmented image |
| CN106997591A (en) * | 2017-03-21 | 2017-08-01 | 南京理工大学 | A kind of super voxel dividing method of RGB D image mutative scales |
| CN108154104A (en) * | 2017-12-21 | 2018-06-12 | 北京工业大学 | A kind of estimation method of human posture based on depth image super-pixel union feature |
-
2019
- 2019-09-25 CN CN201910909933.5A patent/CN110610505A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103455984A (en) * | 2013-09-02 | 2013-12-18 | 清华大学深圳研究生院 | Method and device for acquiring Kinect depth image |
| CN104574375A (en) * | 2014-12-23 | 2015-04-29 | 浙江大学 | Image significance detection method combining color and depth information |
| CN105469369A (en) * | 2015-11-27 | 2016-04-06 | 中国科学院自动化研究所 | Digital image filtering method and system based on segmented image |
| CN106997591A (en) * | 2017-03-21 | 2017-08-01 | 南京理工大学 | A kind of super voxel dividing method of RGB D image mutative scales |
| CN108154104A (en) * | 2017-12-21 | 2018-06-12 | 北京工业大学 | A kind of estimation method of human posture based on depth image super-pixel union feature |
Non-Patent Citations (3)
| Title |
|---|
| 杨龙等: "基于超像素分割的红外图像细节增强算法", 《红外》 * |
| 涂淑琴等: "RGB-D图像分类方法研究综述", 《激光与光电子学进展》 * |
| 赵轩等: "RGB-D图像中的分步超像素聚合和多模态融合目标检测", 《中国图象图形学报》 * |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111190981A (en) * | 2019-12-25 | 2020-05-22 | 中国科学院上海微系统与信息技术研究所 | Method and device for constructing three-dimensional semantic map, electronic equipment and storage medium |
| CN111190981B (en) * | 2019-12-25 | 2020-11-06 | 中国科学院上海微系统与信息技术研究所 | Method and device for constructing three-dimensional semantic map, electronic equipment and storage medium |
| CN111079713A (en) * | 2019-12-31 | 2020-04-28 | 帷幄匠心科技（杭州）有限公司 | Method for extracting pedestrian color features and terminal equipment |
| CN113542142A (en) * | 2020-04-14 | 2021-10-22 | 中国移动通信集团浙江有限公司 | Portrait anti-counterfeiting detection method and device and computing equipment |
| CN113542142B (en) * | 2020-04-14 | 2024-03-22 | 中国移动通信集团浙江有限公司 | Portrait anti-fake detection method and device and computing equipment |
| WO2021228194A1 (en) * | 2020-05-15 | 2021-11-18 | 上海非夕机器人科技有限公司 | Cable detection method, robot and storage device |
| CN111709483A (en) * | 2020-06-18 | 2020-09-25 | 山东财经大学 | Multi-feature-based super-pixel clustering method and equipment |
| CN112183378A (en) * | 2020-09-29 | 2021-01-05 | 北京深睿博联科技有限责任公司 | Road slope estimation method and device based on color and depth image |
| CN112364730A (en) * | 2020-10-29 | 2021-02-12 | 济南大学 | Hyperspectral ground object automatic classification method and system based on sparse subspace clustering |
| CN112364730B (en) * | 2020-10-29 | 2023-01-17 | 济南大学 | Hyperspectral ground object automatic classification method and system based on sparse subspace clustering |
| CN112880563A (en) * | 2021-01-22 | 2021-06-01 | 北京航空航天大学 | Single-dimensional pixel combination mode equivalent narrow-area-array camera spatial position measuring method |
| CN112880563B (en) * | 2021-01-22 | 2021-12-28 | 北京航空航天大学 | Single-dimensional pixel combination mode equivalent narrow-area-array camera spatial position measuring method |
| CN112785608A (en) * | 2021-02-09 | 2021-05-11 | 哈尔滨理工大学 | Medical image segmentation method for improving SNIC (single noise integrated circuit) based on adaptive parameters |
| CN112991238A (en) * | 2021-02-22 | 2021-06-18 | 上海市第四人民医院 | Texture and color mixing type food image segmentation method, system, medium and terminal |
| CN112991238B (en) * | 2021-02-22 | 2023-08-22 | 上海市第四人民医院 | Food image segmentation method, system and medium based on texture and color mixing |
| CN113199479A (en) * | 2021-05-11 | 2021-08-03 | 梅卡曼德（北京）机器人科技有限公司 | Trajectory generation method and apparatus, electronic device, storage medium, and 3D camera |
| CN113689549A (en) * | 2021-08-03 | 2021-11-23 | 长沙宏达威爱信息科技有限公司 | Modeling method and digital design system |
| CN113689549B (en) * | 2021-08-03 | 2024-04-09 | 长沙宏达威爱信息科技有限公司 | Modeling method and digital design system |
| CN116778095A (en) * | 2023-08-22 | 2023-09-19 | 苏州海赛人工智能有限公司 | Three-dimensional reconstruction method based on artificial intelligence |
| CN116778095B (en) * | 2023-08-22 | 2023-10-27 | 苏州海赛人工智能有限公司 | Three-dimensional reconstruction method based on artificial intelligence |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110610505A (en) | Image segmentation method fusing depth and color information | |
| Hughes et al. | A deep learning framework for matching of SAR and optical imagery | |
| US11727661B2 (en) | Method and system for determining at least one property related to at least part of a real environment | |
| CN109872397B (en) | Three-dimensional reconstruction method of airplane parts based on multi-view stereo vision | |
| CN109655019B (en) | Cargo volume measurement method based on deep learning and three-dimensional reconstruction | |
| CN109146948B (en) | Crop growth phenotype parameter quantification and yield correlation analysis method based on vision | |
| Park et al. | Color image segmentation based on 3-D clustering: morphological approach | |
| CN108052942B (en) | Visual image recognition method for aircraft flight attitude | |
| CN109584281B (en) | Overlapping particle layering counting method based on color image and depth image | |
| CN107369158B (en) | Indoor scene layout estimation and target area extraction method based on RGB-D image | |
| Urban et al. | Finding a good feature detector-descriptor combination for the 2D keypoint-based registration of TLS point clouds | |
| CN111046843A (en) | Monocular distance measurement method under intelligent driving environment | |
| CN110751097A (en) | Semi-supervised three-dimensional point cloud gesture key point detection method | |
| CN116452852A (en) | Automatic generation method of high-precision vector map | |
| CN106709432B (en) | Human head detection counting method based on binocular stereo vision | |
| Schulz et al. | Object-class segmentation using deep convolutional neural networks | |
| Omidalizarandi et al. | Segmentation and classification of point clouds from dense aerial image matching | |
| Zhang et al. | An enhanced multi-view vertical line locus matching algorithm of object space ground primitives based on positioning consistency for aerial and space images | |
| Kim et al. | Multi-view object extraction with fractional boundaries | |
| Novacheva | Building roof reconstruction from LiDAR data and aerial images through plane extraction and colour edge detection | |
| CN115880371A (en) | Method for positioning center of reflective target under infrared visual angle | |
| Deng et al. | Texture edge-guided depth recovery for structured light-based depth sensor | |
| CN111160300B (en) | Deep learning hyperspectral image saliency detection algorithm combined with global prior | |
| Wang et al. | Rgb-guided depth map recovery by two-stage coarse-to-fine dense crf models | |
| Wang et al. | Fast and accurate satellite multi-view stereo using edge-aware interpolation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191224 |
|
| RJ01 | Rejection of invention patent application after publication |