CN109741358B

CN109741358B - Superpixel segmentation method based on adaptive hypergraph learning

Info

Publication number: CN109741358B
Application number: CN201811634332.XA
Authority: CN
Inventors: 王少帆; 林菁; 王立春; 孔德慧; 尹宝才
Original assignee: Beijing University of Technology
Current assignee: Beijing University of Technology
Priority date: 2018-12-29
Filing date: 2018-12-29
Publication date: 2020-11-06
Anticipated expiration: 2038-12-29
Also published as: CN109741358A

Abstract

The invention relates to a superpixel segmentation method based on self-adaptive hypergraph learning, which belongs to an image segmentation technology in the field of computer vision and comprises the following steps of: the method comprises the following steps: image preprocessing, namely downsampling an input RGB image; step two: carrying out image representation based on a hypergraph on the image obtained after the preprocessing; step three: carrying out hypergraph segmentation based on hypergraph learning on the constructed hypergraph; step four: and performing post-processing on the segmentation result corresponding to the input image, wherein the post-processing comprises two parts of connectivity enhancement and superpixel boundary optimization. The invention introduces the idea of hypergraph into superpixel segmentation for the first time so as to encode the high-order relation between pixels; the super edge is constructed by using a self-adaptive neighbor method, so that the connection strength between the vertex and the super edge can be accurately described; and optimizing the super-pixel boundary by combining a Canny operator to ensure that the boundary recall rate is further improved.

Description

Superpixel segmentation method based on adaptive hypergraph learning

The technical field is as follows:

the invention belongs to an image segmentation technology in the field of computer vision.

Background art:

the concept of super-pixel was first proposed by Xiaofeng Ren in 2003 and refers to a set of pixels with certain visual significance, which are composed of adjacent pixels with similar texture, color, brightness and other characteristics. Superpixel segmentation is one of the most important tasks in the field of computer vision, and in recent years, various superpixel segmentation methods have been proposed, which are roughly classified into two categories, i.e., graph theory-based methods and gradient-based methods.

(1) Super-pixel segmentation method based on graph theory

The main idea of these methods is to treat the image as an undirected weighted graph. Each pixel corresponds to a point in the graph, the differences or similarities between pixel features correspond to weights on the edges, and then superpixel segmentation is performed on the graph using various segmentation algorithms.

The graph-based method is firstly proposed by Felzenzwalb, aiming at enabling elements in the same region to be similar as much as possible and elements in different regions to be dissimilar as much as possible, and is mainly realized by clustering nodes on a graph, and the generated superpixel is the minimum spanning tree of a pixel set, and the method has the defect that the quantity and compactness of the superpixel cannot be controlled. The normalized cutting method uses a normalized cutting algorithm, simultaneously measures the difference between regions and the similarity in the regions, and has the advantage that the shape of the segmented superpixel is relatively regular and compact. However, for large images, the computational complexity is relatively high. The super-pixel grid describes a greedy algorithm capable of maintaining an image topological structure, the input of the greedy algorithm is a boundary graph of an image, the optimal path is searched in the vertical and horizontal strip directions, the image is continuously divided into two parts from the vertical and horizontal directions to obtain super-pixels, good performance is maintained in speed and segmentation accuracy, and the shortcoming is that the greedy algorithm is excessively dependent on the boundary graph of the image. The entropy rate is favorable for forming a cluster with compact and uniform structure, so that the obtained superpixels only cover a single target object in the graph, and the balance item is used for ensuring that the cluster has similar size and reducing the number of unbalanced superpixels.

(2) Gradient-based superpixel segmentation method

These methods all start from initial rough clustering, and continuously update clustering by a gradient descent method until convergence, and representative algorithms include a watershed method, a mean shift method, a simple linear iterative clustering method and the like.

The watershed method describes an image with a topological topographic map, when applied to image segmentation, the gray value of each pixel in the image represents the altitude of the point, each local minimum and its area of influence are the watershed basin, and the boundary of the watershed basin forms the watershed. The watershed segmentation can obtain thousands of water collecting basins, and the result is fine, so that the image has a very serious over-segmentation phenomenon. The mean shift-based method is to cluster pixels with the same mode point into the same superpixel, and the method has better stability and noise immunity in practical application but is slow in speed. Among the algorithms, the best known is a simple linear iterative clustering method, which generates superpixels by using a K-means method, converts color images into five-dimensional feature vectors in CIELAB color space and XY coordinates, then constructs distance measurement, and finally completes a clustering process based on the distance measurement.

For different computer vision applications, the generated superpixels need to meet specific requirements, and although it is difficult to define general criteria for satisfying all superpixel segmentation algorithms, the superpixels typically need to meet the following criteria: the boundary of the super pixel is matched with the boundary of the object; one superpixel should belong to only one object; ③ the super-pixels should have similar size and regular shape. The previously proposed methods generally can only well satisfy one or two of the above conditions, and it is difficult to satisfy the above three conditions simultaneously.

The invention content is as follows:

the invention aims to solve the three problems: the superpixel boundary is matched with the object boundary, one superpixel is ensured to belong to only one object, and the segmented superpixels have similar sizes and regular shapes, so that the superpixel segmentation method based on the self-adaptive superpixel learning is provided. FIG. 1 shows a flow chart of a superpixel segmentation method based on adaptive hypergraph learning.

The invention is realized by the following technical scheme:

the method comprises the following steps: image pre-processing

The input is an RGB image, because the number of pixels of the image is large, if the hypergraph is constructed directly, the constructed hypergraph is too large, and the problems of insufficient memory, low operation efficiency and the like are caused, so that the image is firstly downsampled to 1/9 size of the original image, and then the subsequent calculation is carried out. By doing so, not only can efficiency be improved, but also the effect of noise reduction is achieved.

Step two: hypergraph-based image representation

Carrying out image representation based on a hypergraph on an image obtained after preprocessing, wherein the image obtained after preprocessing comprises N pixels, each pixel corresponds to a vertex in the hypergraph, a corresponding hyperedge is constructed in the hypergraph by taking each vertex as a central point c, namely each hyperedge corresponds to one central point c, each hyperedge can be connected with two or more vertices, and N hyperedges are in total; using correlation matrix H_N*N＝(h_v,e) Representing a hypergraph, where v represents a vertex in the hypergraph and e represents a hypergraph in the hypergraphEdge, h_v,eThe connection strength of the vertex v and the super edge e is represented, and the calculation method is as follows:

wherein d is_vcThe distance between the vertex v and the center point c corresponding to the excess edge is represented by the following specific calculation method:

wherein,

[l,a,b,x,y]^Trepresenting the pixel characteristics, [ l, a, b ] in the pre-processed image]^TDenotes the color of the pixel in CIELAB space, [ x, y ]]^TIndicating the position of the pixel, the indices v, c indicating the vertex v, the centre point c, N_colRepresents the maximum color distance and has a value range of [1, 40 ]]，

Representing the maximum position distance, N representing the total number of the image pixels obtained after preprocessing, and K representing the number of the preset super pixels; gamma ray_cFor the regularization parameter, η is the Lagrangian multiplier, calculated by the following formula:

wherein each super edge is connected with a plurality of vertexes,

representing the sum of the distances from all t vertexes connected with the super edge to the center point c corresponding to the super edge, wherein the vertexes connected with the super edge are t vertexes nearest to the center point cPoint composition, vertex t +1 represents the t +1 th vertex closest to the center point c, so d_c,t+1Representing the distance between the center point c and the vertex t +1, the calculation method thereof and d_vcConsistently, t has a value in the range of [10,60 ]]；

In order to improve the calculation efficiency and obtain better segmentation results, we only focus on the local range of the center point c, that is, only the t vertices closest to the center point c can be connected to the center point c, instead of all the vertices being connected to the center point c with a certain probability.

Step three, superpixel segmentation based on hypergraph learning

Carrying out hypergraph segmentation based on hypergraph learning on the constructed hypergraph; first, a hypergraph laplacian matrix is calculated by the following method:

wherein, H represents the correlation matrix corresponding to the hypergraph, I represents the unit matrix, D_VIs a diagonal matrix with elements d on its diagonal_vDegree of vertex v, D_EAlso diagonal matrices, the elements of which on the diagonal_eFor the degree of the excess edge e, the calculation method is as follows:

wherein V represents the vertex set of the hypergraph, E represents the super edge set of the hypergraph,

when the vertex v belongs to the hyper-edge e, q_v,eWhen the vertex v does not belong to the super edge e, q is 1_v,e0; then, the eigenvectors corresponding to the first K minimum eigenvalues are obtained according to the calculated Laplace matrix, and a matrix C ═ C is formed₁,c₂,...,c_K]For each row of matrix C (y)_i)_i＝1,...,NAnd if the ith row of the matrix C belongs to the jth cluster, the ith pixel of the preprocessed image belongs to the jth super pixel, the preprocessed image is subjected to super pixel segmentation according to the steps, and then a segmentation result is converted into a segmentation result corresponding to the input image by using a nearest neighbor interpolation method.

Step four: image post-processing

And performing post-processing on the segmentation result corresponding to the input image, wherein the post-processing comprises two parts of connectivity enhancement and superpixel boundary optimization.

After the super-pixel segmentation is completed through the three steps, the segmented super-pixels are likely to have the problems that the size is too small, a single super-pixel is segmented into a plurality of discontinuous super-pixels and the like, and the problems are solved by enhancing the connectivity.

Due to the fact that the nearest neighbor interpolation method can cause a segmentation result to be jagged, segmentation quality is affected, in the method, a Canny operator is used for optimizing a boundary, and the specific method comprises the following steps: and after the connectivity of the segmentation result is enhanced, obtaining a first boundary graph, simultaneously obtaining a second boundary graph corresponding to the input image by using a Canny operator, comparing the two boundary graphs, and correcting a certain pixel in the first boundary graph into a boundary pixel until all pixels are traversed if the pixel in the second boundary graph is a boundary pixel and the pixel in the first boundary graph is not a boundary pixel in the neighborhood of the ideal superpixel 1/9.

Advantageous effects

The invention introduces the idea of hypergraph into superpixel segmentation for the first time so as to encode the high-order relation between pixels; the super edge is constructed by using a self-adaptive neighbor method, so that the connection strength between the vertex and the super edge can be accurately described; and optimizing the super-pixel boundary by combining a Canny operator to ensure that the boundary recall rate is further improved.

Drawings

FIG. 1: a superpixel segmentation flow chart;

FIG. 2: a hypergraph schematic diagram;

FIG. 3: a correlation matrix H is shown schematically;

FIG. 4: a boundary optimization schematic diagram;

FIG. 5: an original input image;

FIG. 6: dividing the label blocks;

fig. 7 (a): effect diagram of the invention (50 blocks);

fig. 7 (b): a simple linear iterative clustering effect graph;

FIG. 7(c) is a diagram showing effects of the content adaptation method;

fig. 8 (a): effect diagram of the invention (100 blocks);

fig. 8 (b): a simple linear iterative clustering effect graph;

fig. 8 (c): a content adaptive method effect graph;

fig. 9 (a): effect graph of the invention (200 blocks);

fig. 9 (b): a simple linear iterative clustering effect graph;

fig. 9 (c): a content adaptive method effect graph;

FIG. 10: a boundary recall ratio comparison graph;

FIG. 11: segmenting an insufficient error comparison graph;

FIG. 12: a segmentation accuracy comparison map can be achieved.

The specific implementation mode is as follows:

the input images in this embodiment are images in a training set of a berkeley segmented data set (BSDS500), each image having a size of 321 × 481 or 481 × 321, and carrying manually labeled segmented label blocks. Firstly, image preprocessing is carried out, namely, downsampling is carried out on an input RGB image; the pixels in the preprocessed image correspond to vertexes in the hypergraph, each vertex in the hypergraph is taken as a central point, the self-adaptive neighbors are utilized to construct a hyperedge, and the hyperedge is constructed by traversing all vertexes; then, performing superpixel segmentation based on hypergraph learning; and optimizing the boundary by using a Canny operator for the obtained segmentation result, and finally outputting a superpixel segmentation result.

The specific implementation process is as follows:

the method comprises the following steps: image preprocessing, namely, downsampling an input RGB image to 1/9 size of an original image, setting the original image size to 481 × 321, and setting the downsampled image size to 160 × 107;

step two: the image obtained after the preprocessing is represented by a hypergraph image, and the preprocessed image is divided into N17120 pixels and K100 superpixels, and the maximum position distance is assumed

Maximum color distance N_col15, one super edge can be connected with t, 50 vertexes, so as to calculate the connection strength h of the vertex v and the super edge e_v,eSo as to obtain a correlation matrix H of the hypergraph; as shown in FIG. 2, for simplicity, it is assumed that the picture has 6 pixels, the vertex 1 is used as the central point to construct the super edge 1, the super edge is connected with three vertices 1, 4 and 5, the vertex 5 is used as the central point to construct the super edge 5, the super edge is connected with three vertices 2, 5 and 6, for simplicity and clarity of the diagram, only three super edges are drawn in the diagram, actually, the super edges are constructed by using each vertex in the diagram as the central point in sequence to form six super edges, so the correlation matrix H is used_6*6＝(h_v,e) Representing a hypergraph, where v represents the vertices of the hypergraph, e represents the hyper-edges of the hypergraph, h_v,eRepresenting the strength of the connection of vertex v to supercide e, as shown in fig. 3, each column of the matrix represents a supercide and each row represents a vertex.

Step three: carrying out hypergraph segmentation based on hypergraph learning on the constructed hypergraph; firstly, a hypergraph Laplace matrix L is calculated, and then eigenvectors corresponding to the first K-100 minimum eigenvalues are obtained according to the calculated Laplace matrix L to form a matrix C-C₁,c₂,...,c₁₀₀]For each row of matrix C (y)_i)_{i＝1,...,17120}The total number of lines is 17120, which correspond to the pre-processed graphs17120 pixels in the image are clustered by using a K-means algorithm to form K which is 100 clusters, and if the 1 st row of the matrix C belongs to the 2 nd cluster, the 1 st pixel of the image belongs to the 2 nd super-pixel; according to the segmentation result obtained in the above steps, the segmentation result is the result corresponding to the image with the size of 160 × 107 after down sampling, and then the segmentation result is converted into the segmentation result corresponding to the input image with the size of 481 × 321 by using the nearest neighbor interpolation method;

step four: traversing the segmentation result graph obtained in the third step according to the Z-shaped trend, and redistributing discontinuous superpixels and superpixels with the size less than 386 pixels to superpixels with the nearest color in the superpixels at the position; for superpixel boundary optimization, the specific method is as follows: and after the connectivity of the segmentation result is enhanced, obtaining a first boundary graph, obtaining a second boundary graph corresponding to the input image by using a Canny operator, comparing the two boundary graphs, and in the neighborhood of 170 pixels, if a certain pixel in the second boundary graph is a boundary pixel and the pixel in the first boundary graph is not a boundary pixel, modifying the pixel in the first boundary graph into a boundary pixel until all pixels are traversed, wherein the diagram 4 is a boundary optimization diagram.

The method is experimentally verified, and obvious effects are achieved. One of the purposes of introducing superpixels is to improve efficiency, and the division into too many superpixels is meaningless in practical application. Therefore, we segment the image into 50, 100, 200 blocks, i.e. set K50, 100, 200, and compare it with the simple linear iterative clustering method and the content adaptive superpixel segmentation method for qualitative and quantitative evaluation. Fig. 5 is the original input image in the dataset and fig. 6 is the split label block.

(1) Qualitative assessment

Fig. 7 is a result graph of dividing 50 super pixels, fig. 8 is a result graph of dividing 100 super pixels, and fig. 9 is a result graph of dividing 200 super pixels (a, b, c are an adaptive super map learning method (proposed in the present patent), a simple linear iterative clustering method, and a content adaptive super pixel division method, in this order). As can be seen from the figure, the segmentation algorithm proposed in this patent can better fit the object boundary, and as the number of segmented blocks is larger, the boundary processing is better, and the segmentation is finer.

(2) Quantitative assessment

In the experiment, 3 evaluation criteria are adopted to evaluate the superpixel Segmentation algorithm, namely Boundary Recall (BR), insufficient Segmentation Error (UE) and reachable Segmentation Accuracy (ASA), and the s is used_i(i ═ 1,2,3.., m) denotes the ith block of superpixels, g_j(j ═ 1,2,3, ·, n) denotes the jth split tag block, | | | denotes the size of the pixel set, | | | |. | denotes the euclidean distance,

represents an indication function, and satisfies the condition in parentheses of 1 and does not satisfy the condition of 0.

The boundary recall ratio (BR) evaluates the coincidence degree of the boundary of the superpixel segmented by the algorithm and the boundary of the segmented label block, and is calculated by the following formula:

wherein, B(s) and B (g) represent the pixel set of the super pixel boundary and the boundary pixel set of the segmentation label block, respectively. Indicating function

To check if the distance of the nearest pixels in B(s) and B (g), respectively, is within σ, we set σ to 2 in the experiment.

Under-segmentation error (UE) is another evaluation of the degree of boundary conformance, which ensures that a superpixel belongs to only one object, and that if a superpixel overlaps more than one label tile, the UE will grow.

The Achievable Segmentation Accuracy (ASA) represents the ratio of the number of pixels correctly segmented to the total number of all pixels. The higher the ASA value, the more accurate the segmentation is represented.

In the experiment, the self-adaptive hypergraph learning method, the simple linear iterative clustering method and the content self-adaptive method which are proposed by the patent are quantitatively compared according to the three standards. Fig. 10, 11, and 12 are graphs comparing experimental results of Boundary Recall (BR), insufficient segmentation error (UE), and Achievable Segmentation Accuracy (ASA), respectively, where a larger value for the boundary recall indicates a higher boundary goodness of fit; the higher the achievable segmentation precision value is, the more accurate the segmentation is, and the better the segmentation effect is; for under-segmentation errors, the smaller the value, the smaller the error, so the smaller the error, the better. As can be seen from the figure, the segmentation method provided by the patent has certain advantages under three standards.

Claims

1. The superpixel segmentation method based on the adaptive hypergraph learning is characterized by comprising the following steps of:

the method comprises the following steps: image preprocessing, namely downsampling an input RGB image;

step two: performing hypergraph-based image representation on the preprocessed image, specifically comprising: the preprocessed image has N pixels, each pixel corresponds to a vertex in the hypergraph, and corresponding hyperedges are constructed in the hypergraph by taking each vertex as a central point c, namely each hyperedge corresponds to one central point c, each hyperedge can be connected with two or more vertices, and N hyperedges are provided in total; using correlation matrix H_N*N＝(h_v,e) Representing a hypergraph, where v represents the vertices of the hypergraph, e represents the hyper-edges of the hypergraph, h_v,eThe connection strength of the vertex v and the super edge e is represented, and the calculation method is as follows:

wherein,

wherein each super edge is connected with a plurality of vertexes,

represents the sum of the distances from all t vertexes connected with the super edge to the center point c corresponding to the super edge, the vertexes connected with the super edge are composed of t vertexes nearest to the center point c, and the vertex t +1 represents the t +1 th vertex nearest to the center point c, so d_c,t+1Representing the center point c and the topDistance between points t +1, method for calculating the same and d_vcConsistently, t has a value in the range of [10,60 ]]；

Step three: the constructed hypergraph is subjected to hypergraph learning-based superpixel segmentation, and the hypergraph learning-based superpixel segmentation specifically comprises the following contents:

first, a hypergraph laplacian matrix is calculated by the following method:

when the vertex v belongs to the hyper-edge e, q_v,eWhen the vertex v does not belong to the super edge e, q is 1_v,e0; then, the eigenvectors corresponding to the first K minimum eigenvalues are obtained according to the calculated Laplace matrix, and a matrix C ═ C is formed₁,c₂,...,c_K]For each row of matrix C (y)_i)_i＝1,...,NRepresenting that N rows are total, the N rows respectively correspond to N pixels in the preprocessed image, the N pixels are clustered by using a K-means algorithm to form K clusters, if the ith row of the matrix C belongs to the jth cluster, the ith pixel of the preprocessed image belongs to the jth super-pixel, the preprocessed image is subjected to super-pixel segmentation according to the steps, and then nearest neighbor interpolation is used for carrying out super-pixel segmentation on the preprocessed imageConverting the segmentation result into a segmentation result corresponding to the input image by a value method;

step four: performing post-processing on a segmentation result corresponding to an input image, wherein the post-processing comprises two parts of connectivity enhancement and super-pixel boundary optimization, and the post-processing specifically comprises the following steps: according to the segmentation result graph obtained in the step three of traversing the Z-shaped trend, the discontinuous superpixels and the superpixels with the size smaller than the ideal superpixels size 1/4 are redistributed to the superpixels with the nearest color to the nearest neighbor superpixels, wherein the ideal superpixels size is M/K, M represents the total number of the input image pixels, and K represents the number of the preset superpixels; for superpixel boundary optimization, the specific method is as follows: and after the connectivity of the segmentation result is enhanced, obtaining a first boundary graph, simultaneously obtaining a second boundary graph corresponding to the input image by using a Canny operator, comparing the two boundary graphs, and correcting a certain pixel in the first boundary graph into a boundary pixel until all pixels are traversed if the pixel in the second boundary graph is a boundary pixel and the pixel in the first boundary graph is not a boundary pixel in the neighborhood of the ideal superpixel 1/9.