CN104915946B - A kind of object segmentation methods based on conspicuousness suitable for serious degraded image - Google Patents

Abstract

A saliency-based object segmentation method suitable for severely degraded images, including the following steps: (1) generate initialized salient object seeds through saliency maps; (2) generate skeletons based on local autocorrelation; (3) Generate the starting point for expansion on the edge of the object; (4) initialize the direction of expansion for each starting point; (5) set the termination condition of the expansion and complete the expansion operation; (6) mark the starting point where the termination point cannot be found As degraded area markers; (7) Repair and smooth the expanded result; (8) Repair object segmentation results with superpixels according to the degraded area markers. The present invention combines local autocorrelation and superpixels to effectively improve the accuracy and integrity of segmentation results, avoiding the problem of partial loss of segmentation results caused by severe image degradation, and improving the effectiveness of salient object segmentation for image degradation. robustness.

Description

A Saliency-Based Object Segmentation Method for Severely Degraded Images

technical field

The invention relates to technical fields such as computer vision and image processing, especially a saliency-based object segmentation method.

Background technique

Saliency-based object segmentation is a relatively popular research, and its goal is to segment the object of interest in the image. Detecting salient regions in natural images is an imitation of the human visual system, that is, automatically finding regions of interest that humans will focus their gaze on. When people observe a natural image or a real scene, they pay more attention to the whole salient object rather than just a salient region. Therefore, saliency-based object segmentation is a necessary research, which is widely used in many high-level applications, such as object recognition, behavior analysis, object-of-interest segmentation in images, etc. Nevertheless, salient object segmentation methods based on saliency information often fail when the image is severely degraded, especially when the foreground has localized motion blur and the background has uniform motion blur. Degradation phenomena in images can have serious impacts on most computer vision applications. It usually leads to a decrease in the accuracy of the algorithm results, and even causes some algorithms to fail. This problem is common in saliency-based object segmentation for natural images. Salient objects in most degraded images may contain many less salient parts, which cause ambiguity when performing object segmentation. Therefore, the results of saliency-based object segmentation on degraded images are usually incomplete.

Contents of the invention

In order to overcome the problem of partial area loss of object segmentation results caused by severe image degradation, the present invention proposes a saliency-based object segmentation method suitable for severely degraded images, which can effectively improve the accuracy and integrity of segmentation results , which avoids the problem of partial loss of segmentation results caused by severe image degradation, and improves the robustness of salient object segmentation to image degradation.

The technical solution adopted by the present invention to solve its technical problems is:

A saliency-based object segmentation method suitable for severely degraded images, including the following steps:

(1) Salient object seeds initialized by saliency map generation

Find a histogram peak in the range with higher significance in the histogram of the saliency map obtained based on the soft image abstraction method. The range with higher significance is selected as (127,255], and threshold the saliency map through the threshold T Segmentation obtains a binarized segmentation result; mark the connected domains one by one, and use the morphological expansion operation before marking to protect more significant details of the target object, and extract the most important in the already marked disconnected areas The region of is used as the initial salient object seed;

(2) Generate a skeleton based on local autocorrelation

The local autocorrelation equation of a point (x, y) within a local window w centered on this point:

Among them, I(x _k , y _k ) represents the gradient of the point (x _k , y _k ) in the window w(3×3), and Δx and Δy represent the displacements in the x and y directions respectively;

This formula (2.1) is approximated as:

in

Calculate the two eigenvalues of the matrix M obtained from each point, where the eigenvector of the smaller eigenvalue corresponds to the long axis direction of the ellipse, which can be expressed as the extension direction of the point, and convert the value calculated at each point to In the direction space [0,180], each value represents the positive and negative directions on a straight line, and the value on each pixel of the generated motion direction map corresponds to a certain direction;

Normalize the motion direction map to 4 directions {0, 45, 90, 135} by means of average distribution, and the direction with the largest number in the normalized image of the motion direction map will be taken as the direction of the background; then the direction of the background All directions are removed, and the remaining largest connected region is used as a supplementary object seed; two adjacent directions in the background are searched along the background direction to repair the missing part. Linking two neighboring related regions together when they are spatially close to each other;

If the initial salient object seed from step (1) consists of several disconnected regions, it means that the result from step (1) cannot be used to represent the whole skeleton of the target object, in this case, based on The skeleton of local autocorrelation is used to optimize the initial salient object seeds into better salient object seeds. Considering that the skeleton obtained by local autocorrelation is somewhat inflated compared with the target object, a morphological erosion operation is applied to Fix this problem; when the initial object seed consists of several disconnected regions, the skeleton based on local autocorrelation and the initial object seed are fused together as the final salient object seed; otherwise, only the obtained in step (1) The result serves as the final salient object seed;

Holes in salient object seeds need to be filled, except for those holes that account for a percentage of the entire target object above a threshold;

(3) Generate starting points for expansion on the edges of salient object seeds

Each point on the boundary of the salient object seed is taken as the starting point of our extended method, constructing a 3×3 convolution kernel N _s , and let the salient object seed binary map M _b and the convolution kernel N _s is used for convolution operation. When the value of the center point p in the operation window is 1, N _s will be used to calculate a determination factor d:

Among them, p _ij is the binary value of the pixel of the i-th row and j-th column of the salient object seed in the w window, and N _sij represents the value of the i-th row and j-th column in N _s , if n is not equal to 0 or 8, it is determined The factor d is 1, indicating that the values of all points are not the same, and this central point is taken as a starting point on the edge of the salient object seed;

(4) Initialize the direction of expansion for each starting point

Based on the starting point set {P _s }, the direction of each starting point will be calculated. It is assumed that each starting point expands along the normal direction, so it is necessary to calculate the normal direction, divide the expansion direction into 8 types, and use 0 to 7 eight numbers to mark them, a convolution kernel N _d is constructed to determine the normal direction of each starting point, and the formula for calculating the label value of the direction l _d is as follows:

Among them, p _ij is the value of the binary image M _b corresponding to the pixel in the i-th row and j-th column in the convolution window w, Indicates that the pixel value is inverted, if it is 1, it becomes 0, if it is 0, it becomes 1, and p ₁₁ is the value of the pixel in the first row and the first column in the window w, and n ₁ is When p ₁₁ =1, the number of all points satisfying {p _ij =1} except the center point, and n ₂ is the number of all points satisfying {p _ij =1} except the center point when p ₁₁ =0 ;

(5) Set the termination condition of the extension and complete the extension operation

The algorithm of adaptive threshold canny operator is used to detect the boundary of the original image, and a scalable restricted region is specified, which is obtained by three dilations of salient object seeds;

In the restricted area, each point on the boundary has the opportunity to be the termination point of the expansion, and the termination conditions are divided into two categories according to the expansion direction: {0,2,4,6} and {1,3,5,7}, Each type contains several cases;

When a starting point finds its ending point along its normal direction, an operation will be performed to extend the starting point to the corresponding ending point, and the starting point (x, y) and the ending point (x _e , y _e ) points on the extension line are reassigned, and the set of points on the extension line is expressed as the following formula:

{p(x+d _ix Δx,y+d _iy Δy)=1|0≤Δx≤|xx _e |,0≤Δy≤|yy _e |}, (5.1)

Among them, p(x+d _ix Δx,y+d _iy Δy) is the binary value of the point on the extension line, which means that the reassigned point on the extension line in the restricted area will become the point in our final target object; d The value of _i is from the set {(-1,-1),(0,-1),(1,-1),(1,0),(1,1),(0,1),(-1 ,1),(-1,0)}, which corresponds to the 8 direction marker values {0,1,2,3,4,5,6,7} in Figure 3;

(6) Mark the starting point where the end point cannot be found as the degraded area marking point

If a start point does not have an end point in the extended restricted area, mark the start point as a point in the degenerate area, and obtain a set of marked points P ₁ representing the degenerate area;

(7) Repair and smooth the expanded result

Sort these disconnected holes by area, and reassign the points in the holes as points in the target object, except for those holes whose area accounts for more than a percentage of the entire object, and finally, use a Gaussian filter to smooth the results caused by the error caused by rough borders.

(8) Use superpixels to repair object segmentation results according to degraded area marker points

Use the superpixels obtained by the simple linear iterative clustering superpixel segmentation method to patch those degenerated regions that are lost, when the local autocorrelation-based skeleton is combined with the salient object seeds, the process of superpixel-based object fusion is not performed;

Among them, Obj _f represents the final target object, and Obj _e represents the result obtained after expansion; parameter a=0 means that the skeleton based on local autocorrelation is not applied to the salient object seed, and n _li represents the superpixel S _i The number of labeled points, which are obtained from step (6) and belong to the set P _l .

The technical idea of the present invention is to construct a saliency-based object segmentation method using an automatic expansion mechanism, which can be roughly divided into two steps: generation of salient object seeds and segmentation based on salient object seeds. During the generation of salient object seeds, we generate initial salient object seeds based on the results of soft image abstraction methods, and optimize the salient object seeds by incorporating local autocorrelation consistency as the whole skeleton of the target object. Based on the observation that boundaries in degenerated regions are not salient during salient object seed-based segmentation, we propose a novel method called "normal extension". Based on the origins at the salient object edges, we compute the propagation direction to each origin and list the termination conditions. In this way, the salient object seeds are extended to the ground-truth boundaries of the target objects, and those points whose corresponding boundaries cannot be found in the salient object seeds are marked. To inpaint the lost degenerated regions, superpixels are fused into our final object segmentation results according to the densities of labeled points.

The beneficial effects of the present invention are mainly manifested in: using the normal extension method to extend the salient object seed to the real boundary of the target object, combining local autocorrelation and superpixels can effectively improve the accuracy and integrity of the segmentation result, avoiding the The problem of partial area loss in segmentation results caused by severe image degradation improves the robustness of salient object segmentation to image degradation.

Description of drawings

Figure 1 The two eigenvectors of the M matrix obtained according to the local autocorrelation equation are used as the major and minor axes of an ellipse, and the angle formed by the major axis of the ellipse (that is, the eigenvector of the smaller eigenvalue object) and the horizontal line is used as the pixel The extension direction of the point (extended to be the motion direction).

Figure 2 Generating extended onsets on salient object seeds. On the left is the convolution kernel used, and on the right is the process of applying the convolution kernel to a pixel in the salient object seed and its result.

The left side of Figure 3 shows 8 types of extension directions, and the right side uses eight numbers from 0 to 7 to mark these directions.

Specific implementation method

The present invention will be further described below in conjunction with the accompanying drawings.

Referring to Figures 1 to 3, the specific steps of a saliency-based object segmentation method suitable for severely degraded images are as follows:

(1) Salient object seeds initialized by saliency map generation. In order to be able to obtain the whole object by extension, we need a good skeleton of the target object. Typically, salient regions are more likely to be foreground regions. Therefore, we look for a histogram peak in the range with higher saliency in the histogram of the saliency map obtained by the soft image abstraction method. In our method, this range is selected as (127,255], which corresponds to the region with a high degree of saliency. Then, threshold segmentation is performed on the saliency map by the threshold T to obtain a binarized segmentation result. We one by one Mark the connected domain, and use the morphological dilation operation before marking to protect more salient details of the target object. We extract the most important area in the already marked disconnected area as our rough initial salient object seed.

(2) Generate a skeleton based on local autocorrelation. The following formula is the local autocorrelation equation of a point (x, y) within a local window w centered on the point:

Among them, I(x _k , y _k ) represents the gradient of the point (x _k , y _k ) in the window w(3×3), and Δx and Δy represent the displacements in the x and y directions, respectively.

This formula can be approximated as:

in

We compute the two eigenvalues of the matrix M(2×2) obtained from each point. The eigenvector of the smaller eigenvalue corresponds to the major axis direction of the ellipse (as shown in Figure 2), and this direction can be expressed as the extension direction of the point (the direction of motion). We transform the value calculated at each point into the direction space [0,180], so that each value represents both positive and negative directions on a straight line. The value on each pixel of the generated motion direction map corresponds to a certain direction.

We normalize the motion pattern to 4 directions {0, 45, 90, 135} by means of an even distribution. The direction with the largest number in the image normalized by the motion direction map will be taken as the direction of the background. Afterwards, the direction of the background is removed, and the remaining largest connected region is used as a supplementary object seed. Of course, the parts on the object seed that are in the same direction as the background are also lost. We search for two adjacent directions in the background along the background direction to patch the missing parts. Two adjacent related regions are linked together when they are spatially close to each other.

We assume here that if the initial salient object seed from step (1) consists of several disconnected regions, it means that the result from step (1) cannot be used to represent the whole skeleton of the target object. In this case, a skeleton based on local autocorrelation is used to let the initial salient object seeds be optimized into better salient object seeds. Considering that the skeleton obtained by local autocorrelation is somewhat bloated compared to the target object, a morphological erosion operation is applied to correct this problem. When the initial object seed consists of several disconnected regions, we fuse the local autocorrelation based skeleton with the initial object seed as our final salient object seed. Otherwise, just use the result obtained in step (1) as our final salient object seed. Before the next step, the holes in the salient object seeds need to be filled, except for those holes that account for more than 20% of the whole target object (20% is our threshold obtained through extensive experiments).

(3) Generate starting points for extension on the edges of salient objects. By our initialization method, a binary map _Mb of salient object seeds (only values 0 and 1) is generated. We propose a novel method to extend salient object seeds to the ground-truth boundaries of target objects. Every point on the boundary of the salient object seed is taken as a starting point for our extension method. The method of how to establish the starting point set {P _s } will be introduced below.

We construct a 3×3 convolution kernel N _s (as shown in Figure 2), and let the salient object seed map M _b and convolution kernel N _s perform convolution operation. When the value of the center point p in the operating window is 1, N _s is used to calculate a determination factor d.

Among them, p _ij is the binary value of the salient object seed in the i-th row and j-column pixel in the w window, and N _sij represents the value of the i-th row and j-column in N _s . If n is not equal to 0 or 8 (the determination factor d is 1), it means that the values of all points are not the same, so we take this central point as a starting point on the edge of the salient object seed.

(4) Initialize the direction of expansion for each starting point. Based on the set of starting points {P _s }, the direction of each starting point is calculated. We imagine that each starting point extends along the normal direction, so we need to calculate this normal direction. We roughly divide our extension directions into 8 types, and mark them with eight numbers from 0 to 7, as shown in Figure 3. A convolution kernel N _d is constructed to determine the normal direction of each starting point. The formula used to calculate the marker value for direction l _d is as follows:

Where p _ij is the value of the binary image M _b in the convolution window w(3×3) corresponding to the pixel in row i and column j, Indicates that the pixel value is inverted, if it is 1, it becomes 0, if it is 0, it becomes 1, and p ₁₁ is the value of the pixel in the first row and the first column in the window w. n ₁ is when p ₁₁ = 1 all satisfies except the central point , and n ₂ is the number of all points satisfying {p _ij =1} except the central point when p ₁₁ =0.

(5) Set the termination condition of the extension and complete the extension operation. Our method extends the salient object seed to the true boundary of the target object, so a boundary needs to be created to terminate the extension. We have an algorithm called the adaptive threshold canny operator to detect the boundary of the original image and specify a scalable restricted area. This restricted region is obtained by three dilations of salient object seeds.

In a restricted area, every point on the boundary has the opportunity to be the termination point of the extension. According to the extension direction shown in Figure 3, we can divide the termination conditions into two categories: {0,2,4,6} and {1,3,5,7}. Each type contains several cases.

When a start point finds its end point along its normal direction, an operation will be performed to extend the start point to the corresponding end point. We reassign the points on the extension line between the start point (x,y) and the end point (x _e , y _e ). The set of points on the extension line can be expressed as the following formula:

{p(x+d _ix Δx,y+d _iy Δy)=1|0≤Δx≤|xx _e |,0≤Δy≤|yy _e |}, (5.1)

Where p(x+d _ix Δx,y+d _iy Δy) is the binary value of the point on the extension line, which means that the reassigned point on the extension line in the restricted area will become the point in our final target object. The value of d _i can be selected from the set {(-1,-1),(0,-1),(1,-1),(1,0),(1,1),(0,1),( -1,1),(-1,0)}, which corresponds to the 8 direction marker values {0,1,2,3,4,5,6,7} in Figure 3.

(6) Mark the starting point where the end point cannot be found as the degraded area marking point. According to the assumption that the boundary of the degenerate region must be insignificant, if a starting point does not have an end point in the expansion-limited region, we mark the starting point as a point in the degenerate region. Finally, a set of marker points P ₁ representing the degraded area can be obtained.

(7) Repair and smooth the expanded result. Similar to the last step in the process of generating salient object seeds described in step (2), here an operation is required to patch holes in the expanded results. We sort these disconnected holes according to the area, and reassign the points in the holes to the points in the target object, except for those holes whose area accounts for more than 20% of the entire object (the threshold of 20% here is obtained through a large number of experiments of). Finally, we use a Gaussian filter to smooth the rough edges in the result caused by errors.

It must be noted here that the method of initializing the direction in the "normal extension" proposed by us is not applicable to the case of obtaining the normal direction of a straight line. It is only suitable for calculating the normal direction on the boundary of some large regions. And some miscalculations caused by the process of generating the starting point, initializing the direction or calculating the termination condition will not have a great impact on our system, and our system has high robustness.

(8) Use superpixels to repair object segmentation results according to degraded region markers. Based on the assumption that the boundaries in the degenerated regions of the image must be insignificant, we use superpixels obtained by a simple linear iterative clustering superpixel segmentation method to inpaint those degenerated regions that are lost. Considering that a part of the salient object seeds combined with local autocorrelation is larger than the real target object, we do not perform this superpixel-based object fusion procedure when the local autocorrelation based skeleton is combined with the salient object seeds.

Among them, Obj _f represents the final target object, and Obj _e represents the result obtained after expansion. The parameter a = 0 indicates that the skeleton based on local autocorrelation is not applied to the salient object seeds, n _li indicates the number of labeled points in superpixels S _i which are obtained from step (6) and belong to the set _Pl .

The saliency-based object segmentation method applicable to severely degraded images in this embodiment uses a soft image abstraction method to obtain a saliency map for the input image, and then through threshold segmentation, extracts the main connected domain as a rough salient object seed; Local autocorrelation calculates the motion direction of each pixel, as shown in Figure 1, after filtering out the background, the skeleton based on local autocorrelation is obtained; according to the pros and cons of rough salient object seeds, it is decided whether to combine the local autocorrelation based on The final salient object seed point is formed by the characteristic skeleton; the starting point for expansion is found according to the salient object seed, as shown in Figure 2; the normal direction of each starting point is calculated as the expansion direction, and the eight possible expansion directions are as follows: As shown in Figure 3; set the end point for each starting point, find the starting point of the ending point and connect it with the end point; the starting point without finding the ending point is marked as a point in the degenerated area; repair and smooth the expanded result ; Finally, the object segmentation results are patched with superpixels according to the degraded area markers.

Claims (1)

1. a kind of object segmentation methods based on conspicuousness suitable for serious degraded image, it is characterised in that：It is described to be based on showing The object segmentation methods of work property comprise the following steps：

(1) the notable object seed of initialization is generated by notable figure

One is found in the higher scope of significance in the histogram of the notable figure obtained based on soft picture abstraction method directly The peak value of square figure, the higher scope of the significance be chosen for (127,255], row threshold division is entered to notable figure by threshold value T and obtained To the segmentation result of a binaryzation；Connected domain is marked one by one, and using morphological dilation before mark To protect the more significantly details of destination object, topmost region is extracted as first in labeled good not connected region Begin notable object seed；

(2) skeleton based on local autocorrelation is generated

Local autocoorrelation of the point (x, y) in the local window w centered on the point：

<mrow> <mi>f</mi> <mrow> <mo>(</mo> <mi>x</mi> <mo>,</mo> <mi>y</mi> <mo>)</mo> </mrow> <mo>=</mo> <msub> <mi>&amp;Sigma;</mi> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mi>k</mi> </msub> <mo>,</mo> <msub> <mi>y</mi> <mi>k</mi> </msub> <mo>)</mo> <mo>&amp;Element;</mo> <mi>w</mi> </mrow> </msub> <msup> <mrow> <mo>&amp;lsqb;</mo> <mi>I</mi> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mi>k</mi> </msub> <mo>,</mo> <msub> <mi>y</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> <mo>-</mo> <mi>I</mi> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mi>k</mi> </msub> <mo>+</mo> <mi>&amp;Delta;</mi> <mi>x</mi> <mo>,</mo> <msub> <mi>y</mi> <mi>k</mi> </msub> <mo>+</mo> <mi>&amp;Delta;</mi> <mi>y</mi> <mo>)</mo> </mrow> <mo>&amp;rsqb;</mo> </mrow> <mn>2</mn> </msup> <mo>,</mo> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>2.1</mn> <mo>)</mo> </mrow> </mrow>

Wherein, I (x_k,y_k) represent point (x_k,y_k) gradient in window w, Δ x and Δ y are represented in the two directions x and y respectively Displacement；

The formula (2.1) is approximately：

<mrow> <mi>f</mi> <mrow> <mo>(</mo> <mi>x</mi> <mo>,</mo> <mi>y</mi> <mo>)</mo> </mrow> <mo>&amp;cong;</mo> <msub> <mi>&amp;Sigma;</mi> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mi>k</mi> </msub> <mo>,</mo> <msub> <mi>y</mi> <mi>k</mi> </msub> <mo>)</mo> <mo>&amp;Element;</mo> <mi>w</mi> </mrow> </msub> <mo>&amp;lsqb;</mo> <msub> <mi>I</mi> <mi>x</mi> </msub> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mi>k</mi> </msub> <mo>,</mo> <msub> <mi>y</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> <mo>-</mo> <msub> <mi>I</mi> <mi>y</mi> </msub> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mi>k</mi> </msub> <mo>,</mo> <msub> <mi>y</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> <mo>&amp;rsqb;</mo> <msup> <mfenced open = "[" close = "]"> <mtable> <mtr> <mtd> <mi>&amp;Delta;</mi> <mi>x</mi> </mtd> </mtr> <mtr> <mtd> <mi>&amp;Delta;</mi> <mi>y</mi> </mtd> </mtr> </mtable> </mfenced> <mn>2</mn> </msup> <mo>=</mo> <mo>&amp;lsqb;</mo> <mi>&amp;Delta;</mi> <mi>x</mi> <mo>,</mo> <mi>&amp;Delta;</mi> <mi>y</mi> <mo>&amp;rsqb;</mo> <mi>M</mi> <mfenced open = "[" close = "]"> <mtable> <mtr> <mtd> <mi>&amp;Delta;</mi> <mi>x</mi> </mtd> </mtr> <mtr> <mtd> <mi>&amp;Delta;</mi> <mi>y</mi> </mtd> </mtr> </mtable> </mfenced> <mo>,</mo> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>2.2</mn> <mo>)</mo> </mrow> </mrow>

Wherein

<mrow> <mi>M</mi> <mo>=</mo> <msub> <mi>&amp;Sigma;</mi> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mi>k</mi> </msub> <mo>,</mo> <msub> <mi>y</mi> <mi>k</mi> </msub> <mo>)</mo> <mo>&amp;Element;</mo> <mi>w</mi> </mrow> </msub> <mfenced open = "[" close = "]"> <mtable> <mtr> <mtd> <mrow> <msubsup> <mi>I</mi> <mi>x</mi> <mn>2</mn> </msubsup> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mi>k</mi> </msub> <mo>,</mo> <msub> <mi>y</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> </mrow> </mtd> <mtd> <mrow> <msub> <mi>I</mi> <mi>x</mi> </msub> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mi>k</mi> </msub> <mo>,</mo> <msub> <mi>y</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> <msub> <mi>I</mi> <mi>y</mi> </msub> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mi>k</mi> </msub> <mo>,</mo> <msub> <mi>y</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> </mrow> </mtd> </mtr> <mtr> <mtd> <mrow> <msub> <mi>I</mi> <mi>x</mi> </msub> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mi>k</mi> </msub> <mo>,</mo> <msub> <mi>y</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> <msub> <mi>I</mi> <mi>y</mi> </msub> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mi>k</mi> </msub> <mo>,</mo> <msub> <mi>y</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> </mrow> </mtd> <mtd> <mrow> <msubsup> <mi>I</mi> <mi>y</mi> <mn>2</mn> </msubsup> <mrow> <mo>(</mo> <msub> <mi>x</mi> <mi>k</mi> </msub> <mo>,</mo> <msub> <mi>y</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> </mrow> </mtd> </mtr> </mtable> </mfenced> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>2.3</mn> <mo>)</mo> </mrow> </mrow>

The matrix M obtained from each point two characteristic values are calculated, wherein the oval length of the characteristic vector correspondence of smaller characteristic value Direction of principal axis, this direction can be expressed as bearing of trend a little, by each point calculate obtained value be transformed into director space [0, 180] in, each value is represented in positive and negative both direction point-blank, each pixel of direction of motion figure generated Value all corresponds to the direction of some；

Direction of motion figure is normalized into 4 directions { 0,45,90,135 } by way of mean allocation, returned in direction of motion figure The maximum direction of quantity will be taken as the direction of background in image after one change；The direction of background is all removed afterwards, it is remaining Largest connected region is just taken as the object seed of supplement；Two proximal directions are lost to repair along in background direction search background The part of mistake, just connects together them when two neighbouring relevant ranges are spatially near each other；

If obtaining initial significantly object seed from step (1) to be made up of several disconnected regions, this is meant that from step Suddenly the result obtained in (1) can not be used for representing the whole skeleton of destination object, in this case, based on local autocorrelation Skeleton be used to allow initial significantly object seed to be optimized to preferably significantly object seed, it is contemplated that obtained by local autocorrelation The skeleton arrived compares some expansions with destination object, so applying morphologic etching operation to correct this problem；Originally Notable object seed begin when being made up of some disconnected regions, by the skeleton based on local autocorrelation and initial significantly object Seed is merged as final notable object seed；Otherwise, result will be only obtained in step (1) as final to show Write object seed；

Cavity in final notable object seed needs to be filled, except those account for 1/5 threshold value of whole destination object Cavity above；

(3) starting point for extension is generated on the edge of final notable object seed

The starting point of our extended methods is taken as in borderline each point of final notable object seed, one is constructed The convolution kernel N of individual 3 × 3 size_s, and allow final notable object seed binary map M_bWith convolution kernel N_sMake convolution operation, work as behaviour When the value for making central point p in window is 1, N_sWill be by using calculating a factor of determination d：

<mrow> <mi>d</mi> <mo>=</mo> <mfenced open = "{" close = ""> <mtable> <mtr> <mtd> <mrow> <mn>1</mn> <mo>,</mo> </mrow> </mtd> <mtd> <mrow> <mi>n</mi> <mo>&amp;NotEqual;</mo> <mn>0</mn> <mi>a</mi> <mi>n</mi> <mi>d</mi> <mi> </mi> <mi>n</mi> <mo>&amp;NotEqual;</mo> <mn>8</mn> </mrow> </mtd> </mtr> <mtr> <mtd> <mrow> <mn>0</mn> <mo>,</mo> </mrow> </mtd> <mtd> <mrow> <mi>n</mi> <mo>=</mo> <mn>0</mn> <mi>o</mi> <mi>r</mi> <mi> </mi> <mi>n</mi> <mo>=</mo> <mn>8</mn> </mrow> </mtd> </mtr> </mtable> </mfenced> <mo>,</mo> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>3.1</mn> <mo>)</mo> </mrow> </mrow>

<mrow> <mi>n</mi> <mo>=</mo> <msub> <mi>&amp;Sigma;</mi> <mrow> <msub> <mi>p</mi> <mrow> <mi>i</mi> <mi>j</mi> </mrow> </msub> <mo>&amp;Element;</mo> <mi>w</mi> </mrow> </msub> <msub> <mi>p</mi> <mrow> <mi>i</mi> <mi>j</mi> </mrow> </msub> <mo>*</mo> <msub> <mi>N</mi> <mrow> <mi>s</mi> <mi>i</mi> <mi>j</mi> </mrow> </msub> <mo>,</mo> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>3.2</mn> <mo>)</mo> </mrow> </mrow>

Wherein, p_ijThe final notable object seed pixel binary value that the i-th row jth is arranged in w windows, and N_sijRepresent N_sIn the value that arranges of the i-th row jth, if n is not equal to 0 and 8, factor of determination d is 1, represent value a little be not just as, Just it assign this convolution kernel central point as one of starting point on final notable object seed edge；

(4) direction of extension is initialized for each starting point

Based on starting point set { P_s, the direction of each starting point can be calculated, it is contemplated that each starting point expands all along normal direction Exhibition, so needing to calculate this normal direction, is divided into 8 types, and mark it with 0 to 7 eight numeral by propagation direction , a convolution kernel N_dIt is constructed out for determining the normal direction of each starting point, in calculated direction l_dMark value public affairs Formula is as follows:

<mrow> <msub> <mi>N</mi> <mi>d</mi> </msub> <mo>=</mo> <mfenced open = "[" close = "]"> <mtable> <mtr> <mtd> <mn>0</mn> </mtd> <mtd> <mn>1</mn> </mtd> <mtd> <mn>2</mn> </mtd> </mtr> <mtr> <mtd> <mn>7</mn> </mtd> <mtd> <mn>0</mn> </mtd> <mtd> <mn>3</mn> </mtd> </mtr> <mtr> <mtd> <mn>6</mn> </mtd> <mtd> <mn>5</mn> </mtd> <mtd> <mn>4</mn> </mtd> </mtr> </mtable> </mfenced> <mo>,</mo> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>4.2</mn> <mo>)</mo> </mrow> </mrow>

Wherein, p_ijIt is binary map M_bThe value of the i-th row jth row pixel of correspondence in convolution window w,Expression is entered to the pixel value Row inversion operation, is changed into 0 if being originally 1, if 0 is changed into 1, and p₁₁It is then the row picture of the 1st row the 1st in window w The value of element, n₁It is to work as p₁₁When=1 in addition to central point all satisfaction { p_ij=1 } quantity of point, and n₂It is then to work as p₁₁When=0 All satisfaction { p in addition to central point_ij=1 } quantity of point；(5) end condition of extension is set and extended operation is completed

The border of original image is detected using the algorithm of adaptive threshold canny operators, and defines an expansible limitation Region, the restricted area is to carry out triple-expansion by final notable object seed to obtain；

In confined areas, each borderline point has the opportunity to the terminating point as extension, and bar will be terminated according to propagation direction Part is divided into two classes：{ 0,2,4,6 } and { 1,3,5,7 }, each type all contains several situation；

When normal direction of the starting point along it have found its terminating point, starting point will be carried out to corresponding termination One operation of point extension, in starting point (x, y) and terminating point (x_e,y_e) between extension line on point assignment again, extension Point set on line is expressed as formula：

{p(x+d_ixΔx,y+d_iyΔ y)=1 | 0≤Δ x≤| x-x_e|,0≤Δy≤|y-y_e|, (5.1)

Wherein, p (x+d_ixΔx,y+d_iyΔ y) is the binary value of the point on extension line, and it represents extension line in confined areas On again assignment point can turn into our final goal objects in point；d_iValue from set (- 1, -1), (0, -1), (1, - 1), (1,0), (1,1), (0,1), (- 1,1), (- 1,0) } in find, its 8 bearing mark value of correspondence 0,1,2,3,4,5,6, 7}；

(6) mark could not find the starting point of terminating point as degradation regions mark point

If terminating point is not present in extension restricted area in a starting point, by the starting point labeled as one in degradation regions In point, obtain a mark point set P for representing degradation regions₁；

(7) repairing is carried out and smooth to the result after extension

These disconnected cavities are ranked up according to area, and the point in cavity again assignment is turned into destination object Point, except those areas account for more than 20% cavity of whole object, finally, using Gaussian filter come in sharpening result by Coarse border caused by error；

(8) object segmentation is repaired with super-pixel according to degradation regions mark point

The degenerate region that those are lost is repaired using the super-pixel obtained by simple linear iteration cluster superpixel segmentation method Domain, when the skeleton based on local autocorrelation is attached to final notable object seed, the object based on super-pixel is not performed The process of fusion；

<mrow> <msub> <mi>Obj</mi> <mi>f</mi> </msub> <mo>=</mo> <mfenced open = "{" close = ""> <mtable> <mtr> <mtd> <mrow> <msub> <mi>Obj</mi> <mi>e</mi> </msub> <mo>+</mo> <msub> <mo>&amp;Sigma;</mo> <mrow> <msub> <mi>S</mi> <mi>i</mi> </msub> <mo>&amp;Element;</mo> <mi>S</mi> <mo>,</mo> <msub> <mi>n</mi> <mrow> <mi>l</mi> <mi>i</mi> </mrow> </msub> <mo>&gt;</mo> <mn>5</mn> </mrow> </msub> <msub> <mi>S</mi> <mi>i</mi> </msub> <mo>,</mo> <mi>a</mi> <mo>=</mo> <mn>0</mn> </mrow> </mtd> </mtr> <mtr> <mtd> <mrow> <msub> <mi>Obj</mi> <mi>e</mi> </msub> <mo>,</mo> <mi>a</mi> <mo>=</mo> <mn>1</mn> </mrow> </mtd> </mtr> </mtable> </mfenced> <mo>,</mo> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>8.1</mn> <mo>)</mo> </mrow> </mrow>

Wherein, Obj_fThe final destination object of expression, and Obj_eResult obtained by representing after extending；Parameter a=0 represents base It is not applied in the skeleton of local autocorrelation in final notable object seed, n_liRepresent in super-pixel S_iMiddle mark point Quantity, these mark points are obtained from step (6), and belong to set P_l。