CN104835146A - Salient object segmenting method in stereo image based on depth information and image cutting - Google Patents

Salient object segmenting method in stereo image based on depth information and image cutting Download PDF

Info

Publication number
CN104835146A
CN104835146A CN201510174794.8A CN201510174794A CN104835146A CN 104835146 A CN104835146 A CN 104835146A CN 201510174794 A CN201510174794 A CN 201510174794A CN 104835146 A CN104835146 A CN 104835146A
Authority
CN
China
Prior art keywords
map
mrow
saliency
depth
depth information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201510174794.8A
Other languages
Chinese (zh)
Inventor
刘志
范星星
宋杭科
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Shanghai for Science and Technology
Original Assignee
University of Shanghai for Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Shanghai for Science and Technology filed Critical University of Shanghai for Science and Technology
Priority to CN201510174794.8A priority Critical patent/CN104835146A/en
Publication of CN104835146A publication Critical patent/CN104835146A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/10Segmentation; Edge detection
    • G06T7/12Edge-based segmentation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10004Still image; Photographic image
    • G06T2207/10012Stereo images

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a salient object segmenting method in a stereo image based on depth information and image cutting. The method comprises: inputting and segmenting an original image and a depth map in order to acquire a plurality of areas; computing a saliency map of the original image in combination with area-class depths, colors, and spatial domain information; acquiring a threshold value of the saliency map obtained through computation in order to complete initial segmentation of the original image, thereby acquiring an object/background seed point; constructing a map by using the depth map, the saliency map obtained through computation, and a salient weighted histogram, and designing a cost function; and completing salient object segmentation at one time by using a maximum flow minimum cut algorithm. The method reasonably utilizes the depth information and the saliency map, and more accurately and automatically segment the salient object in the stereo image.

Description

Method for segmenting salient object in stereo image based on depth information and image cutting
Technical Field
The invention relates to the technical field of communication, in particular to a method for segmenting a salient object in a stereo image based on depth information and image cutting. The method is mainly considered from the perspective of reasonably utilizing depth information and a saliency map to improve the segmentation result of a salient object, and aims to generate the saliency map by utilizing depth, color and spatial information, construct the map by combining the depth map and a histogram of saliency weighting, design a cost function and improve the segmentation result of the salient object by using a graph cutting method.
Background
Salient object segmentation techniques refer to the separation of objects of user interest in a video or image from the background at the pixel level. In the conventional segmentation technology, a saliency model is mainly constructed by using information such as color, direction, texture and the like, and a salient object is segmented by using a graph cutting method by using a generated saliency map. However, the research result has many defects, and the universality and accuracy of the algorithm are to be improved.
With the continuous development and popularization of the stereo imaging and display technology, the performance of the saliency model and the saliency object segmentation method can be improved to a certain extent by utilizing the depth information obtained by the stereo camera or the Kinect sensor. Fan et al in 2014 proposed a saliency model combined with depth information in "salient region detection of stereoscopic images" published in the 19 th Digital Signal Processing (DSP) conference held in hong kong, which effectively utilizes depth information and greatly improves saliency performance.
Disclosure of Invention
The invention aims to reasonably utilize depth information and a saliency map and further improve a segmentation result of a salient object, and provides a method for segmenting the salient object in a stereo image based on depth information and map segmentation.
In order to achieve the above purpose, the invention adopts the following scheme:
a method for segmenting a salient object in a stereo image based on depth information and image cutting comprises the following specific steps:
inputting an original image and a depth map, pre-dividing the original image and the depth map, and generating a saliency map;
step two, a threshold value is selected from the significance map obtained in the step one to obtain an object/background seed point so as to complete the initial segmentation of the original image;
step three, taking the saliency map, the depth map and the original image as input of map cutting, constructing a map by using the saliency map and combining the depth map and a saliency weighted histogram, and designing a cost function;
and fourthly, completing the segmentation of the salient object at one time by using a maximum flow minimum segmentation algorithm.
Preferably, the step generates a region-level saliency map using a saliency model provided by Fan in conjunction with region-level depth, color, and spatial information.
Preferably, the second step divides the image into two parts by thresholding the saliency map, wherein the part of pixels with saliency values greater than the threshold is used as object seed points and labeled as "obj", and the part of pixels with saliency values less than the threshold is used as background seed points and labeled as "bkg".
Preferably, the third step combines the depth map and the histogram with significance weighting to construct a map and design a cost function, which is expressed as follows:
E(L)=R(L)+λ·B(L)+β·E(θobjbkg)
wherein, L represents a binary vector of pixel point marks, and the marks comprise an object mark and a background mark, which are marked as 'obj' and 'bkg'; r (l) is a data item reflecting the degree of penalty with which each pixel is labeled as an object or background; b (L) is a smoothing term which is mainly used for punishing adjacent pixels obtaining different marks, and the method mainly considers the difference of colors between the adjacent pixels; e (theta)objbkg) Reflecting the difference of the object region and the background region on a color histogram as an appearance overlapping item; λ and β are balance factors.
Preferably, the maximum flow minimum cut algorithm adopted by the maximum flow minimum cut algorithm cut chart in the step four is actually a process of solving a minimum value for the cost function of the formula in the step three.
Compared with the prior art, the invention has the following obvious substantive characteristics and advantages: the invention provides a method for segmenting a salient object in a three-dimensional image based on depth information and image cutting, which comprises the steps of firstly inputting an original image and a depth image, carrying out image segmentation on the original image and the depth image to obtain a plurality of regions, and then generating a saliency map of the original image by combining depth, color and space domain information of the region level; then, a threshold value is selected for the saliency map to obtain an object/background seed point so as to complete the initial segmentation of the original image; then, constructing a graph by using the depth map, the calculated significance map and the significance weighted histogram and designing a cost function; and finally, automatically segmenting the original image by using a graph cutting method. The method reasonably utilizes the depth information, and can more accurately and automatically segment the salient objects for the stereo image. Compared with the prior art, the segmentation method has the following advantages: the depth information and the image significance are reasonably utilized, and good seed points are provided for image cutting; in the graph cutting process, depth information is introduced again, the depth information is combined with a significance graph and a significance weighted histogram to construct a graph and design a cost function, and the maximum flow minimum cut algorithm is used for completing the segmentation, so that a better segmentation result is obtained.
Drawings
Fig. 1 is a flowchart of a method for segmenting a salient object in a stereo image based on depth information and graph cutting according to the present invention;
fig. 2(a) is a schematic diagram of an original image, fig. 2(b) is a schematic diagram of a depth map, fig. 2(c) is a schematic diagram of a saliency map, fig. 2(d) is a schematic diagram of an initial segmentation result, fig. 2(e) is a schematic diagram of a final segmentation result, and fig. 2(f) is a schematic diagram of a salient object template segmented manually.
Detailed Description
Examples of the present invention will be described in further detail below with reference to the accompanying drawings.
The experiment carried out by the invention is realized by programming on a PC test platform with a CPU of 2.39GHz and a memory of 2G.
As shown in fig. 1, the method for segmenting a salient object in a stereo image based on depth information and graph cutting according to the present invention adopts the following technical solutions: firstly, inputting an original image and a depth map, carrying out image segmentation on the original image and the depth map, and then generating a saliency map by a saliency model; then, a threshold value is selected for the saliency map to obtain an object/background seed point so as to complete the initial segmentation of the original image; and then, constructing a graph by using the depth map, the generated saliency map and the saliency weighted histogram, designing a cost function, and automatically segmenting the original image by using a graph cutting method. The method comprises the following specific steps:
step one, inputting an original image and a depth map, pre-dividing the original image and the depth map to obtain a plurality of regions, and generating a saliency map, namely calculating a region-level saliency map by using a saliency model provided by Fan (Fan is a salient region detection technology of a stereo image), as shown in fig. 2 (c). And generating a region-level saliency map by utilizing a saliency model provided by Fan in combination with the depth, color and spatial information of the region level.
And step two, a threshold value T is determined for the saliency map generated in the step one to obtain an object/background seed point so as to complete the initial segmentation of the original image, the image is divided into two parts by a method for determining the threshold value for the saliency map, the part of pixels with the saliency value larger than the threshold value is used as the object seed point and is marked as 'obj', and the part of pixels with the saliency value smaller than the threshold value is used as the background seed point and is marked as 'bkg'.
And step three, taking the saliency map, the depth map and the original image as input of map cutting, constructing a map by using the saliency map and combining the depth map and the histogram weighted by the saliency, designing a cost function, and designing the cost function.
The traditional salient object segmentation method only takes an original image and a saliency map as input, and the invention introduces depth information to improve the segmentation result. Therefore, the method takes the saliency map, the depth map and the original image as input of map cutting, and utilizes the saliency map to combine the depth map and the histogram weighted by the saliency to construct the map. Different from the graph constructed by general graph cutting, the graph constructed by the method is added with K auxiliary nodes A1,A2...Ak...AKAnd K is the level number of the color histogram and connects each pixel in the image with the corresponding auxiliary node.
The depth map and the saliency-weighted histogram are combined to construct a map and design a cost function, which is in the form of equation (1) and includes a data term, a smoothing term, and an appearance overlap term.
E(L)=R(L)+λ·B(L)+β·E(θobjbkg) (1)
Wherein, L represents a binary vector of pixel point marks, and the marks comprise an object mark and a background mark, which are marked as 'obj' and 'bkg'; r (L) is a data item, reflecting that each pixel is labeledMarking the punishment degree of the object or the background; b (L) is a smoothing term which is mainly used for punishing adjacent pixels obtaining different marks, and the method mainly considers the difference of colors between the adjacent pixels; e (theta)objbkg) Reflecting the difference of the object region and the background region on a color histogram as an appearance overlapping item; λ and β are balance factors.
To introduce depth information into the data items of the cost function, the relationship between the depth map and the saliency map needs to be analyzed. From observation of a large number of depth maps and saliency maps, it is found that saliency maps tend to be more reliable than depth maps for two reasons: firstly, the depth information is only a part of information used for generating the saliency map, so that the saliency map contains more comprehensive information; secondly, the depth map is obtained by a depth estimation algorithm, the reliability of the algorithm directly affects the quality of the depth map, and it is sometimes difficult to distinguish a salient object from the background only by depth information.
It can also be seen from the depth map that, as shown in fig. 2(b), the range of the depth value of the salient object region is narrower than that of the background region, so that the invention designs different data items only for different seed points, and obtains the initial mark only after the initial segmentationDepth information is introduced for the pixels of "obj". The definition of the data items R (L) is shown in Table 1.
Table 1 definition table of data items
Wherein,andrespectively representing the initial segmentation and the final segmentation successorA mark obtained by a pixel; s (p) and D (p) represent the saliency value and the depth value of the pixel p in the saliency map S and the depth map D, respectively (both the saliency map and the depth map are normalized to [0, 1 ]]);μSAnd muDThe mean of the entire saliency map and the mean of the entire depth map are represented separately.
As can be seen from the table, the initial mark obtained when the pixel p is reachedFor "obj", if the saliency value and the depth value of the pixel are larger and smaller, then the probability that the pixel belongs to the salient object is larger, and the final mark obtained by the pixel is obtainedThere is a greater tendency to remain unchanged, i.e. "obj", when a smaller penalty value should be obtained. Conversely, if the saliency value of the pixel p is small and the depth value is large, the initial mark is unreliable, the mark needs to be changed, and the final mark is markedAnd more likely to become "bkg" when the pixel gets a smaller penalty value. When it is initially markedFor "bkg", the data item is associated with a significance value only, and a small significance value is finally markedMore likely to be "bkg" and a high significance value is more likely to be "obj" in the final label.
The sliding term is mainly used to penalize adjacent pixels with different labels, and here, the difference of colors between adjacent pixels is mainly considered, and its expression is as follows (2):
<math> <mrow> <mi>B</mi> <mrow> <mo>(</mo> <mi>L</mi> <mo>)</mo> </mrow> <mo>=</mo> <munder> <mi>&Sigma;</mi> <mrow> <mo>{</mo> <mi>p</mi> <mo>,</mo> <mi>q</mi> <mo>}</mo> <mo>&Element;</mo> <mi>N</mi> </mrow> </munder> <mi>d</mi> <msup> <mrow> <mo>(</mo> <mi>p</mi> <mo>,</mo> <mi>q</mi> <mo>)</mo> </mrow> <mrow> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mo>&CenterDot;</mo> <mo>|</mo> <msub> <mi>L</mi> <mi>p</mi> </msub> <mo>-</mo> <msub> <mi>L</mi> <mi>q</mi> </msub> <mo>|</mo> <mo>&CenterDot;</mo> <msup> <mi>e</mi> <mfrac> <mrow> <mo>-</mo> <msup> <mrow> <mo>|</mo> <mo>|</mo> <msub> <mi>c</mi> <mi>p</mi> </msub> <mo>-</mo> <msub> <mi>c</mi> <mi>q</mi> </msub> <mo>|</mo> <mo>|</mo> </mrow> <mn>2</mn> </msup> </mrow> <mrow> <mn>2</mn> <msup> <mi>&sigma;</mi> <mn>2</mn> </msup> </mrow> </mfrac> </msup> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>1</mn> <mo>)</mo> </mrow> </mrow> </math>
wherein d (p, q) represents the Euclidean distance between the positions of two pixel points p and q, | | cp-cqI is the Euclidean distance, σ, of the color between pixels p and q2Is the average of the squares of the color distances of all pixel pairs. It can be seen from equation (2) that the closer the two pixel points are and the closer the color values are, the more likely they will obtain the same mark, and if the marks they obtain are different, a larger penalty value will be obtained in the smoothing term, thereby forcing them to obtain the same mark. Wherein the balance factor lambda is set here9, to apply a moderate mark smoothing effect.
To design the apparent overlap term, a histogram of saliency weighting is first defined, as follows:
will be provided withAndrespectively defined as the number of pixels whose colors belong to the kth level of the color histogram in the initially segmented object seed point and in the background seed point. By cpRepresenting the color of the pixel p, QkRepresenting the kth level in the color histogram, the significance weighted histogram may be defined as the following formula (3):
<math> <mrow> <mi>H</mi> <mrow> <mo>(</mo> <mi>k</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfenced open='{' close=''> <mtable> <mtr> <mtd> <munder> <mi>&Sigma;</mi> <mrow> <mi>p</mi> <mo>&Element;</mo> <mi>&Omega;</mi> </mrow> </munder> <mfrac> <mrow> <mi>S</mi> <mrow> <mo>(</mo> <mi>p</mi> <mo>)</mo> </mrow> </mrow> <mi>T</mi> </mfrac> <mi>&delta;</mi> <mrow> <mo>(</mo> <msub> <mi>c</mi> <mi>p</mi> </msub> <mo>&Element;</mo> <msub> <mi>Q</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> <mo>,</mo> <mi>if</mi> <msubsup> <mi>&theta;</mi> <mi>obj</mi> <mi>i</mi> </msubsup> <mrow> <mo>(</mo> <mi>k</mi> <mo>)</mo> </mrow> <mo>></mo> <msubsup> <mi>&theta;</mi> <mi>bkg</mi> <mi>i</mi> </msubsup> <mrow> <mo>(</mo> <mi>k</mi> <mo>)</mo> </mrow> </mtd> </mtr> <mtr> <mtd> <munder> <mi>&Sigma;</mi> <mrow> <mi>p</mi> <mo>&Element;</mo> <mi>&Omega;</mi> </mrow> </munder> <mfrac> <mrow> <mn>1</mn> <mo>-</mo> <mi>S</mi> <mrow> <mo>(</mo> <mi>p</mi> <mo>)</mo> </mrow> </mrow> <mi>T</mi> </mfrac> <mi>&delta;</mi> <mrow> <mo>(</mo> <msub> <mi>c</mi> <mi>p</mi> </msub> <mo>&Element;</mo> <msub> <mi>Q</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> <mo>,</mo> <mi>otherwise</mi> </mtd> </mtr> </mtable> </mfenced> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>3</mn> <mo>)</mo> </mrow> </mrow> </math>
where (.) is an indication function, when the condition in parentheses is true, the indication function value is 1, and when the condition is false, the indication function value is 0. According to the formula 3, whenWhen the majority of pixels belonging to the k-th order are initially dividedObject labels are obtained later, which have greater significance and are more likely to belong to the object, and therefore, the auxiliary node A is connectedkAnd the capacity of the edges between these pixels can be expanded with significance values to make these edges more difficult to cut, here set to s (p)/T, so that the cost of the optimal cut set of cut edges is defined as the following equation (4):
<math> <mrow> <mi>e</mi> <mrow> <mo>(</mo> <mi>k</mi> <mo>)</mo> </mrow> <mo>=</mo> <mi>min</mi> <mo>[</mo> <munder> <mi>&Sigma;</mi> <mrow> <msubsup> <mi>L</mi> <mi>p</mi> <mi>f</mi> </msubsup> <mo>=</mo> <mmultiscripts> <mi>obj</mi> <mrow> <mo>&prime;</mo> <mo>&prime;</mo> </mrow> <mrow> <mo>&prime;</mo> <mo>&prime;</mo> </mrow> </mmultiscripts> </mrow> </munder> <mfrac> <mrow> <mi>S</mi> <mrow> <mo>(</mo> <mi>p</mi> <mo>)</mo> </mrow> </mrow> <mi>T</mi> </mfrac> <mi>&delta;</mi> <mrow> <mo>(</mo> <msub> <mi>c</mi> <mi>p</mi> </msub> <mo>&Element;</mo> <msub> <mi>Q</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> <mo>,</mo> <munder> <mi>&Sigma;</mi> <mrow> <msubsup> <mi>L</mi> <mi>p</mi> <mi>f</mi> </msubsup> <mo>=</mo> <mmultiscripts> <mi>bkg</mi> <mrow> <mo>&prime;</mo> <mo>&prime;</mo> </mrow> <mrow> <mo>&prime;</mo> <mo>&prime;</mo> </mrow> </mmultiscripts> </mrow> </munder> <mfrac> <mrow> <mi>S</mi> <mrow> <mo>(</mo> <mi>p</mi> <mo>)</mo> </mrow> </mrow> <mi>T</mi> </mfrac> <mi>&delta;</mi> <mrow> <mo>(</mo> <msub> <mi>c</mi> <mi>p</mi> </msub> <mo>&Element;</mo> <msub> <mi>Q</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> <mo>]</mo> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>4</mn> <mo>)</mo> </mrow> </mrow> </math>
as can be seen from equation 4, because the saliency value of the pixel in the background is small, the edge connecting the auxiliary node and the background pixel is more easily cut off when the graph is cut, so as to protect the seed point with higher saliency. When inThen, most pixels are more likely to be background, and thus the auxiliary node A is connectedkAnd the capacity of the edge between the pixels is set to [1-S (p)]T, so that the cost of the edge cut by the optimal cut set is defined as follows (5):
<math> <mrow> <mi>e</mi> <mrow> <mo>(</mo> <mi>k</mi> <mo>)</mo> </mrow> <mo>=</mo> <mi>min</mi> <mo>[</mo> <munder> <mi>&Sigma;</mi> <mrow> <msubsup> <mi>L</mi> <mi>p</mi> <mi>f</mi> </msubsup> <mo>=</mo> <mmultiscripts> <mi>obj</mi> <mrow> <mo>&prime;</mo> <mo>&prime;</mo> </mrow> <mrow> <mo>&prime;</mo> <mo>&prime;</mo> </mrow> </mmultiscripts> </mrow> </munder> <mfrac> <mrow> <mn>1</mn> <mo>-</mo> <mi>S</mi> <mrow> <mo>(</mo> <mi>p</mi> <mo>)</mo> </mrow> </mrow> <mi>T</mi> </mfrac> <mi>&delta;</mi> <mrow> <mo>(</mo> <msub> <mi>c</mi> <mi>p</mi> </msub> <mo>&Element;</mo> <msub> <mi>Q</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> <mo>,</mo> <munder> <mi>&Sigma;</mi> <mrow> <msubsup> <mi>L</mi> <mi>p</mi> <mi>f</mi> </msubsup> <mo>=</mo> <mmultiscripts> <mi>bkg</mi> <mrow> <mo>&prime;</mo> <mo>&prime;</mo> </mrow> <mrow> <mo>&prime;</mo> <mo>&prime;</mo> </mrow> </mmultiscripts> </mrow> </munder> <mfrac> <mrow> <mn>1</mn> <mo>-</mo> <mi>S</mi> <mrow> <mo>(</mo> <mi>p</mi> <mo>)</mo> </mrow> </mrow> <mi>T</mi> </mfrac> <mi>&delta;</mi> <mrow> <mo>(</mo> <msub> <mi>c</mi> <mi>p</mi> </msub> <mo>&Element;</mo> <msub> <mi>Q</mi> <mi>k</mi> </msub> <mo>)</mo> </mrow> <mo>]</mo> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>5</mn> <mo>)</mo> </mrow> </mrow> </math>
thus, the apparent overlap term in combination with the significance weighted histogram is defined as the following equation (6):
<math> <mrow> <mi>E</mi> <mrow> <mo>(</mo> <msub> <mi>&theta;</mi> <mi>obj</mi> </msub> <mo>,</mo> <msub> <mi>&theta;</mi> <mi>bkg</mi> </msub> <mo>)</mo> </mrow> <mo>=</mo> <munderover> <mi>&Sigma;</mi> <mrow> <mi>k</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>K</mi> </munderover> <mi>e</mi> <mrow> <mo>(</mo> <mi>k</mi> <mo>)</mo> </mrow> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>6</mn> <mo>)</mo> </mrow> </mrow> </math>
the balance factor β for adjusting the weight of the appearance overlap term is defined as the following formula (7):
<math> <mrow> <mi>&beta;</mi> <mo>=</mo> <mn>0.8</mn> <mo>&CenterDot;</mo> <mfrac> <mrow> <munder> <mi>&Sigma;</mi> <mrow> <mi>p</mi> <mo>&Element;</mo> <mi>&Omega;</mi> </mrow> </munder> <mi>&delta;</mi> <mrow> <mo>(</mo> <msubsup> <mi>L</mi> <mi>p</mi> <mi>i</mi> </msubsup> <mo>=</mo> <mmultiscripts> <mi>obj</mi> <mrow> <mo>&prime;</mo> <mo>&prime;</mo> </mrow> <mrow> <mo>&prime;</mo> <mo>&prime;</mo> </mrow> </mmultiscripts> <mo>)</mo> </mrow> </mrow> <mrow> <mo>|</mo> <mi>&Omega;</mi> <mo>|</mo> <mo>/</mo> <mn>2</mn> <mo>-</mo> <munderover> <mi>&Sigma;</mi> <mrow> <mi>k</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>K</mi> </munderover> <mi>min</mi> <mo>[</mo> <msubsup> <mi>&theta;</mi> <mi>obj</mi> <mi>i</mi> </msubsup> <mrow> <mo>(</mo> <mi>k</mi> <mo>)</mo> </mrow> <mo>,</mo> <msubsup> <mi>&theta;</mi> <mi>bkg</mi> <mi>i</mi> </msubsup> <mrow> <mo>(</mo> <mi>k</mi> <mo>)</mo> </mrow> <mo>]</mo> </mrow> </mfrac> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>7</mn> <mo>)</mo> </mrow> </mrow> </math>
from the formula 7, ifAndmore overlap of (b), indicating that after the initial segmentation, the color of the object and the color of the background are not well separated, so the effect of the apparent overlap term in combination with the significance weighted histogram needs to be enhanced, where β will take a larger value; otherwise, β will take a smaller value to attenuate the effect of the apparent overlap term.
And step four, finishing final segmentation of the salient object by using a maximum flow minimum segmentation algorithm. The maximum flow minimum cut algorithm adopted by the maximum flow minimum cut algorithm cut chart is actually a process of solving the minimum value of the cost function of the formula in the step three. And finally, realizing minimization of the cost function by using a maximum flow minimum cut algorithm so as to finish graph cutting. The final segmentation result is shown in fig. 2(e), and compared with the initial segmentation result in fig. 2(d), the final segmentation result can segment the whole salient object more completely and maintain good object boundary.
The traditional salient object segmentation method generally utilizes information such as color and saliency maps, and the like, but recently emerging depth information can be used for improving segmentation performance, so that the method firstly utilizes the depth information to combine color and spatial information to generate a saliency map with higher quality, then threshold values are taken for the saliency map to finish initial segmentation, finally, a map cutting method is adopted, the saliency map, the depth map and a saliency weighted histogram are utilized to construct a map, and segmentation of salient objects is finished.

Claims (5)

1. A method for segmenting a salient object in a stereo image based on depth information and image cutting is characterized by comprising the following specific steps:
inputting an original image and a depth map, pre-dividing the original image and the depth map, and generating a saliency map;
step two, a threshold value is selected from the significance map obtained in the step one to obtain an object/background seed point so as to complete the initial segmentation of the original image;
step three, taking the saliency map, the depth map and the original image as input of map cutting, constructing a map by using the saliency map and combining the depth map and a saliency weighted histogram, and designing a cost function;
and fourthly, completing the segmentation of the salient object at one time by using a maximum flow minimum segmentation algorithm.
2. The method for segmenting the salient objects in the stereo image based on the depth information and the graph cutting as claimed in claim 1, wherein the step of generating the region-level saliency map by utilizing a saliency model provided by Fan in combination with the region-level depth, color and spatial information.
3. The method for segmenting the salient objects in the stereo image based on the depth information and the graph cutting as claimed in claim 1, wherein the second step divides the image into two parts by thresholding the saliency map, and the part of the pixels with the saliency value larger than the threshold is used as the object seed points and marked as the object seed pointsAnd the part of the pixels with the significance value smaller than the threshold value is used as a background seed point and is marked as
4. The method for segmenting the salient objects in the stereo image based on the depth information and the graph cutting as claimed in claim 1, wherein the step three combines the depth map and the histogram of the saliency weighting to construct the graph and design the cost function, and the expression is as follows:
wherein,a binary vector representing pixel point markers including an object marker and a background marker, denotedAndreflecting, for a data item, a degree of penalty for each pixel being labeled as an object or background;for the smoothing item, the method is mainly used for punishing adjacent pixels with different marks, and the method mainly considers the difference of colors between the adjacent pixels;reflecting the difference of the object region and the background region on a color histogram as an appearance overlapping item;andis a balance factor.
5. The method for segmenting salient objects in stereoscopic images based on depth information and image segmentation as claimed in claim 4, wherein said step four of maximum-flow minimum-cut algorithm employs maximum-flow minimum-cut algorithm to segment images, which is actually a process of solving a minimum value for the cost function of the formula in the step three.
CN201510174794.8A 2015-04-14 2015-04-14 Salient object segmenting method in stereo image based on depth information and image cutting Pending CN104835146A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510174794.8A CN104835146A (en) 2015-04-14 2015-04-14 Salient object segmenting method in stereo image based on depth information and image cutting

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510174794.8A CN104835146A (en) 2015-04-14 2015-04-14 Salient object segmenting method in stereo image based on depth information and image cutting

Publications (1)

Publication Number Publication Date
CN104835146A true CN104835146A (en) 2015-08-12

Family

ID=53813011

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510174794.8A Pending CN104835146A (en) 2015-04-14 2015-04-14 Salient object segmenting method in stereo image based on depth information and image cutting

Country Status (1)

Country Link
CN (1) CN104835146A (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105787938A (en) * 2016-02-25 2016-07-20 上海大学 Figure segmentation method based on depth map
CN107145892A (en) * 2017-05-24 2017-09-08 北京大学深圳研究生院 A kind of image significance object detection method based on adaptive syncretizing mechanism
CN107169417A (en) * 2017-04-17 2017-09-15 上海大学 Strengthened based on multinuclear and the RGBD images of conspicuousness fusion cooperate with conspicuousness detection method
WO2017173578A1 (en) * 2016-04-05 2017-10-12 华为技术有限公司 Image enhancement method and device
CN107292923A (en) * 2017-06-29 2017-10-24 北京大学深圳研究生院 The back-propagating image vision conspicuousness detection method excavated based on depth map
CN107909577A (en) * 2017-10-18 2018-04-13 天津大学 Fuzzy C-mean algorithm continuous type max-flow min-cut brain tumor image partition method
CN108154150A (en) * 2017-12-18 2018-06-12 北京工业大学 A kind of conspicuousness detection method based on background priori
CN108596919A (en) * 2018-04-24 2018-09-28 重庆邮电大学 A kind of Automatic image segmentation method based on depth map
CN111161299A (en) * 2018-11-08 2020-05-15 深圳富泰宏精密工业有限公司 Image segmentation method, computer program, storage medium, and electronic device

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101840577A (en) * 2010-06-11 2010-09-22 西安电子科技大学 Image automatic segmentation method based on graph cut
CN102592268A (en) * 2012-01-06 2012-07-18 清华大学深圳研究生院 Method for segmenting foreground image
CN104091336A (en) * 2014-07-10 2014-10-08 北京工业大学 Stereoscopic image synchronous segmentation method based on dense disparity map

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101840577A (en) * 2010-06-11 2010-09-22 西安电子科技大学 Image automatic segmentation method based on graph cut
CN102592268A (en) * 2012-01-06 2012-07-18 清华大学深圳研究生院 Method for segmenting foreground image
CN104091336A (en) * 2014-07-10 2014-10-08 北京工业大学 Stereoscopic image synchronous segmentation method based on dense disparity map

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
顾建栋 等: ""结合核密度估计和边缘信息的运动对象分割算法"", 《计算机辅助设计与图形学学报》 *

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105787938A (en) * 2016-02-25 2016-07-20 上海大学 Figure segmentation method based on depth map
WO2017173578A1 (en) * 2016-04-05 2017-10-12 华为技术有限公司 Image enhancement method and device
CN107169417A (en) * 2017-04-17 2017-09-15 上海大学 Strengthened based on multinuclear and the RGBD images of conspicuousness fusion cooperate with conspicuousness detection method
CN107169417B (en) * 2017-04-17 2021-01-12 上海大学 RGBD image collaborative saliency detection method based on multi-core enhancement and saliency fusion
CN107145892A (en) * 2017-05-24 2017-09-08 北京大学深圳研究生院 A kind of image significance object detection method based on adaptive syncretizing mechanism
WO2019000821A1 (en) * 2017-06-29 2019-01-03 北京大学深圳研究生院 Back-propagation image visual significance detection method based on depth map mining
CN107292923A (en) * 2017-06-29 2017-10-24 北京大学深圳研究生院 The back-propagating image vision conspicuousness detection method excavated based on depth map
CN107909577A (en) * 2017-10-18 2018-04-13 天津大学 Fuzzy C-mean algorithm continuous type max-flow min-cut brain tumor image partition method
CN108154150A (en) * 2017-12-18 2018-06-12 北京工业大学 A kind of conspicuousness detection method based on background priori
CN108154150B (en) * 2017-12-18 2021-07-23 北京工业大学 Significance detection method based on background prior
CN108596919A (en) * 2018-04-24 2018-09-28 重庆邮电大学 A kind of Automatic image segmentation method based on depth map
CN108596919B (en) * 2018-04-24 2021-07-13 重庆邮电大学 Automatic image segmentation method based on depth map
CN111161299A (en) * 2018-11-08 2020-05-15 深圳富泰宏精密工业有限公司 Image segmentation method, computer program, storage medium, and electronic device
CN111161299B (en) * 2018-11-08 2023-06-30 深圳富泰宏精密工业有限公司 Image segmentation method, storage medium and electronic device

Similar Documents

Publication Publication Date Title
CN104835146A (en) Salient object segmenting method in stereo image based on depth information and image cutting
Campbell et al. Automatic 3d object segmentation in multiple views using volumetric graph-cuts
CN108537239B (en) Method for detecting image saliency target
CN104200461B (en) The remote sensing image registration method of block and sift features is selected based on mutual information image
CN103745468B (en) Significant object detecting method based on graph structure and boundary apriority
CN106529573A (en) Real-time object detection method based on combination of three-dimensional point cloud segmentation and local feature matching
CN106504276A (en) The combinations matches cost algorithms of non local Stereo Matching Algorithm and parallax joint filling algorithm
CN106203430A (en) A kind of significance object detecting method based on foreground focused degree and background priori
JP2017004480A (en) Conspicuity information acquisition device and conspicuity information acquisition method
CN103080979B (en) From the system and method for photo synthesis portrait sketch
CN107330861B (en) Image salient object detection method based on diffusion distance high-confidence information
CN110276264A (en) A kind of crowd density estimation method based on foreground segmentation figure
CN104680510A (en) RADAR parallax image optimization method and stereo matching parallax image optimization method and system
WO2019071976A1 (en) Panoramic image saliency detection method based on regional growth and eye movement model
CN104835175A (en) Visual attention mechanism-based method for detecting target in nuclear environment
CN106997478B (en) RGB-D image salient target detection method based on salient center prior
CN104657980A (en) Improved multi-channel image partitioning algorithm based on Meanshift
CN104050675B (en) Feature point matching method based on triangle description
CN112288758B (en) Infrared and visible light image registration method for power equipment
JP2018088151A (en) Boundary line estimating apparatus
CN109255357A (en) A kind of RGBD image collaboration conspicuousness detection method
CN113610905B (en) Deep learning remote sensing image registration method based on sub-image matching and application
WO2011131029A1 (en) Method for detecting similar units based on outline belt graph
CN108520534A (en) A kind of adaptive multimodality fusion Stereo Matching Algorithm
CN106530336A (en) Stereo matching algorithm based on color information and graph-cut theory

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
EXSB Decision made by sipo to initiate substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20150812

WD01 Invention patent application deemed withdrawn after publication