CN107452010B - Automatic cutout algorithm and device - Google Patents

Abstract

An automatic matting method and device relate to the field of digital image processing, comprising: acquiring an original image to be matted, calculating its matting visual saliency; using spatial domain filtering and threshold segmentation algorithms to separate a foreground area and a background area, and combining morphological Learn the operation to obtain a tripartite map; perform gradient calculation on each pixel in the unknown area, and obtain the foreground and background sample point sets of the current unknown area pixel by sampling according to the gradient direction and saliency size; calculate the opacity and confidence of each sample point , and take the sample pair with the highest confidence as the best sample pair for final matting. The local area of opacity is smoothed to obtain the final estimated opacity; finally, according to the final estimated opacity and the color of the best sample pair, a matting operation is performed in the original image to extract the foreground target. The invention also discloses an automatic drawing device. The embodiments of the present invention have the advantages of no user interaction, simple and convenient use, and high matting accuracy and success rate.

Description

A kind of automatic matting algorithm and device

technical field

The invention relates to the field of digital image processing, in particular to an automatic image matting algorithm and device.

Background technique

In real life, the target of interest is extracted from a background image and used as an independent material or synthesized with a new background image in order to obtain a complete and realistic background replacement effect. This technology has been widely used in image editing, Film and television special effects and other fields have widely penetrated into people's daily life. Digital image matting technology has become a hot spot in the field of computer vision research in recent years due to its good application prospects and commercial value.

Digital matting algorithms model each pixel in a natural image as a linear model of foreground and background colors, namely:

I=αF+(1-α)B (1)

Among them, I represents the color value in the actual image, F represents the foreground color value, B represents the background color value, α is called the foreground opacity, and the value range is [0, 1], where the opacity of the foreground area α=1, The opacity α of the background area is 0, and in the unknown area, that is, the edge area of the foreground object, α takes a value between (0, 1). The so-called matting is the process of obtaining the foreground F, the background B and the opacity α when the actual image I is known. I, F, and B are all three-dimensional vectors, and the equation needs to find the remaining seven unknowns based on the three known quantities, so it is a highly underconstrained problem.

At present, the matting technology that has been widely used in film and media production companies is blue screen matting. The blue screen cutout operation is simple, but it has great restrictions on the background, and when the foreground appears blue, it will not be able to completely cut out the target.

At present, the natural image matting algorithms mainly studied by scholars can be roughly divided into two categories, namely:

(A) Sampling-based algorithm. The method assumes that the image is locally continuous, and estimates the foreground and background components of the current pixel with known sample points near the unknown region. For example, the invention CN105225245 proposes a natural image matting method based on texture distribution assumption and regularization strategy, and improves the Bayesian matting method, but the defect of the sampling-based method is that the obtained alpha map has poor connectivity and often requires image priors knowledge and a lot of user markup;

(2) Algorithms based on propagation. This method requires the user to first mark (such as points, lines, etc.), identify the foreground and background, and then regard the unknown area as a field, and the edge of the field corresponds to the known area. The Laplace matrix is established to describe the relationship of the alpha value, Converting the matting process into the Laplace matrix solution process has the disadvantage of a large amount of calculation and is not effective for non-connected regions.

In addition, there are algorithms that combine sampling and propagation, in order to take advantage of the two, such as robust matting algorithms, etc., but the algorithms still generally have the defects of complex user interaction, too many prior assumptions of images, and large amount of calculation. , thus limiting the scope of application and increasing the difficulty of use.

SUMMARY OF THE INVENTION

In order to solve the problems existing in the prior art, the present invention provides an automatic matting algorithm and device, which calculates the visual saliency of matting according to the input image, without limiting the background and image prior knowledge, from natural scene images Complete automatic matting without user interaction, while ensuring high matting accuracy and success rate.

The technical scheme adopted by the present invention to solve the technical problem is as follows:

An automatic image matting algorithm, the method comprises the following steps:

Step 1: Obtain the original image to be matted, and calculate its matting visual saliency;

Step 2: According to the cutout visual saliency map, use spatial domain filtering and threshold segmentation algorithms to separate the foreground area and the background area, and combine morphological operations to obtain a tripartite map;

Step 3: Perform gradient calculation on each pixel of the unknown area according to the trisector map, and obtain the foreground and background sample point sets of the pixels in the current unknown area by sampling according to the gradient direction and saliency size;

Step 4: Calculate the opacity and confidence of each sample point according to the foreground and background sample point sets of the pixels in the current unknown area, and take the sample pair with the highest confidence as the best sample pair for final matting. Then smooth the local area of opacity to get the final estimated opacity;

Step 5: According to the final estimated opacity and the color value of the best sample pair, perform a matting operation in the original image to extract the foreground target.

An automatic map-out device, the device comprising:

The image acquisition module is used to collect the color value of a single image;

A cutout visual saliency calculation module for calculating the cutout visual saliency of the image according to the image color value obtained by the image acquisition module;

The tripartite map calculation module is used to separate the foreground area and the background area according to the cutout visual saliency map obtained by the cutout visual saliency calculation module. get a three-point map;

The sample point set acquisition module performs gradient calculation on each pixel in the unknown area according to the trisectoral graph obtained by the tripartite graph calculation module, and obtains the foreground and background sample points of the pixel in the current unknown area by sampling according to the gradient direction and the saliency size. set;

The opacity calculation module calculates the opacity and confidence of each sample point according to the foreground and background sample point sets obtained by the sample point set acquisition module, and takes the sample pair with the highest confidence as the best sample for final matting right. Then smooth the local area of opacity to get the final estimated opacity;

The foreground keying module is used to perform a keying operation in the original image according to the final estimated opacity and the color value of the best sample pair to extract the foreground target.

The beneficial effects of the present invention are as follows: the method for calculating the visual saliency of the cutout proposed by the present invention simulates the visual attention mechanism of the human eye, can automatically extract the foreground target, avoids complex user interaction operations, completes the automatic cutout process, and is simple to operate Convenient; by limiting the number of sample point pairs, the matting time is shortened; both the saliency map and the opacity are smoothed, and the matting accuracy is improved.

Description of drawings

Fig. 1 is the schematic flow chart of a kind of automatic matting algorithm of the present invention

FIG. 2 is a schematic flowchart of calculating the regional saliency of the present invention

Fig. 3 is a structural schematic diagram of an automatic map-out device of the present invention

Detailed ways

The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

1 is a schematic flowchart of an embodiment of an automatic image matting algorithm of the present invention. An embodiment of the present invention provides an automatic image matting method. The method can be executed by any image matting device with image storage and display functions. The device It can be various terminal devices, such as personal computers, mobile phones, tablet computers, etc., or digital cameras, video cameras, etc., which can be specifically implemented by software and/or hardware. As shown in Figure 1, the method of this embodiment includes:

Step 1: Obtain the original image to be matted, and calculate the matting visual saliency of the original image.

Considering that the foreground objects that usually need to be extracted have the following characteristics: the foreground objects have a complete target area, and the contrast with the surrounding background is relatively obvious; the color distribution is relatively uniform; most areas have high brightness; Therefore, the visual matting saliency calculation method proposed in the embodiment of the present invention takes into account the characteristics of the foreground target, such as color, brightness, and regional integrity. Assuming that the acquired image is in rgb format, the grayscale image needs to be calculated first according to the r, g, and b color channels. I _gray , the specific formula is:

I _gray = (r+g+b)/3 (2)

The original image may also be in any format such as YUV, the embodiment of the present invention does not limit the output image format of the camera, and the corresponding formula for converting color to gray scale also needs to be adjusted.

Then perform low-pass filtering and down-sampling on I _gray , that is, take the original grayscale image I _gray as the 0th scale layer of the pyramid. The first scale layer is convolved with the low-pass filter for the 0th scale layer, and then sampled by 1/2 in the x and y directions respectively, and the rest of the layers are analogous, and the resolution of each layer is half of the previous layer. . The low-pass filter here may be a Gaussian filter, a Laplacian filter, or a Gabor filter, and the embodiment of the present invention does not specifically limit the form of the low-pass filter for generating the scale pyramid.

Corresponding to the visual characteristics of the human eye, it is difficult to attract the attention of the human eye in the place where the image brightness is very low, so it is necessary to perform threshold suppression on the brightness component of the scale pyramid, that is, the area lower than 5% of the maximum brightness value I _{gray_max} , the brightness The component is set to 0, which can effectively suppress the dark background interference. Although the points with low local brightness on the edge of the target may also be eliminated, on the one hand, these parts will be retained in the regional saliency map, and on the other hand, the edge area will be divided into unknown areas, so the final matting is complete. Sex doesn't matter.

After building the scale pyramid, the luminance saliency map is then calculated. The specific calculation method is: take the image on the fine scale c as the central area of vision, and the image on the coarse scale s as the peripheral area of vision, let the central scale c∈{2,3,4}, the peripheral scale s=c+δ , δ∈{3,4}, six center-periphery combinations can be obtained as {2-5, 2-6, 3-6, 3-7, 4-7, 4-8}. Interpolate the image at scale s to the scale c, and subtract the interpolated image from the image at scale c according to the formula I(c,s)=|I(c)ΘI(s)|(3) to get Brightness difference map. where I(σ) is the image pyramid, σ=0,1,...,8 represents different scales, and Θ represents the center-periphery difference operator. These feature maps represent the difference in brightness between a certain position in the image and its local neighborhood. The greater the difference, the higher the brightness saliency in the local range, and the easier it is to attract the attention of the human eye. After calculating 6 sets of brightness difference maps, these difference maps are fused, redundant features are discarded, and the final brightness saliency map is generated. Since at different scales, the absolute value of the difference map cannot reflect the saliency information, so the difference maps at different scales cannot be simply added. In the embodiment of the present invention, the difference map is normalized, and the specific steps of the normalization function N(·) are:

(1) Normalize all 6 difference maps to the interval [0,1];

(2) Calculate the local variance of each difference map separately;

(3) The fusion weight is positively correlated with the local variance, that is, the larger the local variance, the greater the amount of information contained in the disparity map, and the disparity map should be given a larger weight in the weighted combination.

(4)加权组合亮度差异图，得到亮度显著性图。其中

表示跨尺度相加因子。Then, according to the formula

(4) The brightness difference map is weighted and combined to obtain the brightness saliency map. in

Represents a cross-scale additive factor.

The color saliency map is then calculated. According to Ewald's "color pair" model, in the center of the visual receptive field of the human eye, if the neuron is activated by the R color, it is inhibited by the G color, and if it is activated by the B color, it is inhibited by the Y color. Therefore, according to formula (21~24) Convert the image from three channels of rgb to four channels of RGBY:

R=r-(g+b)/2 (5)

G=g-(r+b)/2 (6)

B=b-(g+r)/2 (7)

Y=(r+g)/2-|r-g|/2-b (8)

求取RG的具体过程为：分别在尺度c和尺度s上，逐像素对图像的R、G通道相减求绝对值，然后将s尺度上的计算结果插值到c尺度上，最后与c尺度上的原结果逐像素作差。BY的求取过程类似。这样多次相减，可分别得到6幅RG和BY上的颜色差异显著性图。Then according to the characteristics of human visual cells, the red-green and blue-yellow anti-color groups RG and BY are calculated, that is: RG(c,s)=|R(c)-G(c)|Θ|G(s)- R(s)(9),

The specific process of obtaining RG is as follows: on the scale c and scale s respectively, subtract the R and G channels of the image pixel by pixel to obtain the absolute value, then interpolate the calculation result on the s scale to the c scale, and finally compare the c scale with the c scale. The original result on the above is pixel-by-pixel difference. The process of obtaining BY is similar. In this way, 6 saliency maps of color difference on RG and BY can be obtained respectively.

(11)加权组合颜色差异图，得到颜色显著性图。Then, according to the formula

(11) Weighting and combining the color difference maps to obtain a color saliency map.

Then calculate the regional saliency map. The foreground target is divided into superpixels, and then the normalized color histogram of each superpixel is counted, and the superpixels are clustered according to the color histogram, and the image is divided into several regions {r ₁ , r ₂ , . . . .,r _k }, the cluster center of each region is ce _i , i=1,...,k. Then the regional _saliency map VA _r of the region ri can be calculated as follows:

Among them, _w (ri) represents the weight of the area of _ri , and the calculation method is:

Among them, PN(r _i ) represents the number of pixels in the region _ri . The above formula indicates that the larger the area, the greater the weight, that is, the larger area around the region _ri has a greater influence on its significance than the smaller area.

Dr(r _j , _ri ) represents the distance between the area r _j and the cluster center of the area _ri , namely:

Among them, ce _i (m) and ce _j (n) represent the m and n color components of the color histograms ce _i and ce _j , and Dc(m,n) represent the m and n colors in the LAB Euclidean distance in color space.

The embodiment of the present invention proposes a regional saliency extraction method combining superpixel segmentation and clustering. The specific implementation steps will be described in FIG. 2 below.

Step A) Perform superpixel segmentation on the input image. The optional superpixel segmentation methods include normalized segmentation (NC) algorithm, graph cut (GS) algorithm, fast drift (QS) algorithm, simple linear iterative clustering (SLIC) algorithm and so on. Considering the rapidity and practicability of the algorithm, the embodiment of the present invention selects the SLIC algorithm, and the specific steps are:

1) Assuming that the total number of pixels of the image is N, it is pre-segmented into K*K superpixels. First, the entire image is evenly divided into K*K small blocks, and the center of each small block is taken as the initial point. Calculate the pixel gradient in the 3*3 neighborhood of each initial point, the point with the smallest gradient is the initial center O _i of the superpixel segmentation algorithm, i=0,1,...K*K-1, assign each initial center a separate label;

2) Express each pixel as a five-dimensional vector {l,a,b,x,y} of CIELAB color space and XY coordinates, and calculate the distance between each pixel and its closest center. The distance calculation formula is:

Among them, d _lab is the color difference value, d _xy is the position difference value, S is the center distance, m is the balance parameter, and Dis is the distance between pixels. For each pixel, assign the label of the closest center to it;

3) Recalculate the centers of pixels with different labels, update O _i , and calculate the difference between the old and new centers, if the difference is less than the threshold, the algorithm ends, otherwise, return to step 2).

Step B: Count the normalized color histogram of each superpixel. That is, each dimension of the Lab color space is divided into several bins, the probability that the pixel color in the superpixel falls in each bin is counted, and finally the statistical histogram is normalized.

Step C: Cluster the superpixels to divide the image into several continuous regions. The optional clustering method includes any one of division, model, hierarchy, grid, and density-based clustering. In this embodiment, the density-based DBSCAN clustering algorithm is used. The specific steps are: select one of the superpixels as a seed point, use the given error thresholds Eps and MinPts to search for all its density-reachable superpixels and determine whether the point is a core point. If it is a core point, it forms a clustering area with its density reachable points; if it is not a core point or a boundary point, re-select other points as seed points and repeat the above steps; if it is not a core point but a boundary point, it is considered that is the noise point and discarded. Repeat the above steps until all points are retrieved, and finally form several divided areas. DBSCAN can effectively deal with noise points and can be divided into clusters of arbitrary shapes, so it is suitable for this embodiment.

Step D: Calculate the regional saliency. According to the superpixel clustering results and formulas (12) (13) (14), the regional saliency of each region is calculated.

Finally, according to the importance of color information, area information and luminance information, the formulas VA = α _g VA _g + α _c VA _c +α _r VA _r (18) and α _g + α _c +α _r =1 (19) The three types of saliency maps are fused to obtain the final matting visual saliency map. In this embodiment, α _c =0.5, α _r =0.3, and α _g =0.2.

Step 2: According to the visual saliency map of the matting, use spatial domain filtering and threshold segmentation algorithms to separate the foreground area and the background area, and combine the morphological operations to obtain a tripartite map.

Firstly, the spatial domain filter is used to smooth the matting visual saliency map to remove noise, and then a threshold segmentation algorithm (such as the Otsu method) is used to calculate the threshold T _va , then in the saliency map, the pixels with a saliency value greater than T _va correspond to For the foreground area, the pixels smaller than T _va correspond to the background area, so as to obtain the coarse three-part map It _tc .

The spatial domain filtering is to ensure that the singularity of saliency does not appear in the local area, and at the same time to smooth the saliency value, you can choose median filter, bilateral filter or Gaussian filter. In order to ensure the computational efficiency of the algorithm, in this embodiment, a Gaussian filter is selected as the smoothing filter of the visual saliency map for matting, and the filter window is 3*3.

Since the brightness saliency, color saliency and regional saliency are comprehensively considered in the matting visual saliency map to ensure that the saliency of the foreground area is much greater than that of the background area, there is no strict requirement for the performance of the threshold segmentation algorithm. , the threshold T _va can be valued in a wide range without affecting the results of foreground segmentation. In this embodiment, we choose the Otsu threshold algorithm, and the specific steps are:

(1) Assume that the gray level of the matting visual saliency map VA is 0, 1, ..., L-1, the total number of pixels in the image is N, and the gray level histogram is calculated, that is, the gray level in the VA is assumed to be i, i=0,1,..., the number of pixels of L-1 is N _i , then the value of the corresponding gray level i in the grayscale histogram is N _i /N;

(2) Traverse the threshold T _va from 0 to L-1, divide the pixels into two categories: less than T _va and greater than or equal to T _va , and count the inter-class difference between the two categories:

g=ω ₀ (μ ₀ -μ) ² +ω ₁ (μ ₁ -μ) ² (20)

Among them, ω ₀ and ω ₁ are the proportions of the number of pixels less than T _va and greater than or equal to T _va , respectively, and μ ₀ and μ ₁ are the average values of pixels less than T _va and greater than or equal to T _va , respectively.

(3) After traversing, find the corresponding T _va when the inter-class difference g takes the maximum value as the final segmentation threshold.

The shape and size of the morphological operator should be selected according to the image content, such as image resolution, size and shape of the foreground area, and in this embodiment, the default is a circle to ensure uniformity in all directions. When performing morphological operations, in order to avoid small holes and burrs after threshold segmentation, the following morphological operations are performed on I _tc : first, an opening operation is performed to connect local discontinuous areas and remove holes; then the size is r. The corrosion operation of _e can obtain the foreground area F _g of the tripartite map; as an expansion operation of size r _d , the background part _Bg of the tripartite map can be obtained, and the area between the foreground and the background is the unknown area, so that the to-be-cut area is obtained. Subdivision tripartite map It of image _I.

Assuming that the gray value of the foreground area is 1, the gray value of the background area is 0, and the morphological operator is used to convolve the binary image. For the white foreground area, corrosion can shrink its boundaries, while for the black background area , the expansion can shrink its boundary, and finally the unknown area is between the foreground and the background.

Step 3: Perform gradient calculation on each pixel of the unknown area according to the trisector map, and obtain the foreground and background sample point sets of the pixels in the current unknown area by sampling according to the gradient direction and saliency size.

For each pixel I(x _i , y _i ) in the unknown area, calculate its gradient value Gra _i , the direction of the gradient is denoted as θ, and the calculation formula of θ is

In this embodiment, a reference pair of foreground and background sample points is searched on a straight line in the gradient direction, and around the pair of sample points, the size of the search area is determined according to the visual saliency value of the pixel point at the current position. The greater the saliency, the smaller the search range, which means that around the pixel with high saliency, the real foreground and background points are closer to the pixel. Then, according to the spatial distance and visual saliency, 5 pairs of foreground and background samples are respectively searched. The specific process is as follows:

1) Let the initial value of the search radius _rs be 1, and the count count=0;

2) On the circle with the reference foreground/background sample point as the center and _rs as the radius, calculate whether the condition is satisfied for all pixel points p: |VA(p)-VA(p ₀ )|<T _vap , where p ₀ is the center of the search area, that is, the reference foreground or background sample point, p is the point on the search circle, if the conditions are met, count+1;

3) Determine whether count>5, if so, stop the search; if otherwise, _rs ++ and return to step 2).

Such a sampling strategy, on the one hand, can generate fewer sampling point pairs, reducing the complexity of subsequent matting calculations, and on the other hand, sampling along the gradient direction can ensure that the foreground and background points are located in different texture areas with a high probability. At the same time, the sampling according to the saliency value of the neighborhood can ensure that there is a certain degree of spatial and saliency similarity between the sampling points, so the sampling method can ensure that the real sampling point pair is included with a high probability, thereby improving the accuracy of matting

Step 4: Calculate the opacity and confidence of each sample point according to the foreground and background sample point sets of the pixels in the current unknown area, and take the sample pair with the highest confidence as the best sample pair for final matting. Then smooth the local area of opacity to get the final estimated opacity;

Choose a point from the foreground sample point set and the background sample point set. According to the imaging linear model, the estimated opacity is

和

分别表示第m个前景样本点和第n个背景样本点的颜色值。这样对于每个未知区域像素，一共获得25个不同的不透明度估计，随后需要从中选出置信度最高的不透明度用于抠出前景目标。in

and

Represent the color values of the mth foreground sample point and the nth background sample point, respectively. In this way, for each unknown region pixel, a total of 25 different opacity estimates are obtained, and then the opacity with the highest confidence needs to be selected for the foreground target.

The requirements for the optimal pair of foreground and background points are: 1) the linear model of formula (1) has the smallest error; 2) the foreground sample point and the background sample point have a large color difference; 3) the foreground or background sample point is different from The color value of the current pixel is close; 4) The spatial distance between the foreground and background sample points and the current pixel is small.

According to Criterion 1) and Criterion 2), the linear-color difference similarity is defined as

According to criterion 3), the color similarity is defined as

According to criterion 4), the spatial distance similarity is defined as

Define the confidence function as

分别是线性-色差相似度、颜色相似度、空间距离相似度，D_i是未知像素的前背景采样半径，由其抠图视觉显著度决定，视觉显著度越大则D_i越小，σ₁、σ₂和σ₃用于调整不同相似度之间的权值。选取置信度最高的α作为当前未知像素的不透明度的估计，而对应的前景、背景样本对则作为最终抠图使用的最佳前景、背景样本对。in,

are the linear-color difference similarity, color similarity, and spatial distance similarity, respectively, D _i is the front and background sampling radius _of the unknown pixel, which is determined by its matting visual saliency _. , σ ₂ and σ ₃ are used to adjust the weights between different similarities. The α with the highest confidence is selected as the estimation of the opacity of the current unknown pixel, and the corresponding pair of foreground and background samples is used as the best pair of foreground and background samples for final matting.

For each pixel of the unknown region, the opacity is estimated point-by-point as before. In the end, it is possible that the confidence of some pixels is too low, resulting in a large error in the estimated opacity, and finally chromatic aberration in the matting. Therefore, it is necessary to locally smooth the opacity of the unknown region. The factors that need to be considered when smoothing are: color value difference, spatial position difference, and saliency difference, that is, the larger the local color difference, the farther the local spatial position, and the larger the saliency pixel, the smaller the weight. For this reason, in order to balance the influence of the spatial domain, the color value domain, and the saliency value domain, the opacity smoothing method in the present invention is as follows:

Among them, P _i , P _j represent the coordinates of the two points i and j, I _i , I _j represent the colors of the two points i and j, VA _i , VA _j represent the visual significance of the matting of the two points i and j, σ _p , σ _c and σ _va are used to adjust the weight between the three. The opacity calculated in this way fully considers the influence of spatial position, color and saliency, so that the closer the spatial position, the more similar the color, and the closer the saliency is, the closer the opacity of the pixels is, which is similar to the subjective feeling of the human eye. Consistent, thus effectively eliminating the singular point of opacity and improving the precision of matting.

According to the four criteria of the optimal foreground and background point pairs, the metric function is determined as shown in formula (30), which comprehensively considers the linear model conformity, the color difference between the foreground and background, the color difference between the target pixel and the foreground and background, and the spatial distance. four indicators. By setting the values of σ ₁ , σ ₂ , and σ ₃ , it is used to adjust the weights of different degrees of similarity. Then, according to formula (31), the opacity is smoothed to obtain the opacity for final matting. In this embodiment, the smoothing operation comprehensively considers color difference, spatial position difference and saliency difference, and by changing σ _p , σ _c and σ _va , the proportions of these three in the weighting coefficient can be adjusted. For example, if the saliency information is emphasized, that is, σ _va should be greater than σ _p and σ _c .

Step 5: According to the final estimated opacity and the color value of the best sample pair, perform a matting operation in the original image to extract the foreground target.

The specific operation steps are as follows: create a new image with the same size as the original image as the background, use the calculated opacity and foreground pixel values, and synthesize it with the new background according to formula (1) to obtain the final matting result.

In the example of the present invention, the natural image is cut out, and the specific content of the foreground target and the background is not limited, as long as the foreground and the background have a clearly distinguishable boundary that can be discerned by the naked eye.

The embodiment of the present invention provides an automatic image matting algorithm. By acquiring the original image to be matted, the matting visual saliency of the original image is calculated; then according to the matting visual saliency map, spatial domain filtering and threshold segmentation algorithms are used, Separate the foreground area and the background area, and combine the morphological operations to obtain a tripartite map; according to the tripartite map, perform gradient calculation on each pixel of the unknown area, and sample the foreground and background of the current unknown area pixel according to the gradient direction and saliency size. Sample point set: Calculate the opacity and confidence of each sample point according to the foreground and background sample point sets of pixels in the current unknown area, and take the sample pair with the highest confidence as the best sample pair for final matting. Then smooth the local area of opacity to obtain the final estimated opacity; according to the final estimated opacity and the color value of the best sample pair, perform a matting operation in the original image to extract the foreground target. The method for calculating the visual saliency of map cutout proposed by the invention simulates the visual attention mechanism of the human eye, can automatically extract foreground objects, eliminates complex user interaction operations, completes the automatic map cutout process, and the operation is simple and convenient; The number of saliency maps and the opacity are smoothed, which shortens the matting time; the saliency map and opacity are smoothed to improve the matting accuracy.

3 is a schematic structural diagram of an automatic map-out device provided by an embodiment of the present invention, and the device includes:

The image acquisition module is used to collect the color value of a single image;

A cutout visual saliency calculation module for calculating the cutout visual saliency of the image obtained by the image acquisition module;

The tripartite map calculation module is used to separate the foreground area and the background area according to the cutout visual saliency map obtained by the cutout visual saliency calculation module. get a three-point map;

The sample point set acquisition module performs gradient calculation on each pixel of the unknown area according to the tri-partite map obtained by the tri-partite map calculation module, and obtains the foreground and background sample points of the current unknown area pixel by sampling according to the gradient direction and the saliency size. set;

The opacity calculation module calculates the opacity and confidence of each sample point according to the foreground and background sample point sets obtained by the sample point set acquisition module, and takes the sample pair with the highest confidence as the best sample for final matting right. Then smooth the local area of opacity to get the final estimated opacity;

The foreground keying module is used to perform a keying operation in the original image according to the final estimated opacity and the color value of the best sample pair to extract the foreground target.

Specifically, the cutout visual saliency calculation module includes:

The scale pyramid generation unit is used to perform smoothing and downsampling according to the acquired image to be matted to generate a scale pyramid;

The luminance saliency calculation unit is used to generate the scale pyramid obtained by the unit according to the scale pyramid, taking the image on the fine scale as the central area of vision, and the image on the coarse scale as the peripheral area of vision, and calculates the luminance saliency map;

The color saliency calculation unit is used to generate the scale pyramid obtained by the scale pyramid generation unit, taking the image on the fine scale as the central area of vision, and the image on the coarse scale as the peripheral area of vision, and calculates the color saliency map;

The regional saliency calculation unit is used to perform superpixel segmentation on the foreground target according to the to-be-mashed image obtained by the image acquisition module, and cluster the superpixels according to the color histogram, and calculate the color saliency of each clustered region;

The saliency fusion unit is used to fuse the brightness saliency map obtained by the brightness saliency calculation unit, the color saliency map obtained by the color saliency calculation unit, and the regional saliency map obtained by the regional saliency calculation unit to obtain the image to be keyed. Cutout visual saliency map.

The three-part graph calculation module includes:

The spatial domain filtering unit is used to select an appropriate spatial domain filtering method to smooth the visual saliency map of the matting;

The threshold segmentation unit is used to obtain a rough three-part map by using a threshold segmentation algorithm to segment and obtain the foreground area and the background area according to the smooth matting visual saliency map obtained by the spatial domain filtering unit;

Morphological operation unit: It is used to perform morphological operations on the rough tripartite map obtained by the threshold segmentation unit to fill the holes, and obtain the foreground, background and unknown areas, that is, the precise tripartite map.

The sample point set acquisition module includes:

a gradient calculation unit, configured to obtain the gradient of each unknown pixel according to the grayscale value of the image to be matted;

The sampling unit is used to draw a straight line according to the gradient direction obtained by the gradient calculation unit, and take the first intersection pair of the straight line and the foreground area and the background area as the initial search point. In the neighborhood of the search point, search from near to far and unknown Sample points whose pixel saliency value difference is less than the threshold.

The opacity calculation module includes:

Linear-color difference similarity calculation unit: used to obtain the sample point set obtained by the module according to the sample point set, take out the sample points one by one, and calculate their linear-color similarity;

Color similarity calculation unit: used to obtain the sample point set obtained by the module according to the sample point set, take out the sample points one by one, and calculate their color similarity;

Spatial distance similarity calculation unit: used to obtain the sample point set obtained by the module according to the sample point set, take out the sample points one by one, and calculate their spatial distance similarity;

Sample screening unit: used to calculate the confidence of each pair of sample points relative to the current unknown pixel according to the similarity values obtained from the linear-color difference similarity calculation unit, the color similarity calculation unit and the spatial distance similarity calculation unit; select the confidence The highest degree of opacity is used as an estimate of the opacity of the current loxel.

Smoothing unit: It is used to locally smooth the opacity obtained by the sample screening unit.

The embodiment of the present invention provides an automatic matting device, which calculates the matting visual saliency of the original image by acquiring the original image to be matted; and then uses spatial domain filtering and threshold segmentation algorithms according to the matting visual saliency map, Separate the foreground area and the background area, and combine the morphological operations to obtain a tripartite map; according to the tripartite map, perform gradient calculation on each pixel of the unknown area, and sample the foreground and background of the current unknown area pixel according to the gradient direction and saliency size. Sample point set: Calculate the opacity and confidence of each sample point according to the foreground and background sample point sets of pixels in the current unknown area, and take the sample pair with the highest confidence as the best sample pair for final matting. Then smooth the local area of opacity to obtain the final estimated opacity; according to the final estimated opacity and the color value of the best sample pair, perform a matting operation in the original image to extract the foreground target. The visual saliency calculation method for map cutout proposed by the invention simulates the visual attention mechanism of human eyes, can automatically extract foreground targets, eliminates complex user interaction operations, completes the automatic map cutout process, and the operation is simple and convenient; The number of saliency maps and the opacity are smoothed to shorten the matting time; the saliency map and the opacity are smoothed to improve the matting accuracy.

The embodiment of the present invention also provides a computer program product for automatic image cutout.

Claims (8)

1. an automatic matting algorithm, is characterized in that, this algorithm comprises the steps: Step 1: Obtain the original image to be matted, and calculate its matting visual saliency; Step 2: According to the cutout visual saliency map described in Step 1, use spatial domain filtering and threshold segmentation algorithms to separate the foreground area and the background area, and combine morphological operations to obtain a tripartite map; Step 3: perform gradient calculation on each pixel of the unknown region of the three-part map described in Step 2, and obtain the foreground and background sample point sets of the pixels of the current unknown region by sampling according to the gradient direction and the saliency size; Step 4: Calculate the opacity and confidence of each sample point according to the foreground and background sample point sets of the pixels in the current unknown area described in Step 3, and take the sample pair with the highest confidence as the best sample pair for final matting. , and then smooth the local area of opacity to obtain the final estimated opacity; the opacity calculation method includes: Select a point from the set of foreground sample points and the set of background sample points. According to the imaging linear model, the estimated opacity is

in

and

represent the color values of the mth foreground sample point and the nth background sample point respectively; I _i is the color value of the unknown area pixel i; thus, for each unknown area pixel i, a total of 25 different opacity estimates are obtained, and then The opacity with the highest confidence needs to be selected for the foreground target; The linear-color difference similarity is defined as

Define color similarity as

The spatial distance similarity is defined as

Define the confidence function as

in

are linear-color difference similarity, color similarity, and spatial distance similarity, respectively. Di is the front-background sampling radius of unknown pixel i, which is determined by its matting visual saliency. _The greater the visual saliency, the smaller Di is. σ ₂ and σ ₃ are used to adjust the weights between different degrees of similarity; α with the highest confidence is selected as the estimation of the opacity of the currently unknown pixel, and the corresponding pair of foreground and background samples are used as the foreground and background samples used in the final matting. pair of background samples; x _i is the position of unknown pixel i in the image,

is the position of the mth foreground sample point in the image,

is the position of the mth background sample point in the image; Finally, smooth the opacity;

Among them, P _i , P _j represent the coordinates of the two points i and j, I _i , I _j represent the colors of the two points i and j, VA _i , VA _j represent the visual significance of the matting of the two points i and j, σ _p , σ _c and σ _va are used to adjust the weight between the three; ω _ij is the weighted weight of the two points i and j on the opacity; Step 5: According to the final estimated opacity and the color value of the best sample pair described in Step 4, perform a matting operation in the original image to extract the foreground target. 2. a kind of automatic matting algorithm according to claim 1, is characterized in that, the concrete step of calculating the matting visual salience of original image is: Step (1), calculate the grayscale image I _gray , and perform layer-by-layer smoothing and downsampling to I _gray to generate a scale pyramid of n layers; Step (2), take the image on the fine scale c as the central area of vision, and the image on the coarse scale s as the peripheral area of vision, first calculate the brightness difference feature map: I(c,s)=|I(c)ΘI(s)| where I(c), I(s) are the image pyramids, c∈{2,3,4} is the central scale, s=c+δ, δ∈{3,4} is the surrounding scale, and Θ is the central-peripheral difference operator, and finally get 6 brightness difference feature maps; the brightness saliency map is the normalized weighted sum of the 6 brightness difference feature maps:

in,

represents the normalization function,

represents a cross-scale addition factor; Step (3), calculate the image on the fine scale c as the central area of vision, and the image on the coarse scale s as the peripheral area of vision, first convert the image from the rgb channel to the RGBY quaternary color channel, and then calculate the red and green channels. Color difference map RG and blue-yellow channel color difference map BY: RG(c,s)=|R(c)-G(c)|Θ|G(s)-R(s)| BY(c,s)=|B(c)-Y(c)|Θ|Y(s)-B(s)| where R(c), G(c), B(c), Y(c) are the RGBY components of the image on scale c, respectively, R(s), G(s), B(s), Y(s) are the RGBY components of the image on scale s, respectively; The color saliency map is the normalized weighted sum of 12 color difference feature maps:

Step (4), perform superpixel segmentation on the foreground target, then count the normalized color histogram of each superpixel, and cluster the superpixels according to the color histogram, and divide the image into several regions {r1, r2,...,rk}, the cluster center of each region is ce _i , i=1,...,k; then the regional significance value VAr of the region ri can be calculated as follows:

Among them, w(r _i ) represents the weight value of the area area, and Dr(rj,ri) represents the distance between the cluster centers of the area rj and the area ri; Step (5), synthesizing brightness saliency, color saliency and regional saliency: VA=α _g VA _g +α _c VA _c +α _r VA _r α _g +α _c +α _r =1 Among them, α _g , α _c , and α _r are weighting coefficients of different saliency. 3. a kind of automatic matting algorithm according to claim 1, is characterized in that, obtains the method for three-part graph, and concrete steps are: First, use the spatial domain filter to smooth the cutout visual saliency map to remove noise, and then use the threshold segmentation algorithm to calculate the threshold value Tva, so as to obtain the coarse three-part map Itc; then perform the following morphological operations on Itc: first do a Open operation to connect local discontinuous areas and remove holes; then perform an etching operation of size re to obtain the foreground area Fg of the three-part map; perform an expansion operation of size rd to obtain the background part of the three-part map Bg, foreground The region between the background and the background is an unknown region, and the subdivision tripartite map It of the image I to be matted is obtained. 4. a kind of automatic matting algorithm according to claim 1, is characterized in that, described sample point set acquisition method, specifically comprises: For each pixel I(xi,yi) in the unknown area, calculate its gradient value Grai, the direction of the gradient is recorded as θ, and the calculation formula of θ is

Draw a straight line along the θ direction to obtain the first intersection of the straight line with the foreground area and the background area, respectively, as the initial search center; in the neighborhood of the intersection, search for 5 points whose difference is less than the threshold Tvap from near to far. Finally, a total of 5*5=25 sample point pairs are generated. 5. an automatic drawing device, it is characterised in that the device comprises: The image acquisition module is used to collect the color value of a single image; A cutout visual saliency calculation module for calculating the cutout visual salience of an image according to the image color value obtained by the image acquisition module; The tripartite map calculation module is used to separate the foreground area and the background area according to the cutout visual saliency map obtained by the cutout visual saliency calculation module. get a three-point map; The sample point set acquisition module performs gradient calculation on each pixel in the unknown area according to the trisectoral graph obtained by the tripartite graph calculation module, and obtains the foreground and background sample points of the pixel in the current unknown area by sampling according to the gradient direction and the saliency size. set; The opacity calculation module calculates the opacity and confidence of each sample point according to the foreground and background sample point sets obtained by the sample point set acquisition module, and takes the sample pair with the highest confidence as the best sample for final matting Yes; then smooth the local area of opacity to obtain the final estimated opacity; the opacity calculation module includes: Linear-color difference similarity calculation unit: used to obtain the sample point set obtained by the module according to the sample point set, take out the sample points one by one, and calculate their linear-color similarity; Color similarity calculation unit: used to obtain the sample point set obtained by the module according to the sample point set, take out the sample points one by one, and calculate their color similarity; Spatial distance similarity calculation unit: used to obtain the sample point set obtained by the module according to the sample point set, take out the sample points one by one, and calculate their spatial distance similarity; Sample screening unit: used to calculate the confidence of each pair of sample points relative to the current unknown pixel according to the similarity values obtained from the linear-color difference similarity calculation unit, the color similarity calculation unit and the spatial distance similarity calculation unit; select the confidence The highest degree of opacity is used as an estimate of the opacity of the current position pixel; Smoothing unit: used for local smoothing based on the opacity obtained by the sample screening unit; the factors considered during smoothing include: difference in color value, difference in spatial position, and difference in significance; The foreground keying module is used to perform a keying operation in the original image according to the final estimated opacity and the color value of the best sample pair to extract the foreground target. 6. a kind of automatic matting device according to claim 5, is characterized in that, described matting visual salience calculation module comprises: The scale pyramid generation unit is used to perform smoothing and downsampling according to the acquired image to be matted to generate a scale pyramid; The luminance saliency calculation unit is used to generate the scale pyramid obtained by the unit according to the scale pyramid, taking the image on the fine scale as the central area of vision, and the image on the coarse scale as the peripheral area of vision, and calculates the luminance saliency map; The color saliency calculation unit is used to generate the scale pyramid obtained by the scale pyramid generation unit, taking the image on the fine scale as the central area of vision, and the image on the coarse scale as the peripheral area of vision, and calculates the color saliency map; The regional saliency calculation unit is used to perform superpixel segmentation on the foreground target according to the to-be-mashed image obtained by the image acquisition module, and cluster the superpixels according to the color histogram, and calculate the color saliency of each clustered region; The saliency fusion unit is used to fuse the brightness saliency map obtained by the brightness saliency calculation unit, the color saliency map obtained by the color saliency calculation unit, and the regional saliency map obtained by the regional saliency calculation unit to obtain the image to be keyed. Cutout visual saliency map. 7. a kind of automatic matting device according to claim 5, is characterized in that, described tripartite graph calculation module comprises: The spatial domain filtering unit is used to select a suitable spatial domain filtering method to smooth the visual saliency map of the matting; The threshold segmentation unit is used to obtain the foreground area and the background area by using a threshold segmentation algorithm to obtain a rough three-part map according to the smooth matting visual saliency map obtained by the spatial domain filtering unit; Morphological operation unit: It is used to perform morphological operations to fill the holes according to the coarse three-part map obtained by the threshold segmentation unit, and obtain the foreground, background and unknown area, that is, an accurate three-part map. 8. a kind of automatic matting device according to claim 5, is characterized in that, described sample point set acquisition module comprises: The gradient calculation unit is used to obtain the gradient of each unknown pixel according to the gray value of the image to be keyed; The sampling unit is used to draw a straight line according to the gradient direction obtained by the gradient calculation module, and take the first intersection pair of the straight line and the foreground area and the background area as the initial search point. In the neighborhood of the search point, search and unknown Sample points whose pixel saliency value difference is less than the threshold.

Images