CN104318514A - Three-dimensional significance based image warping method - Google Patents

Abstract

The invention discloses a three-dimensional significance based image warping method. The three-dimensional significance based image warping method is characterized by the following steps of 1 obtaining a depth map of a target image through depth data; 2 combining the depth map and a two-dimensional model to construct a three-dimensional significance model; 3 updating a weight between the depth data and the two-dimensional model according to distribution of image gradation in a self-adapting mode; 4 calculating the gradient of an image energy function through the three-dimensional significance model; 5 extracting an image two-dimensional edge and a depth contour feature and generating into a triangular mesh through feature points; 6 establishing a target function and increasing constraint; 7 requesting an extreme value for the target function and constructing a transformational relation. The three-dimensional significance based image warping method can combine the depth information and the two-dimensional significance and increases the warping robustness.

Description

Based on the image warping method of three-dimensional significance

Technical field

The invention belongs to image processing field, relate generally to a kind of image warping method based on three-dimensional significance.

Background technology

Universal along with mobile device such as smart mobile phone and panel computer, the photo upload taken by them is shared with friend by user habit to social network sites.Consider that the model of the mobile device that user uses is different, the photo how making user share can show well on different terminals, is one of heat subject of current computer vision field research.

For the problems referred to above, researchers propose certain methods, but inadequate due to distortion and accuracy of detection, and the distortion of objects in images and the loss of part important information annoying researchers all the time.In 2009, international top-level meeting IEEE International Conference on Image Processing having delivered article " Saliency detection for content-aware image resizing " this article recently proposes to utilize the significance calculated in target image image to be carried out to the constraint of distortion, and the point making significance in distort process large is preserved and sacrifices the little point of significance.But when scene more complicated, texture and the object in image is more, the method of this article is utilized to calculate the significance of many points in image all very large, if retained these points, image just can not get effective distortion, if give up these points, some important profile informations will be cast out.Up to the present, still do not occur that one can ensure that objects in images can not be out of shape or lose, the method for distortion can be carried out again the image of scene more complicated.

Summary of the invention

The present invention is intended to solve Most current image warping method and carries out target image producing distortion, lost part important information to the object in image and effectively can not carrying out the problem of distortion to the image having complex scene in the process of distortion, a kind of image warping method based on three-dimensional significance is proposed, depth information and two-dimentional significance can be combined, and strengthen the robustness of distortion.

The present invention is that technical solution problem adopts following technical scheme:

The feature of a kind of image warping method based on three-dimensional significance of the present invention is carried out as follows:

Step 1: utilize formula (1) computed image size to be the energy function E of each pixel in the target image I of m × n:

$E(x,y)=\left|\frac{\mathrm{\δ}}{\mathrm{\δx}}I(x,y)\right|+\left|\frac{\mathrm{\δ}}{\mathrm{\δy}}I(x,y)\right|---\left(1\right)$

In formula (1), E (x, y) is the energy value of described target image I at pixel (x, y) place; I (x, y) is the gray-scale value of described target image I at pixel (x, y) place; X ∈ (0, m); Y ∈ (0, n);

Step 2: carry out feature extraction to described target image I, obtains two dimensional character matrix X;

Step 3, formula (2) is utilized to obtain the two-dimentional significance S of described target image I _2D:

$S_{2D}=\exp \left(\frac{-{\left|\right|X_i-X_j\left|\right|}^2}{{2\mathrm{\σ}}^2}\right)---\left(2\right)$

In formula (2), X _i, X _jbe respectively described two dimensional character matrix two different rows vectors; σ is constant;

Step 4, formula (3) is utilized to build three-dimensional significance model S _3D:

S _3D＝(1-α)S _2D+α·E _depth (3)

In formula (3), E _depthfor the depth map utilizing 3D camera to obtain described target image I, α is auto-adaptive parameter; And have:

$\mathrm{\α}=\frac{\underset{x,y=0}{\overset{m,n}{\mathrm{\Σ}}}n(x,y)\·{(I(x,y)-\stackrel{\‾}{I(x,y)})}^2}{D_{\max }}---\left(4\right)$

In formula (4), n (x, y) represents the number of pixels equaling pixel (x, y) gray-scale value; D _maxfor constant;

Step 5: utilize formula (1) and (3) that described energy function E is newly defined as E':

E'(x,y)＝E(x,y)·S _3D(x,y) (5)

In formula (5), E'(x, y) for described target image I is at the new energy value at pixel (x, y) place;

Step 6: utilize formula (6) to calculate the image significance S of described target image I:

$S((x_b,n),(x_a,n-1))=\underset{a=1}{\overset{b-1}{\mathrm{\Σ}}}|G_{a,n}^v-G_{a,n}^d|+\underset{a=a+1}{\overset{b}{\mathrm{\Σ}}}\left|G_{a,n}^v{G}_{a-1,n}^d\right|---\left(6\right)$

In formula (6), (x _b, n) be b pixel of the n-th row in described target image I, (x _a, n-1) and be the (n-1)th a pixel arranged in described target image I, a ≠ b, and a, b ∈ (0, m); S ((x _b, n), (x _a, n-1)) to represent in described target image I that n-th arranges b pixel x _ba pixel x is arranged with (n-1)th _aenergy differences; represent the gradient on described target image I horizontal direction v; And have represent the gradient on described target image I diagonal d, and have $G_{a,n}^d=|E_{a,n}^{\′}-E_{a+1,n-1}^{\′}|;$

Step 7, depth map according to described target image I, obtain the profile of body surface in described target image I;

Step 8, Delaunay trigonometry is utilized to be coupled together and build triangle gridding on described target image by each point in described two dimensional character matrix X; Described triangle gridding comprises several triangles t;

Step 9, formula (7) is utilized to carry out distortion to triangles all in described triangle gridding:

$G_t=\left(\begin{array}{cc}{s}_t^{\mathrm{\α}}& 0\\ 0& s_t^{\mathrm{\β}}\end{array}\right)---\left(7\right)$

In formula (7), with be respectively the distortion of described triangle t on transverse direction α and longitudinal direction β; G _trepresent the warp function to any one triangle t;

Step 10, formula (8) is utilized to obtain target equation E _s:

$E_s=\underset{t\∈T}{\mathrm{\Σ}}{S}_t{A}_t{\left|\right|J_t\left(q\right)-G_t\left|\right|}^2---\left(8\right)$

In formula (8), T is the set of described triangle t; A _tfor the area of described triangle t, J _tq () expression carries out the Jacobin matrix after distortion to described triangle t; S _tfor the significance of any one triangle t;

Step 11: utilize formula (9) to define constraint condition E _f:

$E_f=\underset{t\∈T}{\overset{T}{\mathrm{\Σ}}}{\left|\right|(c_{t2}^q-c_{t1}^q)-r_i{R}_i(c_{t1}^q-c_t^q)\left|\right|}^2---\left(9\right)$

In formula (9), for three summits without distortion on described triangle t, for three summits of three summits on described triangle t after distortion; r _iit is summit and summit between limit and summit and summit between the side ratio on limit, R _tit is summit and summit between limit and summit and summit between the rotation matrix that forms of limit;

Step 12: utilize formula (10) to obtain distortion matrix F:

F＝λE _s+(1-λ)E _f (10)

In formula (10), λ is coefficient; Translation or the distortion of rotation realization to described target image I is carried out according to the value that distortion matrix F is corresponding with triangle t each in described target image I.

Compared with prior art, beneficial effect of the present invention is embodied in:

1, the method for target image being carried out to energy function calculating based on L1 normal form of classics combines with depth information and proposes a kind of three-dimensional significance newly by the present invention, both ensure that the advantage of original two-dimentional significance, and with the addition of again depth information and make still can well carry out distortion when scene more complicated in target image.

2, the parameter in the present invention between two-dimentional significance and depth information is according to the adaptive adjustment of the distribution of gray scale in image, when intensity profile in image is more single and image scene is more single, the weight of two dimension significance, by the weight higher than depth information, therefore remains the advantage of conventional two-dimensional significance; And when intensity profile is more extensive and image scene more complicated time, the weight of depth information, higher than two-dimentional significance, therefore better can carry out distortion to the image of complex scene.

3, because the precision of two-dimentional significance is subject to the comparatively large and precision of depth information of the impact of ambient light photograph hardly by extraneous illumination effect, therefore the inventive method to external world environment there is stronger noise immunity.

4, the present invention is owing to the addition of depth information and triangle gridding as constraint, both the size of the significance of each point had been considered, consider the integrality at objects in images edge simultaneously, the large significance of two somes significance on such hypothesis same limit is little, and final distortion result also can not make this edge that the situations such as fracture occur; Thus ensure that the robustness of scalloping.

Accompanying drawing explanation

Fig. 1 is target image of the present invention;

Fig. 2 is the distortion result figure of the present invention to target image.

Embodiment

In the present embodiment, a kind of image warping method based on three-dimensional significance is the depth map utilizing depth data to obtain target image; Then depth map and two dimensional model are combined and build three-dimensional significance model; Secondly according to the weight between the adaptive renewal depth data of the distribution of gradation of image and two dimensional model; Again utilize three-dimensional significance model computed image energy function gradient; Extract two-dimensional image edge and depth profile feature afterwards, utilize unique point to generate triangle gridding; Final establishing target function is asked for extreme value and adds constraint, obtains distortion matrix; Carry out as follows specifically:

Step 1: utilize formula (1) computed image size to be the energy function E of each pixel in the target image I of m × n: this energy function is obtained by the gradient utilizing L1 normal form and calculate each pixel of target image; Gradient information reflects the edge of objects in images usually, can effectively ensure the complete of image in tailoring process, target image as shown in Figure 1:

$E(x,y)=\left|\frac{\mathrm{\δ}}{\mathrm{\δx}}I(x,y)\right|+\left|\frac{\mathrm{\δ}}{\mathrm{\δy}}I(x,y)\right|---\left(1\right)$

In formula (1), E (x, y) is for target image I is at the energy value at pixel (x, y) place; I (x, y) is for target image I is at the gray-scale value at pixel (x, y) place; X ∈ (0, m); Y ∈ (0, n);

Step 2: feature extraction is carried out to target image I, the method of image being carried out to feature extraction is unrestricted, and the feature extracting methods such as Local Binary Pattern (LBP) or SIFT such as can be adopted can to obtain obtaining two dimensional character matrix X;

Step 3, formula (2) is utilized to obtain the two-dimentional significance S of target image I _2D:

$S_{2D}=\exp \left(\frac{-{\left|\right|X_i-X_j\left|\right|}^2}{{2\mathrm{\σ}}^2}\right)---\left(2\right)$

In formula (2), X _i, X _jbe respectively two dimensional character matrix two different rows vectors; σ is constant;

Step 4, formula (3) is utilized to build three-dimensional significance model S _3D: depth information combines by model on original two-dimentional significance basis, and according to the difference of image, the weights relation between Automatic adjusument two dimension significance and depth information;

S _3D＝(1-α)S _2D+α·E _depth (3)

In formula (3), E _depthfor the depth map utilizing 3D camera to obtain target image I, α is auto-adaptive parameter; And have:

$\mathrm{\α}=\frac{\underset{x,y=0}{\overset{m,n}{\mathrm{\Σ}}}n(x,y)\·{(I(x,y)-\stackrel{\‾}{I(x,y)})}^2}{D_{\max }}---\left(4\right)$

In formula (4), n (x, y) represents the number of pixels equaling pixel (x, y) gray-scale value; D _maxfor constant; When scene more complicated in image, the distribution of its gray-scale value is by more concentrated, and in contrast, when the scene in image is fairly simple, the distribution of its gray-scale value is just more sparse; When the scene of image is fairly simple, utilizes traditional two-dimentional significance just can calculate the significance of image well, and add that depth information has come to calculate the significance of image when scene more complicated with regard to needing;

Step 5: utilize formula (1) and (3) that energy function E is newly defined as E':

E'(x,y)＝E(x,y)·S _3D(x,y) (5)

In formula (5), E'(x, y) for target image I is at the new energy value at pixel (x, y) place;

Step 6: utilize formula (6) to calculate the image significance S of target image I:

$S((x_b,n),(x_a,n-1))=\underset{a=1}{\overset{b-1}{\mathrm{\Σ}}}|G_{a,n}^v-G_{a,n}^d|+\underset{a=a+1}{\overset{b}{\mathrm{\Σ}}}\left|G_{a,n}^v{G}_{a-1,n}^d\right|---\left(6\right)$

In formula (6), (x _b, n) be b pixel of the n-th row in target image I, (x _a, n-1) and be the (n-1)th a pixel arranged in target image I, a ≠ b, and a, b ∈ (0, m); S ((x _b, n), (x _a, n-1)) to represent in target image I that n-th arranges b pixel x _ba pixel x is arranged with (n-1)th _aenergy differences; represent the gradient on target image I horizontal direction v; And have represent the gradient on target image I diagonal d, and have by calculating the difference of the energy function in each pixel horizontal direction and diagonal, calculating the significance of each point thus obtaining the significance of whole image;

Step 7, according to the depth information in the depth map of target image I, gradient is asked for the depth information of each point, obtain the profile of body surface in target image I;

Step 8, Delaunay trigonometry is utilized to be coupled together and build triangle gridding on target image by point each in two dimensional character matrix X; Triangle gridding comprises several triangles t; Delaunay trigonometry utilizes the character of discrete point, the while of being linked to be leg-of-mutton, ensure that immediate three discrete points and calculates and all can obtain identical triangular mesh from which point;

Step 9, formula (7) is utilized to carry out distortion to triangles all in triangle gridding:

$G_t=\left(\begin{array}{cc}{s}_t^{\mathrm{\α}}& 0\\ 0& s_t^{\mathrm{\β}}\end{array}\right)---\left(7\right)$

In formula (7), with be respectively the distortion of triangle t on transverse direction α and longitudinal direction β; G _trepresent the warp function to any one triangle t;

Step 10, formula (8) is utilized to obtain target equation E _s:

$E_s=\underset{t\∈T}{\mathrm{\Σ}}{S}_t{A}_t{\left|\right|J_t\left(q\right)-G_t\left|\right|}^2---\left(8\right)$

In formula (8), T is the set of triangle t; A _tfor the area of triangle t, J _tq () expression diabolo t carries out the Jacobin matrix after distortion; S _tfor the significance of any one triangle t, this target equation is minimized, just can obtain optimum transformation matrix;

Step 11: the profiling object surface obtained to prevent depth information can not because of conversion distortion too many, therefore need on the basis of above-mentioned objective function, add a constraint: utilize formula (9) to define constraint condition E _f:

$E_f=\underset{t\∈T}{\overset{T}{\mathrm{\Σ}}}{\left|\right|(c_{t2}^q-c_{t1}^q)-r_i{R}_i(c_{t1}^q-c_t^q)\left|\right|}^2---\left(9\right)$

In formula (9), for three summits without distortion on triangle t, for three summits of the summit of three on triangle t after distortion; r _iit is summit and summit between limit and summit and summit between the side ratio on limit, R _tit is summit and summit between limit and summit and summit between the rotation matrix that forms of limit; By minimizing this objective function, make the shape of former three points as far as possible similar to the shape of three points after conversion;

Step 12: utilize formula (10) to obtain distortion matrix F:

F＝λE _s+(1-λ)E _f (10)

In formula (10), λ is coefficient; Carry out translation or the distortion of rotation realization to target image I with triangle t each in target image I according to the value that distortion matrix F is corresponding, thus both saved the high point of significance in image, in turn ensure that validity.

Claims (1)

1., based on an image warping method for three-dimensional significance, it is characterized in that carrying out as follows:

Step 1: utilize formula (1) computed image size to be the energy function E of each pixel in the target image I of m × n:

$E(x,y)=\left|\frac{\mathrm{\δ}}{\mathrm{\δx}}I(x,y)\right|+\left|\frac{\mathrm{\δ}}{\mathrm{\δy}}I(x,y)\right|---\left(1\right)$

In formula (1), E (x, y) is the energy value of described target image I at pixel (x, y) place; I (x, y) is the gray-scale value of described target image I at pixel (x, y) place; X ∈ (0, m); Y ∈ (0, n);

Step 2: carry out feature extraction to described target image I, obtains two dimensional character matrix X;

Step 3, formula (2) is utilized to obtain the two-dimentional significance S of described target image I _2D:

$S_{2D}=\exp \left(\frac{-{\left|\right|X_i-X_j\left|\right|}^2}{2{\mathrm{\σ}}^2}\right)---\left(2\right)$

In formula (2), X _i, X _jbe respectively described two dimensional character matrix two different rows vectors; σ is constant;

Step 4, formula (3) is utilized to build three-dimensional significance model S _3D:

S _3D＝(1-α)S _2D+α·E _depth (3)

In formula (3), E _depthfor the depth map utilizing 3D camera to obtain described target image I, α is auto-adaptive parameter; And have:

$\mathrm{\α}=\frac{\underset{x,y=0}{\overset{m,n}{\mathrm{\Σ}}}n(x,y)\·{(I(x,y)-\stackrel{\‾}{I(x,y)})}^2}{D_{\max }}---\left(4\right)$

In formula (4), n (x, y) represents the number of pixels equaling pixel (x, y) gray-scale value; D _maxfor constant;

Step 5: utilize formula (1) and (3) that described energy function E is newly defined as E':

E'(x,y)＝E(x,y)·S _3D(x,y) (5)

In formula (5), E'(x, y) for described target image I is at the new energy value at pixel (x, y) place;

Step 6: utilize formula (6) to calculate the image significance S of described target image I:

$S((x_b,n),(x_a,n-1))=\underset{a=1}{\overset{b-1}{\mathrm{\Σ}}}|G_{a,n}^v-G_{a,n}^d|+\underset{a=a+1}{\overset{b}{\mathrm{\Σ}}}\left|G_{a,n}^v{G}_{a-1,n}^d\right|---\left(6\right)$

In formula (6), (x _b, n) be b pixel of the n-th row in described target image I, (x _a, n-1) and be the (n-1)th a pixel arranged in described target image I, a ≠ b, and a, b ∈ (0, m); S ((x _b, n), (x _a, n-1)) to represent in described target image I that n-th arranges b pixel x _ba pixel x is arranged with (n-1)th _aenergy differences; represent the gradient on described target image I horizontal direction v; And have represent the gradient on described target image I diagonal d, and have $G_{a,n}^d=|E_{a,n}^{\′}-E_{a+1,n-1}^{\′}|;$

Step 7, depth map according to described target image I, obtain the profile of body surface in described target image I;

Step 8, Delaunay trigonometry is utilized to be coupled together and build triangle gridding on described target image by each point in described two dimensional character matrix X; Described triangle gridding comprises several triangles t;

Step 9, formula (7) is utilized to carry out distortion to triangles all in described triangle gridding:

$G_t=\left(\begin{array}{cc}{s}_t^{\mathrm{\α}}& 0\\ 0& s_t^{\mathrm{\β}}\end{array}\right)---\left(7\right)$

In formula (7), with be respectively the distortion of described triangle t on transverse direction α and longitudinal direction β; G _trepresent the warp function to any one triangle t;

Step 10, formula (8) is utilized to obtain target equation E _s:

$E_s=\underset{t\∈T}{\mathrm{\Σ}}{S}_t{A}_t{\left|\right|J_t\left(q\right)-G_t\left|\right|}^2---\left(8\right)$

In formula (8), T is the set of described triangle t; A _tfor the area of described triangle t, J _tq () expression carries out the Jacobin matrix after distortion to described triangle t; S _tfor the significance of any one triangle t;

Step 11: utilize formula (9) to define constraint condition E _f:

$E_f=\underset{t\∈T}{\overset{T}{\mathrm{\Σ}}}{\left|\right|(c_{t2}^q-c_{t1}^q)-r_i{R}_i(c_{t1}^q-c_t^q)\left|\right|}^2---\left(9\right)$

In formula (9), for three summits without distortion on described triangle t for three summits of three summits on described triangle t after distortion; r _iit is summit and summit between limit and summit and summit between the side ratio on limit, R _tit is summit and summit between limit and summit and summit between the rotation matrix that forms of limit;

Step 12: utilize formula (10) to obtain distortion matrix F:

F＝λE _s+(1-λ)E _f (10)

In formula (10), λ is coefficient; Translation or the distortion of rotation realization to described target image I is carried out according to the value that distortion matrix F is corresponding with triangle t each in described target image I.