CN103955960B - Image viewpoint transformation method based on single input image - Google Patents

Abstract

The invention discloses an image viewpoint transformation method based on a single input image, which comprises the following steps: step 1, interactively deducting the cube structure contained in the image and marking the feature lines in the image; step 2, using the imaging principle of a pinhole camera to combine The spatial characteristics of cube objects reconstruct a 3D scene, which can completely simulate the real 3D scene when the viewpoint is changed; Step 3, without changing the focal length, reproject the reconstructed 3D scene to the new The image plane corresponding to the viewpoint; step 4, perform constrained Delaunay triangular meshing on the image according to the interaction constraints; step 5, use the scene theme structural constraints obtained by reprojection in step 3 as the deformation driver, and use the method of solving the energy equation to 4. Solve the grid deformation energy equation formed; step 6, use the target grid obtained from the solution to map the corresponding grid textures in the original image one by one to generate the target image.

Description

An Image Viewpoint Transformation Method Based on a Single Input Image

technical field

The invention relates to an image viewpoint transformation method based on a single input image, which belongs to the field of computer image processing and multimedia information technology processing.

Background technique

With the improvement of digital camera hardware and the development of image editing software, people can more easily obtain high-quality images. However, the photos taken by a large number of photographers are often unnatural due to the selection of the angle of view. For example, a photographer may take a picture of a building that was originally perpendicular to the ground, making the shooting effect unnatural. How to correct such a tilted photo of the main structure has a very good application value. In this case, people began to study the problem of image viewpoint conversion. The traditional method is to first use multiple images of the same scene for registration, then perform 3D reconstruction, and then use the existing 3D structure to reproject under the new viewpoint to render the image under the new viewpoint. The obvious disadvantage of this method is that it requires multiple input images or requires a lot of human interaction to accurately reconstruct the 3D scene, which is not feasible for a slightly complex 3D scene, so the applicability and efficiency of this type of method are very low, and the effect Also not good.

Contents of the invention

Purpose of the invention: The present invention provides an image viewpoint transformation method based on a single input image, so that the user can obtain the viewpoint-transformed image through minimal and convenient interaction and simple calculation.

Technical solution: The present invention discloses an image viewpoint transformation method based on a single input image, which mainly includes the following steps:

Step 1, image interaction: extract the cube structure in the image through manual interaction, specifically by marking the image projection pixels corresponding to six points of the cube, denoted as (p ₀ ,p ₁ ,p ₂ ,p ₃ , p ₄ ,p ₅ ), where the three-dimensional plane corresponding to (p ₀ ,p ₁ ,p ₂ ,p ₃ ) is perpendicular to the three-dimensional plane corresponding to (p ₀ ,p ₁ ,p ₅ ,p ₄ ), and p ₀ p ₁ corresponds to The space straight line is the intersection line of two planes, and the feature lines in the image are obtained interactively, including feature lines that affect image content, horizontal and vertical constraint line segments, and specified image boundary lines;

Step 2: Reconstruct a 3D scene by using the imaging principle of the pinhole camera combined with the spatial characteristics of the cube structure object. When the viewpoint changes, the scene structure is used to simulate the real 3D scene structure. In the assumed 3D scene, one of the vertices of the cube structure and its On the basis of the coincidence of image points on the image, use the geometric constraints contained in the cube to calculate the corresponding cube space point in the 3D scene of (p ₀ ,p ₁ ,p ₂ ,p ₃ ,p ₄ ,p ₅ ) coordinates(P ₀ ^T ,P ₁ ^T ,P ₂ ^T ,P ₃ ^T ,P ₄ ^T ,P ₅ ^T );

Step 3, use the cube corresponding to the spatial coordinates of the six points calculated in step 2 to reproject under the new viewpoint to represent the reconstructed 3D scene: take the origin O where the center of the simulated scene is located as the center of the sphere, and take the center of the simulated scene to A new viewpoint O' is defined on the spherical plane whose distance |OF| is the radius of the camera, and the reconstructed three-dimensional cube structure (P ₀ ^T , P ₁ ^T , P ₂ ^T , P ₃ ^T , P ₄ ^T , P ₅ ^T ) is re-projected to the image plane corresponding to the new viewpoint;

Step 4, image triangulation: According to the interactive structure of step 1, use the Delaunay triangulation with constraints to triangulate the image; the algorithm of Delaunay triangulation can be found in "Computational Geometry (3rd Edition) Berger, Published by World Book Publishing House on October 1, 2013, ISBN: 9787510061776”;

Step 5, grid parameterization: use conformal mapping and various feature-preserving constraint principles to establish grid parameterization equations, use the scene cube frame obtained by re-projection in step 4 to construct hard constraints for projection mapping, and solve parametric energy equations , get the target grid;

Step 6, target image generation: According to the grid obtained in step 5, the grid texture corresponding to the target grid in the original image is mapped to the corresponding target grid one by one by bilinear interpolation method to generate the target image.

In step 1, the hidden cube structure and feature lines of the image, which occupy the main structure of the image, are obtained through manual interaction.

Step 1 includes the following steps:

Step 11, the single input image contains a cubic structure hidden in the scene, manually and interactively extract the projection of the cubic structure in the image, and use the image projection pixels corresponding to six points of the cube (p ₀ , p ₁ ,p ₂ ,p ₃ ,p ₄ ,p ₅ ) to represent the cube structure, where (p ₀ ,p ₁ ,p ₂ ,p ₃ ) corresponds to a three-dimensional space plane orthogonal to (p ₀ ,p ₁ , p ₅ , p ₄ ) correspond to the three-dimensional plane, and the space line corresponding to p ₀ p ₁ is the intersection line of the two planes;

Step 12, according to the cube structure (p ₀ ,p ₁ ,p ₂ ,p ₃ ,p ₄ ,p ₅ ) obtained in step 11, convert the cube structure (p ₀ ,p ₁ ,p ₂ ,p ₃ ,p ₄ , p ₅ ) are marked as hard-constrained feature lines, and the triangular mesh vertices are uniformly extracted on these hard-constrained feature lines. The specific method is as follows: for a specific hard-constrained feature line, start from one end , take the next point on the line at intervals of 25 pixels along the direction of the line until the other end of the line. For the grid vertices obtained on all hard constraint feature lines, denote as {v _c1 ,v _c2 ,.. .,v _ck }, where ck represents the total number of vertices extracted here;

Step 13, interactively obtain the feature lines in the image, including feature lines that affect the content of the image, horizontal and vertical constraint line segments, and specified image boundary lines, and evenly take points on these lines. The specific method is as follows: constrain the line to a specific feature line, Take a point from one end, and take the next point on the line every 25 pixels along the line until the other end of the line;

Step 14, for the characteristic curve in the image, such as the original circular sun, extract the triangular grid points on the curve in turn, the specific method is as follows: start from one end of the characteristic curve, follow the elongation direction of the characteristic curve, and divide by about 25 pixel distance as the point interval until reaching the other end of the characteristic curve. .

In step 2, a simulated three-dimensional cube structure model is established by using the imaging principle of the pinhole camera and the spatial geometry of the cube object.

Step 2 specifically includes the following steps:

Step 21, take the center of the image as the origin O(0,0,0), O to the camera point F(0,0,f) is the positive direction of the Z axis, and the horizontal and vertical axes of the image plane are the XY directions to establish a three-dimensional The world coordinate system, where f is the focal length of the camera, refer to Figure 3 for details;

Step 22, calculate and obtain the intersection c 1 of the straight line p ₀ p ₃ and the straight line p ₁ p ₂ , the intersection c ₂ of the straight line p ₀ p ₄ and the straight line p _{1 p 5} , obviously in three-dimensional space, the straight line Fc ₁ and the straight line P ₀ ^T P ₃ ^T , P ₁ ^T P ₂ ^T intersect at infinity, obviously there is Fc ₁ flat and straight line P ₀ ^T P ₃ ^T , the same straight line Fc ₂ and straight line P ₀ ^T P ₄ ^T , P ₁ ^T P ₅ ^T intersect at infinity, obviously there is Fc ₂ flat and straight line P ₀ ^T P ₄ ^T , according to P ₀ ^T P ₃ ^T ⊥P ₀ ^T P ₅ ^T , we get Therefore, the solution obtains the focal length f;

Step 23, write down the six points on the cuboid in the real scene corresponding to the cuboid structure (p ₀ , p ₁ , p ₂ , p ₃ , p ₄ , p ₅ ) in the image as (Q ₀ , Q ₁ , Q ₂ , Q ₃ , Q ₄ , Q ₅ ), according to the pinhole imaging model: k=0,1,...,5, where representative vector and _Similarly , _{for (P 0 T ,} ^P _{1 T , P 2 T} ^{, P} ₃ ^T , P ₄ ^T , P ₅ ^T ), there must also be k=0,1,...,5 is established, where rk represents a vector and The ratio between, the value of fixed r ₀ is 1, using and Solve r ₁ , r ₂ , r ₃ , r ₄ , r ₅ ; when , P _i and Q _i are completely coincident, at this time we can completely calculate the real coordinates of the cube structure corresponding to (p ₀ ,p ₁ ,p ₂ ,p ₃ ,p ₄ ,p ₅ ) in the real scene, otherwise there is P _i ^T P _j ^T //Q _i Q _j is established, that is, the model formed by (P ₀ ^T ,P ₁ ^T ,P ₂ ^T ,P ₃ ^T ,P ₄ ^T ,P ₅ ^T ) is (Q ₀ ,Q ₁ , Q ₂ , Q ₃ , Q ₄ , Q ₅ ) constitute a scaling of the model;

Step 24, according to the ratios r ₀ , r ₁ , r ₂ , r ₃ , r ₄ , r ₅ obtained in step 23, calculate the corresponding cubic structure (p ₀ , p ₁ , p ₂ , p ₃ , p ₄ , p ₅ ) cube structure (P ₀ ^T , P ₁ ^T , P ₂ ^T , P ₃ ^T , P ₄ ^T , P ₅ ^T ), set P ₀ ^T to coincide with p ₀ , and define the center of the scene as Through the projection point of the vertex P ₀ ^T on the main optical axis in the cube structure in the scene, here is exactly the origin O, that is, the center of the simulated scene falls on the origin, corresponding to the projection point of Q ₀ on the main optical axis in the real scene , the real scene center O _Q .

In step 3, the simulated 3D cube structure is reprojected under the new viewpoint.

Step 3 specifically includes the following steps:

Step 31, according to the cubic structure (P ₀ ^T , P ₁ ^T , P ₂ ^T , P ₃ ^T , P ₄ ^T , P ₅ ^T ) obtained in step 24, on the spherical surface with O as the center |OF| Randomly pick a new viewpoint O', where and The included angle does not exceed 45 degrees. After simulating the center O of the scene, a plane I' perpendicular to OO' is constructed to obtain the point (P ₀ ^T , P ₁ ^T , P ₂ ^T , P ₃ ^T , P ₄ ^T , P ₅ ^T ) The intersection point (p' ₀ ,p ₁ ',p' ₂ ,p' ₃ ,p' ₄ ,p' ₅ ) of the connection line with O' and the plane I' completes the simulation of the cube structure in the simulation scene under the new viewpoint projection;

Step 32, the normal vector Cross product with the vertical vector (0,1,0) to get the unit vector x', and x' with The cross product obtains the unit vector y'. On the image plane I', take O as the origin, the direction of the vector x' is the positive direction of the abscissa, and the direction of the vector y' is the positive direction of the ordinate to establish a two-dimensional image coordinate system. i=0,1,2,...,5 are respectively projected onto the two directions of vectors x' and y' to obtain intersection points (p' ₀ ,p ₁ ',p' ₂ ,p' ₃ ,p' ₄ , The two-dimensional coordinates of p' ₅ ) on the image plane I' are recorded as (q ₀ , q ₁ , q ₂ , q ₃ , q ₄ , q ₅ ), so we get six points on the cube structure in the original image 2D coordinates projected on the target image under the new viewpoint.

In step 4, use the extracted mesh vertices to perform Delaunay triangulation on the image.

Step 4 specifically includes the following steps:

Step 41, on the basis of the feature grid points obtained in step 1, randomly and evenly pick points for the remaining regions of the image as grid vertices;

Step 42, in the process of triangulation, use the feature line of the image interactive in step 1, take the connection of adjacent vertices on the same feature line as the edge of the triangle mesh, and use the constrained Delaunay triangulation to triangulate the image, The algorithm of Delaunay triangulation can be found in "Computational Geometry (3rd Edition) published by Berg World Book Press on October 1, 2013, ISBN: 9787510061776".

In step 5, energy constraints for image deformation are constructed.

Step 5 specifically includes the following steps:

Step ₅₁ , use the pixel points _{( q 0 , q 1} , _{q 2 ,} q ₃ ,q ₄ ,q ₅ ), using Calculate the point v' _ci corresponding to the point v' _ci , where v _ci is the grid vertex on the original image's hard-constrained feature line, p _s , p _e is the two endpoints on the hard-constrained feature line where v _ci is located, q _s , q _e are points corresponding to p _s and p _e respectively, that is, {v _c1 ,v _c2 ,...,v _ck } is mapped to {v' _c1 ,v' _c2 ,. ..,v' _ck }, record its corresponding hard constraint relationship as F _C (v' _c1 ,v' _c2 ,...,v' _ck )=0, indicating that v _i is mapped to v during image deformation mapping _i ';

Step 52, define shape constraints: record the affine transformation of the two-dimensional triangular mesh as M:(x,y)→(x',y'), according to the Cauchy-Riemann equation:

$\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}$

Where i is the imaginary unit. Regarding the Cauchy-Riemann equation, please refer to "LSCM: Least squares conformal maps for automatic texture atlas generation, ACM Transactions on Graphics, 21(3): 362-372." The Jacobian corresponding to the mapping M The matrix has the following form:

$\mathrm{<span\; class="notranslate">\; J</span>}<span\; class="notranslate">\; =</span>\left[\begin{array}{cc}\mathrm{<span\; class="notranslate">\; a</span>}& \mathrm{<span\; class="notranslate">\; b</span>}\\ \mathrm{<span\; class="notranslate">\; b</span>}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; a</span>}\end{array}\right]<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Among them, the values of a and b are arbitrary, which are used to represent the relationship between the elements in the matrix J that are equal and opposite to each other;

Corresponding to the transformation from the original grid to the target grid, we use radial transformation; record the original triangle as T(v _ti ,v _tj ,v _tk ), v _ti ,v _tj ,v _tk are its three vertices, and the corresponding viewpoint transformation The deformed triangle to be solved is T'(v' _ti , v' _tj , v' _tk ), and v' _ti , v' _tj , v' _tk are three corresponding to v _ti , v _tj , v _tk Vertex, A=T'T ^-1 is an affine transformation matrix from T to T', T ^-1 has the following form:

${\mathrm{<span\; class="notranslate">\; T</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\left[\begin{array}{ccc}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}& {\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}& {\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\\ {\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}& {\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}& {\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ {\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}& {\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}& {\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}\end{array}\right]<span\; class="notranslate">\; =</span>{\left[\begin{array}{ccc}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; ti</span>}}& \stackrel{}{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; tj</span>}}}& {\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; tk</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; ti</span>}}& {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; tj</span>}}& {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; tk</span>}}\\ {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; ti</span>}}& {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; tj</span>}}& {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; tk</span>}}\end{array}\right]}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

Among them, x _ti , x _tj , x _tk are the abscissa coordinates of points v _ti , v _tj , v _tk on the image, y _ti , y _tj , y _tk are the vertical coordinates of points v _ti , v _tj , v _tk on the image Coordinates, z _ti , z _tj , z _tk are the vertical coordinates used to represent v _ti , v _tj , v _tk , where the uniform value is 1; a ₁ , a ₂ , a ₃ , b ₁ , b ₂ , b ₃ , c ₁ , c ₂ , and c ₃ are the parameter representations of each element in the matrix T ^-1 , and the values are constants obtained from calculation. The following energy equation of the conformal mapping is obtained from formulas (1) and (2):

E _TJ1 ＝a ₁ x' _ti +a ₂ x' _tj +a ₃ x' _tk +(b ₁ y' _ti +b ₂ y' _tj +b ₃ y' _tk ) (3)

E _TJ2 ＝b ₁ x' _ti +b ₂ x' _tj +b ₃ x' _tk -(a ₁ y' _ti +a ₂ y' _tj +a ₃ y' _tk )

Where x' _ti , x' _tj , x' _tk are the abscissa values of v' _ti , v' _tj , v' _tk respectively, y' _ti , y' _tj , y' _tk are respectively v' _ti , v' _tj , the ordinate value of v' _tk , combine them together to get:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; TJ</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; TJ</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

Define feature line constraints: record (v _i , v _j , v _k ) as three consecutive points on a feature line, keep and The ratio between r _lj and the rotation angle θ _lj , define the following objective equation:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; li</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; lj</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; lk</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; lk</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; lj</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; lj</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; lj</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; lj</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; li</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

in $R_{\mathrm{lj}}=\left(\begin{array}{cc}\cos {\mathrm{\&theta;}}_{\mathrm{lj}}& -\sin {\mathrm{\&theta;}}_{\mathrm{lj}}\\ \sin {\mathrm{\&theta;}}_{\mathrm{lj}}& \cos {\mathrm{\&theta;}}_{\mathrm{lj}}\end{array}\right),$

Define vertical constraints: record the grid points on the vertical line segment lv as {v _lv1 ,v _lv2 ,...,v _lvm }, and the vertical constraints are expressed as:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; V</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; lv</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\underset{\mathrm{<span\; class="notranslate">\; nv</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; lvm</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; nv</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; lv</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

Among them, x' _nv represents the abscissa of point v' _nv , and x' _lv1 represents the abscissa of point v' _lv1 .

Define horizontal constraints: record the grid points on the horizontal line segment lh as {v _lh1 ,v _lh2 ,...,v _lhm }, and the vertical constraints are expressed as: ${\mathrm{E.}}_h=\underset{\mathrm{lh}}{\mathrm{\&Sigma;}}\underset{\mathrm{no}=1}{\overset{\mathrm{lhm}}{\mathrm{\&Sigma;}}}{(x_{\mathrm{no}}^{\&prime;}-x_{\mathrm{lh}1}^{\&prime;})}^2---\left(7\right),$

Among them, y' _nv represents the abscissa of point v' _nv , and y _l ' _h1 represents the abscissa of point v' _lh1 .

According to user interaction designation, part or all of the boundary of the original image is designated as a non-rigid feature line constraint, and (v _bi , v _bj , v _bk ) are three consecutive feature vertices on the boundary feature line, keeping and The ratio between r _bj defines the boundary constraints E _B of the image as follows;

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; B</span>}}<span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bi</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bj</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bk</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bk</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bj</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; bj</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bj</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bi</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 8</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

where r _bj is and ratio between.

Step 53, integrating step 51 and step 52, expresses the energy equation of grid parameterization as follows:

argmaxλ _S E _S +λ _L E _L +λ _V E _V +λ _H E _H +λ _B E _B (9)

stF _C (v ₁ ',v' ₂ ,...,v' _k )＝0

Among them, λ _S , λ _L , λ _V , λ _H , and λ _B are the weights corresponding to each energy item. In the actual operation process, there are certain adjustments for different picture parameters. After multiple sets of experimental pictures, a set of experience is given. The parameter values are as follows: λ _S takes 1, λ _L and λ _B take 100, and λ _V and λ _H take 10.

In step 6, the target image is generated using the grid map.

Step 6 specifically includes the following steps:

Step 61, according to equation (9), solve the target triangular grid corresponding to each triangular grid in the original image, and map the image texture in the original triangular grid to the corresponding target triangular grid by bilinear interpolation method , forming the target image.

Beneficial effect: the present invention comprises the following advantages:

(1) It can handle the case of only a single scene graph and a simpler operation process, without requiring multiple scene graphs as input and avoiding complex image registration operations.

(2) Relatively small and convenient user interaction. Compared with the method of manually simulating viewpoint transformation, the interactive workload of the method of the present invention is greatly reduced, and only the cube structure of the main body and a small number of feature lines need to be manually marked interactively, which ensures the ease of use of the method.

(3) The authenticity of the effect. Since a single real photo is used as input and the resulting image is generated through image deformation, the resulting image has a good effect. In addition, the present invention supports the user to manually mark the feature lines that need to maintain the shape, so the resulting image is not prone to prominent deformation defects.

(4) Fast processing speed and robustness. Excluding user interaction, the method of the present invention can obtain the result image in about 0.1 second.

Description of drawings

The advantages of the above and/or other aspects of the present invention will become clearer as the present invention will be further described in detail with reference to the accompanying drawings and specific embodiments.

Fig. 1 is the basic flowchart of the method of the present invention.

Fig. 2 is a schematic diagram of an implicit cube structure for image interactive extraction described in the present invention.

Fig. 3 is a schematic diagram of the consistency between the simulated three-dimensional structure described in the present invention and the real three-dimensional reprojection.

Fig. 4 is an example flowchart of the implementation of the method of the present invention.

Fig. 5 is a comparison effect diagram between the results of the method of the present invention and the method realized by Adobe Systems (Adobe Systems) Lightroom software.

FIG. 6 is an effect diagram of the method of the present invention for simulating the observation of an image scene by a viewing sphere.

detailed description:

The process flow of this method is shown in Figure 1 . First, the cube structure contained in the image and the characteristic curve in the marked image are interactively deducted; then the simulated 3D structure is restored by using the imaging principle of the pinhole camera and the spatial characteristics of the cube object; Project to the image plane corresponding to the new viewpoint; triangulate the image according to the interactive constraints; use the result obtained in step 3 reprojection as the deformation driver, and use the method of solving the energy equation to solve the grid parametrically; finally through the texture map Interpolate to generate the target image. The present invention can perform three-dimensional simulation observation on objects of a single input image within a certain range of spherical field of view to form a continuous real-scene simulation. For details, refer to the example flow chart in FIG. 4 .

Specifically, as shown in Figure 1, an image viewpoint transformation method based on a single input image:

Step 1, image interaction: extract the cube structure in the image through manual interaction, specifically by marking the image projection pixels corresponding to six points of the cube (p ₀ , p ₁ , p ₂ , p ₃ , p ₄ , p ₅ ), where the three-dimensional plane corresponding to (p ₀ , p ₁ , p ₂ , p ₃ ) is perpendicular to the three-dimensional plane corresponding to (p ₀ , p ₁ , p ₅ , p ₄ ), and p ₀ p ₁ is the Intersection lines, as shown in Figure 2, interactively obtain feature lines in the image, including feature lines that affect image content, horizontal and vertical line segments, and specified image boundary lines;

Step 2: Reconstruct a 3D scene by using the imaging principle of the pinhole camera combined with the spatial characteristics of the cube structure object. When the viewpoint changes, the scene structure is used to simulate the real 3D scene structure. In the assumed 3D scene, one of the vertices of the cube structure and its On the basis of the coincidence of image points on the image, use the geometric constraints contained in the cube to calculate the corresponding cube space point in the 3D scene of (p ₀ ,p ₁ ,p ₂ ,p ₃ ,p ₄ ,p ₅ ) coordinates(P ₀ ^T ,P ₁ ^T ,P ₂ ^T ,P ₃ ^T ,P ₄ ^T ,P ₅ ^T );

Step 3, use the cube corresponding to the spatial coordinates of the six points calculated in step 2 to represent the reconstructed 3D scene: take the center of the simulated scene, that is, the origin O as the center of the sphere, and take the center of the simulated scene to The camera distance |OF| defines a new viewpoint O' on a spherical plane with a radius, which will reconstruct a three-dimensional cube structure (P ₀ ^T , P ₁ ^T , P ₂ ^T , P ₃ ^T , P ₄ ^T , P ₅ ^T ) Reproject to the image plane corresponding to the new viewpoint;

Step 4, image triangulation: According to the interactive structure of step 1, use the Delaunay triangulation with constraints to triangulate the image; the algorithm of Delaunay triangulation can be found in "Computational Geometry (3rd Edition) Berger, Published by World Book Publishing House on October 1, 2013, ISBN: 9787510061776";

Step 5, grid parameterization: use the principle of conformal mapping and various feature preservation constraints to establish grid parameterization equations, use the scene cube frame obtained by reprojection in step 4 to construct hard constraints for projection mapping, and solve the parameterized energy equation , get the target grid;

Step 6, target image generation: According to the grid obtained in step 5, the grid texture corresponding to the target grid in the original image is mapped to the corresponding target grid one by one by bilinear interpolation method to generate the target image.

In step 1, the hidden cube structure and feature lines of the image, which occupy the main structure of the image, are obtained through manual interaction.

Step 1 includes the following steps:

Step 11, as shown in Figure 2, extracts the projection structure of the hidden cube structure in the image through manual interaction. We generally select such six points (p ₀ ,p ₁ ,p ₂ ,p ₃ ,p ₄ ,p ₅ ), which corresponds to 6 points (Q ₀ , Q ₁ , Q ₂ , Q ₃ , Q ₄ , Q ₅ ) on the cuboid in the real scene, where (Q ₀ , Q ₁ , Q ₂ , Q ₃ ) are in On the same cube surface, (Q ₀ ,Q ₁ ,Q ₃ ,Q ₄ ) are on the same cube surface, and there are Q ₀ Q ₃ //Q ₁ Q ₂ , Q ₀ Q ₄ //Q ₁ Q ₅ , Q ₀ Q ₄ ⊥Q ₀ Q ₁ , Q ₀ Q ₃ ⊥Q ₀ Q ₁ ;

Step 12, according to the cube structure (p ₀ ,p ₁ ,p ₂ ,p ₃ ,p ₄ ,p ₅ ) obtained in step 11, convert the cube structure (p ₀ ,p ₁ ,p ₂ ,p ₃ ,p ₄ , p ₅ ) are marked as hard-constrained feature lines, and the triangular mesh vertices are uniformly extracted on these hard-constrained feature lines. The specific method is as follows: for a specific hard-constrained feature line, start from one end , take the next point on the line at intervals of 25 pixels along the direction of the line until the other end of the line. For the grid vertices obtained on all hard constraint feature lines, denote as {v _c1 ,v _c2 ,.. .,v _ck }, where ck represents the total number of vertices extracted from the hard-constrained mesh;

Step 13, interactively obtain the feature lines in the image, including feature line segments that affect image content, horizontal and vertical constraint line segments, and specified image boundary lines, and evenly take points on these line segments. The specific method is as follows: For a specific feature line, we start from Points are taken from one end of the line, and the next point on the line is taken every 25 pixels along the line until the other end of the line;

Step 14, for the characteristic curve in the image, such as the original circular sun, extract triangular grid points on the curve in turn, the specific method is as follows: start from one end of a characteristic curve, follow the elongation direction of the characteristic curve, and move by about The distance of 25 pixels is taken as the point interval until reaching the other end of the characteristic curve.

In step 2, a simulated three-dimensional cube structure model is established by using the imaging principle of the pinhole camera and the spatial geometry of the cube object.

Step 2 specifically includes the following steps:

Step 21, take the center of the image as the origin O(0,0,0), O to the camera point F(0,0,f) is the positive direction of the Z axis, and the horizontal and vertical axes of the image plane are the XY directions to establish a three-dimensional The world coordinate system, where f is the focal length of the camera, refer to Figure 3 for details;

Step 22, calculate and obtain the intersection c 1 of the straight line p ₀ p ₃ and the straight line p ₁ p ₂ , the intersection c ₂ of the straight line p ₀ p ₄ and the straight line p _{1 p 5} , obviously in three-dimensional space, the straight line Fc ₁ and the straight line P ₀ ^T P ₃ ^T , P ₁ ^T P ₂ ^T intersect at infinity, obviously there is Fc ₁ flat and straight line P ₀ ^T P ₃ ^T , the same straight line Fc ₂ and straight line P ₀ ^T P ₄ ^T , P ₁ ^T P ₅ ^T intersect at infinity, obviously there is Fc ₂ flat and straight line P ₀ ^T P ₄ ^T , according to P ₀ ^T P ₃ ^T ⊥P ₀ ^T P ₅ ^T , we get Therefore, the solution obtains the focal length f;

Step 23, record the six points on the cube in the real scene corresponding to the cube structure (p ₀ , p ₁ , p ₂ , p ₃ , p ₄ , p ₅ ) in the original image as the cube structure (Q ₀ , Q ₁ , Q ₂ ,Q ₃ ,Q ₄ ,Q ₅ ), obviously we have k=0,1,...,5, where representative vector and _{The ratio} ^between , for the _cube structure _{( P 0 T ,} P ₁ ^T ,P ₂ ^T ,P ₃ ^T ,P ₄ ^T ,P ₅ ^T ), must also have k=0,1,...,5 where r _k represents the vector and The ratio between, the value of fixed r ₀ is 1, using Solve r ₁ , r ₂ , r ₃ , r ₄ , r ₅ ; when , P _i and Q _i are completely coincident, at this time we can completely calculate the real coordinates of the cube structure corresponding to (p ₀ ,p ₁ ,p ₂ ,p ₃ ,p ₄ ,p ₅ ) in the real scene, otherwise there is P _i P _j //Q _i Q _j is established, that is, the model formed by (P ₀ ^T ,P ₁ ^T ,P ₂ ^T ,P ₃ ^T ,P ₄ ^T ,P ₅ ^T ) is (Q ₀ ,Q ₁ ,Q ₂ ,Q ₃ ,Q ₄ ,Q ₅ ) constitute a scaling of the model;

Step 24, according to the ratios r ₀ , r ₁ , r ₂ , r ₃ , r ₄ , and r ₅ obtained in step 23, calculate the corresponding cubic structure (p ₀ , p ₁ , p ₂ , p ₃ , p ₄ ,p ₅ ) cube structure (P ₀ ^T ,P ₁ ^T ,P ₂ ^T ,P ₃ ^T ,P ₄ ^T ,P ₅ ^T ), set P ₀ ^T to coincide with p ₀ , define the center of the scene as over The projection point of the vertex P ₀ ^T on the main optical axis in the cube structure in the scene, here is the origin O, that is, the center of the simulated scene falls on the origin, corresponding to the projection point of Q ₀ on the main optical axis in the real scene, the real The scene center O _Q .

In step 3, the simulated 3D cube structure is reprojected under the new viewpoint.

Step 3 specifically includes the following steps:

Step 31, according to the simulated cubic structure (P ₀ ^T , P ₁ ^T , P ₂ ^T , P ₃ ^T , P ₄ ^T , P ₅ ^T ) obtained in step 2, we use O as the center of the circle |OF| Randomly pick a new viewpoint O' on the sphere, where and The included angle does not exceed 45 degrees, and the plane I' perpendicular to OO' is constructed by simulating the center O of the scene to obtain the point (P ₀ ^T , P ₁ ^T , P ₂ ^T , P ₃ ^T , P ₄ ^T , P ₅ ^T ) The new intersection point (p' ₀ ,p ₁ ',p' ₂ ,p' ₃ ,p' ₄ ,p' ₅ ) of the connection line with O' and the plane I' completes the cube structure in the simulation scene under the new viewpoint the projection;

Step 32, the normal vector Cross product with the vertical vector (0,1,0) to get the unit vector x', and x' with The cross product obtains the unit vector y'. On the image plane I', take O as the origin, the direction of the vector x' is the positive direction of the abscissa, and the direction of the vector y' is the positive direction of the ordinate to establish a two-dimensional image coordinate system. k=0,1,2,...,5 are respectively projected onto the two directions of vectors x' and y' to obtain the intersection point (p' ₀ ,p ₁ ',p' ₂ ,p' ₃ ,p' ₄ , The two-dimensional coordinates of p' ₅ ) on the image plane I' are recorded as points (q ₀ , q ₁ , q ₂ , q ₃ , q ₄ , q ₅ ), so that we get six The 2D coordinates of the point projected on the target image under the new viewpoint.

In step 4, use the extracted mesh vertices to perform Delaunay triangulation on the image.

Step 4 specifically includes the following steps:

Step 41, on the basis of the feature grid points obtained in step 1, randomly and evenly pick points for the remaining regions of the image as grid vertices;

Step 42, in the process of triangulation, use the feature line of the image interactive in step 1, take the connection of adjacent vertices on the same feature line as the edge of the triangle mesh, and use the constrained Delaunay triangulation to triangulate the image, The algorithm of Delaunay triangulation can be found in "Computational Geometry (3rd Edition) published by Berg World Book Press on October 1, 2013, ISBN: 9787510061776".

In step 5, energy constraints for image deformation are constructed.

Step 5 specifically includes the following steps:

Step ₅₁ , use the pixel points _{( q 0 , q 1} , _{q 2 ,} q ₃ ,q ₄ ,q ₅ ), using Calculate the point v' _ci corresponding to the point v _ci , where v _ci is the grid vertex on the original image hard-constrained feature line, p _s , _pe are the two endpoints on the hard-constrained feature line where v _ci is located, q _s , q _e are points corresponding to p _s and p _e respectively, that is, {v _c1 ,v _c2 ,...,v _ck } is mapped to {v' _c1 ,v' _c2 ,.. .,v' _ck }, record its corresponding hard constraint relationship as F _C (v' _c1 ,v' _c2 ,...,v' _ck )=0, indicating that v _ci is mapped to v' during image deformation mapping _ci ;

Step 52, define shape constraints: record the affine transformation of the two-dimensional triangular mesh as M:(x,y)→(x',y'), according to the Cauchy-Riemann equation:

$\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}$

Where i is the imaginary unit. Regarding the Cauchy-Riemann equation, please refer to "LSCM: Least squares conformal maps for automatic texture atlas generation, ACM Transactions on Graphics, 21(3): 362-372." The Jacobian corresponding to the mapping M The matrix has the following form:

Among them, a and b take any value, which is used to represent the relationship between the elements in the matrix J that are equal and opposite to each other;

Corresponding to the transformation from the original grid to the target grid, we use radial transformation; record the original triangle as T(v _ti ,v _tj ,v _tk ), v _ti ,v _tj ,v _tk are its three vertices, and the corresponding viewpoint transformation The deformed triangle to be solved is T'(v' _ti , v' _tj , v' _tk ), and v' _ti , v' _tj , v' _tk are three corresponding to v _ti , v _tj , v _tk Vertex, A=T'T ^-1 is an affine transformation matrix from T to T', T ^-1 has the following form:

${\mathrm{<span\; class="notranslate">\; T</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\left[\begin{array}{ccc}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}& {\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}& {\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\\ {\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}& {\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}& {\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ {\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}& {\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}& {\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}\end{array}\right]<span\; class="notranslate">\; =</span>{\left[\begin{array}{ccc}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; ti</span>}}& \stackrel{}{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; tj</span>}}}& {\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; tk</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; ti</span>}}& {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; tj</span>}}& {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; tk</span>}}\\ {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; ti</span>}}& {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; tj</span>}}& {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; tk</span>}}\end{array}\right]}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

x _ti , x _tj , x _tk are the abscissas of v _ti , v _tj , v _tk on the image, y _ti , y _tj , y _tk are the ordinates of v _ti , v _tj , v _tk on the image, z _ti , z _tj , z _tk are the vertical coordinates used to represent v _ti , v _tj , v _tk , where the uniform value is 1; a ₁ , a ₂ , a ₃ , b ₁ , b ₂ , b ₃ , c ₁ , c ₂ , and c ₃ are the parameter representations of each element in the matrix T-1, and the values are all constants obtained by calculation. The following energy equation of the conformal mapping is deduced from formulas (1) and (2):

E _TJ1 ＝a ₁ x' _ti +a ₂ x' _tj +a ₃ x' _tk +(b ₁ y' _ti +b ₂ y' _tj +b ₃ y' _tk ) (3)

E _TJ2 ＝b ₁ x' _ti +b ₂ x' _tj +b ₃ x' _tk -(a ₁ y' _ti +a ₂ y' _tj +a ₃ y' _tk )

Where x' _ti , x' _tj , x' _tk are the abscissa values of v' _ti , v' _tj , v' _tk respectively, y' _ti , y' _tj , y' _tk are respectively v' _ti , v' _tj , the ordinate value of v' _tk , combine them together to get:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; TJ</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; TJ</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

Define feature line constraints: record (v _i , v _j , v _k ) as three consecutive points on a feature line, keep and The ratio between r _lj and the rotation angle θ _lj , define the following objective equation:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; li</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; lj</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; lk</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; lk</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; lj</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; lj</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; lj</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; lj</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; li</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

in $R_{\mathrm{lj}}=\left(\begin{array}{cc}\cos {\mathrm{\&theta;}}_{\mathrm{lj}}& -\sin {\mathrm{\&theta;}}_{\mathrm{lj}}\\ \sin {\mathrm{\&theta;}}_{\mathrm{lj}}& \cos {\mathrm{\&theta;}}_{\mathrm{lj}}\end{array}\right),$

Define vertical and horizontal constraints: In the target image, we should strive to maintain the original vertical or horizontal straight line, without loss of generality, assuming that the grid points on the vertical line segment lv are recorded as {v _lv1 ,v _lv2 ,...,v _lvm } , the vertical constraint can be expressed as:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; V</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; lv</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\underset{\mathrm{<span\; class="notranslate">\; nv</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; lvm</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; nv</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; lv</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

Among them, x' _nv represents the abscissa of point v' _nv , and x' _lv1 represents the abscissa of point v' _lv1 . Define horizontal constraints: record the grid points on the horizontal line segment lh as {v _lh1 ,v _lh2 ,...,v _lhm }, and the vertical constraints are expressed as:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; lh</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; lhm</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; lh</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

Among them, y' _nv represents the abscissa of point v' _nv , and y _l ' _h1 represents the abscissa of point v' _lh1 .

According to user interaction designation, part or all of the boundary of the original image is designated as a non-rigid feature line constraint, and (v _bi , v _bj , v _bk ) are three consecutive feature vertices on the boundary feature line, keeping and The ratio between r _bj defines the boundary constraints E _B of the image as follows;

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; B</span>}}<span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bi</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bj</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bk</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bk</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bj</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; bj</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bj</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; bi</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

(8) where r _bj is and ratio between.

Step 53, integrating step 51 and step 52, expresses the energy equation of grid parameterization as follows:

argmaxλ _S E _S +λ _L E _L +λ _V E _V +λ _H E _H +λ _B E _B (9)

stF _C (v ₁ ',v' ₂ ,...,v' _k )＝0

Among them, λ _S , λ _L , λ _V , λ _H , and λ _B are the weights corresponding to each energy item. In the actual operation process, there are certain adjustments for different picture parameters. After multiple sets of experimental pictures, a set of experience is given. The parameter values are as follows: λ _S takes 1, λ _L and λ _B take 100, and λ _V and λ _H take 10.

In step 6, the target image is generated using the grid map.

Step 6 specifically includes the following steps:

Step 61, according to equation (9), solve the target triangular grid corresponding to each original triangular grid in the original image, and the picture texture in the triangular grid is mapped to the corresponding triangular grid by bilinear interpolation method to form the target image.

Example

The hardware environment used for testing in this embodiment is: Intel Core i3-2100CPU3.1GHz main frequency, 4G memory. The software environment is Microsoft Visual Studio software version 2008, Matlab software version 2012a and Microsoft Windows7 operating system flagship version. For the specific process, refer to the implementation example in FIG. 4 .

In this example, the calculation and solution time varies from one hundred milliseconds to several hundred milliseconds depending on the grid density generated by the input image. As shown in Figure 4, the method of the present invention firstly interacts with the input image, the bold solid line marks the hidden cube body structure obtained through interaction, and the rest of the dotted lines are the feature lines manually marked by the user and the specified image boundaries; On the basis of the interactive graph, on the one hand, the Delaunay method with constraints is used to triangulate the image; on the other hand, the cube main structure projected under the new viewpoint is obtained by building a simulated 3D model and then reprojected; combined with the grid parameterization energy item And the target structure constraints obtained by reprojection are solved to obtain the target grid image; the boundary editing is performed on the generated grid image to obtain the result image under the new viewpoint. In addition, Fig. 5 shows the effect comparison diagram between the result generated by the method of the present invention and the Adobe Lightroom software. Compared with the flaws marked with circles in the Lightroom results, the method of the present invention has avoided the failure of this type of situation well. Occurrence; Fig. 6 has shown the result of scene observation with multiple new viewpoints in the spherical field of view of the present invention method, and the figure of center is input image, and remaining 12 scenes are the result images that are generated by respective corresponding viewpoint rendering, through Viewpoint interpolation is then used to obtain grid vertex position interpolation. After calculating the key viewpoint structure, the grid interpolation method can be used to generate scene graphs of different viewpoints under the current spherical view in real time.

The method of the present invention takes a real shot image as input, restores the three-dimensional structure of the main body line of the image through simple interaction and calculation, and uses the method of image deformation to generate a brand new image after viewpoint transformation, which ensures the authenticity of the resulting image, and at the same time The physical rationality of the result is guaranteed, and because the method of the present invention does not require accurate three-dimensional scene reconstruction, the method has high efficiency.

Claims (7)

1. the image viewpoint change method based on single width input picture, is characterized in that, comprises the following steps:

Step 1, image is mutual: extract the cube structure in image, wherein six corresponding image projection of point of square body of markPixel, and obtain alternately the characteristic curve in image, characteristic curve comprises the characteristic curve that affects picture material, horizontal and vertical is approximatelyBunch section and specify image boundary line;

Step 2, utilizes pinhole camera image-forming principle to rebuild a three-dimensional scenic in conjunction with the spatial character of cube structure object, is lookingWhen point transformation, by this three-dimensional scene structure Reality simulation three-dimensional scene structure, in hypothesis three-dimensional scenic, cube structure whereinOn the basis that the picture point on image overlaps with it, a summit, the geometrical constraint that the side's of utilization body comprises, calculates in step 1The three dimensions point coordinates of six some correspondences;

Step 3, utilizes the square body corresponding to space coordinates of six points that under new viewpoint, re-projection step 2 calculates, and representsThe three-dimensional scenic of rebuilding: using the central point of image as step 2 reconstruction of three-dimensional scene center, be designated as O; Taking O as the centre of sphere, this sceneCenter is to the distance of camera | in the ball plane that OF| is radius, define a new viewpoint O', in the three-dimensional scenic that step 2 is rebuildCube structureAgain project to picture plane corresponding to new viewpoint;

Step 4, image triangle gridding: according to the result of step 1, utilize the Delaunay trigonometric ratio of belt restraining to carry out imageTriangle gridding;

Step 5, mesh parameterization: utilize conformal projection principle and feature to keep constraint to set up the equation of mesh parameterization, profitThe hard constraint that obtains scene side's body framework structure projection mapping with step 3 re-projection, solves parametrization energy equation, obtains netLattice;

Step 6, target image generates: solve the grid obtaining according to step 5, by net corresponding with target gridding in original imageCheck reason utilizes bilinear interpolation method to be mapped to one by one target gridding, generates target image.

2. a kind of image viewpoint change method based on single width input picture according to claim 1, described step 1 toolBody comprises the following steps:

Step 11, described single width input picture, implicit cube structure in its scene, mark out alternately this cube structure at imageIn projection, with six point (p₀,p₁,p₂,p₃,p₄,p₅) expression cube structure, wherein (p₀,p₁,p₂,p₃) four points are correspondingSpace three-dimensional planar quadrature is in (p₀,p₁,p₅,p₄) four corresponding three-dimensional planars of point, p₀p₁Corresponding space line is two flatThe intersection of face;

Step 12, the cube structure (p obtaining according to step 11₀,p₁,p₂,p₃,p₄,p₅), by cube structure (p in image₀,p₁, p₂,p₃,p₄,p₅) between the line that comprises be all labeled as hard constraint characteristic curve, on these hard constraint characteristic curves, extract equably threeAngle grid vertex, concrete grammar is as follows: to a specific hard constraint characteristic curve, start to get a little from one end, along rectilinear direction every25 pixel distances are got the next point on this straight line, until the other end of straight line, for getting on all hard constraint characteristic curvesThe grid vertex obtaining, is designated as { v_c1,v_c2,...,v_ck, the hard constraint grid vertex sum that wherein ck representative is extracted;

Step 13, obtains the characteristic straight line in image alternately, comprises the Eigenvector that affects picture material, horizontal and vertical constraintEvenly get a little on described characteristic straight line line segment and specify image boundary line, and concrete grammar is as follows: straight to a special characteristicLine, starts to get a little from one end, gets the next point on this straight line along rectilinear direction every 25 pixel distances, until straight line is anotherOne end;

Step 14 for the indicatrix in image, is extracted triangulation network lattice point successively on curve, and concrete grammar is as follows: from spyThe one end of levying curve starts, along indicatrix prolonging direction, using 25 pixel distances as getting an interval, until arrive featureThe other end of curve.

3. a kind of image viewpoint change method based on single width input picture according to claim 2, described step 2 toolBody comprises the following steps:

Step 21, taking picture centre as initial point O (0,0,0), O to camera loca F (0,0, f) be Z axis forward, the plane of delineationTransverse axis and y direction are that XY direction of principal axis is set up three-dimensional world coordinate system, and wherein f is camera focus;

Step 22, finds out respectively plane (p₀,p₁,p₂,p₃) and plane (p₀,p₁,p₅,p₄) the vanishing point c that forms on as plane₁With c₂, obtain according to vanishing point scaling algorithmObtain focal distance f so solve;

Step 23, the cube structure in note true three-dimension scene is corresponding to the cube structure (p of six points₀,p₁,p₂,p₃,p₄,p₅)Point be cube structure (Q₀,Q₁,Q₂,Q₃,Q₄,Q₅), have according to pin-hole imaging model:WhereinRepresentation vectorWithBetween ratio; For in simulated scenario and cube structure (p six points₀,p₁,p₂,p₃, p₄,p₅) corresponding cube structureFormulaSet up wherein r_kGenerationTable vectorWithBetween ratio, fixing r₀Value be 1, utilize WithSolve r₁,r₂,r₃,r₄,r₅；

Step 24, the r obtaining according to step 23₀,r₁,r₂,r₃,r₄,r₅, calculate in simulated scenario corresponding to cube structure (p₀, p₁,p₂,p₃,p₄,p₅) cube structureSetWith p₀Overlap, definition scene center is interludeSummit in cube structure in scapeSubpoint on primary optical axis is initial point O herein, and simulated scenario center is dropped on initial point,Corresponding to Q in real scene₀Subpoint on primary optical axis, i.e. real scene center O_Q。

4. a kind of image viewpoint change method based on single width input picture according to claim 3, described step 3 toolBody comprises the following steps:

Step 31, obtains cube structure according to step 24Taking O as the center of circle | OF| is as radiusOn sphere, get at random a new viewpoint O', whereinWithAngle be no more than 45 degree, through simulated scenario center O structurePerpendicular to the planar I of OO' ', try to achieve cube structureWith the line of new viewpoint O', remember itself and planeThe new intersection point of I' is (p'₀,p′₁,p'₂,p'₃,p'₄,p'₅), complete the throwing of cube structure under new viewpoint O' in simulated scenarioShadow;

Step 32, by normal vectorObtain unit vector x' with the multiplication cross of vertical vector (0,1,0), by x' withMultiplication cross obtainsTo unit vector y', on plane of delineation I' taking O as initial point, vector x ' direction is abscissa forward, vectorial y' direction is sat for verticalMark forward is set up image two-dimensional coordinate system, willProject to respectively vector x ', on y' both direction, obtainIntersection point (p'₀,p′₁,p'₂,p'₃,p'₄,p'₅) two-dimensional coordinate on plane of delineation I', be designated as point (q₀,q₁,q₂,q₃,q₄,q₅)，Obtain in original image six points on cube structure and under new viewpoint, be projected in the two-dimensional coordinate on target image.

5. a kind of image viewpoint change method based on single width input picture according to claim 4, described step 4 toolBody comprises the following steps:

Step 41, on the basis of the triangulation network lattice point obtaining in step 1, evenly adopts a conduct at random for the remaining area of imageGrid vertex;

Step 42, in the process of trigonometric ratio, with the mutual characteristics of image line of step 1, gets adjacent vertex on same characteristic curve and connectsLine, as triangle gridding limit, utilizes the Delaunay trigonometric ratio of belt restraining to carry out triangle gridding to image.

6. a kind of image viewpoint change method based on single width input picture according to claim 5, described step 5 toolBody comprises the following steps:

Step 51, the two-dimensional coordinate point (q that utilizes step 32 to obtain₀,q₁,q₂,q₃,q₄,q₅), according to formulaCalculate a v_ciCorresponding some v'_ci, wherein v_ciFor the grid vertex on original image hard constraint characteristic straight line, p_s，p_eFor v_ciTwo end points on the hard constraint characteristic straight line of place, q_s，q_eFor being respectively and p_s，p_eCorresponding point, i.e. { v_c1,v_c2,...,v_ck}Under new viewpoint O', be mapped as { v'_c1,v'_c2,...,v'_ck, remember that it is F that its corresponding hard constraint closes_C(v'_c1,v'_c2,..., v'_ck)=0, shows v in the time that anamorphose is shone upon_ciBe mapped to v'_ci；

Step 52, definition shape constraining: two-dimentional triangle gridding affine transformation is designated as to M:(x, y) → (x', y'), according to Cauchy-Riemann's equation:

$\frac{\&part;M}{\&part;x}+i\frac{\&part;M}{\&part;y}=0,$

Wherein i is imaginary unit, and Jacobian matrix corresponding to mapping M has following form:

$J=\left[\begin{array}{cc}a& b\\ b& -a\end{array}\right]---\left(1\right),$

Wherein a, b value is any, in order to equate and the relation of opposite number each other between each element in representing matrix J;

Corresponding former grid, to the conversion of target gridding, adopts affine transformation; Remember that former triangle is T (v_ti,v_tj,v_tk)，v_ti,v_tj, v_tkFor its three summits, after its corresponding viewpoint change, deforming triangle to be solved is T'(v'_ti,v'_tj,v′_tk)，v'_ti,v'_tj, v′_tkFor with v_ti,v_tj,v_tkThree summits one to one, A=T'T^-1The affine transformation matrix from T to T', T^-1Have as followsForm:

$T^{-1}=\left[\begin{array}{ccc}{a}_1& b_1& d_1\\ a_2& b_2& d_2\\ a_3& b_3& d_3\end{array}\right]={\left[\begin{array}{ccc}{x}_{ti}& x_{tj}& x_{tk}\\ y_{ti}& y_{tj}& y_{tk}\\ z_{ti}& z_{tj}& z_{tk}\end{array}\right]}^{-1}---\left(2\right),$

Wherein, x_ti,x_tj,x_tkFor a v_ti,v_tj,v_tkAbscissa on image, y_ti,y_tj,y_tkFor a v_ti,v_tj,v_tkAt imageOn ordinate, z_ti,z_tj,z_tkFor for representing v_ti,v_tj,v_tkOrdinate, unified value is 1 here; a₁,a₂,a₃,b₁,b₂, b₃,c₁,c₂,c₃It is matrix T^-1In the Parametric Representation of each element, obtained the following energy side of conformal projection by formula (1) and formula (2)Journey:

$\begin{array}{c}{E}_{TJ1}=a_1{x}_{ti}^{\&prime;}+a_2{x}_{tj}^{\&prime;}+a_3{x}_{tk}^{\&prime;}+(b_1{y}_{ti}^{\&prime;}+b_2{y}_{tj}^{\&prime;}+b_3{y}_{tk}^{\&prime;})\\ E_{TJ2}=b_1{x}_{ti}^{\&prime;}+b_2{x}_{tj}^{\&prime;}+b_3{x}_{tk}^{\&prime;}-(a_1{y}_{ti}^{\&prime;}+a_2{y}_{tj}^{\&prime;}+a_3{y}_{tk}^{\&prime;})\end{array}---\left(3\right),$

Wherein x'_ti,x'_tj,x'_tkBe respectively v'_ti,v'_tj,v′_tkAbscissa value, y'_ti,y'_tj,y'_tkBe respectively v'_ti,v'_tj, v′_tkOrdinate value, their gangs are obtained:

$E_S=\underset{T}{\&Sigma;}(E_{TJ1}^2+E_{TJ2}^2)---\left(4\right)$

Defined feature line constraint: note (v_li,v_lj,v_lk) be continuous three points on a characteristic curve, keepWithBetweenRatio r_ljAnd anglec of rotation θ_lj, be defined as follows target equation:

$E_L=\underset{(v_{li},v_{lj},v_{lk})}{\&Sigma;}\left|\right|(v_{lk}^{\&prime;}-v_{lj}^{\&prime;})-r_{lj}{R}_{lj}(v_{lj}^{\&prime;}-v_{li}^{\&prime;})|{|}^2---\left(5\right),$

Wherein

Definition vertical constraint: remember that the mesh point on vertical line segment lv is designated as { v_lv1,v_lv2,...,v_lvm, vertical constraint is expressed as:

$E_V=\underset{lv}{\&Sigma;}\underset{nv=1}{\overset{lvm}{\&Sigma;}}{(x_{nv}^{\&prime;}-x_{lv1}^{\&prime;})}^2---\left(6\right),$

Wherein x'_nvRepresent some v'_nvAbscissa, x'_lv1Represent some v'_lv1Abscissa;

Definition horizontal restraint: the mesh point on note horizontal line section lh is designated as { v_lh1,v_lh2,...,v_lhm, vertical constraint is expressed as:

$E_H=\underset{lh}{\&Sigma;}\underset{nh=1}{\overset{lhm}{\&Sigma;}}{(y_{nh}^{\&prime;}-y_{lh1}^{\&prime;})}^2---\left(7\right),$

Wherein y'_nvRepresent some v'_nvAbscissa, y'_lh1Represent some v'_lh1Abscissa;

Specify according to user interactions, non-rigid Characteristic constraint line is appointed as in part or all of original image border, note (v_bi,v_bj, v_bk) be continuous three the feature summits on boundary characteristic line, keepWithBetween ratio r_bjThe limit of definition imageBound constrained E_BAs follows;

$E_B=\underset{(v_{bi},v_{bj},v_{bk})}{\&Sigma;}\left|\right|(v_{bk}^{\&prime;}-v_{bj}^{\&prime;})-r_{bj}(v_{bj}^{\&prime;}-v_{bi}^{\&prime;})|{|}^2---\left(8\right),$

Wherein r_bjForWithBetween ratio;

Step 53, according to step 51 and step 52, represents that the energy equation of mesh parameterization is as follows:

$\begin{array}{c}\arg {\mathrm{max\&lambda;}}_S{E}_S+{\mathrm{\&lambda;}}_L{E}_L+{\mathrm{\&lambda;}}_V{E}_V+{\mathrm{\&lambda;}}_H{E}_H+{\mathrm{\&lambda;}}_B{E}_B\\ s.t.F_C(v_{c1}^{\&prime;},v_{c2}^{\&prime;},\mathrm{...},v_{ck}^{\&prime;})=0\end{array}---\left(9\right),$

Wherein λ_S,λ_L,λ_V,λ_H,λ_BFor weights corresponding to each energy term, λ_SGet 1, λ_LWith λ_BGet 100, λ_VAnd λ_HGet 10.

7. a kind of image viewpoint change method based on single width input picture according to claim 6, described step 6 toolBody comprises the following steps:

Step 61, according to formula (9), solves and the target triangle gridding that in original image, each triangle gridding is corresponding, by former trianglePicture texture in grid adopts the method for bilinear interpolation to be mapped to corresponding target triangle gridding, forms target image.