CN106910253B - Stereo image cloning method based on different camera distances - Google Patents

Stereo image cloning method based on different camera distances Download PDF

Info

Publication number
CN106910253B
CN106910253B CN201710097015.8A CN201710097015A CN106910253B CN 106910253 B CN106910253 B CN 106910253B CN 201710097015 A CN201710097015 A CN 201710097015A CN 106910253 B CN106910253 B CN 106910253B
Authority
CN
China
Prior art keywords
image
viewpoint image
representing
grid
parallax
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710097015.8A
Other languages
Chinese (zh)
Other versions
CN106910253A (en
Inventor
侯春萍
刘琦
阎维青
陈华
王致芃
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tianjin University
Original Assignee
Tianjin University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tianjin University filed Critical Tianjin University
Priority to CN201710097015.8A priority Critical patent/CN106910253B/en
Publication of CN106910253A publication Critical patent/CN106910253A/en
Application granted granted Critical
Publication of CN106910253B publication Critical patent/CN106910253B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/128—Adjusting depth or disparity
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Architecture (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Abstract

The invention belongs to the technical field of image processing, and provides a stereo image cloning algorithm for inconsistent camera distances. The technical scheme adopted by the invention is that the method for cloning the three-dimensional image based on different camera distances comprises the steps of firstly carrying out image preprocessing, then processing the outline drawn by a user so as to carry out subsequent grid deformation, then adjusting the shape and size of a cloning region of the three-dimensional image by adjusting parallax, and finally fusing the adjusted cloning region of the three-dimensional image into a target image. The invention is mainly applied to the image processing occasion.

Description

Stereo image cloning method based on different camera distances
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a stereo image cloning algorithm based on different camera distances.
Background
With the increase of 3D media such as 3D movies and televisions, 3D contents have received wide attention from users. Therefore, once a user has convenient access to a 3D camera and a 3D display device, it is desirable to be able to edit a 3D image as if it were editing a 2D image, however, applying the method of 2D image editing directly to a 3D image often does not produce a comfortable and pleasant visual experience.
Currently, there are many mature algorithms for 2D image cloning, however, 3D image cloning faces many new challenges: 1) the 3D image increases depth constraint compared to the 2D image, and therefore, in order to ensure depth consistency in the stereoscopic image cloning process, the size and shape of the cloning region need to be adjusted. 2) For a comfortable visual experience, it is necessary to maintain the correspondence between the left view image and the right view image.
At present, some researchers have conducted studies on stereo image cloning. Lo et al propose a 3D copy and paste method that requires accurate segmentation of the object to be cloned in the source image and pasting it into the destination image. Luo et al propose a seamless stereoscopic image cloning algorithm based on perspective perception deformation, which does not require accurate segmentation of a region to be cloned in a source image and can process cloned objects with fuzzy boundaries and difficult segmentation. However, in the process of stereoscopic image cloning, the above studies assume that the camera pitches of the stereoscopic cameras that capture the source image and the target image are identical. However, in practice, the camera pitches of the stereo cameras for capturing the source image and the target image often do not coincide, and therefore the problem of cloning the stereo images when the camera pitches do not coincide is urgently solved.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide a stereo image cloning algorithm aiming at the inconsistent camera spacing. The technical scheme adopted by the invention is that the method for cloning the three-dimensional image based on different camera distances comprises the steps of firstly carrying out image preprocessing, then processing the outline drawn by a user so as to carry out subsequent grid deformation, then adjusting the shape and size of a cloning region of the three-dimensional image by adjusting parallax, and finally fusing the adjusted cloning region of the three-dimensional image into a target image.
The image preprocessing comprises the specific steps of calculating disparity maps of a source image and a target image, and constructing quadrilateral grids for the source image and the target image in the following construction mode: first using a uniform grid Ml＝(Vl,El,Fl) A left viewpoint image representing a source image, wherein VlMesh vertices representing left viewpoint image, ElMesh edge representing left viewpoint image, FlThe grid quadrangle of the left viewpoint image is represented, and then the grid M of the right viewpoint image of the source image is constructed according to the disparity map of the source imager＝(Vr,Er,Fr) Wherein V isrMesh vertices representing right viewpoint image, ErMesh edge, F, representing right view imagerA grid quadrangle representing the right view image.
The contour processing method comprises the specific steps of adjusting any contour drawn by a user by combining an initial uniform grid of a left viewpoint image of a source image to enable a new contour to be a minimum rectangular contour containing the contour drawn by the user, wherein four corner points of the rectangular contour are grid vertexes in the initial uniform grid of the left viewpoint image of the source image, after the new contour of the left viewpoint image of the source image is obtained, the new contour of a right viewpoint image of the source image is obtained according to a source image parallax image, and a contour surrounding area is a source image cloning area.
Iteratively adjusting the shape and size of the stereoscopic image clone region by adjusting the parallax, namely iteratively optimizing deformation,
1) parallax adjustment
The adjustment mode is as follows: adopting the parallax map gradient field of the source image clone region as a guide field G for solving the following formula0Meanwhile, the parallax of the edge of the adjusted source image cloning region is forced to be the same as the parallax of the edge of the pasting region of the target stereo image, and the parallax image of the adjusted source image cloning region is obtained by solving the minimum value of the weighted Poisson equation
Figure BDA0001230583870000022
Figure BDA0001230583870000023
Wherein
Figure BDA0001230583870000024
Is a weight function formulated for the source image clone region for penalizing a smoother region, DTA disparity map of the target image is represented,
Figure BDA0001230583870000025
represents the outline of the source image clone region,
Figure BDA0001230583870000026
a gradient field of a disparity map representing the adjusted source image clone region;
2) optimized deformation based on quadrilateral grids
2.1 relationship between scene object size and disparity
Assuming that the cameras of the stereo camera are arranged in parallel, and deducing the relation between the depth z of the scene object and the distance b between the cameras according to the geometric relation and the similar triangle theorem as follows:
Figure BDA0001230583870000027
where b denotes a stereo camera pitch, f denotes a camera focal length, z denotes a depth of a scene object, and d ═ xL-xRRepresenting image parallax, xLAnd xRCorresponding abscissas of the same scene object in the left viewpoint image and the right viewpoint image are represented;
according to the geometric relation and the similar triangle theorem, the relation between the size of the scene object on the stereo image plane and the depth of the scene object is deduced as
Wherein L represents the size of the scene object, x represents the size of the scene object on the image plane, and the relationship between the size x of the scene object on the stereoscopic image plane and the image parallax d is derived as follows:
Figure BDA0001230583870000029
as can be seen from the above formula, the size x of the scene object L on the stereo image is related to the image parallax d and the stereo camera distance b, but is not related to the stereo camera focal length f;
for the same scene object L1With a size x on the source image and on the target image, respectively1And x2Parallax is d1And d2The distance between the binocular stereo cameras is b1And b2The following formula is obtained:
Figure BDA00012305838700000210
Figure BDA00012305838700000211
according to the two formulas, when the distances between the cameras are inconsistent, the scaling relationship of the same scene object on the source image and the target image is as follows:
Figure BDA0001230583870000031
thus, the size of a scene object on a stereo image is related not only to image parallax, but also to the camera pitch of the stereo camera;
2.2 optimization of the deformation
Guiding quadrilateral mesh deformation by adopting weight combination of perspective scaling item, straight line constraint item, parallax consistent item, vertical alignment item and position fixing item, wherein V islAnd VrRepresenting the left viewpoint image mesh vertices and the right viewpoint image mesh vertices before the optimization deformation,
Figure BDA0001230583870000032
andrepresenting the mesh vertex of the left viewpoint image and the mesh vertex of the right viewpoint image after the optimization deformation;
(1) the perspective zoom term defines a perspective zoom factor of
Wherein the content of the first and second substances,
Figure BDA0001230583870000035
and
Figure BDA0001230583870000036
respectively representing the adjusted parallax and the parallax before the adjustment of the clone region of the source image, bSAnd bTRespectively representing the space between the stereo cameras for shooting the source images and the space between the stereo cameras for shooting the target images;
for any grid edge of left viewpoint image of source image clone regionWherein, Vi lAnd
Figure BDA0001230583870000038
respectively representing the ith and jth vertex coordinates of the left viewpoint image mesh before the optimization deformation, and the scaling factor thereof is defined as
Figure BDA0001230583870000039
Wherein the content of the first and second substances,
Figure BDA00012305838700000310
and
Figure BDA00012305838700000311
respectively representing perspective scaling factors of the ith and jth vertexes of the left viewpoint image grid before the optimization deformation;
perspective zoom term is defined as
Wherein E islAnd ErRespectively representing the sets of left viewpoint image grid edges and right viewpoint image grid edges of a source image clone area; vi lAnd
Figure BDA00012305838700000313
respectively representing the ith and jth vertex coordinates, V, of the left viewpoint image mesh before the optimization deformationi rAnd
Figure BDA00012305838700000314
respectively representing the ith and jth vertex coordinates of the right viewpoint image grid before the optimization deformation;
Figure BDA00012305838700000315
and
Figure BDA00012305838700000316
respectively represent the ith and jth vertex coordinates of the left viewpoint image mesh after the optimized transformation,
Figure BDA00012305838700000317
and
Figure BDA00012305838700000318
respectively representing the ith and jth vertex coordinates of the optimized and deformed right viewpoint image grid; sijA scaling factor representing a grid edge;
(2) linear constraint term
Defining the length ratio of the deformed grid edge to the deformed grid edge of the left viewpoint image of the source image as
Wherein, Vi lAnd
Figure BDA00012305838700000320
respectively represent the ith and jth vertex coordinates of the left viewpoint image mesh before the optimization transformation,
Figure BDA00012305838700000321
and
Figure BDA00012305838700000322
respectively representing the ith and jth vertex coordinates of the optimized and deformed left viewpoint image grid;
defining the length ratio of the right viewpoint image of the source image by the same method
Figure BDA0001230583870000041
The linear constraint term is defined as
Figure BDA0001230583870000042
Wherein E islAnd ErRespectively representing the sets of left viewpoint image grid edges and right viewpoint image grid edges of a source image clone area, Vi lAnd
Figure BDA0001230583870000043
respectively representing the ith and jth vertex coordinates, V, of the left viewpoint image mesh before the optimization deformationi rAnd
Figure BDA0001230583870000044
respectively representing the ith and jth vertex coordinates of the right viewpoint image grid before the optimization deformation;
Figure BDA0001230583870000045
andrespectively represent the ith and jth vertex coordinates of the left viewpoint image mesh after the optimized transformation,
Figure BDA0001230583870000047
and
Figure BDA0001230583870000048
respectively representing the ith and jth vertex coordinates of the optimized and deformed right viewpoint image grid;
(3) disparity coincidence term
For each set of horizontal coordinate components to be calculated
Figure BDA0001230583870000049
The parallax should be consistent with the adjusted parallax so as to make
Figure BDA00012305838700000410
Indicating the adjusted disparity. The disparity-consistent term is defined as
Figure BDA00012305838700000411
Wherein the content of the first and second substances,
Figure BDA00012305838700000412
represents the horizontal coordinates of the left viewpoint image to be calculated,
Figure BDA00012305838700000413
representing a horizontal coordinate to be calculated of the right viewpoint image;
(4) vertical alignment item
For each set of vertical coordinate components to be calculatedShould be adjusted to approach 0 to eliminate the effect of vertical parallax, the vertical alignment term is defined as
Figure BDA00012305838700000415
Wherein the content of the first and second substances,
Figure BDA00012305838700000416
indicates the vertical coordinate of the left view image to be calculated,
Figure BDA00012305838700000417
representing a vertical coordinate to be calculated of the right viewpoint image;
(5) position fixing item
To prevent the central position of the source image cloning region from changing, a position fixing item is introduced to fix the left of the source image cloning regionThe center position of the gridIndicates the center position of the initial left cell, where VlRepresenting the coordinates of the initial grid vertices of the left viewpoint image, Vi lThe ith vertex coordinate representing the initial left viewpoint image mesh. The position fixed item is defined as
Figure BDA00012305838700000419
Wherein the content of the first and second substances,
Figure BDA00012305838700000420
representing the coordinates of the mesh vertexes of the left viewpoint image after the optimization deformation,
Figure BDA00012305838700000421
representing the ith vertex coordinate of the optimized and deformed left viewpoint image grid;
the total energy of the optimization deformation is defined as the weighted sum of the above five energy terms:
Φ＝ωsΦs+ωlΦl+ωdΦd+ωvΦv+ωdΦd
wherein phis、Φl、Φd、ΦvAnd phidRespectively representing perspective zoom energy, straight line constraint energy, parallax coincidence energy, vertical alignment energy and position fixing energy. Omegas、ωl、ωd、ωvAnd ωdThe weight coefficient corresponding to the energy;
an optimized deformation grid is obtained by solving the minimum of the energy function.
The image fusion method specifically comprises the steps of obtaining an optimized deformed grid of a source image clone region, obtaining the deformed source image clone region through bilinear interpolation, and then respectively fusing a left viewpoint image and a right viewpoint image of the deformed source image clone region to a left viewpoint image and a right viewpoint image of a target image by adopting a plane Poisson fusion method to obtain a final stereo image fusion result.
The invention has the characteristics and beneficial effects that:
the stereo image cloning algorithm based on different camera pitches provided by the invention realizes the cloning of stereo images under the condition that the stereo camera pitch for shooting a source image is different from the stereo camera pitch for shooting a target image, and compared with the algorithm provided by the predecessor, the algorithm provided by the invention realizes the cloning of more accurate stereo images.
Description of the drawings:
figure 1 gives a schematic view of the profile modification.
In the figure, a source image left view image outline and b source image right view image outline.
Fig. 2 shows the relationship between the size of the scene object and the parallax.
In the figure, a is a top view, b is a right side view
Fig. 3 comparison of the present invention and a case where the camera pitch is not considered.
a, a source image left viewpoint image;
b, a left viewpoint image of the target image;
c, obtaining a left viewpoint image of the stereo clone image without considering the camera distance;
d, taking the camera distance into consideration to obtain a left viewpoint image of the stereo clone image;
e, a source image right viewpoint image;
f, a right viewpoint image of the target image;
g, a right viewpoint image of the stereo clone image obtained without considering the camera distance;
h a right viewpoint image of the stereoscopic clone image obtained while considering the camera pitch.
Detailed Description
At present, a plurality of stereo image cloning algorithms exist, the existing algorithms all assume that the stereo camera distances of a shooting source image and a shooting target image are consistent, and actually, the camera distances have great influence on the cloning of the stereo images.
The invention provides a stereo image cloning algorithm based on different camera distances. Firstly, image preprocessing is carried out, then, the contour drawn by a user is processed so as to carry out subsequent grid deformation, then, the shape and the size of a stereoscopic image clone area are adjusted by adjusting parallax, and finally, the adjusted stereoscopic image clone area is fused into a target image. The specific technical scheme of the invention comprises the following steps:
1. image pre-processing
In the image preprocessing stage, disparity maps of a source image and a target image need to be calculated, and quadrilateral grids need to be constructed for the source image and the target image, wherein the construction mode is as follows: first using a uniform grid Ml＝(Vl,El,Fl) A left view point diagram representing a source image, wherein VlMesh vertices representing left viewpoint image, ElMesh edge representing left viewpoint image, FlThe grid quadrangle of the left viewpoint image is represented, and then the grid M of the source image right viewpoint image is constructed according to the disparity map of the source imager＝(Vr,Er,Fr) In which V isrMesh vertices representing right viewpoint image, ErMesh edge, F, representing right view imagerA grid quadrangle representing the right view image.
2. Contour processing
In order to facilitate the operation of the user, the user is allowed to draw an arbitrary outline for the left viewpoint image of the source image. Then, in order to perform optimization iterative adjustment, any contour drawn by the user is adjusted by combining the initial uniform grid of the left viewpoint image of the source image. And enabling the new contour to be a minimum rectangular contour containing the user drawing contour, wherein four corner points of the rectangular contour are all grid vertexes in an initial uniform grid of the left viewpoint image of the source image. And after obtaining the new contour of the left viewpoint image of the source image, obtaining the new contour of the right viewpoint image of the source image according to the source image disparity map. The outline modification schematic is shown in fig. 1. And the contour surrounding area is the source image clone area.
3. Iterative optimization of deformation
3.1 parallax adjustment
In stereo image cloning, source image and target imageThe images are usually shot by different stereo cameras, so that the parallax of a source image clone area and the parallax of a target image are usually different to a certain extent, and the phenomenon of parallax mutation of the cloned stereo image is avoided. The parallax of the source image clone area needs to be adjusted in the following manner: adopting the parallax map gradient field of the source image clone region as a guide field G for solving the following formula0Meanwhile, the parallax of the edge of the adjusted source image cloning region is forced to be the same as the parallax of the edge of the pasting region of the target stereo image, and the parallax image of the adjusted source image cloning region is obtained by solving the minimum value of the weighted Poisson equation
Figure BDA0001230583870000061
Figure BDA0001230583870000062
WhereinIs a weight function formulated for the source image clone region for penalizing a smoother region, DTA disparity map of the target image is represented,
Figure BDA0001230583870000065
represents the outline of the source image clone region,
Figure BDA0001230583870000066
a gradient field of the disparity map representing the adjusted region of the source image clone.
3.2 optimized deformation based on quadrilateral meshes
3.2.1 relationship between scene object size and disparity
Generally, as the distance from the object to the viewer increases, the object becomes gradually smaller. The stereo image has depth information, the size of objects in the stereo image is affected by the perceived depth, and the perceived depth of the stereo image is affected by image parallax. Therefore, the size of an object in a stereoscopic image depends on the parallax of the object. Fig. 2 gives a schematic diagram of the relationship between object size and parallax, assuming here that the stereo camera cameras are arranged in parallel, which is a common arrangement of stereo cameras.
According to the geometric relationship and the theorem of similar triangles in fig. 1(a), it can be derived that the relationship between the depth z of the scene object and the camera distance b is:
Figure BDA0001230583870000067
where b denotes a stereo camera pitch, f denotes a camera focal length, z denotes a depth of a scene object, and d ═ xL-xRRepresenting image parallax, xLAnd xRAnd corresponding abscissas of objects of the same scene in the left view image and the right view image are represented.
According to the geometric relationship and the theorem of similar triangles in FIG. 1(b), it can be deduced that the relationship between the size of the scene object on the stereoscopic image plane and the depth of the scene object is
Figure BDA0001230583870000071
Where L represents the size of the scene object and x represents the size of the scene object on the image plane.
It can thus be derived that the relationship between the size x of the scene object in the stereoscopic image plane and the image disparity d is
Figure BDA0001230583870000072
As can be seen from the above formula, the size x of the scene object L on the stereo image is related to the image parallax d and the stereo camera pitch b, and is not related to the stereo camera focal length f.
For the same scene object L1With a size x on the source image and on the target image, respectively1And x2Parallax is d1And d2The distance between the binocular stereo cameras is b1And b2The following formula is obtained:
Figure BDA0001230583870000073
Figure BDA0001230583870000074
according to the two formulas, when the distances between the cameras are inconsistent, the scaling relationship of the same scene object on the source image and the target image is as follows:
Figure BDA0001230583870000075
it can be seen that the size of the scene object on the stereoscopic image is related to not only the image parallax but also the camera pitch of the stereoscopic camera.
3.2.2 optimization of deformation
In order to realize the cloning of stereo images under different stereo camera pitches, the invention adopts the weight combination of a perspective zooming item, a straight line constraint item, a parallax consistency item, a vertical alignment item and a position fixing item to guide the deformation of a quadrilateral mesh, wherein V islAnd VrRepresenting the left viewpoint image mesh vertices and the right viewpoint image mesh vertices before the optimization deformation,
Figure BDA0001230583870000076
and
Figure BDA0001230583870000077
and representing the vertices of the left viewpoint image mesh and the vertices of the right viewpoint image mesh after the optimization deformation.
(1) Perspective zoom item
According to the introduction 3.1.1, the size of scene objects on a stereo image is dependent on image parallax and stereo camera distance. The invention defines a perspective zoom factor as
Figure BDA0001230583870000078
Wherein the content of the first and second substances,
Figure BDA0001230583870000079
andrespectively representing the adjusted parallax and the parallax before the adjustment of the clone region of the source image, bSAnd bTRepresenting respectively the stereo camera pitch of the source image and the stereo camera pitch of the target image, Vi lThe ith vertex coordinate of the left viewpoint image mesh before the optimization deformation is represented.
For any grid edge of left viewpoint image of source image clone region
Figure BDA0001230583870000081
Wherein, Vi lAndrespectively representing the ith and jth vertex coordinates of the left viewpoint image mesh before the optimization deformation, and the scaling factor thereof is defined as
Figure BDA0001230583870000083
Wherein the content of the first and second substances,
Figure BDA0001230583870000084
and
Figure BDA0001230583870000085
respectively representing perspective scaling factors of ith and jth vertexes of the left viewpoint image mesh before the optimization deformation.
Perspective zoom term is defined as
Figure BDA0001230583870000086
Wherein E islAnd ErRespectively representing the sets of left viewpoint image grid edges and right viewpoint image grid edges of a source image clone area; vi lAnd
Figure BDA0001230583870000087
respectively representing the ith and jth vertex coordinates, V, of the left viewpoint image mesh before the optimization deformationi rAnd
Figure BDA0001230583870000088
respectively representing the ith and jth vertex coordinates of the right viewpoint image grid before the optimization deformation;
Figure BDA0001230583870000089
andrespectively represent the ith and jth vertex coordinates of the left viewpoint image mesh after the optimized transformation,and
Figure BDA00012305838700000812
respectively representing the ith and jth vertex coordinates of the optimized and deformed right viewpoint image grid; sijRepresenting the scaling factor of the grid edge.
(2) Straight line zoom item
In the process of optimizing deformation, in order to avoid structural distortion of a cloned object of a source image and force the grid edge to zoom along the original direction, the length ratio of the deformed grid edge to the deformed grid edge of a left viewpoint image of the source image is defined as
Wherein, Vi lAnd
Figure BDA00012305838700000814
respectively representing left viewpoint image net before optimized deformationThe ith and jth vertex coordinates of the lattice,
Figure BDA00012305838700000815
and
Figure BDA00012305838700000816
respectively representing the ith and jth vertex coordinates of the left viewpoint image mesh after the optimized deformation.
Defining the length ratio of the right viewpoint image of the source image by the same method
Figure BDA00012305838700000817
The linear constraint term is defined as
Figure BDA00012305838700000818
Wherein E islAnd ErRespectively representing the sets of left viewpoint image grid edges and right viewpoint image grid edges of a source image clone area, Vi lAnd
Figure BDA00012305838700000819
respectively representing the ith and jth vertex coordinates, V, of the left viewpoint image mesh before the optimization deformationi rAnd
Figure BDA00012305838700000820
respectively representing the ith and jth vertex coordinates of the right viewpoint image grid before the optimization deformation;and
Figure BDA00012305838700000822
respectively represent the ith and jth vertex coordinates of the left viewpoint image mesh after the optimized transformation,
Figure BDA00012305838700000823
and
Figure BDA00012305838700000824
respectively representing the ith and jth vertex coordinates of the optimally deformed right viewpoint image mesh.
(3) Disparity coincidence term
For each set of horizontal coordinate components to be calculated
Figure BDA00012305838700000825
The parallax should be consistent with the adjusted parallax so as to make
Figure BDA00012305838700000826
Indicating the adjusted disparity. The disparity-consistent term is defined as
Figure BDA0001230583870000091
Wherein the content of the first and second substances,
Figure BDA0001230583870000092
represents the horizontal coordinates of the left viewpoint image to be calculated,
Figure BDA0001230583870000093
indicating the horizontal coordinates of the right viewpoint image to be calculated.
(4) Vertical alignment item
For each set of vertical coordinate components to be calculated
Figure BDA0001230583870000094
Should be adjusted to approach 0 to eliminate the effect of vertical parallax, the vertical alignment term is defined as
Figure BDA0001230583870000095
Wherein the content of the first and second substances,
Figure BDA0001230583870000096
indicates the vertical coordinate of the left view image to be calculated,
Figure BDA0001230583870000097
indicating the vertical coordinate of the right viewpoint image to be calculated.
(5) Position fixing item
In order to prevent the central position of the source image cloning region from changing, a position fixing item is introduced to fix the central position of a left grid of the source image cloning region, and the central position of the left grid is adjusted
Figure BDA0001230583870000098
Denotes the center position of the initial left grid, where VlRepresenting the coordinates of the initial grid vertices of the left viewpoint image, Vi lThe ith vertex coordinate representing the initial left viewpoint image mesh. The position fixed item is defined as
Figure BDA0001230583870000099
Wherein the content of the first and second substances,
Figure BDA00012305838700000910
representing the coordinates of the mesh vertexes of the left viewpoint image after the optimization deformation,
Figure BDA00012305838700000911
and (4) representing the ith vertex coordinate of the left viewpoint image grid after the optimized deformation.
The total energy of the optimization deformation is defined as the weighted sum of the above five energy terms:
Φ＝ωsΦs+ωlΦl+ωdΦd+ωvΦv+ωdΦd
wherein phis、Φl、Φd、ΦvAnd phidRespectively representing perspective zoom energy, straight line constraint energy, parallax coincidence energy, vertical alignment energy and position fixing energy. Omegas、ωl、ωd、ωvAnd ωdThe weight coefficient is corresponding to the energy.
An optimized deformation grid is obtained by solving the minimum of the energy function.
4. Image fusion
After the optimized deformation grid of the source image cloning region is obtained, the deformed source image cloning region is obtained through bilinear interpolation, and then a plane Poisson fusion method is adopted to fuse the left viewpoint image and the right viewpoint image of the deformed source image cloning region to the left viewpoint image and the right viewpoint image of the target image respectively, so that the final three-dimensional image fusion result is obtained.
In order to make the objects, technical solutions and advantages of the present invention more apparent, specific implementation processes of the present invention are described below.
The invention carries out simulation based on MATLAB platform, and the specific technical scheme is divided into the following steps:
1. image pre-processing
In the image preprocessing stage, disparity maps of a source image and a target image need to be calculated, and quadrilateral grids need to be constructed for the source image and the target image, wherein the construction mode is as follows: first using a uniform grid Ml＝(Vl,El,Fl) A left viewpoint image representing a source image, wherein VlMesh vertices representing left viewpoint image, ElMesh edge representing left viewpoint image, FlThe grid quadrangle of the left viewpoint image is represented, and then the grid M of the right viewpoint image of the source image is constructed according to the disparity map of the source imager＝(Vr,Er,Fr) Wherein V isrMesh vertices representing right viewpoint image, ErMesh edge, F, representing right view imagerA grid quadrangle representing the right view image.
2. Contour processing
In order to facilitate the operation of the user, the user is allowed to draw an arbitrary outline for the left viewpoint image of the source image. Then, in order to perform optimization iterative adjustment, any contour drawn by the user is adjusted by combining the initial uniform grid of the left viewpoint image of the source image. And enabling the new contour to be a minimum rectangular contour containing any contour drawn by a user, wherein four corner points of the rectangular contour are all grid vertexes in an initial uniform grid of the left viewpoint image of the source image. And after obtaining the new contour of the left view diagram of the source image, obtaining the new contour of the right view diagram of the source image according to the source image disparity map. The outline modification schematic is shown in fig. 1. And the contour surrounding area is the source image clone area.
3. Iterative optimization of deformation
3.1 parallax adjustment
When the stereo images are cloned, a source image and a target image are often shot by different stereo cameras, so that the parallax of a source image cloning region and the parallax of the target image often have certain difference, and the phenomenon of parallax mutation of the cloned stereo images is avoided. The parallax of the source image clone area needs to be adjusted in the following manner: adopting the parallax map gradient field of the source image clone region as a guide field G for solving the following formula0Meanwhile, the parallax of the edge of the adjusted source image cloning region is forced to be the same as the parallax of the edge of the pasting region of the target stereo image, and the parallax image of the adjusted source image cloning region is obtained by solving the minimum value of the weighted Poisson equation
Figure BDA0001230583870000101
Figure BDA0001230583870000102
Figure BDA0001230583870000103
Wherein
Figure BDA0001230583870000104
Is a weight function formulated for the source image clone region for penalizing a smoother region, DTA disparity map of the target image is represented,
Figure BDA0001230583870000105
represents the outline of the source image clone region,
Figure BDA0001230583870000106
representing disparity maps of adjusted source image clone regionsA gradient field.
3.2 optimized deformation based on quadrilateral meshes
3.2.1 relationship between scene object size and disparity
Generally, as the distance from the object to the viewer increases, the object becomes gradually smaller. The stereo image has depth information, the size of objects in the stereo image is affected by the perceived depth, and the perceived depth of the stereo image is affected by image parallax. Therefore, the size of an object in a stereoscopic image depends on the parallax of the object. Fig. 2 gives a schematic diagram of the relationship between object size and parallax, assuming here that the stereo camera cameras are arranged in parallel, which is a common arrangement of stereo cameras.
According to the geometric relationship and the theorem of similar triangles in fig. 1(a), it can be derived that the relationship between the depth z of the scene object and the camera distance b is:
where b denotes a stereo camera pitch, f denotes a camera focal length, z denotes a depth of a scene object, and d ═ xL-xRRepresenting image parallax, xLAnd xRAnd corresponding abscissas of objects of the same scene in the left view image and the right view image are represented.
According to the geometric relationship and the theorem of similar triangles in FIG. 1(b), it can be deduced that the relationship between the size of the scene object on the stereoscopic image plane and the depth of the scene object is
Figure BDA0001230583870000111
Where L represents the size of the scene object and x represents the size of the scene object on the image plane.
It can thus be derived that the relationship between the size x of the scene object in the stereoscopic image plane and the image disparity d is
Figure BDA0001230583870000112
As can be seen from the above formula, the size x of the scene object L on the stereo image is related to the image parallax d and the stereo camera pitch b, and is not related to the stereo camera focal length f.
For the same scene object L1With a size x on the source image and on the target image, respectively1And x2Parallax is d1And d2The distance between the binocular stereo cameras is b1And b2The following formula is obtained:
Figure BDA0001230583870000113
Figure BDA0001230583870000114
according to the two formulas, when the distances between the cameras are inconsistent, the scaling relationship of the same scene object on the source image and the target image is as follows:
Figure BDA0001230583870000115
it can be seen that the size of the scene object on the stereoscopic image is related to not only the image parallax but also the camera pitch of the stereoscopic camera.
3.2.2 optimization of deformation
In order to realize the cloning of stereo images under different stereo camera pitches, the invention adopts the weight combination of a perspective zooming item, a straight line constraint item, a parallax consistency item, a vertical alignment item and a position fixing item to guide the deformation of a quadrilateral mesh, wherein V islAnd VrRepresenting the left viewpoint image mesh vertices and the right viewpoint image mesh vertices before the optimization deformation,
Figure BDA0001230583870000116
and
Figure BDA0001230583870000117
representing the sum of the vertices of the left viewpoint image mesh after the optimization deformationAnd (4) right viewpoint image grid vertex.
(1) Perspective zoom item
According to the introduction 3.1.1, the size of the scene object on the stereo image is dependent on the image disparity and the camera pitch of the stereo camera. The invention defines a perspective zoom factor as
Wherein the content of the first and second substances,and
Figure BDA00012305838700001110
respectively representing the adjusted parallax and the parallax before the adjustment of the clone region of the source image, bSAnd bTRepresenting respectively the stereo camera pitch of the source image and the stereo camera pitch of the target image, Vi lThe ith vertex coordinate of the left viewpoint image mesh before the optimization deformation is represented.
For any grid edge of left viewpoint image of source image clone region
Figure BDA00012305838700001111
Wherein, Vi lAnd
Figure BDA00012305838700001112
respectively representing the ith and jth vertex coordinates of the left viewpoint image mesh before the optimization deformation, and the scaling factor thereof is defined as
Figure BDA0001230583870000121
Wherein the content of the first and second substances,
Figure BDA0001230583870000122
and
Figure BDA0001230583870000123
respectively represent the left side before the optimized deformationPerspective scaling factors for the ith and jth vertices of the viewpoint image grid.
Perspective zoom term is defined as
Figure BDA0001230583870000124
Wherein E islAnd ErRespectively representing the sets of left viewpoint image grid edges and right viewpoint image grid edges of a source image clone area; vi lAnd
Figure BDA0001230583870000125
respectively representing the ith and jth vertex coordinates, V, of the left viewpoint image mesh before the optimization deformationi rAnd
Figure BDA0001230583870000126
respectively representing the ith and jth vertex coordinates of the right viewpoint image grid before the optimization deformation;and
Figure BDA0001230583870000128
respectively represent the ith and jth vertex coordinates of the left viewpoint image mesh after the optimized transformation,
Figure BDA0001230583870000129
and
Figure BDA00012305838700001210
respectively representing the ith and jth vertex coordinates of the optimized and deformed right viewpoint image grid; sijRepresenting the scaling factor of the grid edge.
(2) Linear constraint term
In the process of optimizing deformation, in order to avoid structural distortion of a cloned object of a source image and force the grid edge to zoom along the original direction, the length ratio of the deformed grid edge to the deformed grid edge of a left viewpoint image of the source image is defined as
Wherein, Vi lAnd
Figure BDA00012305838700001212
respectively represent the ith and jth vertex coordinates of the left viewpoint image mesh before the optimization transformation,
Figure BDA00012305838700001213
and
Figure BDA00012305838700001214
respectively representing the ith and jth vertex coordinates of the left viewpoint image mesh after the optimized deformation.
Defining the length ratio of the right viewpoint image of the source image by the same method
Figure BDA00012305838700001215
The linear constraint term is defined as
Wherein E islAnd ErRespectively representing the sets of left viewpoint image grid edges and right viewpoint image grid edges of a source image clone area, Vi lAnd
Figure BDA00012305838700001217
respectively representing the ith and jth vertex coordinates, V, of the left viewpoint image mesh before the optimization deformationi rAnd
Figure BDA00012305838700001218
respectively representing the ith and jth vertex coordinates of the right viewpoint image grid before the optimization deformation;
Figure BDA00012305838700001219
and
Figure BDA00012305838700001220
respectively represent the ith and jth vertex coordinates of the left viewpoint image mesh after the optimized transformation,
Figure BDA00012305838700001221
andrespectively representing the ith and jth vertex coordinates of the optimally deformed right viewpoint image mesh.
(3) Disparity coincidence term
For each set of horizontal coordinate components to be calculated
Figure BDA00012305838700001223
The parallax should be consistent with the adjusted parallax, so that
Figure BDA00012305838700001224
Indicating the adjusted disparity. The disparity-consistent term is defined as
Figure BDA00012305838700001225
Wherein the content of the first and second substances,represents the horizontal coordinates of the left viewpoint image to be calculated,
Figure BDA0001230583870000132
indicating the horizontal coordinates of the right viewpoint image to be calculated.
(4) Vertical alignment item
For each set of vertical coordinate components to be calculated
Figure BDA0001230583870000133
Should be adjusted to approach 0 to eliminate the effect of vertical parallax, the vertical alignment term is defined as
Figure BDA0001230583870000134
Wherein the content of the first and second substances,
Figure BDA0001230583870000135
indicates the vertical coordinate of the left view image to be calculated,
Figure BDA0001230583870000136
indicating the vertical coordinate of the right viewpoint image to be calculated.
(5) Position fixing item
In order to prevent the central position of the source image cloning region from changing, a position fixing item is introduced to fix the central position of a left grid of the source image cloning region, and the central position of the left grid is adjustedIndicates the center position of the initial left cell, where VlRepresenting the coordinates of the initial grid vertices of the left viewpoint image, Vi lThe ith vertex coordinate representing the initial left viewpoint image mesh. The position fixed item is defined as
Figure BDA0001230583870000138
Wherein the content of the first and second substances,
Figure BDA0001230583870000139
representing the coordinates of the mesh vertexes of the left viewpoint image after the optimization deformation,
Figure BDA00012305838700001310
and (4) representing the ith vertex coordinate of the left viewpoint image grid after the optimized deformation.
The total energy of the optimization deformation is defined as the weighted sum of the above five energy terms:
Φ＝ωsΦs+ωlΦl+ωdΦd+ωvΦv+ωdΦd
wherein phis、Φl、Φd、ΦvAnd phidRespectively representing perspective zoom energy, straight line constraint energy, parallax coincidence energy, vertical alignment energy and position fixing energy. Omegas、ωl、ωd、ωvAnd ωdThe weight coefficient is corresponding to the energy.
An optimized deformation grid is obtained by solving the minimum of the energy function. The weighting factors employed herein are each ωs＝50，ωl＝50，ωd＝50，ωv100 and ωp1. The total energy Φ is a quadratic optimization function and is programmed to the final solution based on Matlab R2013 b.
4. Image fusion
After the optimized deformation grid of the source image cloning region is obtained, the deformed source image cloning region is obtained through bilinear interpolation, and then a plane Poisson fusion method is adopted to fuse the left viewpoint image and the right viewpoint image of the deformed source image cloning region to the left viewpoint image and the right viewpoint image of the target image respectively, so that the final three-dimensional image fusion result is obtained.
5. Experiment and evaluation of Effect
In order to test the performance of the stereo image cloning algorithm provided by the invention, the invention displays the result of the stereo image cloning. The input source stereo image is shot by a JVC Erio 3D high-definition video camera with the camera interval of 35mm, and the target stereo image is shot by a FUJIFILM REAL 3D W1 camera with the camera interval of 77 mm.
Fig. 3 shows the comparison of the results of the stereo image cloning method with and without taking the camera pitch into account. In fig. 3, (a) and (e) are a left viewpoint image and a right viewpoint image of a source image, respectively; (b) and (f) a left viewpoint image and a right viewpoint image representing the target image; (c) and (g) a left viewpoint image and a right viewpoint image representing a stereoscopic clone image obtained without considering a camera pitch, that is, assuming that camera pitches of a captured source image and a target image are identical; (d) and (g) a left viewpoint image and a right viewpoint image representing a stereoscopic clone image obtained while considering a camera pitch. The existing algorithms assume that the stereo camera distances of the shooting source image and the shooting target image are consistent, however, in the case, the obtained stereo image cloning result is not accurate. When the camera pitch is not considered, i.e., only the influence of parallax on the shape and size of the source image clone area is considered, the source image clone area is enlarged as shown in (c) and (g). However, the source image clone area is reduced when the camera pitch is considered, as in figures (d) and (g). Therefore, the algorithm provided by the invention can realize more accurate scaling.

Claims (4)

1. A stereoscopic image cloning method based on different camera distances is characterized in that image preprocessing is firstly carried out, then a contour drawn by a user is processed so as to carry out subsequent grid deformation, then the shape and the size of a stereoscopic image cloning region are adjusted by adjusting parallax, finally the adjusted stereoscopic image cloning region is fused into a target image, wherein the shape and the size of the stereoscopic image cloning region are iteratively adjusted by adjusting the parallax, namely iterative optimization deformation is carried out, the specific steps are,
1) parallax adjustment
The adjustment mode is as follows: adopting the parallax map gradient field of the source image clone region as a guide field G for solving the following formula0Meanwhile, the parallax of the edge of the adjusted source image cloning region is forced to be the same as the parallax of the edge of the pasting region of the target stereo image, and the parallax image of the adjusted source image cloning region is obtained by solving the minimum value of the weighted Poisson equation
Figure FDA0002227303250000011
WhereinIs a clone region of a source imageA domain-specific weighting function for penalizing smooth regions, DTA disparity map of the target image is represented,
Figure FDA0002227303250000015
represents the outline of the source image clone region,
Figure FDA0002227303250000016
a gradient field of a disparity map representing the adjusted source image clone region;
2) optimized deformation based on quadrilateral grids
2.1 relationship between scene object size and disparity
Assuming that the cameras of the stereo camera are arranged in parallel, and deducing the relation between the depth z of the scene object and the distance b between the cameras according to the geometric relation and the similar triangle theorem as follows:
Figure FDA0002227303250000017
where b denotes a stereo camera pitch, f denotes a camera focal length, z denotes a depth of a scene object, and d ═ xL-xRRepresenting image parallax, xLAnd xRCorresponding abscissas of the same scene object in the left viewpoint image and the right viewpoint image are represented;
according to the geometric relation and the similar triangle theorem, the relation between the size of the scene object on the stereo image plane and the depth of the scene object is deduced as
Figure FDA0002227303250000018
Wherein L represents the size of the scene object, x represents the size of the scene object on the image plane, and the relationship between the size x of the scene object on the stereoscopic image plane and the image parallax d is derived as follows:
Figure FDA0002227303250000019
as can be seen from the above formula, the size x of the scene object L on the stereo image is related to the image parallax d and the stereo camera distance b, but is not related to the stereo camera focal length f;
for the same scene object L1With a size x on the source image and on the target image, respectively1And x2Parallax is d1And d2The distance between the binocular stereo cameras is b1And b2The following formula is obtained:
Figure FDA0002227303250000021
Figure FDA0002227303250000022
according to the two formulas, when the distances between the cameras are inconsistent, the scaling relationship of the same scene object on the source image and the target image is as follows:
Figure FDA0002227303250000023
thus, the size of a scene object on a stereo image is related not only to image parallax, but also to the camera pitch of the stereo camera;
2.2 optimization of the deformation
Guiding quadrilateral mesh deformation by adopting weight combination of perspective scaling item, straight line constraint item, parallax consistent item, vertical alignment item and position fixing item, wherein V islAnd VrRepresenting the left viewpoint image mesh vertices and the right viewpoint image mesh vertices before the optimization deformation,
Figure FDA0002227303250000024
and
Figure FDA0002227303250000025
representing the mesh vertex of the left viewpoint image and the mesh vertex of the right viewpoint image after the optimization deformation;
(1) the perspective zoom term defines a perspective zoom factor of
Figure FDA0002227303250000026
Wherein the content of the first and second substances,
Figure FDA0002227303250000027
and
Figure FDA0002227303250000028
respectively representing the adjusted parallax and the parallax before the adjustment of the clone region of the source image, bSAnd bTRepresenting respectively the stereo camera pitch of the source image and the stereo camera pitch of the target image, Vi lRepresenting the ith vertex coordinate of the left viewpoint image mesh before the optimization deformation;
for any grid edge of left viewpoint image of source image clone region
Figure FDA0002227303250000029
Wherein, Vi lAndrespectively representing the ith and jth vertex coordinates of the left viewpoint image mesh before the optimization deformation, and the scaling factor thereof is defined as
Figure FDA00022273032500000211
Wherein the content of the first and second substances,
Figure FDA00022273032500000212
and
Figure FDA00022273032500000213
respectively representing perspective scaling factors of the ith and jth vertexes of the left viewpoint image grid before the optimization deformation;
perspective zoom term is defined as
Figure FDA00022273032500000214
Wherein E islAnd ErRespectively representing the sets of left viewpoint image grid edges and right viewpoint image grid edges of a source image clone area; vi lAnd
Figure FDA00022273032500000215
respectively representing the ith and jth vertex coordinates, V, of the left viewpoint image mesh before the optimization deformationi rAnd
Figure FDA00022273032500000216
respectively representing the ith and jth vertex coordinates of the right viewpoint image grid before the optimization deformation;
Figure FDA00022273032500000217
and
Figure FDA00022273032500000218
respectively represent the ith and jth vertex coordinates of the left viewpoint image mesh after the optimized transformation,
Figure FDA00022273032500000219
and
Figure FDA00022273032500000220
respectively representing the ith and jth vertex coordinates of the optimized and deformed right viewpoint image grid; sijA scaling factor representing a grid edge;
(2) linear constraint term
Defining the length ratio of the deformed grid edge to the deformed grid edge of the left viewpoint image of the source image as
Figure FDA0002227303250000031
Wherein, Vi lAnd
Figure FDA0002227303250000032
respectively represent the ith and jth vertex coordinates of the left viewpoint image mesh before the optimization transformation,
Figure FDA0002227303250000033
and
Figure FDA0002227303250000034
respectively representing the ith and jth vertex coordinates of the optimized and deformed left viewpoint image grid;
defining the length ratio of the right viewpoint image of the source image by the same method
The linear constraint term is defined as
Figure FDA0002227303250000036
Wherein E islAnd ErRespectively representing the sets of left viewpoint image grid edges and right viewpoint image grid edges of a source image clone area, Vi lAnd
Figure FDA0002227303250000037
respectively representing the ith and jth vertex coordinates, V, of the left viewpoint image mesh before the optimization deformationi rAnd
Figure FDA0002227303250000038
respectively representing the ith and jth vertex coordinates of the right viewpoint image grid before the optimization deformation;and
Figure FDA00022273032500000310
respectively represent the ith and jth vertex coordinates of the left viewpoint image mesh after the optimized transformation,andrespectively representing the ith and jth vertex coordinates of the optimized and deformed right viewpoint image grid;
(3) disparity coincidence term
For each set of horizontal coordinate components to be calculated
Figure FDA00022273032500000313
The parallax should be consistent with the adjusted parallax so as to make
Figure FDA00022273032500000314
Indicating the adjusted parallax, and defining the parallax consistent item as
Figure FDA00022273032500000315
Wherein the content of the first and second substances,
Figure FDA00022273032500000316
represents the horizontal coordinates of the left viewpoint image to be calculated,
Figure FDA00022273032500000317
representing a horizontal coordinate to be calculated of the right viewpoint image;
(4) vertical alignment item
For each set of vertical coordinate components to be calculated
Figure FDA00022273032500000318
Should be adjusted to approach 0 to eliminate the effect of vertical parallax, the vertical alignment term is defined as
Figure FDA00022273032500000319
Wherein the content of the first and second substances,
Figure FDA00022273032500000320
indicates the vertical coordinate of the left view image to be calculated,
Figure FDA00022273032500000321
representing a vertical coordinate to be calculated of the right viewpoint image;
(5) position fixing item
In order to prevent the central position of the source image cloning region from changing, a position fixing item is introduced to fix the central position of a left grid of the source image cloning region, and the central position of the left grid is adjusted
Figure FDA0002227303250000041
Indicates the center position of the initial left cell, where VlRepresenting the coordinates of the initial grid vertices of the left viewpoint image, Vi lThe ith vertex coordinate representing the initial left viewpoint image mesh, and the position fixing term is defined as
Wherein the content of the first and second substances,
Figure FDA0002227303250000043
representing the coordinates of the mesh vertexes of the left viewpoint image after the optimization deformation,
Figure FDA0002227303250000044
representing the ith vertex coordinate of the optimized and deformed left viewpoint image grid;
the total energy of the optimized deformation is defined as the weighted sum of five energy terms, wherein the five energy terms are a perspective scaling term, a straight line constraint term, a parallax consistency term, a vertical alignment term and a position fixing term, and the weighted sum is expressed as:
Φ＝ωsΦs+ωlΦl+ωdΦd+ωvΦv+ωdΦd
wherein phis、Φl、Φd、ΦvAnd phidRespectively representing perspective zoom energy, straight line constraint energy, parallax consistent energy, vertical alignment energy and position fixing energy, omegas、ωl、ωd、ωvAnd ωdThe weight coefficient corresponding to the energy; and obtaining the optimized deformation grid by solving the minimum value of the total energy function.
2. The method for cloning stereo images based on different camera pitches as claimed in claim 1, wherein the image preprocessing comprises the specific steps of calculating disparity maps of the source image and the target image, and constructing quadrilateral meshes for the source image and the target image simultaneously in the following manner: first using a uniform grid Ml＝(Vl,El,Fl) A left viewpoint image representing a source image, wherein VlMesh vertices representing left viewpoint image, ElMesh edge representing left viewpoint image, FlA grid quadrangle representing the left viewpoint image is constructed, and then a grid Mr (V) of the right viewpoint image of the source image is constructed according to the disparity map of the source imager,Er,Fr) Wherein V isrMesh vertices representing right viewpoint image, ErMesh edge, F, representing right view imagerA grid quadrangle representing the right view image.
3. The stereoscopic image cloning method based on different camera pitches as claimed in claim 1, wherein the contour processing specific steps are that any contour drawn by a user is adjusted by combining with an initial uniform grid of a left viewpoint image of a source image, so that a new contour is a minimum rectangular contour containing the contour drawn by the user, wherein four corner points of the rectangular contour are grid vertices in the initial uniform grid of the left viewpoint image of the source image, after the new contour of the left viewpoint image of the source image is obtained, the new contour of the right viewpoint image of the source image is obtained according to a source image disparity map, and a contour surrounding area is a source image cloning area.
4. The method for cloning stereo images based on different camera distances as claimed in claim 1, wherein the image fusion specific steps are, after obtaining the optimized deformed grid of the source image cloning region, obtaining the deformed source image cloning region by bilinear interpolation, and then adopting a planar Poisson fusion method to fuse the left viewpoint image and the right viewpoint image of the deformed source image cloning region to the left viewpoint image and the right viewpoint image of the target image respectively to obtain the final stereo image fusion result.
CN201710097015.8A 2017-02-22 2017-02-22 Stereo image cloning method based on different camera distances Active CN106910253B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710097015.8A CN106910253B (en) 2017-02-22 2017-02-22 Stereo image cloning method based on different camera distances

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710097015.8A CN106910253B (en) 2017-02-22 2017-02-22 Stereo image cloning method based on different camera distances

Publications (2)

Publication Number Publication Date
CN106910253A CN106910253A (en) 2017-06-30
CN106910253B true CN106910253B (en) 2020-02-18

Family

ID=59208811

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710097015.8A Active CN106910253B (en) 2017-02-22 2017-02-22 Stereo image cloning method based on different camera distances

Country Status (1)

Country Link
CN (1) CN106910253B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107767339B (en) * 2017-10-12 2021-02-02 深圳市未来媒体技术研究院 Binocular stereo image splicing method
CN111292367B (en) * 2020-02-18 2023-04-07 青岛联合创智科技有限公司 Binocular camera depth map generation method with variable baseline

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1129813A (en) * 1995-08-11 1996-08-28 刘江海 Treatment of negative films of stereo photographs and pictures and products thereof
CN101375315A (en) * 2006-01-27 2009-02-25 图象公司 Methods and systems for digitally re-mastering of 2D and 3D motion pictures for exhibition with enhanced visual quality
CN101479765A (en) * 2006-06-23 2009-07-08 图象公司 Methods and systems for converting 2D motion pictures for stereoscopic 3D exhibition
CN102074014A (en) * 2011-02-23 2011-05-25 山东大学 Stereo matching method by utilizing graph theory-based image segmentation algorithm
CN104519343A (en) * 2013-09-26 2015-04-15 西克股份公司 3D camera in accordance with stereoscopic principle and method of detecting depth maps
CN105701812A (en) * 2016-01-12 2016-06-22 南京工程学院 Visual identification system suitable for cotton picking robot
CN105959595A (en) * 2016-05-27 2016-09-21 西安宏源视讯设备有限责任公司 Virtuality to reality autonomous response method for virtuality and reality real-time interaction

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9323055B2 (en) * 2006-05-26 2016-04-26 Exelis, Inc. System and method to display maintenance and operational instructions of an apparatus using augmented reality

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1129813A (en) * 1995-08-11 1996-08-28 刘江海 Treatment of negative films of stereo photographs and pictures and products thereof
CN101375315A (en) * 2006-01-27 2009-02-25 图象公司 Methods and systems for digitally re-mastering of 2D and 3D motion pictures for exhibition with enhanced visual quality
CN101479765A (en) * 2006-06-23 2009-07-08 图象公司 Methods and systems for converting 2D motion pictures for stereoscopic 3D exhibition
CN102074014A (en) * 2011-02-23 2011-05-25 山东大学 Stereo matching method by utilizing graph theory-based image segmentation algorithm
CN104519343A (en) * 2013-09-26 2015-04-15 西克股份公司 3D camera in accordance with stereoscopic principle and method of detecting depth maps
CN105701812A (en) * 2016-01-12 2016-06-22 南京工程学院 Visual identification system suitable for cotton picking robot
CN105959595A (en) * 2016-05-27 2016-09-21 西安宏源视讯设备有限责任公司 Virtuality to reality autonomous response method for virtuality and reality real-time interaction

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Content-Driven Retargeting of Stereoscopic Images;Jin Woo Yoo 等;《IEEE SIGNAL PROCESSING LETTERS》;20130531;全文 *
立体图像克隆和立体显示串扰的研究;王志远;《中国博士学位论文全文数据库 信息科技辑》;20160815(第8期);全文 *

Also Published As

Publication number Publication date
CN106910253A (en) 2017-06-30

Similar Documents

Publication Publication Date Title
CN113096234B (en) Method and device for generating three-dimensional grid model by using multiple color pictures
Kim et al. Multi-perspective stereoscopy from light fields
US9113043B1 (en) Multi-perspective stereoscopy from light fields
US20230291884A1 (en) Methods for controlling scene, camera and viewing parameters for altering perception of 3d imagery
US8213708B2 (en) Adjusting perspective for objects in stereoscopic images
Shao et al. QoE-guided warping for stereoscopic image retargeting
CN107451952B (en) Splicing and fusing method, equipment and system for panoramic video
CN107680164B (en) Virtual object size adjusting method and device
CN105678687A (en) Stereo image stitching method based on content of images
Du et al. Changing perspective in stereoscopic images
US9578312B2 (en) Method of integrating binocular stereo video scenes with maintaining time consistency
CN108234985B (en) Filtering method under dimension transformation space for rendering processing of reverse depth map
CN106548494A (en) A kind of video image depth extraction method based on scene Sample Storehouse
CN106327454A (en) Method and device of synthesizing image
CN115719407A (en) Distributed multi-view stereo reconstruction method for large-scale aerial images
CN106910253B (en) Stereo image cloning method based on different camera distances
Park et al. Efficient viewer-centric depth adjustment based on virtual fronto-parallel planar projection in stereo 3D images
Wang et al. Comfort-driven disparity adjustment for stereoscopic video
JP6682984B2 (en) Free-viewpoint video display device
CN102695070A (en) Depth consistency fusion processing method for stereo image
CN113034681B (en) Three-dimensional reconstruction method and device for spatial plane relation constraint
CN103686118A (en) Image depth adjustment method and device
Zhou et al. MR video fusion: interactive 3D modeling and stitching on wide-baseline videos
CN116503577A (en) Method and system for quickly collecting three-dimensional model of virtual simulation scene
Liao et al. Depth Map Design and Depth-based Effects With a Single Image.

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant