CN109087250B - Image splicing method based on regular boundary constraint - Google Patents
Image splicing method based on regular boundary constraint Download PDFInfo
- Publication number
- CN109087250B CN109087250B CN201810991548.5A CN201810991548A CN109087250B CN 109087250 B CN109087250 B CN 109087250B CN 201810991548 A CN201810991548 A CN 201810991548A CN 109087250 B CN109087250 B CN 109087250B
- Authority
- CN
- China
- Prior art keywords
- boundary
- constraint
- image
- regular
- energy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 34
- 230000001788 irregular Effects 0.000 claims abstract description 29
- 238000005457 optimization Methods 0.000 claims abstract description 15
- 230000014759 maintenance of location Effects 0.000 claims description 7
- 230000011218 segmentation Effects 0.000 claims description 5
- 238000004321 preservation Methods 0.000 claims 1
- 238000012545 processing Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- RZVAJINKPMORJF-UHFFFAOYSA-N Acetaminophen Chemical compound CC(=O)NC1=CC=C(O)C=C1 RZVAJINKPMORJF-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 208000029618 autoimmune pulmonary alveolar proteinosis Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Studio Devices (AREA)
Abstract
The invention discloses an image splicing method based on regular boundary constraint, which comprises the following steps: s1, performing initial splicing on the images; s2, extracting irregular boundaries of the panorama; s3 is established based on the rule boundary of the segment rectangle; s4 image stitching based on the segmented rectangular boundary constraint. The method defines a rule boundary according to the content and the shape characteristics of the panoramic image and realizes image splicing based on the rule boundary through global energy optimization.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to an image splicing method based on regular boundary constraint.
Background
With the rapid development of image acquisition and editing processing technologies, panoramic images have been widely applied to portable mobile devices such as smart phones, and the wider viewing angle thereof brings better visual experience to users. The problem of irregular boundaries (irregular boundaries) of image splicing in the prior art causes that a panoramic image needs to be cut in a large quantity in the display process, and partial image information is easy to lose.
Although the current image editing technology has made a lot of progress and is widely applied to smart devices, however, since cameras of mobile terminals such as mobile phones and the like can move freely during shooting, most of panoramic images after feature registration have irregular boundaries, and in order to show panoramic images in common display devices, the panoramic images generally need to be cut. Generally speaking, a panorama shot by a mobile phone camera is a result of cutting through a rectangular window, and partial image information is lost at the moment, so that the wide viewing angle experience of the panorama is reduced.
In order to generate a panorama with regular boundaries and avoid image content loss caused by cutting, there are currently two main approaches:
(1) provided is an image content complementing method. The method comprises the steps of firstly obtaining a circumscribed rectangular boundary of a panoramic image, and then completing image contents between the rectangular boundary and an irregular boundary. However, the performance of the image completion method is unstable, and it is difficult to complete content having a certain semantic meaning with high quality.
(2) Kaiming He et al proposed a deformation-based panorama image squaring method in 2013. The method takes a panoramic image with irregular boundary as input, takes a circumscribed rectangle as boundary constraint, combines local shape and straight line maintenance constraint, and obtains the panoramic image with the rectangular boundary through optimization based on deformation.
Although the above two methods can improve the problem of irregular boundary of the panorama, the following problems still exist:
1) image stitching and boundary regularization are used as two independent processes, and the result is difficult to ensure to be optimal.
2) The meshes on the irregular panorama have a problem of holes at the boundary.
3) When the panoramic image has a large amount of missing contents, the result after deformation optimization has large distortion and corresponds to the error of the characteristics.
Therefore, when the gap between the irregular boundary and the circumscribed rectangle is large, the regular boundary constraint needs to be redefined, and the distortion caused by deformation is reduced while the boundary rule is ensured.
Disclosure of Invention
The invention aims to solve the technical problem of providing an image splicing method based on regular boundary constraint, which defines a regular boundary according to the content and shape characteristics of a panoramic image and realizes image splicing and boundary regularization through global energy optimization.
In order to solve the technical problem, the invention adopts the following technical scheme:
an image stitching method based on regular boundary constraint, the image stitching method comprising:
s1, using a plurality of images containing partial overlapping as input, establishing energy optimization with feature correspondence among the images, local grid shape retention and global similarity retention as constraints, and obtaining an initial image splicing result through grid deformation;
s2, taking the mesh vertex of the initial image splicing as input, extracting the boundaries of the deformed image meshes, constructing the boundaries into polygons, and obtaining the irregular boundaries of the image meshes through the union set operation of the polygon Boolean operation, wherein the boundaries are composed of the intersections between partial mesh vertexes and meshes;
s3, taking the irregular boundary as input, segmenting the boundary according to the upper grid intersection point and the vertex of the irregular boundary, clustering adjacent boundaries according to the direction and the number of the vertexes of the segmented boundary, and taking the coordinate mean value of the vertexes of each segment of the clustered boundary as the target boundary condition of the panoramic image so as to generate a segmented rectangular boundary constraint;
s4, according to the irregular boundary and the segment boundary constraint conditions of the step S2 and the step S3, realizing seamless splicing and regular boundary keeping of the panorama according to the following modes: and performing linear combination on the image initialization splicing constraint, the regular boundary constraint and the straight line keeping constraint, then solving energy optimization and obtaining a panoramic image splicing result through deformation.
In a preferred embodiment, step S4 includes: the image initialization splicing constraint, the rule boundary constraint and the straight line keeping constraint are according to the formula
E(v)＝wa(Ea)+ws(Es)+wg(Eg)+wr(Er)+wl(El)
And performing linear combination, solving energy optimization and obtaining a panoramic image splicing result through deformation.
In a preferred embodiment, step S4 further includes: and optimizing the segmentation boundary in an iterative mode, and solving a new deformed grid by taking the segmentation boundary as constraint to obtain a panorama with a more regular boundary, so that the segmentation regular boundary is close to a rectangle and distortion is avoided.
The invention has the following beneficial effects:
1. the image splicing method based on the regular boundary constraint can generate a high-quality panoramic image, and the acceptable distortion of vision is ensured while the boundary is regularized.
2. The image splicing method based on the regular boundary constraint has high efficiency and practicability, can quickly calculate and render a panoramic image, and effectively ensures the content integrity and the boundary regularity of a panoramic shooting result.
3. The image splicing method based on the regular boundary constraint of the invention provides that the regular boundary constraint is added into an optimization framework of image splicing, and the image splicing based on the regular boundary is realized through global optimization, thereby solving the following problems: when the gap between the irregular boundary and the external rectangle is large, the content integrity of panoramic stitching can be ensured to the maximum extent by defining reasonable sectional rectangle boundaries, and the distortion caused by deformation can be effectively controlled.
Drawings
FIG. 1 is a schematic processing flow diagram of an image stitching method based on regular boundary constraint according to an embodiment of the present invention;
fig. 2 is an image stitching system based on the processing method shown in fig. 1.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, the invention discloses an image stitching method based on regular boundary constraint, fig. 1 shows a processing flow diagram of an embodiment of the image stitching method, and the method comprises:
s1, initial splicing of images
Using a plurality of partially overlapped images as input, establishing a feature correspondence { E) between the imagesaRetention of local grid shape { E }sKeep global similarity { E }gAnd (5) performing constrained energy optimization, and generating an initial image splicing result through grid deformation, wherein:
1) the feature points correspond to the APAP method proposed by Zaragoza et al in 2014 to realize accurate correspondence of the feature points.
2) Local shape retention is realized by restricting the deformation of each grid, namely, each four-side grid is divided into two triangles, and the shape feature retention of each right-angled triangle is realized by using the method proposed in 2009 by Igarashi et al.
3) The global feature is kept by adopting a method proposed by Chen et al in 2016, and global similarity transformation is realized by keeping the scale and rotation consistency of the spliced images.
S2, extracting irregular boundaries of panorama
The meshes of the initial stitching of the image are taken as input according to S1, the boundaries of each mesh are extracted and constructed into corresponding polygons, and then a boolean union operation is performed on the plurality of polygons, thereby obtaining irregular boundaries of the panorama. In this case, the boundary includes intersections between the partial mesh vertices and the mesh. Furthermore, grid corner points (namely, the upper corner point, the lower corner point, the left corner point and the right corner point of the grid) contained in the irregular boundary are extracted, the corner point closest to four vertexes of a circumscribed rectangle of the irregular boundary is used as the corner point of the irregular boundary, and the vertexes on the irregular boundary are divided into 4 directions by the extracted corner points.
S3, establishing rule boundary based on segmented rectangle
And segmenting the boundary according to the grid intersection points and the corner points in each direction of the irregular boundary. And then, further, carrying out clustering analysis on the segment boundaries according to the length and the direction of each segment boundary, merging the similar or excessively short segment boundaries, and finally taking the coordinate average value of each clustered segment boundary as a target boundary value. For the grid intersection points on the boundary, firstly, two pairs of grid vertexes forming the intersection points are extracted, the intersection points are represented by linear interpolation values of the two pairs of grid vertexes, and finally boundary constraints of the intersection points are converted into constraints on the interpolation grid vertexes.
S4, image stitching based on segmented rectangular boundary constraint
According to the irregular boundary and the segment boundary constraint conditions of the step S2 and the step S3, seamless splicing and regular boundary keeping of the panoramic image are achieved in the following mode, according to the irregular boundary constraint of the step S3, combined with the initial image splicing constraint of the step S1, global energy optimization is established, image splicing based on segment rectangular boundary constraint is achieved, and the energy constraint comprises feature consistency correspondence { E }aRetention of local shape { E }sKeep global similarity { E }gRetention of straight line { E }l}, rule boundary { ErLinearly combining the energies according to the following formula to obtain a total energy equation E (v), and minimizing E (v) to obtain a deformed grid, wherein in a specific embodiment, the weight of each energy item is wa＝1.5，ws＝10，wg＝0.75，wr＝1000，wl15. Finally, texture mapping is carried out according to the deformed grid position to obtain image mosaicAnd (5) connecting the two parts, and further eliminating the seam at the splicing part through seamless fusion.
The formula is as follows:
E(v)＝wa(Ea)+ws(Es)+wg(Eg)+wr(Er)+wl(El)
in one embodiment, in order to make the segment rule boundary as close to a rectangle as possible and avoid unnecessary distortion, the segment boundary is continuously optimized in an iterative manner, and a new deformed mesh is solved by taking the segment boundary as a constraint, so as to obtain a panorama with a more regular boundary. In each iteration process, firstly, adjacent segment boundaries in each direction are traversed, and the deformed grid with the minimum image splicing energy cost E (v) after the adjacent boundaries are combined is used as the result of the iteration. And if the energy cost difference of two adjacent iterations is too large or no segment boundary for combination exists, ending the iteration.
Referring to fig. 2, the image stitching method based on the regular boundary constraint can effectively solve the problem of irregular boundary of the traditional stitching method, and can obtain the optimal segmented rectangular boundary through the iterative optimization-based method, so that the visual effect of panoramic stitching is effectively improved.
According to the invention, a user can perform high-quality splicing on the images shot by moving through a handheld mobile device, and the normative property of the boundary of the panoramic image is ensured through the regular boundary constraint, so that the loss of image contents generated by cutting is effectively avoided. The method can be applied to deformation-based image and video editing, such as video splicing, video de-jittering and other operations.
It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the present invention have been described with reference to the accompanying drawings, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims.
Claims (3)
1. An image stitching method based on regular boundary constraint is characterized by comprising the following steps:
s1, using multiple images including partial overlap as input, establishing local mesh shape protection based on feature correspondence between images
Maintaining global similarity as constrained energy optimization, and obtaining an initial image splicing result through grid deformation;
s2, taking the mesh vertex of the initial image splicing as input, extracting the boundaries of a plurality of deformed image meshes, and constructing a plurality of images
Polygons, by union of polygon Boolean operations resulting in irregular boundaries of multiple image meshes, the boundaries being formed by partial meshes
The grid vertexes and the intersection points among the grids;
s3, taking the irregular boundary as input, segmenting the boundary according to the mesh intersection points and the vertexes on the irregular boundary, and according to the mesh intersection points and the vertexes on the irregular boundary
Clustering adjacent boundaries according to the direction and length of the boundary after segmentation, and taking the coordinate mean value of the vertex of each segment boundary after clustering as the
Panorama target boundary conditions, thereby generating piecewise rectangular boundary constraints;
s4, implementing panorama according to the irregular boundary and segment boundary constraint conditions of step S2 and step S3
Graph seamless stitching and regular boundary preservation: image initialization stitching constraint, regular boundary constraint, and line-keeping constraint
And performing line linear combination, then solving energy optimization and obtaining a panoramic image splicing result through deformation.
2. The method for image stitching based on regular boundary constraint according to claim 1, wherein the step S4 comprises:
the image initialization splicing constraint, the rule boundary constraint and the straight line keeping constraint are according to the formula
E(v)＝wa(Ea)+ws(Es)+wg(Eg)+wr(Er)+wl(El)
Performing linear combination, solving energy optimization and obtaining a panoramic image splicing result through deformation;
where e (v) represents total energy, Ea represents feature-consistent correspondence energy, Es represents local shape retention energy, Eg represents global similarity retention energy, Er represents regular boundary energy, El represents straight line retention energy, and the weights of the energy terms are wa ═ 1.5, ws ═ 10, wg ═ 0.75, wr ═ 1000, and wl ═ 15, respectively.
3. The method for image stitching based on regular boundary constraint according to claim 2, wherein the step S4 further comprises: optimizing the segment boundary in an iterative mode, and solving a new deformed grid by taking the segment boundary as constraint to obtain a more regular boundary
Panorama to make the segmentation rule boundaries close to rectangles and avoid distortion.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810991548.5A CN109087250B (en) | 2018-08-29 | 2018-08-29 | Image splicing method based on regular boundary constraint |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810991548.5A CN109087250B (en) | 2018-08-29 | 2018-08-29 | Image splicing method based on regular boundary constraint |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109087250A CN109087250A (en) | 2018-12-25 |
| CN109087250B true CN109087250B (en) | 2022-05-24 |
Family
ID=64795084
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810991548.5A Active CN109087250B (en) | 2018-08-29 | 2018-08-29 | Image splicing method based on regular boundary constraint |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109087250B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111243108A (en) * | 2020-01-19 | 2020-06-05 | 浙江传媒学院 | Three-dimensional image splicing method with maintained rectangular boundary |
| CN114189697B (en) * | 2021-12-03 | 2022-10-14 | 腾讯科技（深圳）有限公司 | Video data processing method and device and readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7426455B1 (en) * | 1999-09-13 | 2008-09-16 | Science Applications International Corporation | Optimal boolean set operation generation among polygon-represented regions |
| CN106611421A (en) * | 2016-12-30 | 2017-05-03 | 西安电子科技大学 | SAR image segmentation method based on feature learning and sketch line constraint |
| WO2017124868A1 (en) * | 2016-01-20 | 2017-07-27 | 北京疯景科技有限公司 | Method and apparatus for acquiring panorama image |
| CN107240067A (en) * | 2017-05-11 | 2017-10-10 | 同济大学 | A kind of sequence image method for automatically split-jointing based on three-dimensional reconstruction |
| CN107274337A (en) * | 2017-06-20 | 2017-10-20 | 长沙全度影像科技有限公司 | A kind of image split-joint method based on improvement light stream |
| CN107869955A (en) * | 2017-10-24 | 2018-04-03 | 华朗三维技术（深圳）有限公司 | A kind of laser 3 d scanner system and application method |
-
2018
- 2018-08-29 CN CN201810991548.5A patent/CN109087250B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7426455B1 (en) * | 1999-09-13 | 2008-09-16 | Science Applications International Corporation | Optimal boolean set operation generation among polygon-represented regions |
| WO2017124868A1 (en) * | 2016-01-20 | 2017-07-27 | 北京疯景科技有限公司 | Method and apparatus for acquiring panorama image |
| CN106611421A (en) * | 2016-12-30 | 2017-05-03 | 西安电子科技大学 | SAR image segmentation method based on feature learning and sketch line constraint |
| CN107240067A (en) * | 2017-05-11 | 2017-10-10 | 同济大学 | A kind of sequence image method for automatically split-jointing based on three-dimensional reconstruction |
| CN107274337A (en) * | 2017-06-20 | 2017-10-20 | 长沙全度影像科技有限公司 | A kind of image split-joint method based on improvement light stream |
| CN107869955A (en) * | 2017-10-24 | 2018-04-03 | 华朗三维技术（深圳）有限公司 | A kind of laser 3 d scanner system and application method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109087250A (en) | 2018-12-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11238644B2 (en) | Image processing method and apparatus, storage medium, and computer device | |
| Wei et al. | Fisheye video correction | |
| EP4057234A1 (en) | Method and apparatus for constructing three-dimensional facial model, and device and storage medium | |
| WO2016110239A1 (en) | Image processing method and device | |
| US20240169674A1 (en) | Indoor scene virtual roaming method based on reflection decomposition | |
| US20080043021A1 (en) | Three Dimensional Polygon Mesh Deformation Using Subspace Energy Projection | |
| CN107451952B (en) | Splicing and fusing method, equipment and system for panoramic video | |
| Zhang et al. | Content-preserving image stitching with piecewise rectangular boundary constraints | |
| CN111008927B (en) | Face replacement method, storage medium and terminal equipment | |
| US20180357775A1 (en) | Image processing method and dual camera system | |
| CN106910173B (en) | The method for realizing flake video wicket real time roaming based on correcting fisheye image | |
| CN109087250B (en) | Image splicing method based on regular boundary constraint | |
| WO2024098685A1 (en) | Face driving method and apparatus for virtual character, and terminal device and readable storage medium | |
| Du et al. | Geometric structure preserving warp for natural image stitching | |
| CN113822965A (en) | Image rendering processing method, device and equipment and computer storage medium | |
| CN110910308A (en) | Image processing method, apparatus, device and medium | |
| CN112489225A (en) | Method and device for fusing video and three-dimensional scene, electronic equipment and storage medium | |
| CN111462205A (en) | Image data deformation and live broadcast method and device, electronic equipment and storage medium | |
| CN110619670A (en) | Face interchange method and device, computer equipment and storage medium | |
| WO2024002064A1 (en) | Method and apparatus for constructing three-dimensional model, and electronic device and storage medium | |
| CN110502305B (en) | Method and device for realizing dynamic interface and related equipment | |
| CN111652025B (en) | Face processing and live broadcasting method and device, electronic equipment and storage medium | |
| CN111652807A (en) | Eye adjustment method, eye live broadcast method, eye adjustment device, eye live broadcast device, electronic equipment and storage medium | |
| Favorskaya et al. | Warping techniques in video stabilization | |
| Wang et al. | Image stitching using double features-based global similarity constraint and improved seam-cutting |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |