CN110443838B - Associative graph construction method for stereoscopic image stitching - Google Patents
Associative graph construction method for stereoscopic image stitching Download PDFInfo
- Publication number
- CN110443838B CN110443838B CN201910614586.3A CN201910614586A CN110443838B CN 110443838 B CN110443838 B CN 110443838B CN 201910614586 A CN201910614586 A CN 201910614586A CN 110443838 B CN110443838 B CN 110443838B
- Authority
- CN
- China
- Prior art keywords
- matching
- stereoscopic
- matching point
- association
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000010276 construction Methods 0.000 title abstract description 16
- 238000010586 diagram Methods 0.000 claims abstract description 39
- 238000000034 method Methods 0.000 claims description 20
- 239000011159 matrix material Substances 0.000 claims description 13
- 230000009466 transformation Effects 0.000 claims description 6
- 238000000605 extraction Methods 0.000 description 3
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
The invention discloses a correlation diagram construction method for stereo image stitching. According to the stereoscopic information formed by the left view, the right view and the parallax image of the stereoscopic images, the 2D matching point pairs among the stereoscopic image views are up-scaled to form the 3D matching point pairs, and the probability model based on the 3D matching point pairs is adopted to judge the matching relationship among the stereoscopic images, so that the association graph of the matching relationship among a plurality of stereoscopic images is constructed, and the association graph is used for guiding the stereoscopic images to be automatically spliced. The invention improves the traditional association diagram construction method for splicing the plane images, pioneers and detects the matching relationship among the stereo images from the 3D view angle, provides a simple and effective association diagram construction method, provides reliable matching relationship association diagrams for splicing multiple stereo images, avoids the need of a user to specify the splicing sequence of the stereo images, and provides a necessary premise for automatic stereo image splicing.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a correlation diagram construction method for stereoscopic image stitching.
Background
With the popularization of binocular shooting equipment, even more and more mobile phones start to have double cameras, and people can shoot stereoscopic images more conveniently. However, due to the limitation of the photographing angle of view of the photographing apparatus, people often cannot acquire the full view of the region-of-interest object at one time, but need to photograph multiple times to acquire a stereoscopic image of the region-of-interest object at a wider angle of view. Therefore, a plurality of stereoscopic images photographed at different angles and having overlapping areas are automatically spliced into a complete stereoscopic image, and the method has great practical significance.
The key premise is that matching relation between stereo images can be automatically determined, and the need of a user to specify the splicing sequence of the stereo images is avoided. After the matching relationship is determined, the matching relationship is generally represented by adopting a correlation graph, each node in the correlation graph corresponds to a pair of stereo images, and the connecting edges between the nodes represent that the matching relationship exists between the corresponding stereo images, i.e. an overlapping area exists, so that the matching relationship can be spliced.
The existing association diagram constructing technology has a scheme in AutoStitch, and the scheme can construct an association diagram of a matching relationship for a group of input plane images. The scheme assumes that the overlapping area of the input images can be similar to the same plane, or the shooting optical center is kept unchanged, SIFT feature points of each plane image are extracted and matched, a RANSAC method is adopted to estimate an optimal global homography matrix, matching point pairs are divided into intra-office points and extra-office points, whether the images have a matching relationship or not is judged according to the inequality relationship of the number of the intra-office points and the number of the extra-office points deduced based on a Bernoulli distribution probability model, and accordingly a correlation diagram is constructed according to the judged matching relationship.
The scheme is a correlation diagram construction method for splicing plane pictures, and the method directly used for constructing the correlation diagram required by stereoscopic image splicing has the following problems:
the binocular light centers of the stereoscopic images cannot be kept fixed in the shooting process, the image overlapping area is difficult to ensure to be similar to the same plane, namely, the stereoscopic images can have parallax phenomenon, so that the method for dividing the intra-office point and the extra-office point by estimating the optimal global homography matrix through the RANSAC is easy to be affected by parallax, and the problem of misjudgment of the matching relationship occurs.
In addition, the stereo image is composed of a left view and a right view, and how to integrate the matching relation judgment results of the two views is also important to consider.
Disclosure of Invention
The invention aims to overcome the defects of the existing method and provides a correlation diagram construction method for stereo image stitching. The problems solved by the invention are mainly two: firstly, how to deal with the problem of misjudgment of the matching relation caused by parallax; and secondly, how to integrate the judging result of the matching relation of the left view and the right view.
In order to solve the above problems, the present invention proposes a method for constructing a correlation map for stereoscopic image stitching, the method comprising:
step one, inputting a left view, a right view and a parallax image of a group of stereoscopic images, and giving out internal parameters of a stereoscopic camera;
step two, assigning an index number to each stereoscopic image;
thirdly, constructing a correlation diagram with any two nodes connected by taking the index number as a node;
step four, extracting SIFT feature points in left and right views of the stereoscopic images corresponding to nodes at two ends of each side in the association diagram to obtain 2D matching point pairs between the stereoscopic images;
step five, based on the parallax map and the camera internal parameters, the 2D matching point pair obtained in the step four is up-scaled to form a 3D matching point pair;
step six, selecting one side in the association diagram, detecting whether the stereo images corresponding to the nodes at the two ends of the side have a matching relationship or not by adopting a probability model based on the 3D matching point pair, and deleting the side from the association diagram if the matching relationship does not exist;
step seven, repeating the step six until each edge in the association graph is detected;
step eight, detecting whether isolated points exist in the association diagram, and deleting the isolated points if the isolated points exist;
and step nine, outputting a final association diagram.
Preferably, the probability model adopts a RANSAC method to estimate an optimal global transformation matrix to divide the 3D matching point pair into an intra-office point and an extra-office point, and judges whether the image has the matching relationship according to the inequality relationship of the number of the intra-office point and the number of the extra-office point deduced based on the Bernoulli distribution probability model.
The association diagram construction method for stereo image splicing is improved based on the traditional association diagram construction method for planar image splicing, the matching relationship among stereo images is detected from a 3D view angle in a pioneering mode, a simple and effective association diagram construction method is provided, reliable matching relationship association diagrams are provided for splicing multiple stereo images, the need of a user to specify the splicing sequence of the stereo images is avoided, and a necessary premise is provided for automatic stereo image splicing.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of an associative map construction according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
FIG. 1 is a flowchart of the association diagram construction of an embodiment of the present invention, as shown in FIG. 1, the method includes:
s1, inputting a left view, a right view and a parallax image of a group of stereoscopic images, and giving out internal parameters of a stereoscopic camera;
s2, assigning an index number for each stereoscopic image;
s3, constructing a correlation diagram with any two nodes connected by taking the index number as a node;
s4, extracting SIFT feature points in left and right views of the stereoscopic images corresponding to nodes at two ends of each side in the association graph to obtain 2D matching point pairs between the stereoscopic images;
s5, based on the parallax image and the camera internal parameters, the 2D matching point pair obtained in the S4 is up-scaled to form a 3D matching point pair;
s6, selecting one side in the association graph, detecting whether the stereo images corresponding to the nodes at the two ends of the side have a matching relationship or not by adopting a probability model based on the 3D matching point pair, and deleting the side from the association graph if the matching relationship does not exist;
s7, repeatedly executing S6 until each edge in the association graph is detected;
s8, detecting whether isolated points exist in the association diagram, and deleting the isolated points if the isolated points exist;
s9, outputting a final association diagram;
step S1, specifically, the following steps are performed:
the input stereoscopic image group is { (L) i ,R i ,D i ) I=1, 2,3,..n }, where L i R is the left view of the ith stereoscopic image i Is the right view of the ith image, D i The disparity map for the i-th image, n is the number of input stereoscopic images. The camera intrinsic includes a focal length f, coordinates of the principal point relative to the imaging plane (u 0 ,v 0 ) Binocular base line of stereo camera.
Step S2, specifically, the following steps are performed:
the index number is set as the input sequence number of the stereoscopic image, i.e., the i-th stereoscopic image (L i ,R i ,D i ) The index number of (c) is set to i.
Step S3, specifically, the following steps are performed:
the association diagram is represented by a 01 matrix, wherein 1 is the connected edge of the corresponding row and column index, and 0 is the unconnected edge. The initial association diagram matrix is:
wherein, the row and column numbers of the matrix are n.
Step S4, specifically, the following steps are performed:
let nodes at both ends of a certain edge in the associated graph be i and j, and correspond to the ith stereoscopic image (L i ,R i ,D i ) And the jth stereoscopic image (L j ,R j ,D j ). SIFT feature point extraction is respectively carried out on the left view and the right view of the two stereo images, wherein the SIFT feature point extraction result of the ith stereo image is expressed as
The SIFT feature point extraction result of the j-th stereoscopic image is expressed as
SIFT feature points representing left view of ith stereoscopic image,/and method for generating the same>
Representing the right side of the ith stereoscopic imageSIFT feature points of view, < >>
And->
And the same is true.
Matching the extracted SIFT feature points, and obtaining the following results:
the SIFT feature point matching result of the left view and the right view of the ith stereoscopic image is expressed as follows:
the SIFT feature point matching result of the left view and the right view of the j-th stereoscopic image is expressed as follows:
the matching result of the SIFT feature points of the left view of the ith stereoscopic image and the SIFT feature points of the left view of the jth stereoscopic image is expressed as follows:
the SIFT feature point matching result of the right view of the ith stereoscopic image and the SIFT feature point matching result of the right view of the jth stereoscopic image are expressed as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,
and->
Respectively represent corresponding matching point pairsIs a number of (3).
The 2D matching point pair of the ith stereoscopic image and the jth stereoscopic image consists of two parts.
1) The screen is selected from
2) The screen is selected from
Finally, the 2D matching point pair of the ith stereoscopic image and the jth stereoscopic image is expressed as:
step S5, specifically, the following steps are performed:
obtaining a parallax image D of a corresponding stereoscopic image of two end nodes i and j of one edge in the association image i And D j And 2D matching point pairs
Wherein n is ij Representing the number of matching point pairs. Further, a +>
Can be written as +.>
I.e. < ->
Corresponding ith pairThe two-dimensional coordinates of the pixels in the left view of the stereoscopic image.
In disparity map D i The disparity value of the corresponding coordinates is d ik According to the formula:
two-dimensional pixel points
The rising dimension is formed into a three-dimensional pixel point +.>
Expressed as->
Similarly->
Corresponding two-dimensional pixel point->
The up-dimension is also formed into a three-dimensional pixel point +.>
Expressed as->
So that the 2D matching point pair is up-scaled to a 3D matching point pair with Θ ij Representing the 3D matching point pair after dimension up, namely:
where→represents an up-scaling operation.
Step S6, specifically, the following steps are performed:
for the association diagramTwo end nodes i and j of one edge of the three-dimensional graph are used for obtaining a 3D matching point pair theta of the corresponding three-dimensional graph ij Estimating the optimal global transformation matrix by RANSAC method to match the point pairs theta ij Divided into outliers and intra-office points. Let the number of points in the office be
The total number of matching point pairs is +.>
According to the derivation of Bernoulli distribution probability model in auto-latch, when +.>
And judging that the two stereo images have a matching relationship. Otherwise, judging that no matching relation exists, and deleting the edge from the association graph.
Further, the dimension of the transformation matrix H is 4*4, for any one of the 3D matching point pairs
Its three-dimensional coordinates are +.>
The transformation matrix H satisfies: />
Wherein { a }, a ij I=0, 1,2,3, j=0, 1,2,3} is a parameter value of the corresponding position of the transformation matrix H.
Step S8, specifically, the following steps are performed:
let the associated graph matrix be G, if a certain index number i satisfies
And judging the node i as an isolated point, and deleting the row and the column with the index number i from the association diagram matrix G.
The association diagram construction method for stereo image splicing provided by the embodiment of the invention is improved based on the traditional association diagram construction method for planar image splicing, the matching relationship among stereo images is detected from a 3D view angle in a pioneering way, a simple and effective association diagram construction method is provided, reliable matching relationship association diagrams are provided for splicing multiple stereo images, the need of a user to specify the splicing sequence of the stereo images is avoided, and a necessary premise is provided for automatic stereo image splicing.
Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program to instruct related hardware, the program may be stored in a computer readable storage medium, and the storage medium may include: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.
In addition, the above description has been made in detail on a method for constructing a correlation diagram for stereo image stitching provided by the embodiment of the present invention, and specific examples are applied herein to illustrate the principles and embodiments of the present invention, and the description of the above embodiments is only for helping to understand the method and core ideas of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
Claims (2)
1. A method for constructing a correlation map for stereoscopic image stitching, the method comprising:
step one, inputting a left view, a right view and a parallax image of a group of stereoscopic images, and giving out internal parameters of a stereoscopic camera;
step two, assigning an index number to each stereoscopic image;
thirdly, constructing a correlation diagram with any two nodes connected by taking the index number as a node;
step four, extracting SIFT feature points in left and right views of the stereoscopic images corresponding to nodes at two ends of each side in the association diagram to obtain 2D matching point pairs between the stereoscopic images;
step five, based on the parallax map and the camera internal parameters, the 2D matching point pair obtained in the step four is up-scaled to form a 3D matching point pair;
step six, selecting one side in the association diagram, detecting whether the stereo images corresponding to the nodes at the two ends of the side have a matching relationship or not by adopting a probability model based on the 3D matching point pair, and deleting the side from the association diagram if the matching relationship does not exist;
step seven, repeating the step six until each edge in the association graph is detected;
step eight, detecting whether isolated points exist in the association diagram, and deleting the isolated points if the isolated points exist;
step nine, outputting a final association diagram;
step five, based on the parallax map and the camera internal parameters, the 2D matching point pair obtained in step four is up-scaled to form a 3D matching point pair, which specifically comprises:
obtaining a parallax image D of a corresponding stereoscopic image of two end nodes i and j of one edge in the association image i And D j And 2D matching point pairs
k＝1,2,3,...,n ij N is }, where n ij Representing the number of matching point pairs; further, a +>
Write->
I.e. < ->
The corresponding pixel two-dimensional coordinates in the left view of the ith stereoscopic image; />
In disparity map D i View of corresponding coordinatesThe difference is d ik According to the formula:
two-dimensional pixel points
Up-to-dimension three-dimensional pixel point>
Expressed as->
Similarly->
Corresponding two-dimensional pixel point->
Also up-dimension three-dimensional pixel point>
Expressed as->
Thus, the 2D matching point pair is up-scaled to form a 3D matching point pair by Θ ij Representing the 3D matching point pair after dimension up, namely:
where→represents an up-scaling operation.
2. The method for constructing a correlation map for stereoscopic image stitching according to claim 1, wherein the probability model is a method for estimating an optimal global transformation matrix by using a RANSAC method to divide a 3D matching point pair into an intra-office point and an extra-office point, and determining whether the image has the matching relationship according to the inequality relationship between the number of the intra-office points and the number of the extra-office points derived based on the bernoulli distribution probability model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910614586.3A CN110443838B (en) | 2019-07-09 | 2019-07-09 | Associative graph construction method for stereoscopic image stitching |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910614586.3A CN110443838B (en) | 2019-07-09 | 2019-07-09 | Associative graph construction method for stereoscopic image stitching |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110443838A CN110443838A (en) | 2019-11-12 |
| CN110443838B true CN110443838B (en) | 2023-05-05 |
Family
ID=68429961
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910614586.3A Active CN110443838B (en) | 2019-07-09 | 2019-07-09 | Associative graph construction method for stereoscopic image stitching |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110443838B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105303615A (en) * | 2015-11-06 | 2016-02-03 | 中国民航大学 | Combination method of two-dimensional stitching and three-dimensional surface reconstruction of image |
| CN107424181A (en) * | 2017-04-12 | 2017-12-01 | 湖南源信光电科技股份有限公司 | A kind of improved image mosaic key frame rapid extracting method |
| CN107833179A (en) * | 2017-09-05 | 2018-03-23 | 云南电网有限责任公司昆明供电局 | The quick joining method and system of a kind of infrared image |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20140108828A (en) * | 2013-02-28 | 2014-09-15 | 한국전자통신연구원 | Apparatus and method of camera tracking |
-
2019
- 2019-07-09 CN CN201910614586.3A patent/CN110443838B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105303615A (en) * | 2015-11-06 | 2016-02-03 | 中国民航大学 | Combination method of two-dimensional stitching and three-dimensional surface reconstruction of image |
| CN107424181A (en) * | 2017-04-12 | 2017-12-01 | 湖南源信光电科技股份有限公司 | A kind of improved image mosaic key frame rapid extracting method |
| CN107833179A (en) * | 2017-09-05 | 2018-03-23 | 云南电网有限责任公司昆明供电局 | The quick joining method and system of a kind of infrared image |
Non-Patent Citations (1)
| Title |
|---|
| 基于DPP改进RANSAC算法的图像拼接;汪旌等;《计算机系统应用》;20180515(第05期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110443838A (en) | 2019-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TWI554976B (en) | Surveillance systems and image processing methods thereof | |
| CN101673395B (en) | Image mosaic method and image mosaic device | |
| CN101394573B (en) | Panoramagram generation method and system based on characteristic matching | |
| CN107194965A (en) | Method and apparatus for handling light field data | |
| CN104050662B (en) | Method for directly obtaining depth image through light field camera one-time imaging | |
| CN105744138B (en) | Quick focusing method and electronic equipment | |
| TWI587241B (en) | Method, device and system for generating two - dimensional floor plan | |
| CN101504761A (en) | Image splicing method and apparatus | |
| CN104641625B (en) | Image processing apparatus, camera device and image processing method | |
| CN104616247B (en) | A kind of method for map splicing of being taken photo by plane based on super-pixel SIFT | |
| CN103902953B (en) | A kind of screen detecting system and method | |
| CN106355621A (en) | Method for acquiring depth information on basis of array images | |
| CN111837158A (en) | Image processing method and device, shooting device and movable platform | |
| CN109544621A (en) | Light field depth estimation method, system and medium based on convolutional neural networks | |
| EP3026631A1 (en) | Method and apparatus for estimating depth of focused plenoptic data | |
| CN113808033A (en) | Image document correction method, system, terminal and medium | |
| CN110443838B (en) | Associative graph construction method for stereoscopic image stitching | |
| CN111738061A (en) | Binocular vision stereo matching method based on regional feature extraction and storage medium | |
| CN106296580A (en) | A kind of method and device of image mosaic | |
| CN110381353A (en) | Video scaling method, apparatus, server-side, client and storage medium | |
| CN112991207B (en) | Panoramic depth estimation method and device, terminal equipment and storage medium | |
| CN116051736A (en) | Three-dimensional reconstruction method, device, edge equipment and storage medium | |
| WO2021208630A1 (en) | Calibration method, calibration apparatus and electronic device using same | |
| Skuratovskyi et al. | Outdoor mapping framework: from images to 3d model | |
| CN110896469B (en) | Resolution testing method for three-shot photography and application thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |