CN105957008B - The real-time joining method of panoramic picture and system based on mobile terminal - Google Patents
The real-time joining method of panoramic picture and system based on mobile terminal Download PDFInfo
- Publication number
- CN105957008B CN105957008B CN201610303650.2A CN201610303650A CN105957008B CN 105957008 B CN105957008 B CN 105957008B CN 201610303650 A CN201610303650 A CN 201610303650A CN 105957008 B CN105957008 B CN 105957008B
- Authority
- CN
- China
- Prior art keywords
- present frame
- panoramic picture
- transformation matrix
- frame
- characteristic point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 239000011159 matrix material Substances 0.000 claims abstract description 92
- 230000009466 transformation Effects 0.000 claims abstract description 72
- 238000001514 detection method Methods 0.000 claims abstract description 19
- 238000004364 calculation method Methods 0.000 claims abstract description 12
- 238000012545 processing Methods 0.000 claims description 18
- 238000013507 mapping Methods 0.000 claims description 10
- 238000002156 mixing Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 5
- 235000013399 edible fruits Nutrition 0.000 claims 1
- 230000008569 process Effects 0.000 description 5
- 230000004044 response Effects 0.000 description 5
- 230000004927 fusion Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 2
- 239000010931 gold Substances 0.000 description 2
- 229910052737 gold Inorganic materials 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000010422 painting Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000000527 sonication Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a kind of real-time joining method of panoramic picture based on mobile terminal and systems, it carries out characteristic point detection by the present frame of the video flowing to acquisition, the characteristic point sequence of attributes of the characteristic point sequence of attributes of the obtained present frame and the former frame of the video flowing is subjected to Feature Points Matching, and the transformation matrix between the present frame and the former frame is calculated according to matched result, finally the present frame and the former frame are spliced in real time according to the transformation matrix, it loops back and forth like this, successively obtain preliminary panoramic picture, intermediate panoramic image, final panoramic picture, and preview is carried out to preliminary panoramic picture or intermediate panoramic image or final panoramic picture in real time and is shown, algorithm is simple, calculation amount is small, it can be realized the real-time splicing that panoramic picture is carried out during pan-shot, live preview, and It is lower for hardware configuration requirement, especially suitable for realizing the shooting of high-resolution real time panoramic in the mobile terminals such as the mobile phone that low and middle-end configures.
Description
Technical field
The present invention relates to technical field of image processing, especially a kind of panoramic picture based on mobile terminal side of splicing in real time
The system of method and its application this method.
Background technique
Panoramic mosaic technology in current mobile device is generally by the image for acquiring multiple angles, and according to characteristic point
Matching process carries out the computationally intensive operation such as image registration and fusion to original high-resolution image, generates panoramic picture.
Firstly, since image registration and fusion is computationally intensive, so figure can only be acquired by way of interval sampling
Picture.Since the interval of Image Acquisition is larger, cause lap smaller first, be to feature point alignment it is unfavorable, be easy to produce
The ghost image of panorama sketch.
Secondly as the scenery difference of sampling is larger, the white balance and exposure parameter of usual camera, which have occurred and that, to be compared
Big variation, so post-processing part is generally also needed using some exposure compensatings and complicated method mixed image, so that figure
As transition naturally, to allow user to need to wait a longer time acquisition panoramic picture.
Again, such acquisition mode, which determines, is difficult to allow some intermediate results of mobile device generation pre- in real time for user
It lookes at, makes the unpredictable final splicing result of user.
In addition, since its is computationally intensive, smooth preview effect can not be obtained even if providing the preview of splicing result, and
And it is unable to satisfy the real-time of Preview results, final fusion results usually have the painting of some unnatural sawtooth and integration region
Sense is smeared, moreover, more demanding for hardware configuration.
It is computationally intensive due to real time panoramic, so being realized in the mobile terminals such as the mobile phone of low and middle-end configuration or camera high
Real time panoramic is differentiated to need to overcome biggish technical difficulty.
Summary of the invention
The present invention is to solve the above problems, providing a kind of real-time joining method of the panoramic picture based on mobile terminal and being
System, algorithm is simple, and real-time is good, and it is more preferable to can be realized splicing live preview, user experience in real time.
To achieve the above object, the technical solution adopted by the present invention are as follows:
Firstly, the present invention provides a kind of real-time joining method of the panoramic picture based on mobile terminal comprising following steps:
10. obtaining the video flowing of preview in real time, and characteristic point detection is carried out to the present frame of the video flowing of acquisition, is somebody's turn to do
The characteristic point sequence of attributes of present frame；
20. by the characteristic point sequence of attributes of the characteristic point sequence of attributes of the present frame and the former frame of the video flowing into
Row Feature Points Matching；
30. calculating the transformation matrix between the present frame and the former frame according to the result of the Feature Points Matching；
40. being spliced the present frame and the former frame in real time according to the transformation matrix, preliminary panorama is obtained
Image, and preview is carried out to the preliminary panoramic picture and is shown；
50. repeating step 10,20,30, and according to the transformation matrix by the present frame and the preliminary panoramic picture
Spliced in real time, obtains intermediate panoramic image, and preview is carried out to the intermediate panoramic image and is shown；
60. repeating step 50, and the present frame and the intermediate panoramic image are carried out in fact according to the transformation matrix
When splice, obtain final panoramic picture, and preview is carried out to the final panoramic picture and is shown.
Preferably, in the step 10, the characteristic point detection further comprises:
11. pair present frame carries out gray processing processing, grayscale image is generated；
12. it is down-sampled to gray level image progress, the grayscale image under one group of different scale is obtained, a grayscale image gold word is formed
Tower；
13. the grayscale image of pair different scale carries out the detection of FAST characteristic point, the characteristic point sequence of attributes is obtained.
Preferably, the characteristic point sequence of attributes includes: position attribution, direction attribute, scale properties and corresponding retouches
State son.
Preferably, in the step 20, the Feature Points Matching further comprises:
21. KNN algorithm is utilized, by the feature of the characteristic point sequence of attributes of the present frame and the former frame of the video flowing
Point sequence of attributes is matched, and the quantity matched pair between the present frame and the former frame is obtained；
22. the quantity of the matching pair described in pair carries out threshold calculations, if the quantity is greater than or equal to preset threshold,
With success；Otherwise it fails to match.
Preferably, in the step 30, according to the result of the Feature Points Matching calculate the present frame and it is described before
Transformation matrix between one frame, calculation method are as follows:
31. if calculating the matching pair by RANSAC algorithm the result of the Feature Points Matching is successful match
Optimal homography matrix, and in conjunction with the internal reference matrix of calibration carry out that the transformation matrix is calculated；
32. if calculating the present frame according to gyro data the result of the Feature Points Matching is that it fails to match
With the transformation matrix between the former frame.
Preferably, in the step 40 or 50 or 60, according to the transformation matrix by the present frame with it is described previous
Frame or the preliminary panoramic picture or the intermediate panoramic image are spliced in real time, further comprise:
41. carrying out forward projection to the present frame according to the transformation matrix, and calculate the feature of the present frame
Mapping coordinates of the point on the cylindrical surface or spherical surface that focal length is radius；
42. carrying out back projection to the mapping coordinates according to the transformation matrix, and calculates the mapping coordinates and exist
2D on the cylindrical surface or the tangent perspective plane of the spherical surface is projected in the interpolated coordinates in the present frame；
43. the method by line sampling obtains interpolated coordinates corresponding pixel value in the present frame；
44. according to the interpolated coordinates and its corresponding pixel value, by the present frame and the former frame or it is described just
Step panoramic picture or the intermediate panoramic image are spliced.
Preferably, in the splicing, also further to the present frame and the former frame or the preliminary panorama
Image or the intermediate panoramic image carry out emergence mixed processing, and the method for the emergence mixed processing includes:
71. calculating the offset between the present frame and the former frame；
72. calculating the mixed weight-value between the present frame and the former frame according to the offset；
73. final mixing is calculated according to the mixed weight-value and the interpolated coordinates and its corresponding pixel value
End value.
Secondly, the present invention provides a kind of real-time splicing system of the panoramic picture based on mobile terminal comprising:
Characteristic point detection module is carried out for obtaining the video flowing of preview in real time, and to the present frame of the video flowing of acquisition
Characteristic point detection, obtains the characteristic point sequence of attributes of the present frame；
Feature Points Matching module, for by the former frame of the characteristic point sequence of attributes of the present frame and the video flowing
Characteristic point sequence of attributes carries out Feature Points Matching；
Transformation matrix computing module, for according to the result of the Feature Points Matching calculate the present frame with it is described previous
Transformation matrix between frame；
Panoramic mosaic module, for being spelled the present frame and the former frame in real time according to the transformation matrix
It connects, obtains preliminary panoramic picture；Alternatively, the present frame and the preliminary panoramic picture are carried out in fact according to the transformation matrix
When splice, obtain intermediate panoramic image；Or according to the transformation matrix by the present frame and the intermediate panoramic image into
Row splicing in real time, obtains final panoramic picture；
Panorama preview module, for the preliminary panoramic picture or the intermediate panoramic image or the final panorama
Image carries out preview and shows.
Preferably, the transformation matrix computing module further comprises the first transformation matrix computing module and the second transformation square
Battle array computing module:
If the result of the Feature Points Matching is successful match, the first transformation matrix computing module is called, is passed through
RANSAC algorithm calculates the optimal homography matrix of the matching pair, and the internal reference matrix of calibration is combined to be calculated
The transformation matrix；
If the result of the Feature Points Matching is that it fails to match, the second transformation matrix computing module is called, according to
Gyro data calculates the transformation matrix between the present frame and the former frame.
Preferably, further include emergence mixing module, in splicing to the present frame and the former frame or
The preliminary panoramic picture or the intermediate panoramic image carry out emergence mixed processing.
The beneficial effects of the present invention are:
A kind of real-time joining method of panoramic picture based on mobile terminal of the invention and system, it is pre- by obtaining in real time
The video flowing look at, and characteristic point detection is carried out to the present frame of the video flowing of acquisition, obtain the characteristic point attribute sequence of the present frame
Then column carry out the characteristic point sequence of attributes of the characteristic point sequence of attributes of the present frame and the former frame of the video flowing special
Sign point matching, and the transformation matrix between the present frame and the former frame is calculated according to the result of the Feature Points Matching,
The present frame and the former frame are spliced according to the transformation matrix in real time finally, looped back and forth like this, successively
To preliminary panoramic picture, intermediate panoramic image, final panoramic picture, and in real time to preliminary panoramic picture or intermediate panoramic image
Or final panoramic picture carries out preview and shows that not only algorithm is simple, calculation amount is small, can be realized every frame resolution ratio is 5,000,000
Pixel (2560*1920), the processing speed of frame per second 30fps, to realize the reality for carrying out panoramic picture during pan-shot
When splicing, live preview, and hardware configuration is required it is lower, it is mobile eventually especially suitable for mobile phone for being configured in low and middle-end etc.
The shooting of high-resolution real time panoramic is realized in end；Also, make spliced panoramic picture more smooth by emergence mixed processing
And do not lose the clarity of original image.
Detailed description of the invention
The drawings described herein are used to provide a further understanding of the present invention, constitutes a part of the invention, this hair
Bright illustrative embodiments and their description are used to explain the present invention, and are not constituted improper limitations of the present invention.In the accompanying drawings:
Fig. 1 is that the present invention is based on the general flow charts of the real-time joining method of the panoramic picture of mobile terminal；
Fig. 2 is that the present invention is based on the structural schematic diagrams of the real-time splicing system of the panoramic picture of mobile terminal；
Fig. 3 is the general flow chart of the real-time splicing of panoramic picture of a specific embodiment of the invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.It should be appreciated that specific embodiment described herein is only to solve
The present invention is released, is not intended to limit the present invention.Based on the embodiments of the present invention, those of ordinary skill in the art are not making
Every other embodiment obtained, shall fall within the protection scope of the present invention under the premise of creative work.
As shown in Figure 1, a kind of real-time joining method of panoramic picture based on mobile terminal of the invention comprising following step
It is rapid:
10. obtaining the video flowing of preview in real time, and characteristic point detection is carried out to the present frame of the video flowing of acquisition, is somebody's turn to do
The characteristic point sequence of attributes of present frame；
20. by the characteristic point sequence of attributes of the characteristic point sequence of attributes of the present frame and the former frame of the video flowing into
Row Feature Points Matching；
30. calculating the transformation matrix between the present frame and the former frame according to the result of the Feature Points Matching；
40. being spliced the present frame and the former frame in real time according to the transformation matrix, preliminary panorama is obtained
Image, and preview is carried out to the preliminary panoramic picture and is shown；
50. repeating step 10,20,30, and according to the transformation matrix by the present frame and the preliminary panoramic picture
Spliced in real time, obtains intermediate panoramic image, and preview is carried out to the intermediate panoramic image and is shown；
60. repeating step 50, and the present frame and the intermediate panoramic image are carried out in fact according to the transformation matrix
When splice, obtain final panoramic picture, and preview is carried out to the final panoramic picture and is shown.
In the step 10, the characteristic point detection further comprises:
11. pair present frame carries out gray processing processing, grayscale image is generated；
12. it is down-sampled to gray level image progress, the grayscale image under one group of different scale is obtained, a grayscale image gold word is formed
Tower；
13. the grayscale image of pair different scale carries out the detection of FAST characteristic point, the characteristic point sequence of attributes is obtained.
Wherein, the characteristic point sequence of attributes includes: position attribution, direction attribute, scale properties and corresponding description
Son.The obtaining step of this feature point sequence of attributes specifically includes:
Step1. gray level image pyramid is established；
Step2. each pyramid maximum target detection points are divided, wherein in pyramid each layer with upper one layer
Quantity ratio is with image scaling than consistent；
Step3.FAST Corner Detection goes out candidate feature point；
Step4. first round screening is done according to FAST response；
Step5. since FAST response representativeness can be poor, it is more accurately gradient that only τ, which is examined, on a kind of circumference
Harris response, needs to recalculate its response to the angle point that detected；
Step6. according to the response calculated in step5, last wheel is carried out to candidate angular and is screened；
Step7. by IC (x square and y square), angle point direction is calculated；
Step8. Gaussian Blur and anti-noise sonication are carried out to each tomographic image, obtains denoising image；
Step9. in the denoising image basis of step8, length is selected to extract position for 64 description.The position of extraction
For centered on angle point.
In the step 20, the Feature Points Matching further comprises:
21. KNN algorithm is utilized, by the feature of the characteristic point sequence of attributes of the present frame and the former frame of the video flowing
Point sequence of attributes is matched, and the quantity matched pair between the present frame and the former frame is obtained；
22. the quantity of the matching pair described in pair carries out threshold calculations, if the quantity is greater than or equal to preset threshold,
With success；Otherwise it fails to match.
It is specific as follows:
Step1. distance measure of the hamming distance of description as two characteristic points is calculated, it is assumed that figure A and figure B are
Two field pictures to be registered, scheme A in characteristic point i to figure B in characteristic point j distance, go through all over figure B in characteristic point, obtain figure B in from
Scheme two characteristic points Minj, sndMinj that i characteristic point is nearest in A, note distance is MinDistance, sndMinDistance,
Wherein, MinDistance < sndMinDistance.
If step2. meeting MinDistance < sndMinDistance* (1-threadhold), threshold is default
Threshold value, value are that then Minj matches i to 0-1., save matching result, otherwise it fails to match.
Step3. step step1-step2 is repeated, all characteristic points in figure A is gone through, obtains matched characteristic point.
In the step 30, calculated between the present frame and the former frame according to the result of the Feature Points Matching
Transformation matrix, calculation method are as follows:
31. if calculating the matching pair by RANSAC algorithm the result of the Feature Points Matching is successful match
Optimal homography matrix, and in conjunction with the internal reference matrix of calibration carry out that the transformation matrix is calculated；
32. if calculating the present frame according to gyro data the result of the Feature Points Matching is that it fails to match
With the transformation matrix between the former frame.
Wherein, the step 31 further comprises:
Step1. best homography matrix is found out by RANSAC algorithm to (matching to) using the characteristic point matched
H。
Step2. the camera internal reference matrix K of calibration is obtained, and in conjunction with the best homography matrix H, finds out correspondence
Transformation matrix R, its calculation formula is:
R=Kinv*H*K；
In above-mentioned formula, K indicates that the camera internal reference matrix, H indicate that the best homography matrix, Kinv indicate
The inverse matrix of the internal reference matrix K, R indicate the spin matrix of the present frame Yu the former frame.
The step 32 further comprises:
Step1. mobile terminal is obtained from the present frame to the attitudes vibration information the former frame, the posture
Change information includes: under mobile terminal local Coordinate System, and present frame posture rotates angle [alpha] relative to three axis of reference attitude ',
β ', γ ', former frame posture relative to reference attitude three axis rotate angle [alpha] ", β ", γ ", and from present frame to former frame it
Between time interval Δ t；
Step2. the gyroscope is calculated in the present frame appearance according to α ', β ', γ ', α ", β ", γ " and Δ t
State is relative to the transformation matrix between the former frame posture.
In the step 40 or 50 or 60, according to the transformation matrix by the present frame and the former frame or described
Preliminary panoramic picture or the intermediate panoramic image are spliced in real time, further comprise:
41. carrying out forward projection to the present frame according to the transformation matrix, and calculate the feature of the present frame
Mapping coordinates of the point on the cylindrical surface or spherical surface that focal length is radius；
42. carrying out back projection to the mapping coordinates according to the transformation matrix, and calculates the mapping coordinates and exist
2D on the cylindrical surface or the tangent perspective plane of the spherical surface is projected in the interpolated coordinates in the present frame；
43. the method by line sampling obtains interpolated coordinates corresponding pixel value in the present frame；
44. according to the interpolated coordinates and its corresponding pixel value, by the present frame and the former frame or it is described just
Step panoramic picture or the intermediate panoramic image are spliced.
Wherein, the back projection (map of forward projection (the map forward) and the step 42 of the step 41
It backward is all) a kind of input picture to the location of pixels mapping (warp) between output image, for two dimensional image,
The one-to-one position of each pixel of original image in the target image is exactly calculated, by taking cylindrical surface as an example:
Forward projection is thrown planar view as in upright projection to cylindrical surface, obtaining each point (x, y, z=1) on view plane
Shadow to cylindrical surface coordinate (u, v, w), wherein (xw,yw,zw) it is that the coordinate fastened in camera coordinates is put on view plane, it calculates
Formula is as follows:
V=scale × (π-arccos (w))；
Rear orientation projection is the inverse process of forward projection in fact, passes through coordinate (u, v) on cylindrical surface, upright projection to view plane
On, the coordinate (x, y) corresponding to original image is found out, calculation formula is as follows:
xw=sin (u/scale)；
zw=cos (u/scale)；
yw=v/scale；
In above-mentioned formula, R is spin matrix, and K is the internal reference matrix of camera, and scale is that cylindrical radius is (generally burnt
Away from).
Also, in the present embodiment, also pass through OpenGL or OpenCL or OpenGL and OpenCL collaborative work mode
It is hardware-accelerated to the forward projection and back projection progress, the execution efficiency of algorithm can be greatly improved, thus real
Splice when real and renders.
It further include step 70, further to the present frame and the former frame or described preliminary in the splicing
Panoramic picture or the intermediate panoramic image carry out emergence mixed processing (blend), and the method for the emergence mixed processing includes:
71. calculating the offset between the present frame and the former frame；
72. calculating the mixed weight-value between the present frame and the former frame according to the offset；
73. final mixing is calculated according to the mixed weight-value and the interpolated coordinates and its corresponding pixel value
End value.
Wherein, the calculation method of the mixed weight-value in the step 72 is as follows:
weightcnt=X/L；
weightpre=1-weightcnt；
In above-mentioned formula, weightcntIndicate the mixed weight-value of the present frame, weightpreIndicate the former frame
Mixed weight-value, L indicate the offset, and X indicates the splicing line position between the present frame and the former frame.
The end value of the step 73 finally mixed, calculation method are as follows:
Valueout=weightcnt*Valuecnt+weightpre*Valuepre；
In formula, ValueoutIndicate the end value finally mixed being calculated, weightcntIndicate the present frame
Mixed weight-value, weightpreIndicate the mixed weight-value of the former frame, ValuecntIndicate that the interpolated coordinates of the present frame are corresponding
Pixel value, ValuepreIndicate the corresponding pixel value of the interpolated coordinates of the former frame.
As shown in Fig. 2, the present invention provides a kind of real-time splicing system of the panoramic picture based on mobile terminal comprising:
Characteristic point detection module A is carried out for obtaining the video flowing of preview in real time, and to the present frame of the video flowing of acquisition
Characteristic point detection, obtains the characteristic point sequence of attributes of the present frame；
Feature Points Matching module B, for by the former frame of the characteristic point sequence of attributes of the present frame and the video flowing
Characteristic point sequence of attributes carry out Feature Points Matching；
Transformation matrix computing module C, for according to the result of the Feature Points Matching calculate the present frame and it is described before
Transformation matrix between one frame；
Panoramic mosaic module D, for being spelled the present frame and the former frame in real time according to the transformation matrix
It connects, obtains preliminary panoramic picture；Alternatively, the present frame and the preliminary panoramic picture are carried out in fact according to the transformation matrix
When splice, obtain intermediate panoramic image；Or according to the transformation matrix by the present frame and the intermediate panoramic image into
Row splicing in real time, obtains final panoramic picture；
Panorama preview module E, for the preliminary panoramic picture or the intermediate panoramic image or described final complete
Scape image carries out preview and shows.
The transformation matrix computing module C further comprises the first transformation matrix computing module C1 and the second transformation matrix meter
Calculate module C2:
If the result of the Feature Points Matching is successful match, the first transformation matrix computing module C1 is called, is led to
It crosses RANSAC algorithm and calculates the optimal homography matrix of the matching pair, and the internal reference matrix of calibration is combined calculate
To the transformation matrix；
If the result of the Feature Points Matching is that it fails to match, the second transformation matrix computing module C2, root are called
The transformation matrix between the present frame and the former frame is calculated according to gyro data.
Further include emergence mixing module F in the present embodiment, in splicing to the present frame and described previous
Frame or the preliminary panoramic picture or the intermediate panoramic image carry out emergence mixed processing, so that in splicing effect not
It will appear obvious crenellated phenomena.
The mobile terminal includes: the equipment that mobile phone, digital camera or tablet computer etc. are configured with camera.
It should be noted that all the embodiments in this specification are described in a progressive manner, each embodiment weight
Point explanation is the difference from other embodiments, and the same or similar parts between the embodiments can be referred to each other.
For system embodiments, since it is basically similar to the method embodiment, so being described relatively simple, related place referring to
The part of embodiment of the method illustrates.Also, herein, the terms "include", "comprise" or its any other variant meaning
Covering non-exclusive inclusion, so that the process, method, article or equipment for including a series of elements not only includes that
A little elements, but also including other elements that are not explicitly listed, or further include for this process, method, article or
The intrinsic element of equipment.In the absence of more restrictions, the element limited by sentence "including a ...", is not arranged
Except there is also other identical elements in the process, method, article or apparatus that includes the element.In addition, this field is general
Logical technical staff is understood that realize that all or part of the steps of above-described embodiment may be implemented by hardware, can also pass through
Program instructs the relevant hardware to complete, and the program can store in a kind of computer readable storage medium, above-mentioned to mention
To storage medium can be read-only memory, disk or CD etc..
The preferred embodiment of the present invention has shown and described in above description, it should be understood that the present invention is not limited to this paper institute
The form of disclosure, should not be regarded as an exclusion of other examples, and can be used for other combinations, modifications, and environments, and energy
Enough in this paper invented the scope of the idea, modifications can be made through the above teachings or related fields of technology or knowledge.And people from this field
The modifications and changes that member is carried out do not depart from the spirit and scope of the present invention, then all should be in the protection of appended claims of the present invention
In range.
Claims (10)
1. a kind of real-time joining method of panoramic picture based on mobile terminal, which comprises the following steps:
10. obtaining the video flowing of preview in real time, and characteristic point detection is carried out to the present frame of the video flowing of acquisition, it is current to obtain this
The characteristic point sequence of attributes of frame；
20. the characteristic point sequence of attributes of the characteristic point sequence of attributes of the present frame and the former frame of the video flowing is carried out special
Sign point matching；
30. calculating the transformation matrix between the present frame and the former frame according to the result of the Feature Points Matching；
40. being spliced the present frame and the former frame in real time according to the transformation matrix, preliminary panoramic picture is obtained,
And preview is carried out to the preliminary panoramic picture and is shown；
50. repeating step 10,20,30, and the present frame and the preliminary panoramic picture are carried out according to the transformation matrix
Splicing in real time obtains intermediate panoramic image, and carries out preview to the intermediate panoramic image and show；
60. repeating step 50, and the present frame and the intermediate panoramic image are spelled in real time according to the transformation matrix
It connects, obtains final panoramic picture, and preview is carried out to the final panoramic picture and is shown.
2. the real-time joining method of a kind of panoramic picture based on mobile terminal according to claim 1, it is characterised in that: institute
In the step 10 stated, the characteristic point detection further comprises:
11. pair present frame carries out gray processing processing, grayscale image is generated；
12. it is down-sampled to gray level image progress, the grayscale image under one group of different scale is obtained, a grayscale image pyramid is formed；
13. the grayscale image of pair different scale carries out the detection of FAST characteristic point, the characteristic point sequence of attributes is obtained.
3. the real-time joining method of a kind of panoramic picture based on mobile terminal according to claim 1 or 2, feature exist
In: the characteristic point sequence of attributes includes: position attribution, direction attribute, scale properties and corresponding description.
4. the real-time joining method of a kind of panoramic picture based on mobile terminal according to claim 1 or 2, feature exist
In: in the step 20, the Feature Points Matching further comprises:
21. KNN algorithm is utilized, by the characteristic point category of the characteristic point sequence of attributes of the present frame and the former frame of the video flowing
Property sequence is matched, and the quantity matched pair between the present frame and the former frame is obtained；
22. described in pair matching pair quantity carry out threshold calculations, if the quantity be greater than or equal to preset threshold, matching at
Function；Otherwise it fails to match.
5. the real-time joining method of a kind of panoramic picture based on mobile terminal according to claim 4, it is characterised in that: institute
In the step 30 stated, the transformation square between the present frame and the former frame is calculated according to the result of the Feature Points Matching
Battle array, calculation method are as follows:
31. if calculating the matching pair most by RANSAC algorithm the result of the Feature Points Matching is successful match
Excellent homography matrix, and in conjunction with the internal reference matrix of calibration carry out that the transformation matrix is calculated；
32. if calculating the present frame and institute according to gyro data the result of the Feature Points Matching is that it fails to match
State the transformation matrix between former frame.
6. the real-time joining method of a kind of panoramic picture based on mobile terminal according to claim 1, it is characterised in that: institute
In the step 40 stated or 50 or 60, according to the transformation matrix by the present frame and the former frame or the preliminary panorama sketch
Picture or the intermediate panoramic image are spliced in real time, further comprise:
41. carrying out forward projection to the present frame according to the transformation matrix, and the characteristic point for calculating the present frame exists
The cylindrical surface or the mapping coordinates on spherical surface that focal length is radius；
42. carrying out back projection to the mapping coordinates according to the transformation matrix, and the mapping coordinates are calculated described
2D on cylindrical surface or the tangent perspective plane of the spherical surface is projected in the interpolated coordinates in the present frame；
43. the method by line sampling obtains interpolated coordinates corresponding pixel value in the present frame；
44. according to the interpolated coordinates and its corresponding pixel value, by the present frame and the former frame or described preliminary complete
Scape image or the intermediate panoramic image are spliced.
7. the real-time joining method of a kind of panoramic picture based on mobile terminal according to claim 6, it is characterised in that: institute
It states in splicing, also further to the present frame and the former frame or the preliminary panoramic picture or the intermediate panoramic
Image carries out emergence mixed processing, and the method for the emergence mixed processing includes:
71. calculating the offset between the present frame and the former frame；
72. calculating the mixed weight-value between the present frame and the former frame according to the offset；
73. the knot finally mixed is calculated according to the mixed weight-value and the interpolated coordinates and its corresponding pixel value
Fruit value.
8. a kind of real-time splicing system of panoramic picture based on mobile terminal characterized by comprising
Characteristic point detection module carries out feature for obtaining the video flowing of preview in real time, and to the present frame of the video flowing of acquisition
Point detection, obtains the characteristic point sequence of attributes of the present frame；
Feature Points Matching module, for by the feature of the characteristic point sequence of attributes of the present frame and the former frame of the video flowing
Point sequence of attributes carries out Feature Points Matching；
Transformation matrix computing module, for according to the result of the Feature Points Matching calculate the present frame and the former frame it
Between transformation matrix；
Panoramic mosaic module is obtained for being spliced the present frame and the former frame in real time according to the transformation matrix
To preliminary panoramic picture；Alternatively, the present frame and the preliminary panoramic picture are spelled in real time according to the transformation matrix
It connects, obtains intermediate panoramic image；Or the present frame and the intermediate panoramic image are carried out in fact according to the transformation matrix
When splice, obtain final panoramic picture；
Panorama preview module, for the preliminary panoramic picture or the intermediate panoramic image or the final panoramic picture
Preview is carried out to show.
9. the real-time splicing system of a kind of panoramic picture based on mobile terminal according to claim 8, it is characterised in that: institute
Stating transformation matrix computing module further comprises the first transformation matrix computing module and the second transformation matrix computing module:
If the result of the Feature Points Matching is successful match, the first transformation matrix computing module is called, is passed through
RANSAC algorithm calculates the optimal homography matrix of the matching pair, and the internal reference matrix of calibration is combined to be calculated
The transformation matrix；
If the result of the Feature Points Matching is that it fails to match, the second transformation matrix computing module is called, according to gyro
Instrument data calculate the transformation matrix between the present frame and the former frame.
10. the real-time splicing system of a kind of panoramic picture based on mobile terminal according to claim 8, it is characterised in that:
Further include emergence mixing module, is used in splicing to the present frame and the former frame or the preliminary panoramic picture
Or the intermediate panoramic image carries out emergence mixed processing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610303650.2A CN105957008B (en) | 2016-05-10 | 2016-05-10 | The real-time joining method of panoramic picture and system based on mobile terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610303650.2A CN105957008B (en) | 2016-05-10 | 2016-05-10 | The real-time joining method of panoramic picture and system based on mobile terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105957008A CN105957008A (en) | 2016-09-21 |
| CN105957008B true CN105957008B (en) | 2019-08-16 |
Family
ID=56914332
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610303650.2A Active CN105957008B (en) | 2016-05-10 | 2016-05-10 | The real-time joining method of panoramic picture and system based on mobile terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105957008B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111651268A (en) * | 2020-05-14 | 2020-09-11 | 武汉兰丁智能医学股份有限公司 | Microscopic image rapid processing system |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106548446B (en) * | 2016-09-29 | 2019-08-09 | 北京奇艺世纪科技有限公司 | A kind of method and device of the textures on Spherical Panorama Image |
| CN108022204B (en) * | 2016-10-31 | 2019-09-13 | 央视国际网络无锡有限公司 | A kind of method that cylinder panorama video is converted to spherical panoramic video |
| CN106504196B (en) * | 2016-11-29 | 2018-06-29 | 微鲸科技有限公司 | A kind of panoramic video joining method and equipment based on space spherical surface |
| CN106846249A (en) * | 2017-01-22 | 2017-06-13 | 浙江得图网络有限公司 | A kind of panoramic video joining method |
| CN107071281A (en) * | 2017-04-19 | 2017-08-18 | 珠海市魅族科技有限公司 | Panorama shooting method and device |
| CN107659769B (en) * | 2017-09-07 | 2019-07-26 | 维沃移动通信有限公司 | A kind of image pickup method, first terminal and second terminal |
| CN113375555A (en) * | 2018-07-02 | 2021-09-10 | 广西电网有限责任公司北海供电局 | Power line clamp measuring method and system based on mobile phone image |
| CN110569927A (en) * | 2019-09-19 | 2019-12-13 | 浙江大搜车软件技术有限公司 | Method, terminal and computer equipment for scanning and extracting panoramic image of mobile terminal |
| CN110648281B (en) * | 2019-09-23 | 2021-03-16 | 华南农业大学 | Method, device and system for generating field panorama, server and storage medium |
| CN111131698B (en) * | 2019-12-23 | 2021-08-27 | RealMe重庆移动通信有限公司 | Image processing method and device, computer readable medium and electronic equipment |
| CN111445416B (en) * | 2020-03-30 | 2022-04-26 | 南京泓众电子科技有限公司 | Method and device for generating high-dynamic-range panoramic image |
| CN113747044B (en) * | 2020-05-29 | 2023-05-02 | 华为技术有限公司 | Panoramic shooting method and equipment |
| WO2022032538A1 (en) * | 2020-08-12 | 2022-02-17 | 深圳市大疆创新科技有限公司 | Panorama photographing method, apparatus and system, and computer-readable storage medium |
| CN113301274B (en) * | 2021-07-28 | 2021-11-09 | 北京海兰信数据科技股份有限公司 | Ship real-time video panoramic stitching method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104519340A (en) * | 2014-12-30 | 2015-04-15 | 余俊池 | Panoramic video stitching method based on multi-depth image transformation matrix |
| CN105262949A (en) * | 2015-10-15 | 2016-01-20 | 浙江卓锐科技股份有限公司 | Multifunctional panorama video real-time splicing method |
| CN105376530A (en) * | 2015-10-27 | 2016-03-02 | 南京泓众电子科技有限公司 | Panorama video image generation device based on fixed position |
| CN105389787A (en) * | 2015-09-30 | 2016-03-09 | 华为技术有限公司 | Panorama image stitching method and device |
-
2016
- 2016-05-10 CN CN201610303650.2A patent/CN105957008B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104519340A (en) * | 2014-12-30 | 2015-04-15 | 余俊池 | Panoramic video stitching method based on multi-depth image transformation matrix |
| CN105389787A (en) * | 2015-09-30 | 2016-03-09 | 华为技术有限公司 | Panorama image stitching method and device |
| CN105262949A (en) * | 2015-10-15 | 2016-01-20 | 浙江卓锐科技股份有限公司 | Multifunctional panorama video real-time splicing method |
| CN105376530A (en) * | 2015-10-27 | 2016-03-02 | 南京泓众电子科技有限公司 | Panorama video image generation device based on fixed position |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111651268A (en) * | 2020-05-14 | 2020-09-11 | 武汉兰丁智能医学股份有限公司 | Microscopic image rapid processing system |
| CN111651268B (en) * | 2020-05-14 | 2023-01-31 | 武汉兰丁智能医学股份有限公司 | Quick processing system for microscopic image |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105957008A (en) | 2016-09-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105957008B (en) | The real-time joining method of panoramic picture and system based on mobile terminal | |
| CN110211043B (en) | Registration method based on grid optimization for panoramic image stitching | |
| CN107301620B (en) | Method for panoramic imaging based on camera array | |
| US11631155B2 (en) | Equatorial stitching of hemispherical images in a spherical image capture system | |
| US6778207B1 (en) | Fast digital pan tilt zoom video | |
| CN104699842B (en) | Picture display method and device | |
| US9049396B2 (en) | Creating composite images based on image capture device poses corresponding to captured images | |
| CN105210368B (en) | Background difference extraction element and background difference extracting method | |
| CN107833179A (en) | The quick joining method and system of a kind of infrared image | |
| CN108564617A (en) | Three-dimensional rebuilding method, device, VR cameras and the panorama camera of more mesh cameras | |
| US20060078214A1 (en) | Image processing based on direction of gravity | |
| US20060078215A1 (en) | Image processing based on direction of gravity | |
| CN104322052A (en) | A system for mixing or compositing in real-time, computer generated 3D objects and a video feed from a film camera | |
| Huang et al. | Panoramic imaging: sensor-line cameras and laser range-finders | |
| CN107516294A (en) | The method and apparatus of stitching image | |
| CN107154014A (en) | A kind of real-time color and depth Panorama Mosaic method | |
| Jung et al. | Deep360Up: A deep learning-based approach for automatic VR image upright adjustment | |
| CN107133918A (en) | A kind of method that optional position in three-dimensional scenic generates panorama sketch | |
| CN109003235A (en) | Bearing calibration, computer readable storage medium and the electric terminal of light field image | |
| CN107851329A (en) | Object is shown based on multiple models | |
| US20220198737A1 (en) | Method and device for displaying details of a texture of a three-dimensional object | |
| CN108958469A (en) | A method of hyperlink is increased in virtual world based on augmented reality | |
| CN109035134A (en) | Panorama Mosaic method, apparatus, electronic equipment and storage medium | |
| CN107563959A (en) | Panoramagram generation method and device | |
| CN110517211A (en) | A kind of image interfusion method based on gradient domain mapping |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |