CN110009570A - A kind of virtual reality panorama sketch intelligent connecting method based on automaton study - Google Patents
A kind of virtual reality panorama sketch intelligent connecting method based on automaton study Download PDFInfo
- Publication number
- CN110009570A CN110009570A CN201910316905.2A CN201910316905A CN110009570A CN 110009570 A CN110009570 A CN 110009570A CN 201910316905 A CN201910316905 A CN 201910316905A CN 110009570 A CN110009570 A CN 110009570A
- Authority
- CN
- China
- Prior art keywords
- image
- camera
- images
- shooting
- coordinate system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 30
- 238000002156 mixing Methods 0.000 claims abstract description 4
- 230000009466 transformation Effects 0.000 claims description 11
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000011161 development Methods 0.000 claims description 5
- 238000001514 detection method Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 claims description 3
- 238000013528 artificial neural network Methods 0.000 claims description 2
- 238000006243 chemical reaction Methods 0.000 claims description 2
- 230000001419 dependent effect Effects 0.000 claims description 2
- 238000007781 pre-processing Methods 0.000 claims description 2
- 238000002360 preparation method Methods 0.000 claims description 2
- 230000000007 visual effect Effects 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 8
- 230000000694 effects Effects 0.000 abstract description 6
- 238000012545 processing Methods 0.000 abstract description 3
- 238000007796 conventional method Methods 0.000 abstract description 2
- 238000010801 machine learning Methods 0.000 description 5
- 239000000463 material Substances 0.000 description 4
- 241000208340 Araliaceae Species 0.000 description 2
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 2
- 235000003140 Panax quinquefolius Nutrition 0.000 description 2
- 235000008434 ginseng Nutrition 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000007654 immersion Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/35—Determination of transform parameters for the alignment of images, i.e. image registration using statistical methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Processing Or Creating Images (AREA)
- Image Processing (AREA)
- Studio Devices (AREA)
Abstract
The present invention provides a kind of virtual reality panorama sketch intelligent connecting method based on automaton study, belong to automaton study technology and image mosaic technology field, the image sequence that the present invention can shoot mobile phone or traditional camera splices, in conjunction with traditional images processing technique, learn library using automaton, blending algorithm is automatically generated, realizes splicing effect more better than conventional method.
Description
Technical field
The present invention relates to automaton study technology and image mosaic technologies, more particularly to a kind of automaton that is based on to learn
Virtual reality panorama sketch intelligent connecting method.
Background technique
In recent years, virtual reality technology (VR, Virtual Reality) development is very burning hot, have already appeared in the market
The more mature household virtual implementing helmet of several moneys such as HTC Vive, Oculus Rift, Sony PlayStation etc..
With the gradually perfect and marketization of hardware facility, corresponding software auxiliary facility development is very powerful, and major game company strives
The VR version of hot game is mutually issued, many network medias are also proposed the VR video of high quality production.However due to VR software
Cost of manufacture is higher, and development speed is still much unable to satisfy the market demand.
VR software development cost is mainly derived from expensive panorama shooting device.A performance is stablized, pretty good complete of effect
The price is very expensive for scape camera.High price hinder panorama shooting device it is popular with it is market-oriented, cause virtual reality ecological
Circle develops slowly.
Panorama sketch applied to virtual reality is different from the panorama sketch that people are understood on common meaning, virtual reality device
Used in panorama sketch be distortion, be substantially the rectangle expanded view of spherical surface.Therefore compared to common image mosaic
Technical difficulty is higher.
In the case where image sequence quality is relatively poor, traditional algorithm is often carried out using SIFT or SURF characteristic point
Image registration infers the relative positional relationship between different images according to the registration relationship of image, and derives transformation matrix with this
And affine transformation is carried out to separate picture, fusion and global white balance finally are carried out to image mosaic region again, finally obtain void
The spherical expanded view of quasi- reality panorama figure.Traditional algorithm has had been provided with very high accuracy in terms of image registration, but
The performance of traditional algorithm is barely satisfactory in terms of image co-registration.Machine learning techniques emerging in the recent period are provided for one preferably
Solution, syncretizing effect can be fine to pixel scale, while guaranteeing fine and smooth syncretizing effect, moreover it is possible to guarantee global
Image balance.
Automaton study technology is grown up on machine learning techniques basis.In the weight of machine learning work
Be the selection of machine learning model and the adjustment of parameter again, this generally requires machine learning engineer and carries on the back with extremely strong technology
Scape and project experiences.Automaton study technology can carry out automatically selecting and oneself for model according to the methods of Bayes's tune ginseng
It is dynamic to adjust ginseng, compared to needing a large amount of manpowers and calculate the grid search of power to terminate an agreement more resources.
Summary of the invention
In order to solve the above technical problems, the invention proposes a kind of virtual reality panorama sketch based on automaton study
Intelligent connecting method, the image sequence that can be shot to mobile phone or traditional camera splice, in conjunction with traditional images processing technique,
Learn library using automaton, automatically generate blending algorithm, realizes splicing effect more better than conventional method.
The technical scheme is that
A kind of virtual reality panorama sketch intelligent connecting method based on automaton study, mainly includes the following steps:
S1. the acquisition of image sequence.Image of the present invention obtains low in cost, it is only necessary to which common mobile phone camera is aided with
Mobile phone pan and tilt head and tripod shoot 720 ° of panoramic spaces.
In the preparation stage, need using pan and tilt head to pass through holder hand cradle and rotate phase so that the principal point of camera remains stationary
The shooting angle of machine.
Camera is being adjusted, after the equipment such as pan and tilt head and tripod, into formal photographing phase.General point four of shooting
Wheel carries out, and the main line direction of first round camera is parallel to the ground, then shoots to 360 ° of spaces around.The secondary figure of every shooting one
As just rotating clockwise certain angle, need to keep general 10% to 20% lap between the adjacent image of shooting, directly
360 ° of all scenes around are enumerated to these images.
Second wheel shooting is similar to the first round, and unique variation is that the main line of camera is no longer parallel with horizontal line, but compares
Horizontal line is higher by certain angle.The angle being higher by is specifically dependent upon the absolute visual field angle of camera in vertical direction, generally can be with
It is arranged at 40 ° or so.
The main line of camera is then adjusted downwards 40 ° or so by third round.
Fourth round is " mending day " and " mending ground ", i.e., camera main line is exactly perpendicularly to horizon, and camera lens is downward with camera lens upwards
Respectively one image of shooting.All images in 720 ° of spaces around are contained by four-wheel all images obtained in this way, after
These images are known as image sequence by text.
S2. basic image preprocessing is carried out to images all in image sequence, such as noise reduction, luminance balance, color are put down
Weighing apparatus etc..These operations have adaptive implementation method in OpenCV, simple to call.The image attributes of image is all after processing
It is roughly the same, registration accuracy and later period syncretizing effect between image can be improved in this way.
S3. the SIFT feature of SIFT feature detection algorithm detection each image is utilized.According to these characteristic point informations, make
All images in image sequence are matched adjacent to ratio method with secondary with closest, detect that the matching between image is closed
System.But the pairing of these characteristic points is simultaneously not all correct, it is therefore desirable to be matched using the characteristic point that RANSAC algorithm rejects mistake.?
To after the match information between all images, by the coordinate information of characteristic point, the transformation matrix between image is derived.
S4. this stage is responsible for image sequence carrying out projective transformation, forms the rectangle expanded view of panorama sketch.
A sphere can be established first, and the transformation matrix for then being derived all images in the S3 stage according to it is thrown
On shadow to spherical surface.It is last again that development of a sphere is rectangular, obtain panorama sketch.
In above-mentioned conversion process, shares two class three-dimensional system of coordinates and us is assisted to convert: world coordinate system and phase
Machine coordinate system.World coordinate system only one, origin faces upward in the centre of sphere, z-axis perpendicular to level.Camera coordinates system has very much
A, every piece image in image sequence has all corresponded to a camera coordinates system, and origin is identical as world coordinate system, but phase
The z-axis of machine coordinate system is parallel to angle corresponding to each image shooting phase owner line.Each image will be in respective camera
It is projected under coordinate system.Image sequence will obtain a spherical image after all projecting.By all of the spherical shape image
Pixel coordinate downconverts under world coordinate system from camera coordinates system, and is unfolded under world coordinate system, can obtain complete
Scape image.
Above-mentioned projective transformation can also be fully completed under world coordinate system completely, and the introducing of camera coordinates system only reduces
The programming difficulty of image projection, simplifies logic flow.
S5. learn library using AutoKeras automaton, it is made to train neural network to come blending image junction automatically
Point, and carry out the balancing work of image.
Detailed description of the invention
Fig. 1 is workflow schematic diagram of the invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments, based on the embodiments of the present invention, those of ordinary skill in the art
Every other embodiment obtained without making creative work, shall fall within the protection scope of the present invention.
A kind of virtual reality panorama sketch intelligent connecting method based on automaton study of the invention, using iPhone5s
Carry out following steps:
S1. iPhone5s is used, the equipment such as pan and tilt head and tripod choose some fixed location, to around 720 °
Space is shot.In the horizontal direction, rotary head carries out 360 ° of shootings, and 10 ° or so are probably kept between image and image
Lap.Three layers of shooting of image point in vertical direction finally carry out mending day and mend ground.
S2. image sequence is imported into OpenCV, noise reduction process is carried out to image, and adjust the brightness value of all images
With contrast to roughly the same.
S3. it is realized using the SIFT algorithm of OpenCV, detects the characteristic point of each image.Then between all images
Characteristic point using it is closest with time close on ratio method and matched, the match point of mistake is removed using RANSAC, to calculate
The transformation relation of every image, to derive transformation matrix.
S4. by every image projection to a virtual panorama ball, the virtual ball the form of data presence without
Establish entity.In projection process, to two images for having lap, it imported into AutoKeras and utilizes robotics
The mode of habit is merged and is balanced to image.
S5. the image that splicing is completed is imported into the 3D engine for supporting virtual reality device, establishes a material ball,
The panorama sketch of generation is added on material ball in a manner of material.It is put into a binocular camera at the center of material ball, connection is empty
Quasi- real world devices, realize the virtual reality experience of immersion.
The foregoing is merely presently preferred embodiments of the present invention, is only used to illustrate the technical scheme of the present invention, and is not intended to limit
Determine protection scope of the present invention.Any modification, equivalent substitution, improvement and etc. done all within the spirits and principles of the present invention,
It is included within the scope of protection of the present invention.
Claims (10)
1. a kind of virtual reality panorama sketch intelligent connecting method based on automaton study, which is characterized in that
Mainly include the following steps:
The acquisition of S1, image sequence；
S2, basic image preprocessing is carried out to images all in image sequence；
S3, the SIFT feature of SIFT feature detection algorithm detection each image is utilized；
S4, it is responsible for image sequence carrying out projective transformation, forms the rectangle expanded view of panorama sketch；
S5., learn library using AutoKeras automaton, it is made to train neural network to carry out blending image interface portion automatically, and
Carry out the balancing work of image.
2. the method according to claim 1, wherein
In S1 step,
In the preparation stage, need using pan and tilt head to pass through holder hand cradle rotating camera so that the principal point of camera remains stationary
Shooting angle；Shooting divides four-wheel to carry out.
3. according to the method described in claim 2, it is characterized in that,
The main line direction of first round camera is parallel to the ground, then shoots to 360 ° of spaces around；One sub-picture of every shooting
Just certain angle is rotated clockwise, needs to keep 10% to 20% lap between the adjacent image of shooting, until these
Image enumerates 360 ° of all scenes around.
4. according to the method described in claim 3, it is characterized in that,
Second wheel shooting is similar to the first round, and unique variation is that the main line of camera is no longer parallel with horizontal line, but compares horizontal
Line is higher by an angle；The angle being higher by is specifically dependent upon the absolute visual field angle of camera in vertical direction.
5. according to the method described in claim 4, it is characterized in that,
The main line of camera is then adjusted downwards 40 ° by third round.
6. according to the method described in claim 5, it is characterized in that,
Fourth round, that is, camera main line is exactly perpendicularly to horizon, and camera lens respectively shoots downwards an image with camera lens upwards；It passes through in this way
It crosses four-wheel all images obtained and contains all images in 720 ° of spaces around, these images are known as image sequence.
7. method according to claim 1 or 6, which is characterized in that
In the S3 step, according to these characteristic point informations, using closest and time neighbouring ratio method, to all in image sequence
Image is matched, and detects the matching relationship between image；The characteristic point pairing of mistake is rejected using RANSAC algorithm；It obtains
After match information between all images, by the coordinate information of characteristic point, the transformation matrix between image is derived.
8. the method according to the description of claim 7 is characterized in that
In S4 step,
A sphere can be established first, and the transformation matrix for then being derived all images in the S3 stage according to it projects to
On spherical surface；It is last again that development of a sphere is rectangular, obtain panorama sketch.
9. according to the method described in claim 8, it is characterized in that,
In above-mentioned conversion process, shares two class three-dimensional system of coordinates auxiliary and converted: world coordinate system and camera coordinates system；
World coordinate system only one, origin faces upward in the centre of sphere, z-axis perpendicular to level；
Camera coordinates system has several, and every piece image in image sequence has all corresponded to a camera coordinates system, origin and generation
Boundary's coordinate system is identical, but the z-axis of camera coordinates system is parallel to angle corresponding to each image shooting phase owner line；
Each image will project under respective camera coordinates system.
10. according to the method described in claim 9, it is characterized in that,
Image sequence all will obtain a spherical image after projection, by all pixels coordinate of the spherical shape image from camera
Coordinate system downconverts under world coordinate system, and is unfolded under world coordinate system, and panoramic picture can be obtained.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910316905.2A CN110009570B (en) | 2019-04-19 | 2019-04-19 | Intelligent virtual reality panorama splicing method based on automatic machine learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910316905.2A CN110009570B (en) | 2019-04-19 | 2019-04-19 | Intelligent virtual reality panorama splicing method based on automatic machine learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110009570A true CN110009570A (en) | 2019-07-12 |
| CN110009570B CN110009570B (en) | 2022-12-20 |
Family
ID=67173113
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910316905.2A Active CN110009570B (en) | 2019-04-19 | 2019-04-19 | Intelligent virtual reality panorama splicing method based on automatic machine learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110009570B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111080804A (en) * | 2019-10-23 | 2020-04-28 | 贝壳技术有限公司 | Three-dimensional image generation method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120274739A1 (en) * | 2009-12-21 | 2012-11-01 | Huawei Device Co.,Ud. | Image splicing method and apparatus |
| CN109272445A (en) * | 2018-10-29 | 2019-01-25 | 中国航空无线电电子研究所 | Panoramic video joining method based on Sphere Measurement Model |
| CN109407547A (en) * | 2018-09-28 | 2019-03-01 | 合肥学院 | Multi-camera in-loop simulation test method and system for panoramic visual perception |
-
2019
- 2019-04-19 CN CN201910316905.2A patent/CN110009570B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120274739A1 (en) * | 2009-12-21 | 2012-11-01 | Huawei Device Co.,Ud. | Image splicing method and apparatus |
| CN109407547A (en) * | 2018-09-28 | 2019-03-01 | 合肥学院 | Multi-camera in-loop simulation test method and system for panoramic visual perception |
| CN109272445A (en) * | 2018-10-29 | 2019-01-25 | 中国航空无线电电子研究所 | Panoramic video joining method based on Sphere Measurement Model |
Non-Patent Citations (1)
| Title |
|---|
| 杨镇豪: "基于柱面变换的图像拼接技术", 《现代计算机(专业版)》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111080804A (en) * | 2019-10-23 | 2020-04-28 | 贝壳技术有限公司 | Three-dimensional image generation method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110009570B (en) | 2022-12-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101000461B (en) | Method for generating stereoscopic panorama by fish eye image | |
| CN109658365B (en) | Image processing method, device, system and storage medium | |
| CN107431796B (en) | The omnibearing stereo formula of panoramic virtual reality content captures and rendering | |
| CN106462944B (en) | High-resolution panorama VR generator and method | |
| CN101616237B (en) | Image processing apparatus, image processing method | |
| CN112085659B (en) | Panorama splicing and fusing method and system based on dome camera and storage medium | |
| Barazzetti et al. | 3D modelling with the Samsung Gear 360 | |
| CN106162203A (en) | Panoramic video player method, player and wear-type virtual reality device | |
| CN104952075A (en) | Laser scanning three-dimensional model-oriented multi-image automatic texture mapping method | |
| CN104881526B (en) | Article wearing method based on 3D and glasses try-on method | |
| CN106534670B (en) | It is a kind of based on the panoramic video generation method for connecting firmly fish eye lens video camera group | |
| JP6921686B2 (en) | Generator, generation method, and program | |
| CN108200360A (en) | A kind of real-time video joining method of more fish eye lens panoramic cameras | |
| CN107317998A (en) | Full-view video image fusion method and device | |
| CN106303497A (en) | A kind of virtual reality content generating method and device | |
| CN115641401A (en) | Construction method and related device of three-dimensional live-action model | |
| Zhang et al. | Research and implementation of digital 3D panoramic visual communication technology based on virtual reality | |
| CN115330594A (en) | Target rapid identification and calibration method based on unmanned aerial vehicle oblique photography 3D model | |
| CN106485781A (en) | Three-dimensional scene construction method based on live video stream and its system | |
| CN108269234A (en) | A kind of lens of panoramic camera Attitude estimation method and panorama camera | |
| CN107197135A (en) | A kind of video generation method, player method and video-generating device, playing device | |
| CN205430421U (en) | A controllable angle of pitch panoramic photography system for preparation of film virtualization | |
| CN114926612A (en) | Aerial panoramic image processing and immersive display system | |
| CN110009570A (en) | A kind of virtual reality panorama sketch intelligent connecting method based on automaton study | |
| CN105681638A (en) | Controllable pitch angle panoramic photography system used for film virtual production |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20221124 Address after: 250100 building S02, No. 1036, Langchao Road, high tech Zone, Jinan City, Shandong Province Applicant after: Shandong Inspur Scientific Research Institute Co.,Ltd. Address before: 250100 First Floor of R&D Building 2877 Kehang Road, Sun Village Town, Jinan High-tech Zone, Shandong Province Applicant before: JINAN INSPUR HIGH-TECH TECHNOLOGY DEVELOPMENT Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |