CN106648109A - Real scene real-time virtual wandering system based on three-perspective transformation - Google Patents
Real scene real-time virtual wandering system based on three-perspective transformation Download PDFInfo
- Publication number
- CN106648109A CN106648109A CN201611271591.1A CN201611271591A CN106648109A CN 106648109 A CN106648109 A CN 106648109A CN 201611271591 A CN201611271591 A CN 201611271591A CN 106648109 A CN106648109 A CN 106648109A
- Authority
- CN
- China
- Prior art keywords
- pictures
- picture
- real
- module
- visual angle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000009466 transformation Effects 0.000 title claims abstract description 11
- 230000000007 visual effect Effects 0.000 claims abstract description 19
- 238000000034 method Methods 0.000 claims abstract description 17
- 238000013507 mapping Methods 0.000 claims description 14
- 239000011159 matrix material Substances 0.000 claims description 8
- 238000012937 correction Methods 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 230000004927 fusion Effects 0.000 claims description 5
- 238000000844 transformation Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 3
- 230000035772 mutation Effects 0.000 claims description 2
- 239000011521 glass Substances 0.000 claims 1
- 238000011161 development Methods 0.000 abstract description 2
- 238000011144 upstream manufacturing Methods 0.000 abstract description 2
- 238000007796 conventional method Methods 0.000 abstract 1
- 230000003287 optical effect Effects 0.000 abstract 1
- 238000009877 rendering Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- UAJUXJSXCLUTNU-UHFFFAOYSA-N pranlukast Chemical compound C=1C=C(OCCCCC=2C=CC=CC=2)C=CC=1C(=O)NC(C=1)=CC=C(C(C=2)=O)C=1OC=2C=1N=NNN=1 UAJUXJSXCLUTNU-UHFFFAOYSA-N 0.000 description 1
- 229960004583 pranlukast Drugs 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention relates to a real scene real-time virtual wandering system based on three-perspective transformation and belongs to the field of computer vision. The system provided by the invention solves the problem that by photographing three pictures of a same scene in different positions and different angles by using a common camera 21, a user wears a virtual reality helmet 23 to realize virtual wandering of a real scene without requiring 3D modeling of a conventional method or detailed geometrical information. The core part of the main algorithm of the system provided by the invention lies in that the three pictures are transformed to a same plane at the same time and a middle perspective is calculated in real time. Based on a conventional backward transformation algorithm of two pictures, the invention further provides a backward transformation algorithm of three pictures, so that the method can be suitable for the condition of a large optical parallax. The system provided by the invention is connected to upstream data acquisition and downstream visual control technology in the computer vision field, so that development of applications such as virtual reality and streetscape services is greatly accelerated.
Description
Technical field
The present invention relates to a kind of real scene real-time virtual roaming system for being based on three view transformations, is related to computer vision
In field of virtual reality.
Background technology
In the field of virtual reality of computer vision, render (the Image-based Rendering) based on picture is one
Plant effective method.This method just can produce new Intermediate View by shooting a small amount of photo and being fused together them
Angle picture.
The existing rendering intent based on picture can be divided into three major types：1st, rendering completely without geological information.2、
Need rendering for a small amount of geological information.3rd, rendering for detailed geological information is needed.Although first method does not need any geometry
Information, however it is necessary that with the picture of big quantity as cost.The detailed geological information that the third method needs is only using special
Equipment just can be obtained, and limit its range of application.The system belongs to second method, has not both needed detailed geological information,
Also without with substantial amounts of picture as cost, and can in real time be calculated according to the change of user's head position and angle and newly regarded
The picture at angle, to realize the effect of virtual roaming.
The content of the invention
The invention discloses a kind of real scene real-time virtual roaming system for being based on three view transformations, according to same field
Three photos that scape shoots in different angles and positions, calculate in real time the picture of middle visual angle.The technology is connected to meter
The data acquisition of calculation machine visual field upstream and the visual spatial attention technology in downstream, are greatly promoted virtual reality, street view service
Deng the development of application, and to provide at least advantages described below.
Than conventional system, common a camera and computer need to be only used, system composition is fairly simple.
Excessive manual operation is not needed, automaticity is higher, New Century Planned Textbook reduction precision is higher.
New multi-perspective picture can in real time be calculated according to the visual angle of user.
To achieve these goals, the real-time scene Virtual Wandering System for being based on three view transformations disclosed by the invention, its
It is characterised by comprising the following steps：
(1) three pictures of same scene diverse location, different angles are shot using camera.
(2) forward transform, by original image the picture after approximately the same plane is corrected is projected to.
(3) mapping relations of picture after correction are calculated, and in the position interpolation of mapping relations mutation.
(4) picture after three corrections of linear fusion, obtains the picture at middle visual angle.
(5) backward conversion, previous step obtained by middle visual angle picture projection to original image visual angle.
Description of the drawings
Illustrate technical solution of the present invention is further understood for providing, and constitute a part for specification, with
The enforcement of the present invention for explaining technical scheme, does not constitute the restriction to technical solution of the present invention together.Accompanying drawing
It is described as follows：
Fig. 1 is the flow chart of the system complete procedure, and Fig. 2 is the hardware composition figure of the system.
Specific embodiment
Embodiments of the present invention are described in detail below with reference to accompanying drawing, whereby to the present invention how application technology means
Carry out solve problem, and reach technique effect realize that process can fully understand and implement according to this.
Module one, data acquisition.With camera 21 from different perspectives, diverse location shoot three photos of same scene.
Module two, data processing.The module mainly includes step：13rd, forward transform；14th, mapping relations interpolation；15th, line
Property fusion；16th, backward conversion.The calculating of the module is completed in a common computer 22.
13rd, forward transform：Three original images are divided into two groups, per group of two pictures.Using the matching of SIFT feature point
Relation calculates two groups of projection matrixes, and respectively by every group of picture projection to approximately the same plane.Further according to the mapping between picture
They, while correcting three pictures, are projected to approximately the same plane by relation.
Assume that three original pictures are I1, I2, I3, it is grouped into I1, I2And I1, I3, parallel picture group I ' can be obtained1,
I′2With I "1, I "3.The mapping relations of two groups of pictures are obtained using most accurate perspective view matching process TSGO at presentWithAssume that E is I '1With I '2Intermediate image, to E and I3Try again correction, you can obtains him
Between correction matrix H6And H5.Two groups of correction matrixs with reference to before, it is possible to obtain correcting the projection of three pictures simultaneously
Matrix.
14th, mapping relations interpolation：The mapping relations of two groups of pictures have been obtained in previous stepWithWith reference to the transformation relation of projection matrixCan be obtained by parallel
Three pictures between mapping relations, be designated asWith
Due toOrSaltus step can cause next step merge picture when produce larger black hole, fill up these black holes
Need to spend additional time interpolation, and interpolation is inversely proportional to the size in black hole, so this step shifts to an earlier date in mapping relations
Upper interpolation, can both shorten the time-consuming of online part, and can improve the quality for generating picture.
15th, linear fusion：According to existing theory, the linear fusion of parallel views meets perspective geometry theory.In being based on
Between visual angle position of centre of gravity λ=(λ1, λ2, λ3) define a mapping functionFor merging three parallel picturesRoot
According to above-mentioned steps,
16th, backward conversion：In the step of above, parallel picture has been obtainedMiddle visual angleFinally
The visual angle for projecting this at original image is needed, the location of pixels of interpolation original image and intermediate picture I of color is obtaineds.I
Extend after existing two pictures to scaling method is become, and propose after three pictures under new big parallax conditions to conversion
Algorithm.
According to, to scaling method is become, in the case of extrapolating three pictures, projection matrix is H after existing two picturest=H6
(H6 -1Hs)t, H heres=H5H1[(H5H1)-1(H5H2)]s.It is verified, the H in the case where original image visual angle difference is largertNo
Convergence, therefore in this case this method can not obtain effective result.
In the case where original image visual angle differs greatly, it is proposed that a kind of succinct effectively backward change scaling method：Ht
=(1-s) H6+tHs, H heres=(1-s) H5H1+sH5H2.The method obtains each projection matrix linear combination well
Effect.
Ht -1Act onOn can obtain normal viewing angle picture Is.Because first three step can be completed offline,
Only this step is needed in line computation, so with the help of GPU parallel computations, it is possible to reach speed of the 0.05s per frame
Degree, i.e., 20 frames are per second, reached real-time requirement.
Module three, virtual roaming.User wears virtual implementing helmet 23 (such as Oculus Rift), virtual implementing helmet
Position and the directional information of user's head are caught, above-mentioned λ=(λ is translated into by data processing module1, λ2, λ3) and it is real
When calculate the picture at middle visual angle.Therefore, user can experience the virtual roaming of real-world scene.
Those skilled in the art should be understood that above-mentioned system architecture of the invention and each step can be with general
Realizing, they can be concentrated on single computing device, or are distributed in multiple computing device groups for camera and computing device
Into network on, alternatively, they can be realized with the executable program code of computing device, it is thus possible to they are deposited
Storage is performed in the storage device by computing device, or they is fabricated to respectively each integrated circuit modules, or by it
In multiple modules or step be fabricated to single integrated circuit module to realize.So, the present invention is not restricted to any specific
Hardware and software combine.
Although the embodiment shown or described by the present invention is as above, described content is only to facilitate understand this
The embodiment invented and adopt, is not limited to the present invention.Technical staff in any the technical field of the invention,
Without departing from disclosed herein spirit and scope on the premise of, can implement formal and details on do any repairing
Change and change, but the scope of patent protection of the present invention, still must be defined by the scope of which is defined in the appended claims.
Claims (3)
1. a kind of real scene real-time virtual roaming system for being based on three view transformations, is characterized in that comprising following main modular：
Module one, data acquisition：Directly with camera from different perspectives, diverse location shoot three photos of same scene.
Module two, data processing：The module mainly includes following four step：
Step one：Forward transform, i.e., three photographic projections for module one being gathered are to approximately the same plane.
Step 2：Mapping relations interpolation, that is, calculate the mapping relations of three pictures of approximately the same plane, and in mapping relations mutation
Position interpolation, obtain continuous mapping relations.
Step 3：Linear fusion, the i.e. position by the pictures of interpolation same plane three and color, obtain their middle visual angle
Picture.
Step 4：Backward conversion, i.e., the picture projection obtained by step 3 to normal visual angle, the centre for obtaining original image is appointed
Meaning visual angle.
Module three, virtual roaming：User takes virtual reality glasses, it is possible to experience the virtual roaming of real scene.
2. step one is characterised by the module two of method described in claim 1：We simultaneously project to three pictures
Approximately the same plane.First we are divided into two groups three pictures, per group of two pictures, and every group of picture are projected to respectively same
Individual plane.Specifically, the formula used by the projection is：
Wherein, I1, I2, I3Three original images that camera shoots are represented,Expression projects to the picture of approximately the same plane.
H1, H2It is respectively I1, I2Projection matrix, H5, H6It is respectively I1, I2Middle multi-perspective picture E and I3Projection matrix.By two
Two correction picture groups, and two groups of pictures are linked together by middle multi-perspective picture, we obtain three parallel pictures.
3. step 4 is characterised by the module two of method described in claim 1：The middle multi-perspective picture that step 3 is obtained
It is based on parallel picture groupTherefore it is not visual angle when taking pictures, needs to project to it by backward transformation matrix
Normal visual angle, i.e. I1, I2, I3Visual angle.Specifically, formula used is：
Little parallax situation：
Big parallax situation：。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611271591.1A CN106648109A (en) | 2016-12-30 | 2016-12-30 | Real scene real-time virtual wandering system based on three-perspective transformation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611271591.1A CN106648109A (en) | 2016-12-30 | 2016-12-30 | Real scene real-time virtual wandering system based on three-perspective transformation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106648109A true CN106648109A (en) | 2017-05-10 |
Family
ID=58838201
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611271591.1A Pending CN106648109A (en) | 2016-12-30 | 2016-12-30 | Real scene real-time virtual wandering system based on three-perspective transformation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106648109A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115619986A (en) * | 2022-09-30 | 2023-01-17 | 北京百度网讯科技有限公司 | Scene roaming method, device, equipment and medium |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1694512A (en) * | 2005-06-24 | 2005-11-09 | 清华大学 | Synthesis method of virtual viewpoint in interactive multi-viewpoint video system |
| CN101330632A (en) * | 2008-07-25 | 2008-12-24 | 浙江大学 | Method for rendering virtual viewpoint image of three-dimensional television system |
| CN101799939A (en) * | 2010-04-02 | 2010-08-11 | 天津大学 | Rapid and self-adaptive generation algorithm of intermediate viewpoint based on left and right viewpoint images |
| CN101873508A (en) * | 2010-05-25 | 2010-10-27 | 东南大学 | Intermediate view synthesis method based on improved BP (Belief Propagation) algorithm |
| CN102308590A (en) * | 2009-02-05 | 2012-01-04 | 富士胶片株式会社 | Three-dimensional image output device and three-dimensional image output method |
| CN103248810A (en) * | 2012-02-10 | 2013-08-14 | 索尼公司 | Image processing device, image processing method, and program |
| CN103353941A (en) * | 2013-06-13 | 2013-10-16 | 西安电子科技大学 | Natural marker registration method based on viewpoint classification |
| CN103945209A (en) * | 2014-04-28 | 2014-07-23 | 华南理工大学 | DIBR method based on block projection |
| CN104048601A (en) * | 2014-06-19 | 2014-09-17 | 湖北汽车工业学院 | Integrally-imaging image mapping method based on coordinate transformation |
| US20150036014A1 (en) * | 2010-12-14 | 2015-02-05 | Pelican Imaging Corporation | Systems and Methods for Synthesizing High Resolution Images Using Images Captured by an Array of Independently Controllable Imagers |
| CN104809719A (en) * | 2015-04-01 | 2015-07-29 | 华南理工大学 | Virtual view synthesis method based on homographic matrix partition |
| CN105184861A (en) * | 2015-10-20 | 2015-12-23 | 南京大学 | Automatic human three-dimensional (3D) reconstruction system |
-
2016
- 2016-12-30 CN CN201611271591.1A patent/CN106648109A/en active Pending
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1694512A (en) * | 2005-06-24 | 2005-11-09 | 清华大学 | Synthesis method of virtual viewpoint in interactive multi-viewpoint video system |
| CN101330632A (en) * | 2008-07-25 | 2008-12-24 | 浙江大学 | Method for rendering virtual viewpoint image of three-dimensional television system |
| CN102308590A (en) * | 2009-02-05 | 2012-01-04 | 富士胶片株式会社 | Three-dimensional image output device and three-dimensional image output method |
| CN101799939A (en) * | 2010-04-02 | 2010-08-11 | 天津大学 | Rapid and self-adaptive generation algorithm of intermediate viewpoint based on left and right viewpoint images |
| CN101873508A (en) * | 2010-05-25 | 2010-10-27 | 东南大学 | Intermediate view synthesis method based on improved BP (Belief Propagation) algorithm |
| US20150036014A1 (en) * | 2010-12-14 | 2015-02-05 | Pelican Imaging Corporation | Systems and Methods for Synthesizing High Resolution Images Using Images Captured by an Array of Independently Controllable Imagers |
| CN103248810A (en) * | 2012-02-10 | 2013-08-14 | 索尼公司 | Image processing device, image processing method, and program |
| CN103353941A (en) * | 2013-06-13 | 2013-10-16 | 西安电子科技大学 | Natural marker registration method based on viewpoint classification |
| CN103945209A (en) * | 2014-04-28 | 2014-07-23 | 华南理工大学 | DIBR method based on block projection |
| CN104048601A (en) * | 2014-06-19 | 2014-09-17 | 湖北汽车工业学院 | Integrally-imaging image mapping method based on coordinate transformation |
| CN104809719A (en) * | 2015-04-01 | 2015-07-29 | 华南理工大学 | Virtual view synthesis method based on homographic matrix partition |
| CN105184861A (en) * | 2015-10-20 | 2015-12-23 | 南京大学 | Automatic human three-dimensional (3D) reconstruction system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115619986A (en) * | 2022-09-30 | 2023-01-17 | 北京百度网讯科技有限公司 | Scene roaming method, device, equipment and medium |
| CN115619986B (en) * | 2022-09-30 | 2024-02-06 | 北京百度网讯科技有限公司 | Scene roaming method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9438878B2 (en) | Method of converting 2D video to 3D video using 3D object models | |
| CN111047510B (en) | Large-field-angle image real-time splicing method based on calibration | |
| TWI534755B (en) | A method and apparatus for building a three dimension model | |
| US9241147B2 (en) | External depth map transformation method for conversion of two-dimensional images to stereoscopic images | |
| CN100355272C (en) | Synthesis method of virtual viewpoint in interactive multi-viewpoint video system | |
| US20230291884A1 (en) | Methods for controlling scene, camera and viewing parameters for altering perception of 3d imagery | |
| CN107038724A (en) | Panoramic fisheye camera image correction, synthesis and depth of field reconstruction method and system | |
| US9443338B2 (en) | Techniques for producing baseline stereo parameters for stereoscopic computer animation | |
| CN104599317B (en) | A kind of mobile terminal and method for realizing 3D scanning modeling functions | |
| Du et al. | Changing perspective in stereoscopic images | |
| WO2019219014A1 (en) | Three-dimensional geometry and eigencomponent reconstruction method and device based on light and shadow optimization | |
| CN111047709B (en) | Binocular vision naked eye 3D image generation method | |
| CN104809719A (en) | Virtual view synthesis method based on homographic matrix partition | |
| CN108053373A (en) | One kind is based on deep learning model fisheye image correcting method | |
| CN109118578A (en) | A kind of multiview three-dimensional reconstruction texture mapping method of stratification | |
| CN107451952A (en) | A kind of splicing and amalgamation method of panoramic video, equipment and system | |
| CN105354808A (en) | Fisheye image correction method | |
| WO2023207452A1 (en) | Virtual reality-based video generation method and apparatus, device, and medium | |
| CN105979241B (en) | A kind of quick inverse transform method of cylinder three-dimensional panoramic video | |
| CN106648109A (en) | Real scene real-time virtual wandering system based on three-perspective transformation | |
| CN111915735B (en) | Depth optimization method for three-dimensional structure outline in video | |
| Zhou et al. | MR video fusion: interactive 3D modeling and stitching on wide-baseline videos | |
| CN108053376A (en) | A kind of semantic segmentation information guiding deep learning fisheye image correcting method | |
| CN108198132A (en) | The method of integration imaging image reconstruction based on Block- matching | |
| CN111915739A (en) | Real-time three-dimensional panoramic information interactive information system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20170510 |