CN107169924B - Method and system for establishing three-dimensional panoramic image - Google Patents
Method and system for establishing three-dimensional panoramic image Download PDFInfo
- Publication number
- CN107169924B CN107169924B CN201710448347.6A CN201710448347A CN107169924B CN 107169924 B CN107169924 B CN 107169924B CN 201710448347 A CN201710448347 A CN 201710448347A CN 107169924 B CN107169924 B CN 107169924B
- Authority
- CN
- China
- Prior art keywords
- image
- images
- panoramic image
- user
- hemisphere
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 21
- 230000000007 visual effect Effects 0.000 claims description 14
- 230000001960 triggered effect Effects 0.000 claims description 10
- 230000009466 transformation Effects 0.000 claims description 9
- 238000013459 approach Methods 0.000 claims description 8
- 230000004927 fusion Effects 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 8
- 230000006835 compression Effects 0.000 claims description 7
- 238000007906 compression Methods 0.000 claims description 7
- 230000006837 decompression Effects 0.000 claims description 6
- 238000012216 screening Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 5
- 238000012544 monitoring process Methods 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 2
- 230000009191 jumping Effects 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 14
- 230000008859 change Effects 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000007654 immersion Methods 0.000 description 1
- 238000011065 in-situ storage Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 210000000697 sensory organ Anatomy 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/08—Projecting images onto non-planar surfaces, e.g. geodetic screens
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Studio Devices (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a method and a system for establishing a three-dimensional panoramic image, which relate to the technical field of virtual reality. Because every device of shooing is from outside to inside to shoot the hemisphere central direction from different angles simultaneously, consequently, can realize that the user shows the three-dimensional panorama view that user observation angle corresponds when circling around the central direction of hemisphere in real time, the user can change the observation position like this at will, has increased user's observation activity area's scope greatly, improves user experience degree. The problem that the movable area of an observer is very limited and the experience effect is poor in the prior art can be solved.
Description
Technical Field
The invention relates to the technical field of virtual reality, in particular to a method and a system for establishing a three-dimensional panoramic image.
Background
The three-dimensional panoramic image is obtained by capturing image information of the whole environment scene through a camera, splicing and integrating the images by using software, and processing a plane image. The three-dimensional panoramic image can simulate a two-dimensional plane image into a real three-dimensional space, and the effect of simulating and reproducing a real environment scene is achieved.
Along with the development of computer software and hardware technology, wearable equipment of intelligence is gradually popularized, and wear-type virtual reality equipment shows virtual environment image in the observer's eye through image display screen, builds an experience of putting into virtual environment for the observer.
However, the viewing angle of the three-dimensional panoramic image currently used in the head-mounted virtual reality device is that the position of the observer is not moved, the three-dimensional panoramic image rotates in situ by 360 degrees to be observed outwards by taking the position of the observer as a center, and the head can be observed in an upward view and a downward view. The scene presented under the observation visual angle makes the movable area of the observer very limited, and the experience effect is poor.
Disclosure of Invention
In order to solve the above problems, the present invention provides a method and a system for creating a three-dimensional panoramic image, which can enlarge an observation angle and increase a user's sense of immersion.
The invention provides a method for establishing a three-dimensional panoramic image, which comprises the following steps:
the photographing devices are symmetrically and uniformly arranged on the hemispherical surface in a spherical surface position relationship with each other, and photograph towards the center direction of the hemisphere, wherein images photographed by the photographing devices adjacent to each other in spatial position are partially overlapped with each other;
the image splicing device is respectively connected with the plurality of photographing devices, receives images sent by the plurality of photographing devices and splices the images sent by the plurality of photographing devices into a panoramic image, and the panoramic image is a hemispherical panoramic image in the center direction;
the virtual reality equipment is connected with the image splicing device, receives the panoramic image spliced by the image splicing device, maps the panoramic image to the appearance of the corresponding virtual hemispherical surface model to display the three-dimensional panoramic image, and the virtual hemispherical surface model corresponds to the hemispherical surface.
Optionally, the hemisphere is divided into a plurality of portions by a spherical latitude line and a spherical longitude line, each portion is intersected by an adjacent longitude line and latitude line to obtain at least 3 intersection points, and the plurality of photographing devices include a wide-angle camera disposed at each intersection point.
Optionally, the system further comprises:
the image compression device is used for compressing the panoramic image spliced by the image splicing device, and is arranged in the image splicing device or externally arranged on the image splicing device and connected with the image splicing device;
and the image decompression device is used for decompressing the panoramic image compressed by the image compression device and sending the decompressed panoramic image to the virtual reality equipment for map display, and the image decompression device can be arranged in the virtual reality equipment.
Optionally, the image stitching device converts images sent by the plurality of photographing devices under different projection planes into the same plane by adopting homography transformation between plane projections, monitors and extracts feature points of each image after the projection planes are unified, performs matching screening of the feature points on two images with an overlapping region, establishes a matching point pair between the two images with the overlapping region, performs image registration and fusion, exposure compensation and ghost removal processing on the images with the matched feature points, and stitches the images shot by the plurality of photographing devices into a panoramic image.
Optionally, the virtual reality device acquires the position and view angle information of the user when the user circles around the center direction of the hemisphere, and displays the three-dimensional image of the image on the hemispherical surface model at the corresponding position and view angle according to the position and view angle information of the user; when a viewpoint height adjusting instruction is triggered, the corresponding viewpoint height position is raised, and the corresponding viewpoint height, horizontal positioning and images on the hemispherical model in the view angle direction are subjected to three-dimensional image display; when a user approaches to the center direction of the hemisphere, the resolution of the displayed three-dimensional image is improved by a preset multiple, and the displayed three-dimensional image is clear and is not distorted.
The invention also provides a method for establishing the three-dimensional panoramic image, which comprises the following steps:
the method comprises the steps of obtaining images shot by a plurality of shooting devices towards the center direction of a hemisphere at the same time, wherein the plurality of shooting devices are symmetrically and uniformly arranged on the hemisphere surface in a sphere position relationship with each other, and the images shot by the shooting devices adjacent in spatial position are partially overlapped with each other;
splicing images shot by a plurality of shooting devices into a panoramic image, wherein the panoramic image is a panoramic image in the center direction of a hemisphere;
and pasting the spliced panoramic picture on the surface of the corresponding virtual hemispherical surface model to display the three-dimensional panoramic picture, wherein the virtual hemispherical surface model corresponds to a hemispherical surface.
Optionally, the hemisphere is divided into a plurality of portions by a spherical latitude line and a spherical longitude line, each portion is intersected by an adjacent longitude line and latitude line to obtain at least 3 intersection points, and the plurality of photographing devices include a wide-angle camera disposed at each intersection point.
Optionally, the method further comprises:
converting images under different projection planes sent by a plurality of photographing devices to the same plane by adopting homography transformation among plane projections;
carrying out feature point monitoring and extraction on each image after the projection planes are consistent, carrying out matching screening on feature points on two images with an overlapping region, and establishing a matching point pair between the two images with the overlapping region;
and carrying out image registration and fusion, exposure compensation and de-ghosting treatment on the images after the feature points are matched, and splicing the plurality of images into a panoramic image.
Optionally, the method further comprises:
when a user circles around the center direction of the hemisphere, position and visual angle information of the user is acquired, and three-dimensional image display is carried out on an image on the hemispherical surface model at the corresponding position and visual angle according to the position and visual angle information of the user;
when a viewpoint height adjusting instruction is triggered, the corresponding viewpoint height position is raised, and the corresponding viewpoint height, horizontal positioning and images on the hemispherical model in the view angle direction are subjected to three-dimensional image display;
when a user approaches to the center direction of the hemisphere, the resolution of the displayed three-dimensional image is improved by a preset multiple, and the displayed three-dimensional image is clear and is not distorted.
Optionally, the method further comprises:
when it is monitored that a button which is positioned on the virtual reality equipment and used for triggering the viewpoint height adjusting instruction is triggered, triggering the viewpoint height adjusting instruction; or
And when the preset times of triggering the viewpoint height adjusting instruction by continuous jumping of the user wearing the virtual reality equipment are monitored, triggering the viewpoint height adjusting instruction.
According to the embodiment of the invention, a plurality of photographing devices which are symmetrically and uniformly arranged on the hemispherical surface in a spherical surface position relationship with each other photograph the central direction of the hemispherical surface at the same time, images photographed by the plurality of photographing devices are spliced into a panoramic image and sent to virtual reality equipment, and the panoramic image is pasted on the outer surface of the corresponding virtual hemispherical surface model to be displayed in a three-dimensional panoramic image. Because the images shot by each shooting device are partially overlapped with each other (the local overlapping degree is high), the image coverage density is higher, the full-angle panoramic image coverage can be realized, and the effect is smoother and closer to a real scene when the partially overlapped images are spliced into a three-dimensional panoramic image. Furthermore, each photographing device simultaneously photographs the hemispherical center direction (scene) from different angles from outside to inside, so that the three-dimensional panoramic view corresponding to the observation angle of the user can be displayed in real time when the user circles around the hemispherical center direction, the observation position can be changed freely by the user, the range of the observation activity area of the user is greatly enlarged, and the user experience is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a system architecture diagram for building a three-dimensional panoramic image according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a photographing apparatus according to an embodiment of the present invention;
fig. 3 is a schematic flow chart of a method for creating a three-dimensional panoramic image according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the examples of the present invention and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, and "a" and "an" generally include at least two, but do not exclude at least one, unless the context clearly dictates otherwise.
It should be understood that the term "and/or" as used herein is merely one type of association that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
It should be understood that although the terms first, second, third, etc. may be used to describe XXX in embodiments of the present invention, these XXX should not be limited to these terms. These terms are only used to distinguish XXX from each other. For example, a first XXX may also be referred to as a second XXX, and similarly, a second XXX may also be referred to as a first XXX, without departing from the scope of embodiments of the present invention.
It is also noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a good or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such good or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a commodity or system that includes the element.
According to the invention, the same scene is shot simultaneously from different angles by the plurality of shooting devices, the plurality of images overlapping the whole scene are obtained in real time, and more dense image coverage is obtained, so that the corresponding images can be provided for the user according to the orientation requirement of the real-time user, and the user can obtain the experience of real-time stereoscopic vision effect.
Fig. 1 is an architecture diagram of a system for building a three-dimensional panoramic image according to an embodiment of the present invention, as shown in fig. 1, the system according to the embodiment of the present invention includes:
the N photographing devices are symmetrically and uniformly arranged on the hemispherical surface in a spherical surface position relationship with each other, and simultaneously photograph the hemispherical surface in the central direction, wherein images photographed by the photographing devices adjacent to each other in spatial position are partially overlapped with each other; wherein N is greater than 3.
In this embodiment, the hemispherical surface is divided into a plurality of portions by spherical latitude lines and spherical longitude lines, each portion is intersected by adjacent longitude lines and latitude lines to obtain at least 3 intersections, and the plurality of photographing devices include wide-angle cameras disposed at each intersection. In an optional implementation mode, 2 parts of a hemispherical surface can be divided by using a spherical latitude line along a spherical radius central point, 4 parts of the hemispherical surface can be divided by using a spherical longitude line, eight intersection points are obtained by intersecting two adjacent longitude lines and two adjacent latitude lines, the eight intersection points are respectively provided with a wide-angle camera, and the eight intersection points of the two adjacent longitude lines at the north pole of the spherical surface are provided with the wide-angle cameras.
Fig. 2 is a schematic layout diagram of a photographing device according to an embodiment of the present invention, as shown in fig. 2, 9 wide-angle cameras are symmetrically and uniformly arranged on a hemispherical surface, and the 9 wide-angle cameras all photograph toward a center direction of the hemispherical surface, that is, each wide-angle camera photographs from outside to inside in the center direction of the hemispherical surface from different angles, images photographed by adjacent wide-angle cameras overlap each other partially (the local overlapping degree is high), the image coverage density is high, full-angle panoramic image coverage can be achieved, and when the partially overlapped images are spliced into a three-dimensional panoramic image, the effect is smoother and closer to a real scene.
The system of this embodiment further comprises: the image splicing device is respectively connected with the plurality of photographing devices, receives the images sent by the plurality of photographing devices and splices the images sent by the plurality of photographing devices into a panoramic image, and the panoramic image is a panoramic image of an object at the center of the ball;
when the image splicing device specifically realizes image splicing, homography transformation among plane projections can be adopted, images under different projection planes sent by a plurality of photographing devices are converted into the same plane, feature point monitoring and extraction are carried out on each image after the projection planes are unified, matching screening of feature points is carried out on two images with an overlapping area, matching point pairs between the two images with the overlapping area are established, image registration and fusion, exposure compensation and ghost removal processing are carried out on the images after the feature points are matched, and the images shot by the plurality of photographing devices are spliced into a panoramic image.
The system of this embodiment further comprises: the virtual reality equipment is connected with the image splicing device, receives the panoramic image spliced by the image splicing device, maps the panoramic image to the appearance of the corresponding virtual hemispherical surface model to display the three-dimensional panoramic image, and the virtual hemispherical surface model corresponds to the hemispherical surface.
Specifically, when the virtual reality equipment carries out a three-dimensional panoramic image and when monitoring that a user circles around the center direction of a hemisphere, the position and visual angle information of the user is obtained, and the image on the hemispherical surface model at the corresponding position and under the corresponding visual angle is subjected to three-dimensional image display according to the position and visual angle information of the user; when a viewpoint height adjusting instruction is triggered, the corresponding viewpoint height position is raised, and the corresponding viewpoint height, horizontal positioning and images on the hemispherical model in the view angle direction are subjected to three-dimensional image display; when the situation that the user approaches to the center direction of the hemisphere is monitored, the resolution of the displayed three-dimensional image is improved by a preset multiple, and the displayed three-dimensional image is clear and is not distorted.
For this reason, the virtual reality equipment is provided with a sensor, and the position and the change of the visual angle of the user can be monitored.
In an optional implementation manner, the system of this embodiment further includes:
the image compression device is used for compressing the panoramic image spliced by the image splicing device, and is arranged in the image splicing device or externally arranged on the image splicing device and connected with the image splicing device;
and the image decompression device is used for decompressing the panoramic image compressed by the image compression device and sending the decompressed panoramic image to the virtual reality equipment for map display, and the image decompression device can be arranged in the virtual reality equipment.
According to the embodiment of the invention, a plurality of photographing devices which are symmetrically and uniformly arranged on the hemispherical surface in a spherical surface position relationship with each other photograph the central direction of the hemispherical surface at the same time, images photographed by the plurality of photographing devices are spliced into a panoramic image and sent to virtual reality equipment, and the panoramic image is pasted on the outer surface of the corresponding virtual hemispherical surface model to be displayed in a three-dimensional panoramic image. Because the images shot by each shooting device are partially overlapped with each other (the local overlapping degree is high), the image coverage density is higher, the full-angle panoramic image coverage can be realized, and the effect is smoother and closer to a real scene when the partially overlapped images are spliced into a three-dimensional panoramic image. Furthermore, each photographing device shoots the semi-spherical center direction from different angles from outside to inside, so that the three-dimensional panoramic view corresponding to the observation angle of the user can be displayed in real time when the user circles around the semi-spherical center direction, the observation position can be changed at will by the user, the range of the observation activity area of the user is greatly enlarged, and the user experience degree is improved.
Based on the system architecture diagram shown in fig. 1, fig. 3 is a schematic flow chart of a method for establishing a three-dimensional panoramic image according to an embodiment of the present invention, as shown in fig. 3, including:
101. and acquiring images shot by a plurality of shooting devices which are symmetrically and uniformly arranged on the hemispherical surface in a spherical position relationship with each other and aim at the central direction of the hemisphere.
The plurality of photographing devices are symmetrically and uniformly arranged on a hemispherical surface in a spherical position relationship with each other, and images photographed by the photographing devices adjacent to each other in spatial position are partially overlapped with each other;
wherein, a plurality of devices of shooing include in the concrete arrangement of hemisphere: the hemisphere is divided into a plurality of parts by the spherical latitude line and the spherical longitude line, each part is intersected by the adjacent longitude line and latitude line to obtain at least 3 intersection points, and the plurality of photographing devices comprise wide-angle cameras arranged at each intersection point. As shown in fig. 2, 9 wide-angle cameras are symmetrically and uniformly arranged on a hemispherical surface, and the 9 wide-angle cameras all shoot towards the center direction of the hemisphere, that is, each wide-angle camera shoots from different angles from outside to inside in the center direction of the hemisphere at the same time, so that images shot by adjacent wide-angle cameras are partially overlapped with each other (the local overlapping degree is high), the image coverage density is high, full-angle panoramic image coverage can be realized, and when the partially overlapped images are spliced into a three-dimensional panoramic image, the effect is smoother and closer to a real scene.
In the embodiment of the invention, each photographing device is connected with the image splicing device, and the photographed images can be sent to the image splicing device.
102. And splicing the images shot by the plurality of shooting devices into a panoramic image.
Specifically, after the image stitching device receives the image transmitted from each photographing device, the following processing needs to be performed on the image: the method comprises the steps of projection plane unification, feature extraction, feature matching, image registration and fusion, exposure compensation and ghost removal, and finally a panoramic map capable of covering the outer surface of a hemispherical model is obtained.
Wherein the projection plane uniformization processing includes: and images under different projection planes shot by each shooting device are converted to be under the same plane by adopting homography transformation between plane projections.
Wherein the feature extraction comprises: detecting and extracting feature points of each image, for example, extracting the SIFT feature points of each image by adopting a Scale-invariant feature transform (SIFT) method, performing continuous filtering on each image for a plurality of times to obtain a first Scale group image, reducing the width and the height of an original image by half, performing continuous filtering for a plurality of times to obtain a second Scale group image, and continuously repeating the process until the width and the height of the image are less than or equal to a set threshold value; carrying out difference on the Gaussian images in each scale to form a Gaussian difference scale group image; and solving and calculating local extreme points of the images of the Gaussian difference scale groups, namely the local extreme points are the Sift characteristic points. The Sift characteristic points are local characteristics of the image, have good invariance to translation, rotation, scale scaling, brightness change, shielding, noise and the like, and keep certain stability to vision change and affine transformation.
The feature matching comprises the following steps: because the images shot by the adjacent wide-angle cameras are partially overlapped (the local overlapping degree is high), for two images with an overlapping area, RANSAC (an abbreviation of RANdom Sample Consensus, which is an algorithm for obtaining effective Sample data by calculating mathematical model parameters of data according to a group of Sample data sets containing abnormal data) algorithm is used for screening feature points, and then a matched feature point pair between the two images is established by using a Sift/KD-Tree (short for k-dimensional Tree, which is a data structure for segmenting a k-dimensional data space).
Image registration and fusion includes: the image registration is to obtain image space coordinate transformation parameters through the matched feature point pairs; finally, carrying out image registration by the coordinate transformation parameters; the image fusion can use non-multiresolution technology or multiresolution technology to improve the utilization rate of image information, improve the accuracy and reliability of computer interpretation and improve the spatial resolution and spectral resolution of images.
The exposure compensation and de-ghosting includes: according to the optical perspective geometry principle, images shot by two different shooting devices always have certain parallax errors such as ghosting and ghosting, and finally, the spliced images have unacceptable defects in vision, so that the images need to be subjected to ghost removing processing; in addition, because the cameras have different shooting angles, for example, some cameras are located on the light-facing surface, and some cameras are located on the backlight surface, the shot images have inconsistent exposure degrees. Therefore, in the embodiment of the present invention, exposure compensation and de-ghosting processing need to be performed on each image, wherein the exposure compensation can perform exposure adjustment on the image according to the target histogram curve.
And the image splicing device repeats the steps to splice the images shot by each shooting device into a panoramic image.
103. And pasting the spliced panoramic picture on the appearance of the corresponding virtual hemispherical surface model to display the three-dimensional panoramic picture.
The virtual hemispherical model in the embodiment of the invention is a model which is formed by simulating a hemispherical surface in reality through a scientific technology by virtual reality equipment and then superimposing the hemispherical surface on a virtual reality world to be perceived by a user, and is a mode of observing the world by adding a virtualization technology to a sense organ of the user. Therefore, a position mapping relation exists between the virtual hemispherical model and the hemispherical surface.
In the embodiment of the invention, in order to improve the transmission efficiency and reduce network transmission resources, after the images shot by each shooting device are spliced into the panoramic image by the image splicing device, the spliced panoramic image can be subjected to lightweight lossless compression and then transmitted to the virtual reality equipment through the network.
The virtual reality equipment decompresses the received panoramic image, and then correspondingly maps the decompressed panoramic image onto the outer surface of the virtual hemispherical surface model to form a three-dimensional panoramic image based on virtual reality.
In the embodiment of the invention, each photographing device simultaneously photographs from outside to inside in the center direction of the hemisphere from different angles, so that the three-dimensional panoramic view corresponding to the observation angle of the user can be displayed in real time when the user circles around the center direction of the hemisphere, the observation position can be changed freely by the user, the range of the observation activity area of the user is greatly enlarged, and the user experience is improved.
Specifically, when the user circles around the center direction of the hemisphere, the position and view angle information of the user is acquired, and the image on the hemispherical surface model at the corresponding position and view angle is subjected to three-dimensional image display according to the position and view angle information of the user.
When a viewpoint height adjusting instruction is triggered, the corresponding viewpoint height position is raised, and the corresponding viewpoint height, horizontal positioning and images on the hemispherical model in the view angle direction are subjected to three-dimensional image display; the triggering conditions of the viewpoint height adjusting instruction include, but are not limited to, the following examples: when it is monitored that a button which is positioned on the virtual reality equipment and used for triggering the viewpoint height adjusting instruction is triggered, triggering the viewpoint height adjusting instruction; or when the preset times that the user wearing the virtual reality equipment continuously jumps to trigger the viewpoint height adjusting instruction are monitored, the viewpoint height adjusting instruction is triggered.
When a user approaches to the center direction of the hemisphere, the resolution of the displayed three-dimensional image is improved by a preset multiple, and the displayed three-dimensional image is clear and is not distorted. As shown in fig. 2, when a user approaches a scene center point along a radius, in order to enable a virtual reality device to clearly and finely display a virtual reality three-dimensional image with high resolution in different depths of field, a resolution of 1080P is increased when a user viewpoint approaches the scene center point, for example, the resolution is increased by a specified multiple through a neural network, the specified multiple is generally set to be 2-4 times, and the closer a viewpoint is to the scene center point, the higher the multiple is set, so as to ensure the image to be clear and undistorted.
It should be noted that, in the embodiment of the present invention, a sensor is disposed in the virtual reality device, and is used for tracking the head of the user to determine the position and the posture of the head of the user in the real world, so that the viewing angle of the virtual world can be determined.
According to the embodiment of the invention, a plurality of photographing devices which are symmetrically and uniformly arranged on the hemispherical surface in a spherical surface position relationship with each other photograph the central direction of the hemispherical surface at the same time, images photographed by the plurality of photographing devices are spliced into a panoramic image and sent to virtual reality equipment, and the panoramic image is pasted on the outer surface of the corresponding virtual hemispherical surface model to be displayed in a three-dimensional panoramic image. Because the images shot by each shooting device are partially overlapped with each other (the local overlapping degree is high), the image coverage density is higher, the full-angle panoramic image coverage can be realized, and the effect is smoother and closer to a real scene when the partially overlapped images are spliced into a three-dimensional panoramic image. Furthermore, each photographing device shoots the semi-spherical center direction from different angles from outside to inside, so that the three-dimensional panoramic view corresponding to the observation angle of the user can be displayed in real time when the user circles around the semi-spherical center direction, the observation position can be changed at will by the user, the range of the observation activity area of the user is greatly enlarged, and the user experience degree is improved.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (8)
1. A system for creating a three-dimensional panoramic image, comprising:
the photographing devices are symmetrically and uniformly arranged on the hemispherical surface in a spherical surface position relationship with each other, and photograph towards the center direction of the hemisphere, wherein images photographed by the photographing devices adjacent to each other in spatial position are partially overlapped with each other;
the image splicing device is respectively connected with the plurality of photographing devices, receives images sent by the plurality of photographing devices and splices the images sent by the plurality of photographing devices into a panoramic image, and the panoramic image is a hemispherical panoramic image in the center direction;
the virtual reality equipment is connected with the image splicing device, receives the panoramic image spliced by the image splicing device, maps the panoramic image on the surface of the corresponding virtual hemispherical surface model to display the three-dimensional panoramic image, and the virtual hemispherical surface model corresponds to the hemispherical surface;
the hemisphere is divided into a plurality of parts by the spherical latitude line and the spherical longitude line, each part is intersected by the adjacent longitude line and latitude line to obtain at least 3 intersection points, and the plurality of photographing devices comprise wide-angle cameras arranged at each intersection point.
2. The system of claim 1, further comprising:
the image compression device is used for compressing the panoramic image spliced by the image splicing device, and is arranged in the image splicing device or externally arranged on the image splicing device and connected with the image splicing device;
and the image decompression device is used for decompressing the panoramic image compressed by the image compression device and sending the decompressed panoramic image to the virtual reality equipment for map display, and the image decompression device can be arranged in the virtual reality equipment.
3. The system of claim 1, wherein:
the image splicing device adopts homography transformation between planar projections, converts images under different projection planes sent by a plurality of photographing devices into the same plane, monitors and extracts characteristic points of each image after the projection planes are unified, performs matching screening of the characteristic points on two images with an overlapped area, establishes a matching point pair between the two images with the overlapped area, performs image registration and fusion, exposure compensation and ghost removal processing on the images after the characteristic points are matched, and splices the images shot by the plurality of photographing devices into a panoramic image.
4. The system of claim 1, wherein:
the virtual reality equipment acquires the position and visual angle information of the user when the user circles around the center direction of the hemisphere, and displays the three-dimensional image of the image on the hemispherical surface model at the corresponding position and visual angle according to the position and visual angle information of the user; when a viewpoint height adjusting instruction is triggered, the corresponding viewpoint height position is raised, and the corresponding viewpoint height, horizontal positioning and images on the hemispherical model in the view angle direction are subjected to three-dimensional image display; when a user approaches to the center direction of the hemisphere, the resolution of the displayed three-dimensional image is improved by a preset multiple, and the displayed three-dimensional image is clear and is not distorted.
5. A method for establishing a three-dimensional panoramic image is characterized by comprising the following steps:
the method comprises the steps of obtaining images shot by a plurality of shooting devices towards the center direction of a hemisphere at the same time, wherein the plurality of shooting devices are symmetrically and uniformly arranged on the hemisphere surface in a sphere position relationship with each other, and the images shot by the shooting devices adjacent in spatial position are partially overlapped with each other;
splicing images shot by a plurality of shooting devices into a panoramic image, wherein the panoramic image is a panoramic image in the center direction of a hemisphere;
pasting the spliced panoramic picture on the surface of a corresponding virtual hemispherical surface model to display a three-dimensional panoramic picture, wherein the virtual hemispherical surface model corresponds to a hemispherical surface;
the hemisphere is divided into a plurality of parts by the spherical latitude line and the spherical longitude line, each part is intersected by the adjacent longitude line and latitude line to obtain at least 3 intersection points, and the plurality of photographing devices comprise wide-angle cameras arranged at each intersection point.
6. The method of claim 5, further comprising:
converting images under different projection planes sent by a plurality of photographing devices to the same plane by adopting homography transformation among plane projections;
carrying out feature point monitoring and extraction on each image after the projection planes are consistent, carrying out matching screening on feature points on two images with an overlapping region, and establishing a matching point pair between the two images with the overlapping region;
and carrying out image registration and fusion, exposure compensation and de-ghosting treatment on the images after the feature points are matched, and splicing the plurality of images into a panoramic image.
7. The method of claim 5, further comprising:
when a user circles around the center direction of the hemisphere, position and visual angle information of the user is acquired, and three-dimensional image display is carried out on an image on the hemispherical surface model at the corresponding position and visual angle according to the position and visual angle information of the user;
when a viewpoint height adjusting instruction is triggered, the corresponding viewpoint height position is raised, and the corresponding viewpoint height, horizontal positioning and images on the hemispherical model in the view angle direction are subjected to three-dimensional image display;
when a user approaches to the center direction of the hemisphere, the resolution of the displayed three-dimensional image is improved by a preset multiple, and the displayed three-dimensional image is clear and is not distorted.
8. The method of claim 7, further comprising:
when it is monitored that a button which is positioned on the virtual reality equipment and used for triggering the viewpoint height adjusting instruction is triggered, triggering the viewpoint height adjusting instruction; or
And when the preset times of triggering the viewpoint height adjusting instruction by continuous jumping of the user wearing the virtual reality equipment are monitored, triggering the viewpoint height adjusting instruction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710448347.6A CN107169924B (en) | 2017-06-14 | 2017-06-14 | Method and system for establishing three-dimensional panoramic image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710448347.6A CN107169924B (en) | 2017-06-14 | 2017-06-14 | Method and system for establishing three-dimensional panoramic image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107169924A CN107169924A (en) | 2017-09-15 |
| CN107169924B true CN107169924B (en) | 2020-10-09 |
Family
ID=59819323
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710448347.6A Active CN107169924B (en) | 2017-06-14 | 2017-06-14 | Method and system for establishing three-dimensional panoramic image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107169924B (en) |
Families Citing this family (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108476291A (en) * | 2017-09-26 | 2018-08-31 | 深圳市大疆创新科技有限公司 | Image generating method, video generation device and machine readable storage medium |
| CN107833265B (en) * | 2017-11-27 | 2021-07-27 | 歌尔光学科技有限公司 | Image switching display method and virtual reality equipment |
| CN108153412A (en) * | 2017-12-11 | 2018-06-12 | 虚拟现实(深圳)智能科技有限公司 | LAN VR live broadcasting methods and system |
| CN108200330A (en) * | 2017-12-11 | 2018-06-22 | 虚拟现实(深圳)智能科技有限公司 | Double 8K image virtuals reality processing methods and system |
| CN108096836B (en) * | 2017-12-20 | 2021-05-04 | 深圳市百恩互动娱乐有限公司 | Method for making game by real-person real shooting |
| CN108124193A (en) * | 2017-12-25 | 2018-06-05 | 中兴通讯股份有限公司 | Method for processing video frequency and device |
| CN108257089B (en) * | 2018-01-12 | 2019-01-08 | 北京航空航天大学 | A method of the big visual field video panorama splicing based on iteration closest approach |
| CN109099902A (en) * | 2018-06-29 | 2018-12-28 | 中国航空规划设计研究总院有限公司 | A kind of virtual reality panoramic navigation system based on Unity 3D |
| KR102200115B1 (en) * | 2018-12-06 | 2021-01-08 | (주)포스트미디어 | System for providing multi-view 360 angle vr contents |
| CN111669568B (en) * | 2019-03-07 | 2024-05-28 | 阿里巴巴集团控股有限公司 | Multi-angle free view angle interaction method and device, medium, terminal and equipment |
| CN110458964B (en) * | 2019-08-21 | 2021-07-27 | 四川大学 | Real-time calculation method for dynamic illumination of real environment |
| KR102295265B1 (en) * | 2019-11-29 | 2021-08-30 | 주식회사 알파서클 | Apparaturs and method for real-time broardcasting of vr video filmed by several camera |
| CN111160377A (en) * | 2020-03-07 | 2020-05-15 | 深圳移动互联研究院有限公司 | Image acquisition system with key mechanism and evidence-based method thereof |
| CN112073669A (en) * | 2020-09-18 | 2020-12-11 | 三星电子（中国）研发中心 | Method and device for realizing video communication |
| CN113014871B (en) * | 2021-02-20 | 2023-11-10 | 青岛小鸟看看科技有限公司 | Endoscopic image display method and device and endoscopic surgery auxiliary system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1841188A (en) * | 2005-03-30 | 2006-10-04 | 北京大学 | Automatic spherical panorama generation method |
| CN103530907A (en) * | 2013-10-21 | 2014-01-22 | 深圳市易尚展示股份有限公司 | Complicated three-dimensional model drawing method based on images |
| CN104268939A (en) * | 2014-09-28 | 2015-01-07 | 国家电网公司 | Transformer substation virtual-reality management system based on three-dimensional panoramic view and implementation method of transformer substation virtual-reality management system based on three-dimensional panoramic view |
| CN105488775A (en) * | 2014-10-09 | 2016-04-13 | 东北大学 | Six-camera around looking-based cylindrical panoramic generation device and method |
| CN105654502A (en) * | 2016-03-30 | 2016-06-08 | 广州市盛光微电子有限公司 | Panorama camera calibration device and method based on multiple lenses and multiple sensors |
| WO2016163342A1 (en) * | 2015-04-06 | 2016-10-13 | 株式会社リコー | Information processing device, information processing method, and information processing program |
| CN106683045A (en) * | 2016-09-28 | 2017-05-17 | 深圳市优象计算技术有限公司 | Binocular camera-based panoramic image splicing method |
-
2017
- 2017-06-14 CN CN201710448347.6A patent/CN107169924B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1841188A (en) * | 2005-03-30 | 2006-10-04 | 北京大学 | Automatic spherical panorama generation method |
| CN103530907A (en) * | 2013-10-21 | 2014-01-22 | 深圳市易尚展示股份有限公司 | Complicated three-dimensional model drawing method based on images |
| CN104268939A (en) * | 2014-09-28 | 2015-01-07 | 国家电网公司 | Transformer substation virtual-reality management system based on three-dimensional panoramic view and implementation method of transformer substation virtual-reality management system based on three-dimensional panoramic view |
| CN105488775A (en) * | 2014-10-09 | 2016-04-13 | 东北大学 | Six-camera around looking-based cylindrical panoramic generation device and method |
| WO2016163342A1 (en) * | 2015-04-06 | 2016-10-13 | 株式会社リコー | Information processing device, information processing method, and information processing program |
| CN105654502A (en) * | 2016-03-30 | 2016-06-08 | 广州市盛光微电子有限公司 | Panorama camera calibration device and method based on multiple lenses and multiple sensors |
| CN106683045A (en) * | 2016-09-28 | 2017-05-17 | 深圳市优象计算技术有限公司 | Binocular camera-based panoramic image splicing method |
Non-Patent Citations (2)
| Title |
|---|
| 基于图象的室内虚拟漫游系统;马向英，杜威，袁晓君，李华;《中国图象图形学报》;20010131;全文 * |
| 球形全景影像位姿估计的改进ＥＰｎＰ算法;邓非，吴幼丝，胡玉雷，崔红霞;《测绘学报》;20160630;第45卷(第6期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107169924A (en) | 2017-09-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107169924B (en) | Method and system for establishing three-dimensional panoramic image | |
| US11869205B1 (en) | Techniques for determining a three-dimensional representation of a surface of an object from a set of images | |
| US10474227B2 (en) | Generation of virtual reality with 6 degrees of freedom from limited viewer data | |
| KR101986329B1 (en) | General purpose spherical capture methods | |
| US11170561B1 (en) | Techniques for determining a three-dimensional textured representation of a surface of an object from a set of images with varying formats | |
| CN105809701B (en) | Panoramic video posture scaling method | |
| CN106375748B (en) | Stereoscopic Virtual Reality panoramic view joining method, device and electronic equipment | |
| KR101295471B1 (en) | A system and method for 3D space-dimension based image processing | |
| KR20180111798A (en) | Adaptive stitching of frames in the panorama frame creation process | |
| WO2018205623A1 (en) | Method for displaying a virtual image, a virtual image display system and device, a non-transient computer-readable storage medium | |
| TW201915944A (en) | Image processing method, apparatus, and storage medium | |
| CN107079141A (en) | Image mosaic for 3 D video | |
| CN107660337A (en) | For producing the system and method for assembled view from fish eye camera | |
| CN108846796B (en) | Image splicing method and electronic equipment | |
| JP2017520944A (en) | Generation and use of 3D radon images | |
| CN107862718B (en) | 4D holographic video capture method | |
| JP2020008972A (en) | Information processor, information processing method, and program | |
| CN109448105B (en) | Three-dimensional human body skeleton generation method and system based on multi-depth image sensor | |
| JP2019509526A (en) | Optimal spherical image acquisition method using multiple cameras | |
| CN105809729B (en) | A kind of spherical panorama rendering method of virtual scene | |
| CN108564654B (en) | Picture entering mode of three-dimensional large scene | |
| JP6799468B2 (en) | Image processing equipment, image processing methods and computer programs | |
| JP2018033107A (en) | Video distribution device and distribution method | |
| CN113454980A (en) | Panorama shooting method, electronic device and storage medium | |
| Popovic et al. | Design and implementation of real-time multi-sensor vision systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20201015 Address after: 261031 north of Yuqing street, east of Dongming Road, high tech Zone, Weifang City, Shandong Province (Room 502, Geer electronic office building) Patentee after: GoerTek Optical Technology Co.,Ltd. Address before: 266104 Laoshan Qingdao District North House Street investment service center room, Room 308, Shandong Patentee before: GOERTEK TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20221124 Address after: 266104 No. 500, Songling Road, Laoshan District, Qingdao, Shandong Patentee after: GOERTEK TECHNOLOGY Co.,Ltd. Address before: 261031 north of Yuqing street, east of Dongming Road, high tech Zone, Weifang City, Shandong Province (Room 502, Geer electronics office building) Patentee before: GoerTek Optical Technology Co.,Ltd. |
|
| TR01 | Transfer of patent right |