US20180309982A1 - System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view - Google Patents

System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view Download PDF

Info

Publication number
US20180309982A1
US20180309982A1 US15/849,885 US201715849885A US2018309982A1 US 20180309982 A1 US20180309982 A1 US 20180309982A1 US 201715849885 A US201715849885 A US 201715849885A US 2018309982 A1 US2018309982 A1 US 2018309982A1
Authority
US
United States
Prior art keywords
image
video
user
virtual reality
navigable
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
US15/849,885
Inventor
Thomas Seidl
Ron Igra
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sharevr Hawaii LLC
Original Assignee
Sharevr Hawaii LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US201161490656P priority Critical
Priority to PCT/US2012/039572 priority patent/WO2012166593A2/en
Priority to US14/090,132 priority patent/US9007430B2/en
Priority to US14/685,234 priority patent/US9883174B2/en
Application filed by Sharevr Hawaii LLC filed Critical Sharevr Hawaii LLC
Priority to US15/849,885 priority patent/US20180309982A1/en
Publication of US20180309982A1 publication Critical patent/US20180309982A1/en
Assigned to SEIDL, THOMAS reassignment SEIDL, THOMAS ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: IGRA, Ron, TOTAL RECALL TECHNOLOGIES
Assigned to SHAREVR HAWAII, LLC reassignment SHAREVR HAWAII, LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SEIDL, THOMAS
Application status is Pending legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/332Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering
    • G06T15/10Geometric effects
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • H04N13/111Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H04N13/117Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/239Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/243Image signal generators using stereoscopic image cameras using three or more 2D image sensors
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/366Image reproducers using viewer tracking
    • H04N13/383Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2215/00Indexing scheme for image rendering
    • G06T2215/08Gnomonic or central projection

Abstract

The present invention relates to a system and method for capturing video of a real-world scene over a field of view that may exceed the field of view of a user, manipulating the captured video, and then stereoscopically displaying the manipulated image to the user in a head mounted display to create a virtual environment having length, width, and depth in the image. By capturing and manipulating video for a field of view that exceeds the field of view of the user, the system and method can quickly respond to movement by the user to update the display allowing the user to look and pan around, i.e., navigate, inside the three-dimensional virtual environment.

Description

    RELATED APPLICATIONS
  • This application is a continuation of U.S. patent application Ser. No. 14/685,234, filed Apr. 13, 2015, which is a continuation of U.S. patent application Ser. No. 14/090,132, filed Nov. 26, 2013, now U.S. Pat. No. 9,007,430, which is a continuation of International Application No. PCT/US2012/039572, filed May 25, 2012, which claims priority to U.S. Provisional Application No. 61/490,656, filed May 27, 2011. Each of the foregoing applications is incorporated herein by reference.
  • FIELD OF THE INVENTION
  • The present invention relates generally to systems and methods for creating a navigable, panoramic three-dimensional environment of a real-world scene, and more particularly, but not exclusively, to capturing video images that extend beyond the field of view of a user and manipulating the images to create a navigable, panoramic three-dimensional virtual reality environment in a head mounted display that permits the user to control the direction they look in the recorded real world environment.
  • BACKGROUND OF THE INVENTION
  • Existing three-dimensional image capture systems may capture images of limited field of view using two cameras mounted side by side and display these two images separately into each eye of a user by various means. Since the field of view of the originally captured images is limited, such systems do not readily permit the user to turn their head, for example 90° to the left or right, to see what is to the left or right of the originally captured images in the real-world scene. However, capturing video data of wide field of view images poses a number of difficulties which must be overcome in order to present the images to the user in a manner that accurately reflects the real-world scene and the user's movement relative to the real-world scene. Hence, it would be an advance in the state of the art to provide a system and method for creating a navigable, panoramic three-dimensional virtual reality environment of a real-world scene that permits the user to move around inside the environment.
  • SUMMARY OF THE INVENTION
  • In one of its aspects, the present invention relates to systems and methods for creating a navigable, panoramic three-dimensional virtual reality environment in which stereoscopic perception of three-dimensional depth to a user is achieved. (As used herein, a “three-dimensional virtual reality environment” is defined to be one in which stereoscopic perception of three-dimensional depth can be perceived by a user, e.g., as in a 3-D TV.) In this regard, the systems and methods may provide for the recording of a real-world scene of up to a full 360° by 360° field of view in three dimensions, manipulating the recorded images, and then displaying the images to a person using a head mounted display with a tracking device.
  • For example, the present invention may provide a system for creating a navigable, panoramic three-dimensional virtual reality environment, comprising a wide field of view optical imaging device having at least two optical imaging elements configured to image at least two different viewpoints of a scene. (As used herein, the term “wide field of view” is defined to mean a field of view having a full angle greater than 75 degrees in a vertical direction and at the same time having a full angle greater than 170 degrees in the horizontal direction.) The optical imaging device may include a fisheye lens and/or a mirror, for example. The optical imaging device may include at least one optical imaging detector configured to record first and second video image data of the different viewpoints of the scene, respectively. For example, the optical imaging device may be configured to alternately display the images of the at least two different view points on the optical imaging detector to provide the first and second video image data. Alternatively, a separate optical imaging detector may provided for each respective optical imaging element.
  • In addition, the system may include a tracking device configured to track the movement of a user, and a position detector disposed in communication with the tracking device to receive tracking data from the tracking device and configured to determine a direction of view of the user. An image renderer may be provided and disposed in communication with the optical imaging device to receive the first and second video image data. The image renderer may also be disposed in communication with the position detector to receive the direction of view, and may be configured to compute respective regions of interest of a portion of the first and second video image data based on the user's direction of view. A head mounted display may be disposed in communication with the image renderer to receive image data associated with the respective regions of interest and may be configured to display the image data associated with the respective regions of interest the user, whereby a navigable, panoramic three-dimensional virtual reality environment is created.
  • In one of its configurations, the renderer may include a computer usable medium having a computer readable program code embodied therein. The computer readable program code may be adapted to be executed to implement a method for rendering the first and second video image data to create a navigable, panoramic three-dimensional virtual reality environment. The method may also include the steps of creating first and second wireframe spheres, and transforming each of the first and second video images by wrapping the first and second video data onto the first and second wireframe spheres respectively.
  • In another of its aspects, the present invention may provide a computer usable medium comprising a computer readable program code embodied therein. The computer readable program code may be adapted to be executed to implement a method for rendering video image data to create a navigable, panoramic three-dimensional virtual reality environment. The method may include the steps of receiving video image data of a scene having a wide field of view comprising first and second video image streams taken from different viewpoints. The first video image stream may have a plurality of first image frames and the second video image stream may have a plurality of second image frames. The method may synchronize the respective pairs of the first and second image frames to create a video output stream of the synchronized image pairs. For example, the synchronization may be performed by merging respective pairs of the first and second image frames to create a video output stream of the merged image pairs. The step of synchronizing respective pairs of the first and second image frames may also include removing distortion from the first and second image frames.
  • The method may also include receiving position data indicating a direction of sight of a user, calculating a first and second region of interest of the respective first and second image frames based on the position data, and displaying the first and second regions of interest on a stereoscopic display, whereby a navigable three-dimensional virtual reality environment is created of the recorded real world environment.
  • The method may also create first and second wireframe spheres and wrap a respective first and second section of the synchronized or merged image pairs onto the respective first and second wireframe spheres. In addition, the method may receive position data indicating a direction of sight of a user and calculate a first and second region of interest of the respective first and second wireframe spheres based on the position data. The first and second regions of interest may be displayed on a stereoscopic display, The step of displaying the first and second regions of interest may include rotating the first and second wireframe spheres in response to the position data.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The foregoing summary and the following detailed description of the preferred embodiments of the present invention will be best understood when read in conjunction with the appended drawings, in which:
  • FIG. 1 schematically illustrates an exemplary system for creating a navigable, panoramic three-dimensional virtual reality environment in accordance with the present invention;
  • FIGS. 2A, 2B schematically illustrate an exemplary method for creating a navigable, panoramic three-dimensional virtual reality environment in accordance with the present invention; and
  • FIGS. 3A-3D schematically illustrate exemplary configurations for optical imaging devices used in the present invention, with FIGS. 3A, 3B showing side elevational and top views, respectively, of a mirror system, FIG. 3C showing a top view of a lens system have forward and rearward pairs of imaging devices, and FIG. 3D showing a top view of a hexagonally configured system having six imaging devices.
  • DETAILED DESCRIPTION OF THE INVENTION
  • Referring now to the figures, wherein like elements are numbered alike throughout, the present invention relates to systems and methods for creating a navigable, panoramic three-dimensional virtual reality environment having an ultra-wide field of view, FIGS. 1, 2A, 2B. For example, in one of its aspects, the systems and methods provide a navigable, panoramic three-dimensional virtual reality environment by capturing video image data of a scene having a wide field of view. By capturing video data of a scene that extends beyond the field of view of a user and storing and transforming that video data, the systems and methods of the present invention can create a navigable, panoramic three-dimensional virtual reality environment that can rapidly render and present to the user video information of new portions of the scene that come into the user's view as the user moves their head. As a result, the user gets the feeling of being inside a virtual environment, being able to perceive depth and pan and zoom, due to the manipulation and display of the video image data by the system and method of the present invention.
  • For example, as shown in FIG. 1, an exemplary system in accordance with the present invention may include a wide field of view optical imaging device 100 which may include at least two optical elements 106, 108 each oriented to image different viewpoints of a scene. Respective optical imaging detectors 102, 104 may be associated with each optical element 106, 108. Alternatively, a single imaging detector may be provided and configured to alternately receive images from each of the optical elements 106, 108. The optical elements 106, 108 may include any suitable optical imaging device capable of imaging a wide field of view, such as fisheye lenses, Fresnel lenses, mirror lenses, and catadioptric systems, for example. For instance, the two optical elements 106, 108 may be provided in the form of fisheye lenses which can capture video image data over a field of 180° by 180°. (For example, a 185° lens Raynox® Circular Fisheye Conversion Lens DCR-CF187PRO, Yoshida Industry Co., Ltd. Tokyo, Japan.) Alternatively, the optical elements may be provided in the form of parabolic reflectors 306, 308 and associated detectors 302, 304, FIGS. 3A, 3B. (For example, Pano Pro™ MkIIVB, Pano Pro, Ltd, Watlington, UK.) Further, two pairs of optical imaging detectors with associated optical elements may be used to capture video image data having a wider field of view. For example, a pair of forward facing optical imaging detectors 312 and a pair of rearward facing optical imaging detectors 310 may be employed, FIG. 3C. Still further, five (or four) optical imaging detectors and optical elements 314 maybe spaced equidistant around the circle with an optional sixth optical imaging detector and optical element 314 pointing upward out of the plane of the circle, FIG. 3D.
  • The optical imaging detectors 102, 104 may record first and second video image data comprising left and right video image streams 2, 4 and may communicate directly with an image renderer 110 to transmit the respective left and right video image streams 2, 4 acquired by the optical imaging detectors 102, 104 to the image renderer 110. Such a configuration in which the optical imaging detectors 102, 104 communication directly with the image renderer 110 may be used for real-time imaging. Alternatively, the optical imaging device 100 may record and store the left and right video image streams 2, 4 for subsequent download to the image renderer 110. For example, the left and right video image streams 2, 4 may be subsequently loaded on to a hard-drive or other suitable storage device to be accessed by the image renderer 110 at a later date.
  • The image renderer 110 may communicate with a head mounted display 130 (iWear® VR920, Vuzix Corporation, Rochester, N.Y.) worn by a user which may include a tracking device 140 configured to track the movement of the user. A position detector 120 may be disposed in communication with the tracking device 140 to receive tracking data from the tracking device 140 to determine the direction of view of the user. Although the position detector 120 is shown as a separate element in FIG. 1, the position detector 120 may optionally be provided as an integral part of the tracking device 140. (For example, FSRK-USB-1, Hill Crest Labs, Rockville, Md.) The position detector 120 may be disposed in communication with the image renderer 110 to supply the direction of view data to the image renderer 110. By combining knowledge of the direction of view of the user with the left and right video image streams 2, 4, the image renderer 110 may control the rendering and display of selected portions of the left and right video image streams 2, 4 corresponding to the user's direction of view on respective left and right screens 134, 132 of the head mounted display 130 so the user perceives a navigable, panoramic three-dimensional virtual reality environment.
  • Specifically, the image renderer 110 may take the left and right video image streams 2, 4, and a perform polar to rectangular conversion to merge respective temporally matched image frames of the left and right video image streams 2, 4 into a merged image pair 8. That is, for each image frame of the left video image stream 2 the temporally corresponding image frame of the right video image stream 4 is merged thereto to create a series of merged image pairs 8 for all video frames to provide an output stream of the merged image pairs 8. Merging temporally matched image frames 8 from the left and right video image streams 2, 4 ensures that the video image data from each optical imaging detector 102, 104 remains temporally synchronized, and is but one example of how the image streams 2, 4 may be synchronized. That is, data will always be in sync as the left and right video image streams 2, 4 are now merged together in the output stream. With the left and right video image streams 2, 4 now merged in stereo format in the output stream, the output stream can be processed to play in the head mounted display 130. (In addition, in certain configurations of the optical imaging device 100 it is possible that one optical element 106, 108 may image the other, in which case the image renderer 110 may patch the images of the left and right video image streams 2, 4 to obscure the image of the other optical element.)
  • To create the head mounted display output, the image renderer 110 may create virtual wireframe spheres 12, 14 (one for each eye) to provide a framework for the virtual reality environment. As each merged image pair 8 is encountered in the output stream, the merged image pair 8 may be split into respective left and right portions that are then wrapped onto respective left, right wireframe sphere 12, 14. The image renderer 110 may virtually place a camera inside the left sphere 14 to create a feed to the left screen 134 of the head mounted display 130 for display to the left eye of the user. Likewise, the image renderer 110 may virtually place a camera inside the right sphere 12 to create a feed to the right screen 132 of the head mounted display 130 for display to the right eye of the user. As these virtual cameras are looking only at a small section of the inside of the virtual spheres 12, 14 the cameras only see part of the image projected around the virtual spheres 12, 14. As the user turns his head anticlockwise a signal is sent back from the tracking device 140 via the position detector 120 to inform the image renderer 110 that the user's head is moving anticlockwise, in response to which the image renderer 110 rotates both spheres 12, 14 clockwise around the spheres' vertical axes. As a result, the virtual cameras inside the virtual spheres 12, 14 will now see an image that pans to the anticlockwise as the cameras are always fixed looking forward, so the inside of the virtual spheres 12, 14 pan clockwise which gives the feeling to the user of movement turning anticlockwise the opposite way.
  • Turning to FIGS. 2A, 2B, an exemplary process by which the image renderer creates the navigable, panoramic three-dimensional virtual reality environment is illustrated. In one exemplary method in accordance with the present invention, a navigable, panoramic three-dimensional virtual reality environment having ultra-wide field of view may be created using a fisheye stereo side-by-side video publisher for creating the output stream of merged image pairs 8, FIG. 2A, in conjunction with a spherical stereo side-by-side viewer for processing and displaying the output stream to the user, FIG. 2B. Turning first to the publisher, FIG. 2A, the publisher may be implemented using FFMpeg open source libraries for handling multimedia data and Microsoft® DirectX® protocols, though other suitable libraries or algorithms may be used to effect the creation of the output stream of merged image pairs 8. As a first step 200, the video publisher may initialize global variables, which may include initializing the bit rate, frame size, audio channel, and audio rate, as well as selecting the output video stream compression, and output FPS frame rate. (The name of the output video file may also be selected at step 200.) In addition at step 200, the publisher may receive the left and right video image streams 2, 4, with the left video image stream 2 having a plurality of left image frames and the right video image stream 4 having a plurality of right image frames.
  • After global initialization, publication of the merged image pairs 8 may proceed by creating two parallel publication threads for processing the left and right video image streams 2, 4, at steps 210 a and 210 b, respectively. Initialization of variables for each publication thread may include initialization of the DirectX® variables: FilterGraph, SampleGrabber, GraphBuilder, IMediaControl, IVideoWindow, IBasicAudio, and IMediaSeek, steps 210 a, 210 b. In addition, the rendering modes associated with sharpening, brightness, contrast, and noise median may be set, steps 220 a, 220 b. With the variables initialized and rendering modes set, the left and right video image streams 2, 4 may be processed to remove distortion introduced by the optical imaging device 100 using the transformation provided in Table 1, steps 230 a, 230 b. The distortion-free streams may then be rendered to respective left and right bitmap streams, steps 240 a, 240 b, which may then be merged to create the merged image pairs 8 of the output stream, step 250.
  • TABLE 1
    Transformation for each pixel of the current frame:
    x_dst, y_dst - destination coordinates
    x_src, y_src - source coordinates
    distance = FrameHeight * PI
    distance_1 = 1.0 / distance
    distance_2 = distance / 1.1
    phi = x_dst * distance_1
    theta = −y_dst * distance_1 + PI/2
    s = SIN( theta )
    v0 = s * SIN( phi )
    v1 = COS( theta )
    r = SQRT(v1*v1 + v0*v0)
    s = ATAN(r / ( s * COS( phi ))) / r
    x_src = v0 * distance_2 * s
    y_src = v1 * distance_2 * s
  • Having formed the output stream of merged image pairs 8, the image renderer 110 may proceed with transformation of the output stream for display in the head mounted display 130 to create the navigable, panoramic three-dimensional virtual reality environment. Beginning at step 260, the video publisher may be initialized by initializing the DirectX® D3D interfaces (e.g., IDirect3D9, IDirect3DDevice9, IDirect3DVertexBuffer9, and IDirect3DTexture9), FIG. 2B. In addition at step 260, the system may be configured to initialize the DirectShow interface (e.g., FilterGraph, SampleGrabber, GraphBuilder, IMediaControl, IVideoWindow, IBasicAudio, IMediaSeek) as well as common parameters, such as all parameters for the stereo video stream and look-up parameters for left and right eye image buffers. After initialization, rendering may proceed by creating two parallel threads for rendering the output video stream of merged image pairs 8 and operating the tracking device 140, at steps 270 and 290, respectively. Specifically, for the tracking thread, the process may include the steps for initializing the tracking device 140, step 290, and tracking the user's movement using the position detector 140, step 292. Returning to the rendering thread, at step 270 the 3D environment may be initialized by filling quad vertices (D3D), creating the Direct3D interface, creating a D3D device and setting the DirectX D3D display mode, setting a render state for the D3D device, setting light for D3D device, creating a vertex buffer for D3D device, preparing backbuffer flipping, and preparing a D3D scene. Once the 3D environment is initialized, the frame-by-frame process of rendering the video stream may proceed at step 272. Specifically, the process may retrieve a current video frame containing a merged image pair 8 and may split the merged image pairs 8 of the video output stream into respective left and right images for use in a left eye buffer and right eye buffer, step 272. Further, the status of the user's position and direction of view in the parallel tracking thread may be checked and updated. As a next step in creating the navigable, panoramic three-dimensional virtual reality environment, the left and right images generated at step 272 may be wrapped around the virtual spheres 12, 14 using the exemplary code listed in Table 2 in view of the following geometric conventions.
  • In analytic geometry, a sphere with center (x0, y0, z0) and radius, r, is the locus of all points (x, y, z) such that (x−x0)2+(y−y0)2+(z−z0)2=r2. The points on the sphere with radius r can be parameterized via

  • x=x 0 +r sin θ cos φ

  • y=y 0 +r sin θ sin φ (0≤φ≤2π and 0≤θ≤π)

  • z=z 0 +r cos θ.
  • A sphere of any radius centered at zero is an integral surface of the following differential form xdx+ydy+zdz=0. In Cartesian coordinates (x, y, z) on the sphere and (X, Y) on the plane, the projection and its inverse are given by the formulas
  • ( X , Y ) = ( x 1 - z , y 1 - z ) , and ( x , y , z ) = ( 2 X 1 + X 2 + Y 2 , 2 Y 1 + X 2 + Y 2 , - 1 + X 2 + Y 2 1 + X 2 + Y 2 ) .
  • In spherical coordinates (φ, θ) on the sphere (with φ the zenith angle, 0≤φ≤π, and θ the azimuth, 0≤θ≤2π) and polar coordinates (R, Θ) on the plane, the projection and its inverse are
  • ( R , Θ ) = ( sin ϕ 1 - cos ϕ , θ ) , ( ϕ , θ ) = ( 2 arctan ( 1 R ) , Θ ) .
  • Here, φ is understood to have value π when R=0. Also, there are many ways to rewrite these formulas using trigonometric identities. In cylindrical coordinates (r, θ, z) on the sphere and polar coordinates (R, Θ) on the plane, the projection and its inverse are
  • ( R , Θ ) = ( r 1 - z , θ ) , ( r , θ , z ) = ( 2 R 1 + R 2 , Θ , R 2 - 1 R 2 + 1 ) .
  • Having transformed the left and right images from a rectangular to spherical form by wrapping the left and right images around respective spheres 12, 14, further processing can proceed by selecting that portion of the wrapped images for display on each of the left and right eyes of the user, step 276. That is, in response to the user's direction of view as determined at step 292, a region of interest for each of the left and right wrapped images is selected for display in the head mounted display 130, step 276. Specifically, the process may, for example, execute the steps for D3D device render by: preparing the D3D matrix, choosing the left and right regions of interest, and controlling the video stream (play, stop, pause) as well as controlling panning (up, down, left, right, zoom). The left and right regions of interest may then be displayed on the respective left and right displays 134, 132 of the head mounted display 130. In the exemplary DirectX® implementation display may be accomplished by having D3D render left and right image buffers containing the left and right regions of interest to the left and right displays 134, 132 of the head mounted display 130, step 278. After a video frame is thus displayed to each of the left and right displays 134, 132, the process may be repeated for each video frame by sending a signal to the rendered 110 to return to step 272, step 280, to create a navigable, panoramic three-dimensional virtual reality environment having ultra-wide field of view.
  • These and other advantages of the present invention will be apparent to those skilled in the art from the foregoing specification. Accordingly, it will be recognized by those skilled in the art that changes or modifications may be made to the above-described embodiments without departing from the broad inventive concepts of the invention. It should therefore be understood that this invention is not limited to the particular embodiments described herein, but is intended to include all changes and modifications that are within the scope and spirit of the invention as set forth in the claims.
  • TABLE 2
    Simple Transformation:
    width, height - width and height of the window
    Fov - current field of view angle (degrees);
    pan, tilt - camera direction point (degrees).
    panDest, tiltDest - model coordinates of clicked (x, y) point (degrees).
    x, y - screen clicked point coordinates (pixels);
    r - panorama model sphere radius;
    vx, vy, vz - coordinates of vector from origin to picked point in projection
    plane;
    vx2, vy2, vz2 - transformed vector of picked point.
    function radiansToDegree( radian ) { return (radian * 180.0) /
    3.14159265358979323846264; }
    function DegreeToRadians( degree ) { return (degree / 180.0) *
    3.14159265358979323846264; }
    input: x, y, Fov, pan, tilt, width, height
    output: panDest, tiltDest
    r = (sqrt(w*w + h*h) / 2.0) / tan (degreeToRadians(fov/2.0) );
    vx = r;
    vy = w/2.0 − x;
    vz = h/2.0 − y;
    pan = degreeToRadians( pan );
    tilt = degreeToRadians ( −tilt );
    cosp = cos( pan ); sinp = sin( pan );
    cost = cos( tilt ); sint = sin( tilt );
    vx2 = vx*cost*cosp − vy*sinp + vz*cosp*sint;
    vy2 = vx*cost*sinp + vy*cosp + vz*sint*sinp;
    vz2 = −vx*sint + vz*cost;
    sinDest = vy2 / sqrt(vx2*vx2 + vy2*vy2);
    panDest = radinasToDegree( arcsin(sinDest) );
    if( vx2 < = 0 && vy2 > 0 )
    panDest = 180 − panDest;
    else if( vx2 < 0 && vy2 <= 0 )
    panDest = −180 − panDest;
    sinDest = vz2 / sqrt(vx2*vx2 + vy2*vy2 + vz2*vz2);
    tiltDest = radiansToDegree( arcsin(sinDest) );

Claims (13)

What is claimed is:
1. A system for creating a navigable, panoramic three-dimensional virtual reality environment with stereoscopic perception of three-dimensional depth to a user, comprising:
a storage device for storing first and second video image streams corresponding to two different stereoscopic viewpoints of a scene, respectively, each stream comprising image frames;
a tracking device configured to track the movement of a user;
a position detector disposed in communication with the tracking device to receive tracking data from the tracking device and configured to determine a direction of view of the user;
an image renderer disposed in communication with the storage device to receive the first and second video image streams and disposed in communication with the position detector to receive the direction of view, the image renderer configured to create an output stream of merged stereo image frames corresponding to the direction of view of the user, the merged frames comprising image frames from the first video stream which are temporally matched to respective image frames from the second video stream; and
a head mounted display disposed in communication with the image renderer to receive output stream and configured to display the output stream to the user, whereby a navigable, panoramic three-dimensional virtual reality environment having stereoscopic perception of three-dimensional depth is created.
2. The system according to claim 1, comprising an optical imaging device disposed in communication with the storage device and configured to image at least two different viewpoints of the scene, the imaging device comprising at least one optical imaging detector configured to record the first and second video image streams corresponding to the at least two different viewpoints of the scene, respectively.
3. The system according to claim 2, wherein the optical imaging device is configured to display the images of the at least two different view points on the optical imaging detector to provide the first and second video image streams.
4. The system according to claim 1, wherein the image renderer is configured to split the merged frames of the output stream and transform each of the split frames onto respective first and second models.
5. The system according to claim 4, wherein the center of origin of the each model is selected to correspond to a location within the head mounted display at which a respective one of the user's eyes is to be positioned.
6. The system according to claim 4, wherein the model comprises a wireframe model.
7. The system according to claim 4, wherein the model comprises a sphere.
8. The system according to claim 1, wherein the renderer comprises a non-transitory computer usable medium having a computer readable program code embodied therein, the computer readable program code adapted to be executed to implement a method for rendering the first and second video image streams to create a panoramic three-dimensional virtual reality environment having stereoscopic perception of three-dimensional depth comprising the steps of creating first and second wireframe models, and transforming each of the first and second video images by wrapping the first and second video streams onto the first and second wireframe models respectively.
9. A non-transitory computer usable medium comprising a computer readable program code embodied therein, the computer readable program code adapted to be executed to implement a method for rendering video image streams to create a navigable, panoramic three-dimensional virtual reality environment having stereoscopic perception of three-dimensional depth comprising the steps of:
receiving video image streams of a scene comprising first and second video image streams taken from different stereoscopic viewpoints, the first video image stream having a plurality of first image frames and the second video image stream having a plurality of second image frames;
merging respective pairs of the first and second image flames in stereo format to create a video output stream of merged stereoscopic image pairs;
receiving position data indicating a direction of sight of a user;
calculating a first and second region of interest of the merged stereoscopic image pairs based on the position data; and
displaying the first and second regions of interest on a head mounted stereoscopic display,
whereby a navigable, panoramic three-dimensional virtual reality environment having stereoscopic perception of three-dimensional depth is created.
10. A non-transitory computer usable medium comprising a computer readable program code adapted to be executed to implement a method for rendering the first and second video image streams to create a navigable, panoramic three-dimensional virtual reality environment according to claim 9, comprising creating first and second wireframe models, and wrapping the respective first and second section of the merged image pairs onto the first and second wireframe models respectively.
11. A non-transitory computer usable medium comprising a computer readable program code adapted to be executed to implement a method for rendering the first and second video image streams to create a navigable three-dimensional virtual reality environment according to claim 10, wherein the step of displaying the first and second regions of interest comprises rotating the first and second wireframe models in response to the position data.
12. A non-transitory computer usable medium comprising a computer readable program code adapted to be executed to implement a method for rendering the first and second video image streams to create a navigable three-dimensional virtual reality environment according to claim 9, comprising removing distortion from the first and second image frames.
13. A non-transitory computer usable medium comprising a computer readable program code adapted to be executed to implement a method for rendering the first and second video image streams to create a navigable, panoramic three-dimensional virtual reality environment according to claim 9, wherein the step of merging the respective pairs of the first and second image frames comprises temporally synchronizing the first and second image frames.
US15/849,885 2011-05-27 2017-12-21 System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view Pending US20180309982A1 (en)

Priority Applications (5)

Application Number Priority Date Filing Date Title
US201161490656P true 2011-05-27 2011-05-27
PCT/US2012/039572 WO2012166593A2 (en) 2011-05-27 2012-05-25 System and method for creating a navigable, panoramic three-dimensional virtual reality environment having ultra-wide field of view
US14/090,132 US9007430B2 (en) 2011-05-27 2013-11-26 System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view
US14/685,234 US9883174B2 (en) 2011-05-27 2015-04-13 System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view
US15/849,885 US20180309982A1 (en) 2011-05-27 2017-12-21 System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US15/849,885 US20180309982A1 (en) 2011-05-27 2017-12-21 System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US14/685,234 Continuation US9883174B2 (en) 2011-05-27 2015-04-13 System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view

Publications (1)

Publication Number Publication Date
US20180309982A1 true US20180309982A1 (en) 2018-10-25

Family

ID=50432372

Family Applications (3)

Application Number Title Priority Date Filing Date
US14/090,132 Active US9007430B2 (en) 2011-05-27 2013-11-26 System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view
US14/685,234 Active US9883174B2 (en) 2011-05-27 2015-04-13 System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view
US15/849,885 Pending US20180309982A1 (en) 2011-05-27 2017-12-21 System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view

Family Applications Before (2)

Application Number Title Priority Date Filing Date
US14/090,132 Active US9007430B2 (en) 2011-05-27 2013-11-26 System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view
US14/685,234 Active US9883174B2 (en) 2011-05-27 2015-04-13 System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view

Country Status (1)

Country Link
US (3) US9007430B2 (en)

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9336592B2 (en) * 2012-02-03 2016-05-10 The Trustees Of Dartmouth College Method and apparatus for determining tumor shift during surgery using a stereo-optical three-dimensional surface-mapping system
WO2013103870A1 (en) 2012-01-04 2013-07-11 The Trustees Of Dartmouth College Method and apparatus for calibration of stereo-optical three-dimensional surface-mapping system
GB2523740A (en) * 2014-02-26 2015-09-09 Sony Comp Entertainment Europe Image encoding and display
US20150358539A1 (en) * 2014-06-06 2015-12-10 Jacob Catt Mobile Virtual Reality Camera, Method, And System
GB2527503A (en) * 2014-06-17 2015-12-30 Next Logic Pty Ltd Generating a sequence of stereoscopic images for a head-mounted display
US10321126B2 (en) * 2014-07-08 2019-06-11 Zspace, Inc. User input device camera
BE1022580A9 (en) * 2014-10-22 2016-10-06 Parallaxter Method of production of immersive videos and interactive parallax method of viewing video with immersive interactive parallax
US20160182822A1 (en) * 2014-12-19 2016-06-23 Sony Corporation System, method, and computer program product for determiing a front facing view of and centering an omnidirectional image
ES1136432Y (en) * 2015-01-28 2015-05-11 Rubio Pablo Abad Adaptation system of virtual reality glasses for viewing stereoscopic augmented reality envelope
CA2977051A1 (en) 2015-02-17 2016-08-25 Nextvr Inc. Methods and apparatus for generating and using reduced resolution images and/or communicating such images to a playback or content distribution device
WO2016191467A1 (en) * 2015-05-27 2016-12-01 Google Inc. Capture and render of panoramic virtual reality content
US9877016B2 (en) 2015-05-27 2018-01-23 Google Llc Omnistereo capture and render of panoramic virtual reality content
US10126813B2 (en) * 2015-09-21 2018-11-13 Microsoft Technology Licensing, Llc Omni-directional camera
CN105657408B (en) * 2015-12-31 2018-11-30 北京小鸟看看科技有限公司 Implementation method of virtual reality and virtual reality scenes of devices
US10157448B2 (en) * 2016-02-12 2018-12-18 Qualcomm Incorporated Foveated video rendering
US10186075B2 (en) * 2016-11-30 2019-01-22 Adcor Magnet Systems, Llc System, method, and non-transitory computer-readable storage media for generating 3-dimensional video images
JP2019022121A (en) * 2017-07-19 2019-02-07 ソニー株式会社 Image processor and control method of image processor

Family Cites Families (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5130794A (en) * 1990-03-29 1992-07-14 Ritchey Kurtis J Panoramic display system
US6690338B1 (en) * 1993-08-23 2004-02-10 Francis J. Maguire, Jr. Apparatus and method for providing images of real and virtual objects in a head mounted display
US6411266B1 (en) * 1993-08-23 2002-06-25 Francis J. Maguire, Jr. Apparatus and method for providing images of real and virtual objects in a head mounted display
US5796426A (en) 1994-05-27 1998-08-18 Warp, Ltd. Wide-angle image dewarping method and apparatus
US6141034A (en) * 1995-12-15 2000-10-31 Immersive Media Co. Immersive imaging method and apparatus
US5960108A (en) 1997-06-12 1999-09-28 Apple Computer, Inc. Method and system for creating an image-based virtual reality environment utilizing a fisheye lens
US6934461B1 (en) * 1999-01-05 2005-08-23 Interval Research Corporation Low attention recording, with particular application to social recording
US6825875B1 (en) * 1999-01-05 2004-11-30 Interval Research Corporation Hybrid recording unit including portable video recorder and auxillary device
US6563532B1 (en) * 1999-01-05 2003-05-13 Internal Research Corporation Low attention recording unit for use by vigorously active recorder
KR20020025301A (en) * 2000-09-28 2002-04-04 오길록 Apparatus and Method for Furnishing Augmented-Reality Graphic using Panoramic Image with Supporting Multiuser
US6891518B2 (en) 2000-10-05 2005-05-10 Siemens Corporate Research, Inc. Augmented reality visualization device
US7411594B2 (en) 2002-01-15 2008-08-12 Canon Kabushiki Kaisha Information processing apparatus and method
GB2394852B (en) 2002-10-31 2006-12-20 Hewlett Packard Co Image capture systems using motion detection
US20070182812A1 (en) * 2004-05-19 2007-08-09 Ritchey Kurtis J Panoramic image-based virtual reality/telepresence audio-visual system and method
US9344612B2 (en) * 2006-02-15 2016-05-17 Kenneth Ira Ritchey Non-interference field-of-view support apparatus for a panoramic facial sensor
US9101279B2 (en) * 2006-02-15 2015-08-11 Virtual Video Reality By Ritchey, Llc Mobile user borne brain activity data and surrounding environment data correlation system
US9270976B2 (en) * 2005-11-02 2016-02-23 Exelis Inc. Multi-user stereoscopic 3-D panoramic vision system and method
US20080007617A1 (en) * 2006-05-11 2008-01-10 Ritchey Kurtis J Volumetric panoramic sensor systems
US20080174659A1 (en) 2007-01-18 2008-07-24 Mcdowall Ian Wide field of view display device and method
US20100045773A1 (en) * 2007-11-06 2010-02-25 Ritchey Kurtis J Panoramic adapter system and method with spherical field-of-view coverage
US20110112934A1 (en) 2008-06-10 2011-05-12 Junichi Ishihara Sensory three-dimensional virtual real space system
US20100259619A1 (en) 2009-04-10 2010-10-14 Nicholson Timothy J Hmd with elevated camera
JP2010256534A (en) 2009-04-23 2010-11-11 Fujifilm Corp Head-mounted display for omnidirectional image display
JP2011205358A (en) * 2010-03-25 2011-10-13 Fujifilm Corp Head-mounted display device

Also Published As

Publication number Publication date
US9883174B2 (en) 2018-01-30
US20140098186A1 (en) 2014-04-10
US20150054912A9 (en) 2015-02-26
US20150264340A1 (en) 2015-09-17
US9007430B2 (en) 2015-04-14

Similar Documents

Publication Publication Date Title
US7429997B2 (en) System and method for spherical stereoscopic photographing
Uyttendaele et al. Image-based interactive exploration of real-world environments
US6496598B1 (en) Image processing method and apparatus
US20040130501A1 (en) Display apparatus, image processing apparatus and image processing method, imaging apparatus, and program
US9600936B2 (en) System for mixing or compositing in real-time, computer generated 3D objects and a video feed from a film camera
KR100922250B1 (en) Panoramic video system with real-time distortion-free imaging
JP6410918B2 (en) System and method for use in playback of panoramic video content
US7837330B2 (en) Panoramic three-dimensional adapter for an optical instrument and a combination of such an adapter and such an optical instrument
US20050265619A1 (en) Image providing method and device
US20050185047A1 (en) Method and apparatus for providing a combined image
US7180536B2 (en) Method for producing stereoscopic images from monoscopic images
EP2328125B1 (en) Image splicing method and device
Anderson et al. Jump: virtual reality video
US7224382B2 (en) Immersive imaging system
US20160246061A1 (en) Display
US8548269B2 (en) Seamless left/right views for 360-degree stereoscopic video
US20030117488A1 (en) Stereoscopic panoramic image capture device
US20080246759A1 (en) Automatic Scene Modeling for the 3D Camera and 3D Video
US20130321586A1 (en) Cloud based free viewpoint video streaming
Zilly et al. Production rules for stereo acquisition
US20120038635A1 (en) 3-d rendering for a rotated viewer
KR101885979B1 (en) Stereo viewing
CN106464854B (en) Display image encoding and
KR100986045B1 (en) Display device and display method
US9838668B2 (en) Systems and methods for transferring a clip of video data to a user facility

Legal Events

Date Code Title Description
STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

AS Assignment

Owner name: SEIDL, THOMAS, HAWAII

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:IGRA, RON;TOTAL RECALL TECHNOLOGIES;REEL/FRAME:048169/0344

Effective date: 20161014

AS Assignment

Owner name: SHAREVR HAWAII, LLC, HAWAII

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SEIDL, THOMAS;REEL/FRAME:048248/0705

Effective date: 20180622

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED