JP2018523326A - Full spherical capture method - Google Patents

Abstract

  Systems and methods for capturing spherical content are described. The system and method determine a region in a plurality of images captured using a plurality of cameras for converting two-dimensional data to three-dimensional data, and depth values for some of the pixels in the region , Generating a spherical image that includes image data for a portion of the pixels in the region, and using the image data, 3 in a three-dimensional space of computer graphics objects generated by the image processing system Construct spherical surfaces, use image data to generate texture mappings to the surface of computer graphics objects, and send spherical images and texture mappings for display on head mounted display devices Can include.

Description

CROSS REFERENCE TO RELATED APPLICATION This application is a priority of US Provisional Patent Application No. 62 / 219,534, filed September 16, 2015, entitled “General Spherical Capture Methods”. Claiming the priority of US Non-Provisional Patent Application No. 15 / 266,602, entitled “Entire Ball Capture Method”, filed on September 15, 2016, It is a continuation application. Both of these applications are hereby incorporated by reference in their entirety.

TECHNICAL FIELD This description relates generally to methods and apparatus for capturing and processing two-dimensional (2D) and three-dimensional (3D) images.

Background A spherical image can provide a 360 degree view of the scene. Such images can be captured and defined using a specific projection format. For example, a spherical image may be defined in an equirectangular projection format to provide a single image with an aspect ratio of 2: 1 for the width and height of the image. In another example, a spherical image may be defined in a cube projection format to provide an image that is remapped to the six faces of the cube.

Overview A system of one or more computers performs a specific operation or action by having software, firmware, hardware, or combinations thereof installed on the system that causes the system to perform an action during operation Can be configured as follows. One or more computer programs may be configured to perform a particular operation or action by including instructions that, when executed by a data processing device, cause the device to perform an action.

  In one general aspect, these instructions include a step of determining regions in a plurality of images captured using a plurality of cameras for converting two-dimensional data to three-dimensional data. It may include a method to be realized. The step of determining a region for converting the two-dimensional data into the three-dimensional data may be automatically performed based at least in part on a user input detected by the head mounted display. User input may include head rotation, and the 3D data may be used to generate a 3D portion in at least one of the plurality of images corresponding to the view. In another example, the user input may include a change in gaze direction, and the three-dimensional data is used to generate a three-dimensional portion in at least one of the plurality of images on the user's line of sight. Also good.

  The method may also include calculating a depth value for a portion of the pixels in the region and generating a spherical image. The spherical image may include image data for some of the pixels in the region. In some implementations, the portion of the pixel is represented on the surface of the computer graphics object with a radius equal to the corresponding depth value associated with one or more of the portion of the pixel in the region. The method also includes using the image data to construct a three-dimensional surface in a three-dimensional space of the computer graphics object generated by the image processing system; and using the image data, the surface of the computer graphics object Generating a texture mapping to. Texture mapping may include the mapping of image data to the surface of a computer graphics object. The method may also include transmitting a spherical image and texture mapping for display on a head mounted display device. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of these methods.

  In some implementations, the method also generates a left eye view by combining the step of generating additional spherical images and texture mappings associated with the region with a portion of the image data and the spherical image. Generating a right eye view by generating additional image data and combining the additional image data and the additional spherical image, and displaying a left eye view and a right eye view on the head mounted display device And a step of performing. The image data may include depth value data and RGB data for at least some of the pixels in the region.

  In some implementations, the plurality of images includes video content, and the image data includes RGB data and depth value data associated with a portion of the pixels. In some implementations, the method further includes using the image data to convert a two-dimensional version of the region to a three-dimensional version of the region and displaying the region on a head-mounted display device. Providing a three-dimensional version of In some implementations, multiple images are captured using multiple cameras mounted on a spherical camera rig.

  In another general aspect, the method includes obtaining a plurality of images using a plurality of cameras and generating at least two update images for the plurality of images, wherein the at least two update images are predefined. A computer generated by interpolating a viewpoint for at least one virtual camera configured to capture content with a left offset from a centerline and capture content with a right offset from a predefined centerline The method realized by is described. In some implementations, interpolating viewpoints involves sampling a plurality of pixels in a plurality of images, generating virtual content using optical flow, and out of at least two updated images. Placing virtual content in at least one of the interiors.

  The method further includes mapping the first image in the at least two updated images to the first sphere to generate a first spherical image for provision to the left eyepiece of the head mounted display. Mapping a second image in at least two updated images to a second sphere to generate a second spherical image for provision to the right eyepiece of the head mounted display; Displaying the first spherical image on the left eyepiece and displaying the second spherical image on the right eyepiece of the head-mounted display.

  In some implementations, the at least one virtual camera is configured to use content captured using one or more physical cameras and to adapt the content to be provided from a viewpoint. . In some implementations, the mapping of the first image includes applying a texture to the first image by assigning pixel coordinates from the first image to the first sphere, The mapping includes applying a texture to the second image by assigning pixel coordinates from the second image to the second sphere. In some implementations, the at least two spherical images are an RGB image having at least some of the plurality of pixels included in the content captured with the left offset and a content captured with the right offset. And an RGB image having at least a part of the plurality of included pixels. In some implementations, the left and right offsets can be modified and are functional to adapt the display accuracy of the first and second images on the head mounted display.

  Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of these methods.

  The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

1 is a block diagram of an example system 100 for capturing, processing, and rendering 2D and 3D content for virtual reality (VR) space. FIG. 6 illustrates an exemplary spherical camera rig configured to capture an image of a scene for use in generating a 3D portion of video content. FIG. 6 illustrates an exemplary icosahedron camera rig configured to capture an image of a scene for use in generating a 3D portion of video content. FIG. 6 illustrates an exemplary hexagonal sphere camera rig configured to capture an image of a scene for use in generating a 3D portion of video content. 2 is a flowchart illustrating one embodiment of a process for generating video content. FIG. 11 illustrates an example of a computing device and a mobile computing device that can be used to implement the techniques described herein.

Like reference symbols in the various drawings indicate like elements.
DETAILED DESCRIPTION Obtaining image content that can be used to accurately reproduce each part of a scene in 2D and / or 3D is generally achieved using multiple cameras housed in a 3D camera rig. Including capturing images or videos. These cameras may be configured to capture each portion of the scene that surrounds the camera rig at the top, sides, and bottom and any scene content shown therebetween. The systems and methods described in this disclosure can employ camera rigs that are spherical, icosahedral, or 3D polygonal, to name just a few. Such camera rigs include several groups (eg triads) strategically placed on the rig to capture image content for all outwardly captureable areas surrounding the rig. ) Cameras can be housed. The image content may include duplicate image content captured between multiple cameras, and this duplication later generates additional image content and stitches existing image content or visual effects in the image content. (Eg 3D effect) can be used to generate.

  One such visual effect may include generating a 3D region in the image content. Generating 3D regions in image content (eg, video content) at runtime (or near real time) may be accomplished using content captured using the systems and methods described herein. This is because such a camera rig is configured to capture all the areas surrounding a sphere or other 3D shape configured to accommodate the camera. Having access to all possible viewing content in the scene allows calculation for depth, which can be used to modify 2D content back to 3D content. An example of 3D region generation may include determining that a particular area should be shown in 3D based on the object or action shown in the image content.

  For example, if the image content represents an acrobatic performance shown from the ceiling, the systems and methods described in this disclosure can be used to display content above the user in the VR space, for example, when the user looks at the ceiling of the VR space. It can be determined that the 3D effect should be applied. The 3D effect can be automatically applied to image content (eg, video content) and displayed to the user on a VR head mounted display (HMD) device. In some implementations, the 3D effect may be manually configured to shift the area for applying the 3D effect. For example, the 3D effect in the acrobat example may be shifted from stage to ceiling if the performance is scheduled to move from the main stage to the ceiling. That is, the image content (eg, video content) can be configured to shift the 3D effect to the ceiling when the acrobat begins to be performed above the user (eg, the audience). In some implementations, the image 3D effect can be applied to a portion of the image content, while the surrounding portion of the image content remains in a two-dimensional format. In some implementations, the system described herein can be used to apply 3D effects to an entire image or video. In other implementations, the systems described herein are 3D to a scene, a portion of a scene, an image / area in a scene, a user selected or a portion of an image / scene selected by the user. Can be used to apply effects.

  The two-dimensional (2D) to three-dimensional (3D) modification of the image content can be performed because the spherical camera rig is used to capture images from all angles around the spherical camera rig, and this Thus, all possible areas are 3D adjustable. Automatic adjustment may include calculating a dense optical flow to calculate a depth map associated with the image content. Computing the depth map may include computing a number of depth values that represent the distance of various points in the scene relative to the position of the camera. In some examples, two or more images are used to calculate depth values and these depth values are used in addition to 2D image data to estimate 3D image data for a particular scene portion. be able to.

  In some implementations, texture mapping to an object may generate a two-dimensional effect that maps two-dimensional data onto the object. In other implementations that send depth map data (or depth data) and texture map data to the depth plane, the effect may be a three-dimensional effect.

  The systems and methods described in this disclosure may include using an optical flow algorithm, depth map calculation, user input, and / or supervisory input to generate a 3D region in the image content. For example, the described systems and methods can apply 3D effects to selected areas of image content. The 3D effect is calculated strategically and can be placed in near real time within image content such as video content. In some implementations, the 3D effect can be manually installed prior to providing image content in a virtual reality (VR) space. In some implementations, the 3D effect may be placed automatically while the user is viewing image content in VR space, for example, in response to the user facing toward the area of interest. For example, if image content captured using the devices described herein is provided to the user in VR space, the user may turn to an area to view the content in VR space, In response to showing interest in the content, the content can be automatically generated as 3D content.

  In certain implementations, the systems and methods described herein include calculating a dense optical flow field between many triad cameras on a spherical camera rig to compose and display 3D image content. obtain. The calculation and transmission of the flow field (using optical flow interpolation techniques) can be done to reconstruct (at or prior to runtime) the specific 3D view that the user wants to see. These techniques can take into account the tilt and translation of the user's head and allow 3D content to be provided in any selectable area in the scene captured by the spherical camera rig. Good. In some implementations, anterior and posterior head translation may also be performed.

  In some implementations, the systems and methods described herein can employ optical flow and / or stereo matching techniques to obtain a depth value for each pixel of the image. A spherical image (or video) generated using optical flow and / or stereo matching techniques can be transmitted to an HMD device, for example. The spherical image may include RGB (red green blue) pixel data, YUV (lumen and chrominance) data, depth data, or additional calculated or obtainable image data. The HMD device can receive such data and render the image as a texture mapped onto the surface in 3D space defined by the depth component.

  In some implementations, the systems and methods described herein can interpolate many different virtual cameras using optical flow techniques. The resulting optical flow data can be used to generate at least two spherical images (eg, a left RGB spherical image and a right RGB spherical image). Pixels in the left RGB spherical image can be acquired from a virtual camera offset to the left, and pixels in the right RGB spherical image can be acquired from a virtual camera offset to the right. In order to generate accurate 3D effects, the systems and methods described herein can modify the amount of left and right offsets used in a virtual camera. That is, selecting the maximum offset may function to provide accurate 3D image content based on content in the image or video, or based on input from the director. The left and right images can then be textures mapped onto a (constant radius) sphere, for example in an HMD device.

  FIG. 1 is a block diagram of an exemplary system 100 for capturing, processing, and rendering 2D and 3D content for virtual reality (VR) space. In the exemplary system 100, a spherically shaped camera rig 102 may capture still and video images and provide them to the image processing system 106 for analysis and processing either directly through the network 104 or alternatively. Once the image is captured, the image processing system 106 performs a number of calculations and processes on the image, for example providing the processed image to the head mounted display (HMD) device 110 through the network 104 for rendering. obtain. In some implementations, the image processing system 106 may also provide processed images to the mobile device 108 and / or to the computing device 112 for rendering, storage, or further processing.

  The HMD device 110 may represent a virtual reality headset, glasses, eyepiece, or other wearable device that can display virtual reality content. In operation, the HMD device 110 may execute a VR application (not shown) that can play the received and / or processed images to the user. In some implementations, the VR application may be hosted by one or more of the devices 106, 108, or 112 shown in FIG. In one example, the HMD device 110 can generate portions of the scene as 3D video content and can provide video playback in a 3D format of the scene captured by the camera rig 102 at strategically selected locations. .

  The camera rig 102 may be configured for use as a camera (which may also be referred to as a capture device) and / or processing device for collecting image data to render content in VR space. Although the camera rig 102 is shown here as a block diagram described with particular functionality, the rig 102 can take any of the implementations shown in FIGS. In addition, the functionality described for camera rigs may be provided throughout this disclosure. For example, to simplify the description of the functionality of the system 100, FIG. 1 shows a camera rig 102 in which no camera for capturing images is placed around the rig. Other implementations of the camera rig 102 may include any number of cameras that may be placed at any point on a 3D camera rig, such as the rig 102.

  As shown in FIG. 1, the camera rig 102 includes a number of cameras 130 and a communication module 132. The camera 130 may include a still camera or a video camera. In some implementations, the camera 130 may include multiple still cameras or multiple video cameras positioned (eg, seated) along the surface of the spherical rig 102. The camera 130 may include a video camera, an image sensor, a stereoscopic camera, an infrared camera, and / or a mobile device camera. The communication system 132 may be used to upload and download images, instructions, and / or other camera related content. The communication system 132 may be wired or wireless and can be interfaced through a private network or a public network.

  The camera rig 102 may be configured to function as a stationary rig or a rotating rig. Each camera on the rig is arranged (eg, installed) offset from the rotation center of the rig. The camera rig 102 may be configured to rotate 360 degrees, for example, to sweep and capture all or part of a 360 degree spherical view of the scene. In some implementations, the rig 102 can be configured to operate in a stationary position, and in such a configuration, an additional camera is added to the rig to capture an additional outward angle view of the scene. But you can.

  In some implementations, the camera may be configured (eg, set up) to function synchronously to capture video from the camera on the camera rig at a particular point in time. In some implementations, the cameras may be configured to function synchronously to capture specific portions of the video from one or more of the cameras over a period of time. Another example of calibrating a camera rig may include configuring how received images are stored. For example, received images can be stored as individual frames or videos (eg, avi files, mpg files), and such stored images can be uploaded to the Internet, another server or device, or on camera rig 102 Each camera can be stored locally.

  The image processing system 106 includes an interpolation module 114, an optical flow module 116, a stitching module 118, a depth map generator 120, and a 3D generator module 122. Interpolation module 114 represents an algorithm that can be used, for example, to sample a portion of a digital image and video and determine a number of interpolated images likely to occur between adjacent images captured from camera rig 102. In some implementations, the interpolation module 114 may be configured to determine interpolated image fragments, image portions, and / or vertical or horizontal image strips between adjacent images. In some implementations, the interpolation module 114 may be configured to determine a flow field (and / or flow vector) between related pixels in neighboring images. The flow field can be used to compensate for both transformations the image has undergone and processing of the transformed image. For example, the flow field can be used to compensate for the transformation of a particular pixel grid of the acquired image. In some implementations, the interpolation module 114 can generate one or more images that are not part of the captured image by interpolation of the surrounding image, and the captured image Can be interleaved to generate additional virtual reality content for the scene. For example, the interpolation module 114 reconstructs views from virtual cameras between real (eg, physical) cameras and selects the central ray of each view to create one virtual camera image from the center of the sphere. May provide 2D (planar) photo / video sphere stitching.

  The optical flow module 116 may be configured to calculate a dense optical flow between the cameras of each triad. For example, the module 116 may calculate a three-way paired optical flow between a pair of cameras forming a triangle on a spherical camera rig. The optical flow module 116 calculates optical flows between the first camera and the second camera, between the second camera and the third camera, and between the third camera and the first camera. Can do. Each pair of cameras used in the calculation can be considered a stereo pair. In some implementations, optical flow computations may be performed between a pair of cameras when the flow vectors are oriented in any direction to create a 2D quantity or configuration. In some implementations, the optical flow calculation may be performed when the flow vector is limited to one dimension (eg, a horizontal stereo pair where the flow is horizontal).

  Using a spherically shaped camera rig (or other 3D-shaped rig described herein) with multiple triad cameras around the surface of the camera rig, the optical flow module 116 allows the precise scene to surround the rig. Can be generated. For example, the optical flow module 116 may calculate an optical flow field for the particular captured image content and access the stitching module 118 to stitch together a planar panorama for the scene. This may reduce artifacts in the video content. Generation of the planar panorama may include presenting the same image to both eyes of the user. This may seem 2D to the user. In some implementations, the stitching module 118 can stitch together stereoscopic panoramas that can provide a unique and different image for each eye associated with the user, such images being displayed to the user in 3D. May be visible. As used herein, 3D content may be considered stereoscopic presentation content and may show a texture mapped onto the depth plane. Similarly, 2D content may be considered as presentation content in a plan view showing a texture mapped on, for example, a plane or a sphere.

  In some implementations, module 114 and splicing module 118 may instead employ non-centered rays to introduce a panoramic twist or, for example, to introduce a 3D effect in a selected direction. Can be used to generate stereospherical pairs. The panorama twist uses the light beam deflected in the first direction to capture the light beam for the first eye (left eye) and uses the light beam deflected in the opposite direction to generate the light beam for the second eye (right eye). Including capturing.

  In general, the optical flow module 116 uses optical flow techniques to calculate the optical flow between adjacent pairs of cameras in a spherical population of cameras, thereby providing accurate mono and stereo spherical panoramas (eg, omnidirectional stereo or mega A panorama twist for a stereo panorama). A group of cameras may be constrained by camera placement so that each point in space appears to be at least three cameras.

  In some implementations, the camera rig described herein can produce artifacts (eg, stitching errors / artifacts, object discontinuities on the camera boundary, missing data at the boundary, or double near the boundary. Image content, ripped objects, distorted objects, removed content, etc.) may be provided with the advantage of reducing or eliminating. Artifacts can be removed particularly well for video content representing video content. Such artifact removal is possible based on the use of a spherical camera rig with a triad camera that contains duplicate video / image content, where the duplicate video / image content accesses the duplicate image area captured by the camera. However, it can be used to correct splicing errors / artifacts by performing an optical flow approach and recalculating the image area that appears to have provided the artifact / error.

  The systems and methods described herein may be used to generate stereo 3D content at any point that can be captured around a 3D spherical shaped camera rig (or other 3D shaped camera rig). Such widely-acquired content can be obtained by mathematical methods that strategically place stereo 3D effects / video content anywhere in the still or video content while removing 3D elsewhere. Or it does not provide 3D effects and allows to save streaming bandwidth, processing power and / or storage space.

  Depth map generator 120 accesses optical flow data (eg, a flow field) for images captured using camera rig 102 and uses such flow data to calculate a depth map for captured image content. Can do. For example, the depth map generator 120 may use image data from many cameras on the rig 102 pointing in various directions. The depth map generator 120 may access and employ a stereo matching algorithm to calculate a depth value for each pixel represented in the captured image. Views from various cameras and depth values can be combined into one spherical image with R (red), G (green), B (blue) and depth values for each pixel. In the viewer, the depth map generator 120 generates a texture map of the RGB image to the surface in 3D space that is constructed by obtaining a depth value at every pixel so that each point of the sphere has a radius equal to the depth value. You can do it. This approach may differ from a 3D spherical image approach that typically uses stereo pairs rather than depth values and / or depth maps.

  In general, the depth map generator 120 generates a depth map that is transmitted with the spherical image. Sending a depth map with image content has the advantage of allowing the user to view 3D content in all directions, including the poles (eg, north above the user and south below the user). Can be provided. In addition, sending the depth map along with the image content can also allow the user to tilt his head and still see the 3D effect. In one example, since depth information is transmitted with the image content, the user may be able to move a small distance from his nominal location (eg, in the X, Y, and / or Z directions) You may be able to see it moving in the right way with the proper parallax. User movement in the VR space may be referred to as actual movement, and the system 100 can track the user position.

  The calculated optical flow data (including light field transmission data) is combined with the spherical video data and sent to the HMD device (or other device) for left and right access for users accessing the HMD device. Can generate a view of In some implementations, the depth map generator 120 may provide separate and separate spherical images and RGB data for each eye.

  In some implementations, optical flow interpolation can be performed by a computer system in communication with the HMD device 106 and specific image content can be sent to the HMD device. In other implementations, interpolation can be performed locally at the HMD device 106 to modify the 3D image content for display. The flow data can be used to generate left and right views for the left and right eyes accessing the HMD device 106. Interpolation can be performed by the HMD device 106. This is because the system 106 provides combined data (eg, spherical video data and optical flow data) to the runtime.

  In some implementations, the 3D generator module 122 uses the optical flow data and depth map data to generate 3D regions in the image content and provides such 3D effects to the user in VR space. . The 3D effect can be triggered to be installed manually or automatically. For example, a 3D aspect of specific image content may be configured after capture in post-processing at the director's decision. In particular, the director can determine that the scene in his VR space can be configured to provide a sequence of airplanes and helicopters that are simulated in the VR space such that airplanes and helicopters fly over the user's head. The director may access a set of tools including a 3D generator tool (not shown) to apply 3D effects to the video content. In this example, the director can determine that the user will look up in the sky if he or she hears the noise of an airplane or helicopter and adjusts the video image content using a 3D generator tool to provide the airplane and helicopter as 3D content it can. In such an example, the director may provide less use to the user when other surrounding video content is provided as 3D content, as the user may be looking up at the sky until helicopters and airplanes pass by. It can be judged that it may not. Thus, the director can configure the video content to adjust the 3D effect from the sky to another area in the video content when the end of the sequence involving helicopters and airplanes is scheduled.

  Manually selecting a portion of the image content that includes a 3D effect may be triggered, for example, by a VR movie director. The director may configure the image content based on the content or based on the desired user response. For example, a director may want to focus the user's attention somewhere in the content, and to name just a few examples, so as to provide useful access to data, an artistic vision, or a smooth transition. can do. The director can preconfigure 3D changes in the image content and adjust the time that such changes are displayed to the user in the VR space.

  Automatically selecting the portion of the image content that includes the 3D effect may include using user input to trigger the effect. For example, the system 100 can be used to trigger a 3D effect to appear in the image content based on the detected head tilt of a user accessing the content in VR space. Other user movements, content changes, sensors, and location-based effects can be used as input to trigger specific application or removal of 3D effects. In one example, a concert on stage can be represented in 3D in VR space, while the user does not first look back during the concert, so the audience behind the user accessing the concert may remain 2D. . However, if the user chooses to look back, the 3D effect can be shifted from stage / concert image content to audience image content.

  In exemplary system 100, devices 106, 108, and 112 may be laptop computers, desktop computers, mobile computing devices, or game consoles. In some implementations, the devices 106, 108, and 112 may be mobile computing devices that may be located (eg, installed / located) within the HMD device 110. The mobile computing device may include a display device that can be used as a screen for the HMD device 110, for example. Devices 106, 108, and 112 may include hardware and / or software for executing VR applications. In addition, devices 106, 108, and 112 may be installed on HMD device 110 when they are installed in front of HMD device 110 or held within a range of positions relative to HMD device 110. It may include hardware and / or software capable of recognizing, monitoring and tracking 3D movement. In some implementations, devices 106, 108, and 112 may provide additional content to HMD device 110 over network 104. In some implementations, the devices 102, 106, 108, 110, and 112 may be connected / interfaced with one or more of each other paired or connected via the network 104. This connection may be wired or wireless. The network 104 may be a public communication network or a private communication network.

  System 100 may include an electronic storage device. The electronic storage device may include a non-transitory storage medium that stores information electronically. The electronic storage device may be configured to store captured images, acquired images, pre-processed images, post-processed images, and the like. Images captured using any of the disclosed camera rigs can be processed and stored as one or more streams of video, or stored as individual frames. In some implementations, storage occurs at the time of capture and rendering occurs immediately after the portion of the capture, allowing faster access to panoramic stereo content faster than if capture and processing were not simultaneous.

  FIG. 2 is a diagram illustrating an exemplary spherical camera rig 200 configured to capture an image of a scene for use in generating a 3D portion of video content. Camera rig 200 includes a number of cameras 202, 204, 206, 208, 210, 212, 214, 216, and 218. Cameras 202-218 are shown attached to a spherical rig. Additional cameras for other angles of the sphere are not shown in FIG. 2, but are configured to collect image content from such other angles. Cameras 202-218 are arranged so that each of the three cameras can work together to capture image content for each point / area surrounding the sphere. Capturing each point / area includes capturing a still or video image of the scene surrounding the rig 200. Cameras 202-218 may be placed against a sphere (or other shaped rig). In some implementations, the cameras 202-218 (and / or more or fewer cameras) may be placed at an angle with respect to the sphere to capture additional image content.

  In one non-limiting example, the cameras 202, 204, and 206 can be arranged to capture an image of the area of the scene surrounding the sphere. The captured images can be analyzed and combined (eg, stitched together) to form a visible scene for the user in VR space. Similarly, images captured using camera 204 may be combined with images captured using cameras 202 and 208 to provide another area of the visible scene. Images captured using cameras 202, 208, and 210 can be combined in the same manner as cameras 206, 212, and 214. A wider space between the cameras may also be possible. For example, images captured using cameras 210, 212, and 216 can be combined to provide image content for a scene (eg, a point) that is visible from half of the hemisphere of rig 200. Similar combinations can be made using cameras 202, 212, and 218 to provide a visible image from another half of the hemisphere of sphere 200. In some implementations, the diameter 220 of the camera rig 200 may be anywhere from about 0.15 meters to about 1.5 meters. In one non-limiting example, the diameter 220 is about 0.2 to about 0.9 meters. In another non-limiting example, the diameter 220 is about 0.5 to about 0.6 meters. In some implementations, the distance between cameras can be from about 0.05 meters to about 0.6 meters. In one non-limiting example, the distance between cameras is approximately 0.1 meters.

  In some implementations, a collection of cameras can be placed on such a spherical camera rig (or other 3D-shaped rig) in many directions to capture each point in space. That is, each point in space may be captured by at least three cameras. In one example, many cameras can be placed on a sphere as close as possible (eg, at each corner of the icosahedron, at each corner of the geodesic dome, etc.). A number of rig configurations are described below. Each configuration described in this disclosure may be configured with the aforementioned or other diameters and inter-camera distances.

  Referring to FIG. 3, an icosahedron camera rig 300 is shown. Many cameras can be mounted on the camera rig 300. The cameras can be placed at triangular points in the icosahedron, as illustrated by cameras 302, 304, and 306. Alternatively or in addition, the camera may be placed in the center of an icosahedron triangle, as illustrated by cameras 308, 310, 312, and 314. Cameras 316, 318, 320, 322, and 324 are shown around the sides of the icosahedron. Additional cameras may be included around the icosahedron. The camera spacing and diameter 326 may be configured similarly to other camera rigs described throughout this disclosure. In some implementations, the camera may be placed tangential to the camera rig. In other implementations, each camera may be installed at various angles relative to the camera rig.

  The camera rig 300 may be stationary or may be configured with cameras 302 to 324 having a wide field of view. For example, the cameras 302-324 can capture a field of view from about 150 degrees to about 180 degrees. Cameras 302-324 may have fisheye lenses to capture a wider view. In some implementations, adjacent cameras (eg, 302 and 320) can function as a stereo pair, with a third camera 306 paired with each of the cameras 302 and 320, and the cameras 302, 306, and 320. A stereo triad camera can be generated from which the optical flow can be calculated from the images obtained from. Similarly, among other camera combinations not shown in FIG. 3, the following cameras may generate combinable images to generate 3D images: (cameras 302, 312, and 324), (cameras 302, 304 and 324), (cameras 304, 316, and 324), (cameras 302, 306, and 320), (cameras 304, 316, and 318), (cameras 304, 306, and 318), and (camera 310) , 312 and 314).

  In some implementations, the camera rig 300 (and other cameras described herein) may be configured to capture an image of a scene, such as the scene 330. The image may include a portion of the scene 330, a video of the scene 330, or a panoramic video of the scene 330. In operation, the system described herein can retrieve such captured images, process the content, and display specific regions within the captured images in a three-dimensional format. For example, the system described herein can determine regions within a plurality of images captured using a plurality of cameras for converting 2D data to 3D data. Exemplary regions include regions 332, 334, and 336. Such an area may be selected by the user, selected by the director, or automatically selected. In some implementations, the region may be selected after the image is captured and during display of the image on the HMD device. Other regions can be selected throughout the scene 330, with regions 332, 334, and 336 representing exemplary regions. Region 332 shows the region captured by rig 300 using capture paths 338, 340, and 342.

  Referring to FIG. 4, a hexagonal spherical camera rig 400 is shown. Many cameras can be mounted on the camera rig 400. The cameras may be placed at hexagonal points or along hexagonal sides, as illustrated by cameras 402, 404, and 406. Alternatively or additionally, the camera can be placed in the center of the hexagon. Additional cameras may be included around the hexagonal spherical camera rig 400. The camera spacing and diameter 408 may be configured similarly to other camera rigs described throughout this disclosure.

  FIG. 5 is a flowchart illustrating one embodiment of a process 500 for providing an area of 3D image content to a user accessing a VR space. Process 500 can use the captured image to retrieve and / or calculate image data, including but not limited to RGB data, which can be used to generate a subset of pixels in a region of the image. Depth value data associated with can be calculated. The system can use the image data to convert a two-dimensional version of the region into a three-dimensional version of the region to provide a three-dimensional version of the region for display on a head mounted display device.

  At block 502, the system 100 may determine regions in the plurality of images captured using the plurality of cameras for converting the two-dimensional data to the three-dimensional data. The plurality of images may be still images, videos, image portions, and / or video portions. In some implementations, the plurality of images may include video image content captured using a number of cameras mounted on a spherically shaped camera rig. In some implementations, the step of determining an area for converting 2D data to 3D data is based at least in part on user input detected on a head mounted display such as display device 106. Done automatically. User input may include head rotation, gaze direction change, hand gesture, location change, and the like. In some implementations, determining the region may occur manually based on the VR movie director's choice to provide a 3D region within a particular video or image.

  At block 504, the system 100 may calculate depth values for some of the pixels in the region. In some implementations, the system 100 may calculate a depth value for each pixel in the region. The step of calculating a depth value may include comparing a number of regions captured by a plurality of cameras. For example, three images of region 332 may be captured by three cameras (eg, cameras 304, 316, and 324) at different angles with respect to region 332. System 100 may compare pixel intensities and locations in the three images to determine pixel intensity accuracy. Using the comparison, a depth value can be calculated for one or more pixels in region 332. Other reference objects can be compared in the scene to confirm the accuracy of the pixel intensity.

  At block 506, the system 100 may generate a spherical image. Generating the spherical image may include calculating a spherically formatted version of the image using the image data.

  At block 508, the system 100 may use the image data to construct a three-dimensional surface in a three-dimensional space of computer graphics objects generated by the image processing system. For example, a portion of the pixels may be represented on the surface of the computer graphics object with a radius equal to the corresponding depth value associated with one or more of the portions of the pixels in the region. Computer graphics objects may be spheres, icosahedrons, triangles, or other polygons.

  At block 510, the system 100 may use the image data to generate a texture mapping to the surface of the computer graphics object. Texture mapping may include mapping image data to the surface of a computer graphics object. At block 512, the system 100 may transmit a spherical image and texture mapping for display on a head mounted display device.

  At block 512, the system 100 may transmit a spherical image and texture mapping for display on a head mounted display device. In some implementations, the process 500 generates additional spherical images and texture mapping for the region, and generates a left eye view by combining a portion of the image data with the spherical image. May be included. Process 500 may additionally include generating additional image data and generating a right eye view by combining the additional image data and the additional spherical image. Process 500 may additionally include displaying a left eye view and a right eye view on the head mounted display device. In some implementations, the image data includes depth value data and RGB data for at least some of the portions of the pixels in the region.

  The display may include 3D image content in the area. The method also generates an additional spherical image and texture mapping, generates a left eye view by combining a portion of the depth values with the RGB data and the spherical image, generates an additional depth value, Generating a right eye view by combining the additional depth values with the updated RGB data and the additional spherical image; and displaying the left eye view and the right eye view on the head mounted display device. Also good.

  In some implementations, the systems described herein may be configured to acquire images using any number of cameras. For example, cameras 402, 404, and 406 (FIG. 4) can be used to capture specific images. The system described herein may use at least one of the captured images to generate at least two updated images for provision to a head mounted display device. The updated image may be configured to provide 2D or 3D content. The 3D content can be composed of parts of the updated image or all of the updated image. The updated image may be generated using a virtual camera viewpoint generated from images captured from physical cameras, such as cameras 402, 404, and 406, for example. The viewpoint may relate to one or more offsets selected to provide specific 3D content in a specific region of the image.

  In some implementations, the updated image includes image data generated from a specific offset. For example, an updated image may include image content in which some of the pixels in the content are captured from an offset that faces the left side of one or more cameras 402, 404, or 406. Another updated image may include image content in which some of the pixels in the content are captured from an offset that faces the right side of one or more cameras 402, 404, or 406.

  In general, the updated image may include offset image content, virtual content, content from various camera angles, manipulated image content, and combinations thereof. In some implementations, the updated image may be generated by interpolating the viewpoint of at least one virtual camera. Interpolation samples a plurality of pixels in a captured image, generates virtual content using optical flow, and adapts the virtual content to be placed inside at least one of the updated images. And may be included.

  The virtual camera may be configured to capture content with a left offset from a predefined centerline and capture content with a right offset from a predefined centerline. The left and right offsets can be modified and may be functional to adapt the image for accurate display on a head mounted display.

  A virtual camera may be configured to utilize content captured using one or more physical cameras and to adapt the content to be provided from an interpolated viewpoint. In particular, the virtual camera may be adapted to capture any offset (angle) generated by one or more physical cameras. The offset may define the viewpoint. The offset may be defined from the center line of the physical camera or from the center line defined between the two physical cameras. The content interpolation can be adjusted to produce content with any offset from either centerline, and the amount and direction of the offset is accurate for the 3D effect in the images provided in the HMD display device. It can be selected to ensure depiction.

  Upon generating at least two updated images, the system described herein converts the first image to the first spherical surface to generate a first spherical image for provision to the left eyepiece of the head mounted display. May be configured to map to. Similarly, the system described herein is configured to map a second image to a second sphere to generate a second spherical image for provision to the right eyepiece of the head mounted display. May be. The mapping of the first image and the mapping of the second image may include applying a texture to the first image and the second image. Applying the texture includes assigning pixel coordinates from the first image to the first sphere, and assigning pixel coordinates from the second image to the second sphere, as described in detail above. Also good.

  FIG. 6 illustrates an example of a general purpose computing device 600 and a general purpose mobile computing device 650 that may be used with the techniques described herein. The computing device 600 includes a processor 602, a memory 604, a storage device 606, a high speed interface 608 connected to the memory 604 and a high speed expansion port 610, and a low speed bus 614 and a low speed interface connected to the storage device 606. 612. Each of the components 602, 604, 606, 608, 610, and 612 are interconnected using various buses and may be optionally mounted on a common motherboard or in other manners. The processor 602 is capable of processing instructions that are executed within the computing device 600, which instructions display graphics information for the GUI on an external input / output device such as a display 616 coupled to the high speed interface 608. Includes instructions stored in memory 604 or on storage device 606 for display. In other implementations, multiple processors and / or multiple buses may be used as appropriate with multiple memories and multiple types of memory. In addition, multiple computing devices 600 may be connected, each device providing a portion of the required operation (eg, as a server bank, a group of blade servers, or a multiprocessor system).

  Memory 604 stores information within computing device 600. In one implementation, the memory 604 is one or more volatile memory units. In another implementation, the memory 604 is one or more non-volatile memory units. The memory 604 may also be another form of computer readable media such as a magnetic disk or optical disk.

  Storage device 606 can provide mass storage for computing device 600. In one implementation, the storage device 606 is a floppy disk device, hard disk device, optical disk device, or tape device, flash memory or other similar solid state memory device, or storage area network or other configuration. Or may be a computer readable medium, such as an array of devices including the device. A computer program product may be tangibly embodied in an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable or machine-readable medium, such as memory 604, storage device 606, or memory on processor 602.

  The high speed controller 608 manages bandwidth intensive operations for the computing device 600, while the low speed controller 612 manages lower bandwidth intensive operations. Such assignment of functions is exemplary only. In one implementation, the high speed controller 608 is coupled to the memory 604, the display 616 (eg, via a graphics processor or accelerator), and a high speed expansion port 610 that can accept various expansion cards (not shown). Is done. In this implementation, the low speed controller 612 is coupled to the storage device 606 and the low speed expansion port 614. A low-speed expansion port that can include various communication ports (eg, USB, Bluetooth, Ethernet, wireless Ethernet) to one or more input / output devices such as a keyboard, pointing device, scanner, or May be coupled to a networking device, such as a switch or router, via a network adapter, for example.

  The computing device 600 may be implemented in many different forms as shown. For example, it may be implemented multiple times as a standard server 620 or in a group of such servers. It may also be implemented as part of the rack server system 624. In addition, it may be implemented on a personal computer such as a laptop computer 622. Alternatively, components from computing device 600 may be combined with other components in a mobile device (not shown), such as device 650. Each such device may include one or more of the computing devices 600, 650, and the entire system may be comprised of multiple computing devices 600, 650 communicating with each other.

  Computing device 650 includes, among other components, processor 652, memory 664, input / output devices such as display 654, communication interface 666, and transceiver 668. Device 650 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 650, 652, 664, 654, 666, and 668 are interconnected using various buses, some of which are optionally on a common motherboard or otherwise. It may be mounted.

  The processor 652 can execute instructions within the computing device 650, including instructions stored in the memory 664. The processor may be implemented as a chip set of chips that include separate analog and digital processors. The processor may provide coordination between other components of the device 650, such as, for example, a user interface, applications executed by the device 650, and control of wireless communication by the device 650.

  The processor 652 may communicate with the user via a control interface 658 and a display interface 656 coupled to the display 654. The display 654 may be, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display), or an OLED (Organic Light Emitting Diode) display, or other suitable display technology. Display interface 656 may include appropriate circuitry for driving display 654 to present graphical information and other information to the user. The control interface 658 may receive commands from the user and convert them for delivery to the processor 652. In addition, an external interface 662 may be provided in communication with the processor 652 to allow near area communication between the device 650 and other devices. The external interface 662 may provide, for example, wired communication in some implementations, wireless communication in other implementations, and multiple interfaces may also be used.

  Memory 664 stores information within computing device 650. Memory 664 may be implemented as one or more of one or more computer readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 674 is also provided and may be connected to the device 650 via the expansion interface 672, and the expansion interface 672 may include, for example, a SIMM (Single In Line Memory Module) card interface. Such an extended memory 674 may provide extra storage space for the device 650 or may also store applications or other information for the device 650. Specifically, the expanded memory 674 may include instructions for performing or supplementing the above-described process, and may also include secure information. Thus, for example, the expansion memory 674 may be provided as a security module for the device 650 and may be programmed with instructions that allow the device 650 to be used safely. In addition, a secure application may be provided via the SIMM card with additional information, such as placing identification information on the SIMM card in a non-hackable manner.

  The memory may include, for example, flash memory and / or NVRAM memory as described below. In one implementation, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable or machine-readable medium, such as memory 664, expansion memory 674, or memory on processor 652, and may be received through transceiver 668 or external interface 662, for example.

  Device 650 may communicate wirelessly via communication interface 666, which may include digital signal processing circuitry as required. Communication interface 666 communicates under various modes or protocols such as, among others, GSM® voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA®, CDMA2000, or GPRS. May be provided. Such communication may occur via radio frequency transceiver 668, for example. In addition, short range communications may occur using Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 670 may provide additional navigation-related and position-related wireless data to the device 650, which data is applied to the application running on the device 650. May be used as appropriate.

  Device 650 may also communicate in voice using an audio codec 660 that can receive verbal information from the user and convert it to usable digital information. Audio codec 660 may also generate sounds audible to the user, such as via a speaker, for example, in the handset of device 650. Such sounds may include sounds from voice calls, may include recorded sounds (eg, voice messages, music files, etc.), and sound generated by applications running on device 650. May also be included.

  The computing device 650 may be implemented in many different forms as shown in the figure. For example, it may be implemented as a mobile phone 680. It may also be implemented as part of a smart phone 682, a personal digital assistant, or other similar mobile device.

  Various implementations of the systems and techniques described herein include digital electronic circuits, integrated circuits, specially designed application specific integrated circuits (ASICs), computer hardware, firmware, software, And / or a combination thereof. These various implementations may include implementations in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, the processor being dedicated. It may be general purpose or may be coupled to receive and send data and instructions to and from the storage system, at least one input device, and at least one output device.

  These computer programs (also known as programs, software, software applications or code) contain machine instructions for programmable processors, in high level procedural and / or object oriented programming languages, and / or assemblies / machines. Can be implemented in a language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus, and / or device used to provide machine instructions and / or data to a programmable processor. Or refers to a device (eg, magnetic disk, optical disk, memory, programmable logic device (PLD)), and includes machine-readable media that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

  In order to provide user interaction, the systems and techniques described herein provide a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display)) for displaying information to the user. Display) monitor) and a computer having a keyboard and pointing device (eg, a mouse or trackball) that allows the user to provide input to the computer. Other types of devices can also be used to provide interaction with the user, for example, the feedback provided to the user is any form of sensory feedback (eg, visual feedback, audio feedback, or tactile feedback). The input from the user may be received in any form including acoustic, voice, or haptic input.

  The systems and techniques described herein include a back-end component (eg, as a data server), or include a middleware component (eg, an application server), or a front-end component (eg, the system and user described herein) In a computing system comprising a graphical user interface or a client computer having a web browser that enables interaction with an implementation of the technique, or comprising any combination of such backend, middleware, or frontend components Can be realized. The components of the system can be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

  The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship between the client and the server is caused by computer programs that are executed on the respective computers and have a client-server relationship with each other.

  In some implementations, the computing device shown in FIG. 6 may include a sensor that interfaces with a virtual reality (VR headset 690). For example, one or more sensors included on the computing device 650 shown in FIG. 6 or other computing devices can provide input to the VR headset 690 or can generally provide input to the VR space. Sensors can include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 650 can use these sensors to determine the absolute position and / or detected rotation of the computing device in the VR space, which can then be used as an input to the VR space. For example, the computing device 650 may be incorporated into the VR space as a virtual object such as a controller, a laser pointer, a keyboard, or a weapon. Positioning by a user of a computing device / virtual object when incorporated in a VR space may allow a user to position the computing device to view the virtual object in some manner in the VR space. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device as if it were an actual laser pointer. The user can use the device in a manner similar to using a laser pointer, for example, moving the computing device left and right, up and down, and circularly.

  In some implementations, one or more input devices included on or connected to the computing device 650 may be used as input to the VR space. Input device can be touch screen, keyboard, one or more buttons, trackpad, touchpad, pointing device, mouse, trackball, joystick, camera, microphone, earphone or mini earphone with input functionality, gaming controller, or others Can be connected to, but is not limited to. A user interacting with an input device included on the computing device 650 when the computing device is incorporated into the VR space can cause certain actions to occur in the VR space.

  In some implementations, the touch screen of computing device 650 may be rendered as a touchpad in VR space. A user can interact with the touch screen of computing device 650. The interaction is rendered as movement on the rendered touchpad in VR space, for example in VR headset 690. The rendered movement can control the object in VR space.

  In some implementations, one or more output devices included on computing device 650 may provide output and / or feedback to a user of VR headset 690 in VR space. Output and feedback can be visual, tactile, or audio. Output and / or feedback can be to turn vibrations, turn one or more lights or strobe on or off or blink and / or blink, sound an alarm, sound a chime, play a song, and play an audio file Including, but not limited to. Output devices may include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes, and speakers.

  In some implementations, the computing device 650 may look like another object in a computer generated 3D environment. User interaction with computing device 650 (eg, rotating and vibrating the touch screen, touching the touch screen, swiping a finger across the touch screen) is interpreted as an interaction with an object in VR space. Can be done. In the example of a laser pointer in VR space, the computing device 650 looks like a virtual laser pointer in a computer generated 3D environment. As the user operates computing device 650, the user sees the movement of the laser pointer in VR space. The user receives feedback from the interaction with the computing device 650 in the VR space on the computing device 650 or on the VR headset 690.

  In some implementations, one or more input devices (eg, mouse, keyboard) in addition to the computing device may be rendered in a computer generated 3D environment. A rendered input device (eg, rendered mouse, rendered keyboard) can be used as rendered in VR space to control objects in VR space.

  Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 650 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be examples only, so as to limit the implementation of the invention described in this document and / or in the claims. Not intended.

  A number of embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of the specification.

  In addition, the logic flow shown in the drawings does not require the particular order or sequence shown to achieve the desired result. In addition, other steps may be provided to the described flow, or steps may be removed from the flow, other components may be added to, or removed from, the described system. May be. Accordingly, other embodiments are within the scope of the following claims.

In the following examples, further implementation examples are summarized.
Example 1: A computer-implemented method for determining regions in a plurality of images captured using a plurality of cameras for converting two-dimensional data into three-dimensional data; Computing a depth value for a portion of the pixel; generating a spherical image including image data for the portion of the pixel in the region; and computer graphics generated by the image processing system using the image data. Constructing a three-dimensional surface in a three-dimensional space of the object and generating a texture mapping to the surface of the computer graphics object using the image data, the texture mapping comprising the image data into the computer graphics Including mapping to the surface of the object. The method further comprises, for display in the head-mounted display device, comprising the step of transmitting spherical image and texture mapping method.

  Example 2: The part of a pixel is represented on the surface of a computer graphics object with a radius equal to a corresponding depth value associated with one or more of the part of the pixels in the region. Method.

  Example 3: generating additional spherical image and texture mapping associated with a region, generating a left eye view by combining a portion of the image data with a spherical image, and generating additional image data And combining the additional image data and the additional spherical image to generate a right eye view and displaying a left eye view and a right eye view on the head mounted display device, the image data comprising: The method of Example 1 or Example 2, comprising depth value data and RGB data for at least some of the portions of the pixels in the region.

  Example 4: The plurality of images includes video content, the image data includes RGB data and depth value data associated with a portion of the pixel, and the system further uses the image data to generate a two-dimensional version of the region. 4. The method of one of Examples 1-3, comprising converting to a three-dimensional version of the region and providing a three-dimensional version of the region for display on a head mounted display device.

  Example 5: The method according to one of Examples 1-4, wherein the plurality of images are captured using a plurality of cameras mounted on a spherically shaped camera rig.

  Example 6: The step of determining a region for converting 2D data to 3D data is automatically performed based at least in part on user input detected on the head mounted display, Examples 1-5 The method according to one of the above.

  Example 7: The method of example 6, wherein the user input includes head rotation and the three-dimensional data is used to generate a three-dimensional portion in at least one of the plurality of images corresponding to the view.

  Example 8: User input includes a change in gaze direction, and the 3D data is used to generate a 3D portion in at least one of the plurality of images on the user's line of sight, Example 6 or Example 7 The method described in 1.

  Example 9: A computer-implemented system comprising at least one processor and a memory for storing instructions that, when executed by at least one processor, causes the system to perform a plurality of operations, The plurality of operations include determining a region in a plurality of images captured using a plurality of cameras for converting two-dimensional data into three-dimensional data, and determining a depth value for a part of the pixels in the region. Calculating, generating a spherical image that includes image data for a portion of the pixels in the region, and using the image data in a three-dimensional space in a three-dimensional space of computer graphics objects generated by the image processing system Use the image data to construct the surface and use the image data to surface the computer graphics object. Generating texture mapping, texture mapping includes mapping image data to a surface of a computer graphics object, and the plurality of operations further includes a spherical image for display on a head mounted display device. And sending the texture mapping.

  Example 10: Generating additional spherical images and texture mapping, generating a left-eye view by combining a portion of the image data with a spherical image, generating additional image data, and generating additional images The method further includes generating a right eye view by combining the data and the additional spherical image, and displaying the left eye view and the right eye view on the head mounted display device, wherein the image data includes the pixels in the region. The system of example 9, including depth value data and RGB data for at least some of the portions.

  Example 11: Multiple images include video content, image data includes RGB data and depth value data associated with a portion of a pixel, and the system further uses the image data to generate a two-dimensional version of the region The system of example 9 or example 10, comprising converting to a three-dimensional version of the region and providing a three-dimensional version of the region for display on a head mounted display device.

  Example 12: The system of one of Examples 9-11, wherein the plurality of images are captured using a plurality of cameras mounted on a spherically shaped camera rig.

  Example 13: Determining a region for converting 2D data to 3D data is automatically performed based at least in part on user input detected on the head mounted display, Examples 9-12 The system according to one of the above.

  Example 14: The user input includes a change in gaze direction, and the 3D data is used to generate a 3D portion in at least one of the plurality of images on the user's line of sight. system.

  Example 15: A computer-implemented method comprising: obtaining a plurality of images using a plurality of cameras; and generating at least two update images for the plurality of images, wherein at least two update images By interpolating the viewpoint for at least one virtual camera configured to capture content with a left offset from a predefined centerline and capture content with a right offset from a predefined centerline The method further includes mapping the first image in the at least two updated images to the first sphere to generate a first spherical image for provision to the left eyepiece of the head mounted display. And generating a second spherical image for provision to the right eyepiece of the head mounted display For this purpose, mapping the second image in the at least two update images to the second spherical surface, displaying the first spherical image on the left eyepiece of the head-mounted display, and right eyepiece of the head-mounted display Displaying a second spherical image.

  Example 16: The at least one virtual camera is configured to use content captured using one or more physical cameras and to adapt the content to be provided from a viewpoint. the method of.

  Example 17: The mapping of the first image includes applying a texture to the first image by assigning pixel coordinates from the first image to the first sphere, wherein the mapping of the second image is the second 17. The method of example 15 or example 16, comprising applying a texture to the second image by assigning pixel coordinates from the image of the second to the second sphere.

  Example 18: Interpolating viewpoints involves sampling multiple pixels in multiple images, generating virtual content using optical flow, and within at least one of at least two update images 18. The method of one of examples 15-17, comprising installing virtual content.

  Example 19: The at least two spherical images include an RGB image having at least a part of a plurality of pixels included in the content captured at the left offset, and a plurality of pixels included in the content captured at the right offset. 19. The method of example 18, comprising an RGB image having at least a portion of

  Example 20: Of Examples 15-19, the left offset and right offset can be modified and are functional to adapt the display accuracy of the first and second images on the head mounted display The method according to one of the above.

Claims (20)

A computer-implemented method comprising:
Determining regions in a plurality of images captured using a plurality of cameras for converting two-dimensional data to three-dimensional data;
Calculating a depth value for a portion of the pixels in the region;
Generating a spherical image including image data for the portion of pixels in the region;
Using the image data to construct a three-dimensional surface in a three-dimensional space of a computer graphics object generated by an image processing system;
Generating texture mapping to a surface of the computer graphics object using the image data, the texture mapping comprising mapping the image data to the surface of the computer graphics object. The method further comprises:
Transmitting the spherical image and the texture mapping for display on a head-mounted display device.   The portion of pixels is represented on the surface of the computer graphics object with a radius equal to a corresponding depth value associated with one or more of the portions of pixels in the region. The method described in 1. Generating an additional spherical image and texture mapping associated with the region;
Generating a left eye view by combining a portion of the image data and the spherical image;
Generating a right eye view by generating additional image data and combining the additional image data and the additional spherical image;
Displaying the left eye view and the right eye view on the head mounted display device,
The method of claim 1, wherein the image data includes depth value data and RGB data for at least some of the portions of pixels in the region. The plurality of images include video content, the image data includes RGB data and depth value data associated with the portion of pixels, and the method further includes:
Using the image data to convert a two-dimensional version of the region into a three-dimensional version of the region;
Providing the three-dimensional version of the region for display on the head-mounted display device.   The method of claim 1, wherein the plurality of images are captured using a plurality of cameras mounted on a spherically shaped camera rig.   The method of claim 1, wherein the step of determining an area for converting two-dimensional data to three-dimensional data is performed automatically based at least in part on user input detected on a head mounted display. .   The method of claim 6, wherein the user input includes head rotation and the three-dimensional data is used to generate a three-dimensional portion in at least one of the plurality of images corresponding to a view. .   The user input includes a change in gaze direction, and the three-dimensional data is used to generate a three-dimensional portion in at least one of the plurality of images on a user's line of sight. the method of. A system realized by a computer,
At least one processor;
A memory for storing instructions, wherein the instructions, when executed by the at least one processor, cause the system to perform a plurality of operations, the plurality of operations comprising:
Determining regions in a plurality of images captured using a plurality of cameras for converting two-dimensional data to three-dimensional data;
Calculating a depth value for a portion of the pixels in the region;
Generating a spherical image including image data for the portion of the pixels in the region;
Using the image data to construct a three-dimensional surface in a three-dimensional space of a computer graphics object generated by an image processing system;
Generating texture mapping to a surface of the computer graphics object using the image data, the texture mapping comprising mapping the image data to the surface of the computer graphics object. The plurality of operations further includes:
Transmitting the spherical image and the texture mapping for display on a head mounted display device. Generating additional spherical images and texture mappings;
Generating a left-eye view by combining a portion of the image data and the spherical image;
Generating a right eye view by generating additional image data and combining the additional image data and the additional spherical image;
Further displaying the left eye view and right eye view on the head mounted display device,
The system of claim 9, wherein the image data includes depth value data and RGB data for at least some of the portions of pixels in the region. The plurality of images include video content, the image data includes RGB data and depth value data associated with the portion of pixels, and the system further includes:
Using the image data to convert a two-dimensional version of the region to a three-dimensional version of the region;
Providing the three-dimensional version of the region for display on the head-mounted display device.   The system of claim 9, wherein the plurality of images are captured using a plurality of cameras mounted on a spherically shaped camera rig.   10. The system of claim 9, wherein determining an area for converting 2D data to 3D data is automatically performed based at least in part on user input detected on a head mounted display. .   14. The user input includes a change in gaze direction, and the three-dimensional data is used to generate a three-dimensional portion in at least one of the plurality of images on a user's line of sight. System. A computer-implemented method comprising:
Acquiring a plurality of images using a plurality of cameras;
Generating at least two update images for the plurality of images, wherein the at least two update images capture content with a left-side offset from a predefined centerline and from the predefined centerline Generated by interpolating viewpoints for at least one virtual camera configured to capture content with a right offset, the method further comprising:
Mapping a first image in the at least two updated images to a first sphere to generate a first spherical image for provision to a left eyepiece of a head mounted display;
Mapping a second image in the at least two updated images to a second sphere to generate a second spherical image for provision to a right eyepiece of the head mounted display;
Displaying the first spherical image on the left eyepiece of the head-mounted display and displaying the second spherical image on the right eyepiece of the head-mounted display.   16. The at least one virtual camera is configured to use content captured using one or more physical cameras and to adapt the content to be provided from the viewpoint. The method described. Mapping the first image comprises applying a texture to the first image by assigning pixel coordinates from the first image to the first sphere;
The method of claim 15, wherein mapping the second image includes applying a texture to the second image by assigning pixel coordinates from the second image to the second sphere.   Interpolating the viewpoint includes sampling a plurality of pixels in the plurality of images, generating virtual content using an optical flow, and inside the at least one of the at least two update images. 16. The method of claim 15, comprising installing virtual content.   At least two spherical images are included in the RGB image having at least a part of the plurality of pixels included in the content captured at the left offset and the content captured at the right offset. The method of claim 18, comprising: an RGB image having at least a portion of the plurality of pixels.   16. The left offset and the right offset are modifiable and are functional to adapt display accuracy of the first image and the second image on the head mounted display. The method described.