US20160241838A1 - Methods and apparatus for generating and using reduced resolution images and/or communicating such images to a playback or content distribution device - Google Patents
Methods and apparatus for generating and using reduced resolution images and/or communicating such images to a playback or content distribution device Download PDFInfo
- Publication number
- US20160241838A1 US20160241838A1 US15/046,314 US201615046314A US2016241838A1 US 20160241838 A1 US20160241838 A1 US 20160241838A1 US 201615046314 A US201615046314 A US 201615046314A US 2016241838 A1 US2016241838 A1 US 2016241838A1
- Authority
- US
- United States
- Prior art keywords
- image
- map
- images
- resolution
- different
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims abstract description 122
- 230000002829 reductive effect Effects 0.000 title abstract description 51
- 230000007613 environmental effect Effects 0.000 claims description 179
- 238000009877 rendering Methods 0.000 abstract description 54
- 230000009467 reduction Effects 0.000 abstract description 52
- 230000005540 biological transmission Effects 0.000 abstract description 14
- 230000033001 locomotion Effects 0.000 abstract description 14
- 238000001514 detection method Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 30
- 238000013507 mapping Methods 0.000 description 27
- 210000003128 head Anatomy 0.000 description 26
- 238000012545 processing Methods 0.000 description 24
- 230000008859 change Effects 0.000 description 17
- 230000009471 action Effects 0.000 description 16
- 238000004891 communication Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 9
- 238000003384 imaging method Methods 0.000 description 9
- 238000007906 compression Methods 0.000 description 8
- 230000006835 compression Effects 0.000 description 8
- 238000005192 partition Methods 0.000 description 7
- 238000005070 sampling Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 230000004044 response Effects 0.000 description 5
- 238000004088 simulation Methods 0.000 description 4
- 230000003068 static effect Effects 0.000 description 4
- 101000829367 Homo sapiens Src substrate cortactin Proteins 0.000 description 3
- 102100023719 Src substrate cortactin Human genes 0.000 description 3
- 238000013459 approach Methods 0.000 description 3
- 239000000872 buffer Substances 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 239000007787 solid Substances 0.000 description 3
- 238000000638 solvent extraction Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 101100400452 Caenorhabditis elegans map-2 gene Proteins 0.000 description 1
- 101150064138 MAP1 gene Proteins 0.000 description 1
- 102100028379 Methionine aminopeptidase 1 Human genes 0.000 description 1
- 101710161855 Methionine aminopeptidase 1 Proteins 0.000 description 1
- 102100023174 Methionine aminopeptidase 2 Human genes 0.000 description 1
- 108090000192 Methionyl aminopeptidases Proteins 0.000 description 1
- 230000002411 adverse Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Images
Classifications
-
- H04N13/0059—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/189—Recording image signals; Reproducing recorded image signals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/04—Texture mapping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
-
- H04N13/0029—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N13/117—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/139—Format conversion, e.g. of frame-rate or size
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/194—Transmission of image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
- H04N13/279—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals the virtual viewpoint locations being selected by the viewers or determined by tracking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/136—Incoming video signal characteristics or properties
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
- H04N21/2387—Stream processing in response to a playback request from an end-user, e.g. for trick-play
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/426—Internal components of the client ; Characteristics thereof
- H04N21/42653—Internal components of the client ; Characteristics thereof for processing graphics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/437—Interfacing the upstream path of the transmission network, e.g. for transmitting client requests to a VOD server
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/8146—Monomedia components thereof involving graphical data, e.g. 3D object, 2D graphics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
- H04N7/0117—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving conversion of the spatial resolution of the incoming video signal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2215/00—Indexing scheme for image rendering
- G06T2215/12—Shadow map, environment map
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N2013/0074—Stereoscopic image analysis
- H04N2013/0088—Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
Definitions
- the present invention relates to methods and apparatus for capturing, streaming and/or playback of content, e.g., content which can be used to simulate an environment.
- content e.g., content which can be used to simulate an environment.
- Display devices which are intended to provide an immersive experience normally allow a user to turn his head and experience a corresponding change in the scene which is displayed.
- Head mounted displays sometimes support 360 degree viewing in that a user can turn around while wearing a head mounted display with the scene being displayed changing as the user's head position changes.
- a user should be presented with a scene that was captured in front of a camera position when looking forward and a scene that was captured behind the camera position when the user turns completely around. While a user may turn his head to the rear, at any given time a user's field of view is normally limited to 120 degrees or less due to the nature of a human's ability to perceive a limited field of view at any given time.
- a 360 degree scene may be captured using multiple cameras with the images being combined to generate the 360 degree scene which is to be made available for viewing.
- a 360 degree view includes a lot more image data than a simple forward view which is normally captured, encoded for normal television and many other video applications where a user does not have the opportunity to change the viewing angle used to determine the image to be displayed at a particular point in time.
- the images corresponding to the environment which are communicated to a playback device exceed the area a user can view at a given time so that content is available in the event the user changes his/her viewing angle by, for example, moving his/her head.
- the playback device has enough information to provide images should the user's viewing angle change without the playback device having to wait for new images or other content corresponding to a portion of the environment which the user was not previously viewing.
- the environment is represented using a mesh model. Images are captured and encoded into frames. At the playback device the encoded images are decoded and applied to a surface of the environmental model, e.g., as a texture.
- the mapping of an image to the surface of the environmental model is in accordance with a texture map also sometimes referred to as a UV map.
- a segment of a UV map corresponds to a segment of the 3D mesh model.
- a UV map is applied to the image and the segments of the image are then applied to the corresponding segments of the 3D model as a texture.
- a UV map can be used to map a portion of a received image onto a corresponding portion of a model of an environment.
- this process is used in some embodiments to map images corresponding to a left eye view onto the 3D model with the result being displayed to a user's left eye.
- An image corresponding to a right eye view is mapped onto the 3D model to generate an image which is displayed to the user's right eye. Differences between the left and right eye views in 3D embodiments result in a user perceiving images in 3D.
- a user's ability to detect the quality of an image decreases with regard to portions of an image the user is not directly looking at.
- a user In the case of a scene of an environment a user is likely to be focused on viewing the area of action in the environment, e.g., the portion of the environment where a ball is during a sporting game or where the actors are on a stage or within the environment.
- the methods and apparatus of the present invention take advantage of this fact to selectively allocate resolution to the image being communicated. While reducing the resolution of images which are less likely to be viewed while maintaining the resolution of portions of images corresponding to an environment which are likely to be viewed, it is possible to make efficient use of limited bandwidth available for streaming image data to a playback device.
- images of an environment are captured and selective reduction of resolution is applied.
- the reduction in resolution may be, and sometimes is, applied to portions of an image perceived to correspond to less important portions of an environment.
- an environmental model may remain fixed, in various embodiments the resolution reduction applied to captured images may change as the portion of the environment of high importance changes. For example, while at the start of a soccer game the center field may be considered the important area of the environment since that is where the kickoff occurs, as the ball moves to the left end of the field from the viewers perspective the left end may become more important than the other portions of the field. As the ball moves to the right end of the field the right end of the field may be more important from the viewers perspective than the left and center portions where the ball is not located.
- a resolution allocation is made based on the relative importance of different portions of an environment at a given time with more resolution being allocated to portions of images corresponding to areas of an environment perceived to be of high importance than areas of low importance.
- the relative importance may be based on motion detected in captured video providing the images being communicated, from user input such as by tracking where users are looking during the capture of images and/or through control of an operator of the encoding and/or streaming system.
- a set of different resolution allocations are supported. Down-sampling or another resolution reduction technique is applied to portions of an image the selected resolution allocation indicates are to be subject to resolution reduction while other portions of the image may be left at full resolution or subject to a lesser amount of resolution reduction.
- a different texture map is used for different resolution allocations.
- the texture map UV map
- different UV maps in combination with selective resolution reduction can be used to allocate different amounts of resolution to different portions of an image of an environment depending on which portion of the environment is considered important at a given point in time while the same environmental model is used despite the different allocations of resolution.
- a set of UV maps corresponding to a portion of the simulated environment may be communicated to the playback device with the streaming device then indicating which UV map is to be used for a communicated image or image pair.
- An image of an entire 360 degree world view including sky and ground images is communicated in a single frame to a playback device in some embodiments.
- images to be used as textures for different portions of an environment are communicated as separate frames.
- an image of the ground may be sent separately, and not updated as frequently, as an image to be used for a 360 degree horizontal portion of the environment while another image may be sent for a sky view.
- the resolution allocation selection and indication of a corresponding UV map may be, and sometimes is, performed for each portion of the environment included in the stream as a separate image.
- the UV maps also referred to herein as texture maps and sometimes as image maps, are normally communicated to the playback device before they are required for rendering. They can be communicated in the content stream with which they are to be used or sent separately. Once communicated to a playback device, the UV maps can be, and sometimes are, stored. After a UV map is stored the streaming device can indentify the map by communicating a map identifier in the content stream along with the image or images to which the UV map is applied.
- the encoding device and/or streaming device Since the resolution allocation is made prior to encoding, the encoding device and/or streaming device normally associates in the image stream the UV map and/or a map identifier with the communicated image or images. In this way the playback device knows which UV map to use when mapping a received image as part of a rendering operation.
- Rendered images are displayed to a user with left eye images of an image pair being displayed to a user's left eye and right eye images displayed to a user's right eye.
- UV maps in combination with selective resolution reduction, different resolutions can be allocated to portions of an environment in a relatively easy to implement manner without requiring changes to be made to the encoder being used to encode images which have been subject to different resolution allocations and without requiring special decoding of images.
- the methods are well suited for 3D applications where left and right eye images are communicated to provide an stereoscopic image pair, the methods may be and sometimes are used for non-stereoscopic embodiments with selective resolution allocation and corresponding UV maps being used in cases where a single image stream is communicated to a playback device, e.g., with the individual images being decoded and rendered onto the environmental map but with the same image being displayed to both eyes of a user.
- the methods can be used for embodiments where a single stream of images is communicated to the playback device and the playback device uses computational processing to generate a pair of eye views from a single stream of received images, e.g., by receiving an image and generating a left eye image and a different right eye image from the single received image.
- FIG. 1 illustrates an exemplary system implemented in accordance with some embodiments of the invention which can be used to capture content, stream content, and output content to one or more users playback devices in accordance with any of the embodiments described herein.
- FIG. 2A illustrates an exemplary stereoscopic scene, e.g., a full 360 degree stereoscopic scene which has not been partitioned.
- FIG. 2B illustrates an exemplary stereoscopic scene which has been partitioned into 3 exemplary scenes in accordance with one exemplary embodiment.
- FIG. 2C illustrates an exemplary stereoscopic scene which has been partitioned into 4 scenes in accordance with one exemplary embodiment.
- FIG. 3 illustrates an exemplary process of encoding an exemplary 360 degree stereoscopic scene in accordance with one exemplary embodiment.
- FIG. 4 illustrates an example showing how an input image portion is encoded using a variety of encoders to generate different encoded versions of the same input image portion.
- FIG. 5 illustrates stored encoded portions of an input stereoscopic scene that has been partitioned into 3 portions.
- FIG. 6 illustrates the combination of FIGS. 6A and 6B .
- FIG. 6A illustrates a first part of a flowchart illustrating the steps of an exemplary method of streaming content in accordance with an exemplary embodiment implemented using the system of FIG. 1 in which selective resolution allocation and different UV maps are used at different times.
- FIG. 6B illustrates a second part of a flowchart illustrating the steps of an exemplary method of streaming content in accordance with an exemplary embodiment implemented using the system of FIG. 1 in which selective resolution allocation and different UV maps are used at different times.
- FIG. 7 illustrates an exemplary content delivery system with resolution allocation selection, resolution reduction and encoding capability that can be used to encode and stream content, along with corresponding UV maps, in accordance with the features of the invention.
- FIG. 8 illustrates an exemplary content playback device that can be used to receive, decode and display the content streamed by the system of FIG. 7 and may use the UV maps shown and described with reference to FIG. 24 and/or various other figures to allow different UV maps to be used for images having different resolution allocations.
- FIG. 9 illustrates the combination of FIGS. 9A and 9B .
- FIG. 9A illustrates the first part of an exemplary method of operating a content playback device in accordance with the present invention.
- FIG. 9B illustrates the second part of an exemplary method of operating a content playback device in accordance with the present invention.
- FIG. 10 illustrates an exemplary method of communicating information to be used to represent an environment in accordance with the present invention.
- FIG. 11 illustrates an exemplary image capture and content streaming method in accordance with an exemplary embodiment in which different resolution allocations can be used for images corresponding to the same environmental portion at different times.
- FIG. 12 illustrates a method of operating a playback device or system, e.g., a rendering device, which can be used in the system of FIG. 1 , to receive and render images using UV maps and an environmental model in accordance with one exemplary embodiment.
- a playback device or system e.g., a rendering device, which can be used in the system of FIG. 1 , to receive and render images using UV maps and an environmental model in accordance with one exemplary embodiment.
- FIG. 13 illustrates a camera rig including multiple camera pairs for capturing left and right eye images corresponding to different sectors of a 360 degree field of view along with a camera or cameras directed towards the sky to capture a sky view.
- FIG. 14 shows how 5 different environmental mesh maps, corresponding to different camera views, can be combined to create a complete spherical view/environment onto which captured images can be projected, e.g., onto the inner surface, as part of a playback operation.
- FIG. 15 shows the full assembly of 5 meshes shown in FIG. 14 to create a spherical simulated environment which can be viewed from a user as if he/she were located at the center of the environment, e.g., sphere.
- FIG. 16 shows a left eye view image and a right eye view image captured by left and right eye cameras, with fisheye lenses, corresponding to a sector of the camera rig shown in FIG. 13 .
- FIG. 17A shows an exemplary mesh model of an environment in accordance with the invention.
- FIG. 17B shows a UV map which can be used to map portions of a 2D image onto surfaces of the mesh model shown in FIG. 17A .
- FIG. 18 shows how captured left and right eye view images of FIG. 16 may appear after cropping prior to encoding and transmission to one or more playback devices.
- FIG. 19 shows an environmental mesh model corresponding to one sector of the camera rig with one of the images shown in FIG. 18 applied, e.g., projected, onto the environmental mesh.
- FIG. 20 shows application of images captured by cameras corresponding to each of the sectors as well as the sky and ground cameras of the camera rig can be combined and projected onto the modeled environment to simulate a complete 360 environment in the form of a sphere.
- FIG. 21 shows how selective resolution can be used with regard to a frame which maps to an environmental grid with different resolutions being used for different portions of the image to be mapped to the environmental model, e.g., with smaller portions of the transmitted image being mapped to corresponding portions of the sky and ground mesh segments than the segments of the middle portion of the environment resulting in lower resolution being allocated to the top and bottom portions of the environment than the middle portion of the environment.
- FIG. 22 shows a first captured image of a first portion of an environment, a first resolution adjusted image generated using a first resolution allocation from the first captured image, and a first UV map corresponding to the first resolution allocation.
- FIG. 23 shows a second captured image of the first portion of the environment, a second resolution adjusted image generated using a second resolution allocation from the second captured image, and a second UV map corresponding to the second resolution allocation.
- FIG. 24 shows a third captured image of the first portion of the environment, a third resolution adjusted image generated using a third resolution allocation from the third captured image, and a third UV map corresponding to the second resolution allocation.
- FIG. 25 illustrates the combination of FIGS. 25A and 25B .
- FIG. 25A shows a first part of a method of operating a content processing and delivery system in accordance with an exemplary embodiment.
- FIG. 25B shows a second part of a method of operating a content processing and delivery system in accordance with an exemplary embodiment.
- FIG. 26 illustrates an exemplary embodiment of a method of playing back content in accordance with the invention.
- FIG. 27 illustrates an example of how a playback device, such as the playback device or devices shown in any of the other figures, can perform image rendering using a UV map corresponding to the resolution allocation that was used to generate the image to be rendered.
- FIG. 28 illustrates an example of how a playback device, such as the playback device or devices shown in any of the other figures, can perform image rendering using a UV map corresponding to the resolution allocation that was used to generate the image to be rendered.
- FIG. 29 illustrates an example of how a playback device, such as the playback device or devices shown in any of the other figures, can perform image rendering using a UV map corresponding to the resolution allocation that was used to generate the image to be rendered.
- FIG. 1 illustrates an exemplary system 100 implemented in accordance with some embodiments of the invention.
- the system 100 supports content delivery, e.g., imaging content delivery, to one or more customer devices, e.g., playback devices/content players, located at customer premises.
- the system 100 includes the exemplary image capturing device 102 , a content delivery system 104 , a communications network 105 , and a plurality of customer premises 106 , . . . , 110 .
- the image capturing device 102 supports capturing of stereoscopic imagery.
- the image capturing device 102 captures and processes imaging content in accordance with the features of the invention.
- the communications network 105 may be, e.g., a hybrid fiber-coaxial (HFC) network, satellite network, and/or internet.
- HFC hybrid fiber-coaxial
- the content delivery system 104 includes an image processing, calibration and encoding apparatus 112 and a content delivery device, e.g. a streaming server 114 .
- the image processing, calibration and encoding apparatus 112 is responsible for performing a variety of functions including camera calibration based on one or more target images and/or grid patterns captured during a camera calibration process.
- Content delivery device 114 may be implemented as a server with, as will be discussed below, the delivery device responding to requests for content with image calibration information, optional environment information, and one or more images captured by the camera rig 102 which can be used in simulating a 3D environment.
- Streaming of images and/or content may be and sometimes is a function of feedback information such as viewer head position and/or user selection of a position at the event corresponding to a camera 102 which is to be the source of the images.
- a user may select or switch between images from a camera rig positioned at center line to a camera rig positioned at the field goal with the simulated 3D environment and streamed images being changed to those corresponding to the user selected camera rig.
- a single camera rig 102 is shown in FIG.
- multiple camera rigs may be present in the system and located at different physical locations at a sporting or other event with the user being able to switch between the different positions and with the user selections being communicated from the playback device 122 to the content server 114 . While separate devices 112 , 114 are shown in the image processing and content delivery system 104 , it should be appreciated that the system may be implemented as a single device including separate hardware for performing the various functions or with different functions being controlled by different software or hardware modules but being implemented in or on a single processor.
- Encoding apparatus 112 may, and in some embodiments does, include one or a plurality of encoders for encoding image data in accordance with the invention.
- the encoders may be used in parallel to encode different portions of a scene and/or to encode a given portion of a scene to generate encoded versions which have different data rates. Using multiple encoders in parallel can be particularly useful when real time or near real time streaming is to be supported.
- the content streaming device 114 is configured to stream, e.g., transmit, encoded content for delivering the encoded image content to one or more customer devices, e.g., over the communications network 105 .
- the content delivery system 104 can send and/or exchange information with the devices located at the customer premises 106 , 110 as represented in the figure by the link 120 traversing the communications network 105 .
- the encoding apparatus 112 and content delivery server are shown as separate physical devices in the FIG. 1 example, in some embodiments they are implemented as a single device which encodes and streams content.
- the encoding process may be a 3D, e.g., stereoscopic, image encoding process where information corresponding to left and right eye views of a scene portion are encoded and included in the encoded image data so that 3D image viewing can be supported.
- the particular encoding method used is not critical to the present application and a wide range of encoders may be used as or to implement the encoding apparatus 112 .
- Each customer premise 106 , 110 may include a plurality of devices/players, e.g., decoding apparatus to decode and playback/display the image content streamed by the content streaming device 114 .
- Customer premise 1 106 includes a decoding apparatus/playback device 122 coupled to a display device 124 while customer premise N 110 includes a decoding apparatus/playback device 126 coupled to a display device 128 .
- the display devices 124 , 128 are head mounted stereoscopic display devices.
- decoding apparatus 122 , 126 present the imaging content on the corresponding display devices 124 , 128 .
- the decoding apparatus/players 122 , 126 may be devices which are capable of decoding the imaging content received from the content delivery system 104 , generate imaging content using the decoded content and rendering the imaging content, e.g., 3D image content, on the display devices 124 , 128 .
- Any of the decoding apparatus/playback devices 122 , 126 may be used as the decoding apparatus/playback device 800 shown in FIG. 8 .
- a system/playback device such as the one illustrated in FIG. 8 can be used as any of the decoding apparatus/playback devices 122 , 126 .
- FIG. 2A illustrates an exemplary stereoscopic scene 200 , e.g., a full 360 degree stereoscopic scene which has not been partitioned.
- the stereoscopic scene may be and normally is the result of combining image data captured from multiple cameras, e.g., video cameras, often mounted on a single video capture platform or camera mount.
- N 3 exemplary portions, e.g., a front 180 degree portion, a left rear 90 degree portion and a right rear 90 degree portion in accordance with one exemplary embodiment.
- FIGS. 2B and 2C show two exemplary partitions, it should be appreciated that other partitions are possible.
- multiple partitions are grouped together and encoded as a group. Different groups of partitions may be encoded and streamed to the user with the size of each group being the same in terms of total degrees of scene but corresponding to a different portion of an image which may be streamed depending on the user's head position, e.g., viewing angle as measured on the scale of 0 to 360 degrees.
- FIG. 3 illustrates an exemplary process of encoding an exemplary 360 degree stereoscopic scene in accordance with one exemplary embodiment.
- the input to the method 300 shown in FIG. 3 includes 360 degree stereoscopic image data 302 captured by, e.g., a plurality of cameras arranged to capture a 360 degree view of a scene.
- the stereoscopic image data 302 e.g., stereoscopic video
- the scene data 302 is partitioned into data corresponding to different scene areas, e.g., N scene areas corresponding to different viewing directions.
- scene areas e.g., N scene areas corresponding to different viewing directions.
- the 360 degree scene area is portioned into three partitions a left rear portion corresponding to a 90 degree portion, a front 180 degree portion and a right rear 90 degree portion.
- the different portions may have been captured by different cameras but this is not necessary and in fact the 360 degree scene may be constructed from data captured from multiple cameras before being dividing into the N scene areas as shown in FIG. 2B and 2C .
- step 306 the data corresponding to the different scene portions is encoded in accordance with the invention.
- each scene portion is independently encoded by multiple encoders to support multiple possible bit rate streams for each portion.
- step 308 the encoded scene portions are stored, e.g., in the content delivery system 104 , for streaming to the customer playback devices.
- FIG. 4 is a drawing 400 illustrating an example showing how an input image portion, e.g., a 180 degree front portion of a scene, is encoded using a variety of encoders to generate different encoded versions of the same input image portion.
- an input image portion e.g., a 180 degree front portion of a scene
- an input scene portion 402 e.g., a 180 degree front portion of a scene
- a plurality of encoders for encoding.
- the plurality of K encoders include a high definition (HD) encoder 1 404 , a standard definition (SD) encoder 2 406 , a reduced frame rate SD encoder 3 408 , . . . , and a high compression reduced frame rate SD encoder K 410 .
- the HD encoder 1 404 is configured to perform full high definition (HD) encoding to produce high bit rate HD encoded image 412 .
- the SD encoder 2 406 is configured to perform low resolution standard definition encoding to produce a SD encoded version 2 414 of the input image.
- the reduced frame rate SD encoder 3 408 is configured to perform reduced frame rate low resolution SD encoding to produce a reduced rate SD encoded version 3 416 of the input image.
- the reduced frame rate may be, e.g., half of the frame rate used by the SD encoder 2 406 for encoding.
- the high compression reduced frame rate SD encoder K 410 is configured to perform reduced frame rate low resolution SD encoding with high compression to produce a highly compressed reduced rate SD encoded version K 420 of the input image.
- control of spatial and/or temporal resolution can be used to produce data streams of different data rates and control of other encoder settings such as the level of data compression may also be used alone or in addition to control of spatial and/or temporal resolution to produce data streams corresponding to a scene portion with one or more desired data rates.
- FIG. 5 illustrates stored encoded portions 500 of an input stereoscopic scene that has been partitioned into 3 exemplary portions.
- the stored encoded portions may be stored in the content delivery system 104 , e.g., as data/information in the memory.
- the stored encoded portions 500 of the stereoscopic scene includes 3 different sets of encoded portions, with each portion corresponding to a different scene area and each set including a plurality of different encoded versions of the corresponding scene portion.
- Each encoded version is a version of encoded video data and thus represents multiple frames which have been coded.
- each encoded version 510 , 512 , 516 is video that corresponds to multiple periods of time and that when streaming, the portion, e.g., frames, corresponding to the period of time being played back will be used for transmission purposes.
- each scene portion e.g., front, rear scene portions
- each scene portion may be encoded using a plurality of different encoders to produce K different versions of the same scene portion.
- the outputs of each encoder corresponding to a given input scene are grouped together as a set and stored.
- the first set of encoded scene portions 502 corresponds to the front 180 degree scene portion, and includes encoded version 1 510 of the front 180 degree scene, encoded version 2 512 , . . . , and encoded version K 516 .
- the second set of encoded scene portions 504 corresponds to the scene portion 2, e.g., 90 degree left rear scene portion, and includes encoded version 1 520 of the 90 degree left rear scene portion, encoded version 2 522 , . . . , and encoded version K 526 of the 90 degree left rear scene portion.
- the third set of encoded scene portions 506 corresponds to the scene portion 3, e.g., 90 degree right rear scene portion, and includes encoded version 1 530 of the 90 degree right rear scene portion, encoded version 2 532 , . . . , and encoded version K 536 of the 90 degree right rear scene portion.
- the various different stored encoded portions of the 360 degree scene can be used to generate various different bit rate streams for sending to the customer playback devices.
- the content delivery system 104 can support a large number of concurrent users since, the encoding process allows the N portions of a scene to be transmitted and processed differently to different users without having to encode the content separately for each individual user.
- the encoding process allows the N portions of a scene to be transmitted and processed differently to different users without having to encode the content separately for each individual user.
- a number of parallel encoders may be used to support real time encoding to allow for real or near real time streaming of sports or other events, the number of encoders used tends to be far less than the number of playback devices to which the content is streamed.
- the portions of content are described as portions corresponding to a 360 degree view it should be appreciated that the scenes may, and in some embodiments do, represent a flattened version of a space which also has a vertical dimension.
- the playback device is able to map the scene portions using a model of the 3D environment, e.g., space, and adjust for vertical viewing positions.
- the 360 degrees which are discussed in the present application refer to the head position relative to the horizontal as if a user changed his viewing angle left or right while holding his gaze level.
- FIG. 6 which comprises FIGS. 6A and 6B is a flowchart 600 illustrating the steps of an exemplary method of providing image content, in accordance with an exemplary embodiment.
- FIG. 6A illustrates the first part of the flowchart 600 .
- FIG. 6B illustrates the second part of flowchart 600 .
- the method of flowchart 600 is implemented in some embodiments using the capturing system shown in FIG. 1 .
- the method 600 commences in start step 602 shown in FIG. 6A . Operation proceeds from step 602 to step 604 . In step 604 , a captured image is received. Operation proceeds from step 604 to step 606 .
- step 606 the resolution allocation to be used is selected. The selection may be made for example based on motion. Operation proceeds from step 606 to decision step 608 . In decision step 608 , if a determination is made that the selected resolution is different from the previous resolution allocation then operation proceeds to step 610 . Otherwise operation proceeds to step 612 .
- step 610 new downsampling and/or filtering information corresponding to the selected resolution allocation used to control resolution reduction is loaded. Operation proceeds from step 610 to step 612 .
- step 612 a resolution reduction operation is performed on the received captured image based on the determined resolution allocation to be used.
- the resolution reduction operation outputs a reduced resolution image 614 with at least some different image portions having different resolutions. Operation proceeds to step 616 .
- step 616 the reduced resolution image is encoded using an encoder which supports compression, e.g., entropy encoding, run length encoding, motion vectors and/or other encoding techniques. Operation proceeds from step 616 to step 618 .
- compression e.g., entropy encoding, run length encoding, motion vectors and/or other encoding techniques. Operation proceeds from step 616 to step 618 .
- a UV map corresponding to the resolution allocation to be used for rendering the image subjected to determined resolution allocation is indicated.
- the playback device is provided with information which allows the communicated image to be applied to the 3D model of the environment taking into consideration which portions of the transmitted image were downsampled prior to being communicated to the playback device. Operation proceeds from step 618 to decision step 622 shown on FIG. 6B via connection node A 620 .
- step 622 a determination is made as to whether the UV map corresponding to the applied resolution allocation has been communicated to the playback device. If the determination is that the UV map corresponding to the applied resolution allocation has not been communicated to the playback device then operation proceeds to step 624 . If the determination is that the UV map corresponding to the applied resolution allocation has been communicated to the playback device then operation proceeds to step 626 .
- step 624 the UV map corresponding to the applied resolution allocation is communicated to the playback device. Operation proceeds from step 624 to step 626 .
- step 626 information indicating the UV map to use is communicated to the playback device. Operation proceeds from step 626 to step 628 .
- step 628 the encoded image is communicated to the playback device. This method may be executed with respect to each received captured image.
- FIG. 7 illustrates an exemplary content delivery system 700 with encoding capability that can be used to encode and stream content in accordance with the features of the invention.
- the system may be used to perform encoding, storage, and transmission and/or content output in accordance with the features of the invention.
- the system 700 or the elements therein perform the operation corresponding to the process illustrated in FIG. 6 .
- the content delivery system 700 may be used as the system 104 of FIG. 1 . While the system shown in FIG. 7 is used for encoding, processing and streaming of content, it should be appreciated that the system 700 may also include the ability to decode and display processed and/or encoded image data, e.g., to an operator.
- the system 700 includes a display 702 , input device 704 , input/output (I/O) interface 706 , a processor 708 , network interface 710 and a memory 712 .
- the various components of the system 700 are coupled together via bus 709 which allows for data to be communicated between the components of the system 700 .
- the memory 712 includes various modules, e.g., routines, which when executed by the processor 708 control the system 700 to implement the partitioning, encoding, storage, and streaming/transmission and/or output operations in accordance with the invention.
- routines which when executed by the processor 708 control the system 700 to implement the partitioning, encoding, storage, and streaming/transmission and/or output operations in accordance with the invention.
- the memory 712 includes various modules, e.g., routines, which when executed by the processor 707 control the computer system 700 to implement the immersive stereoscopic video acquisition, encoding, storage, and transmission and/or output methods in accordance with the invention.
- the memory 712 includes control routines 714 , a partitioning module 716 , encoder(s) 718 , a streaming controller 720 , received input images 732 , e.g., 360 degree stereoscopic video of a scene, encoded scene portions 734 , and timing information 736 .
- the modules are, implemented as software modules.
- the modules are implemented in hardware, e.g., as individual circuits with each module being implemented as a circuit for performing the function to which the module corresponds.
- the modules are implemented using a combination of software and hardware.
- the control routines 714 include device control routines and communications routines to control the operation of the system 700 .
- the partitioning module 716 is configured to partition a received stereoscopic 360 degree version of a scene into N scene portions in accordance with the features of the invention.
- the encoder(s) 718 may, and in some embodiments do, include a plurality of encoders configured to encode received image content, e.g., 360 degree version of a scene and/or one or more scene portions in accordance with the features of the invention.
- encoder(s) include multiple encoders with each encoder being configured to encode a stereoscopic scene and/or partitioned scene portions to support a given bit rate stream.
- each scene portion can be encoded using multiple encoders to support multiple different bit rate streams for each scene.
- An output of the encoder(s) 718 is the encoded scene portions 734 which are stored in the memory for streaming to customer devices, e.g., playback devices.
- the encoded content can be streamed to one or multiple different devices via the network interface 710 .
- UV maps 740 are stored in memory 712 of the content delivery system 700 .
- the UV maps 740 correspond to different resolution allocations and/or areas of the environment.
- the first UV map 1 742 corresponds to a first resolution allocation
- the second UV map 2 744 corresponds to a second resolution allocation
- the third UV map 746 corresponds to a third resolution allocation.
- UV maps with different resolution allocations can correspond to the same area of an environment.
- Different UV maps corresponding to other areas of the environment can be stored in the memory 712 .
- Multiple UV maps may correspond to the environmental model.
- the mesh model of the environment where the received images were captured is stored in memory 712 of the content delivery system 700 , e.g., 3D environmental mesh model 738 . Multiple mesh models may be stored in the memory 712 .
- the streaming controller 720 is configured to control streaming of encoded content for delivering the encoded image content to one or more customer devices, e.g., over the communications network 105 .
- various steps of the flowchart 600 are implemented by the elements of the streaming controller 720 .
- the streaming controller 720 includes a request processing module 722 , a data rate determination module 724 , a current head position determination module 726 , a selection module 728 and a streaming control module 730 .
- the request processing module 722 is configured to process a received request for imaging content from a customer playback device.
- the request for content is received in various embodiments via a receiver 713 in the network interface 710 .
- the request for content includes information indicating the identity of requesting playback device.
- the request for content may include data rates supported by the customer playback device, a current head position of the user, e.g., position of the head mounted display.
- the request processing module 722 processes the received request and provides retrieved information to other elements of the streaming controller 720 to take further actions. While the request for content may include data rate information and current head position information, in various embodiments the data rate supported by the playback device can be determined from network tests and other network information exchange between the system 700 and the playback device.
- the data rate determination module 724 is configured to determine the available data rates that can be used to stream imaging content to customer devices, e.g., since multiple encoded scene portions are supported the content delivery system 700 can support streaming content at multiple data rates to the customer device.
- the data rate determination module 724 is further configured to determine the data rate supported by a playback device requesting content from system 700 .
- the data rate determination module 724 is configured to determine data rates for delivery of image content based on network measurements.
- the current head position determination module 726 is configured to determine a current viewing angle and/or a current head position of the user, e.g., position of the head mounted display, from information received from the playback device. In some embodiments the playback device periodically sends current head position information to the system 700 where the current head position determination module 726 receives and processes the information to determine the current viewing angle and/or a current head position.
- the selection module 728 is configured to determine which portions of a 360 degree scene to stream to a playback device based on the current viewing angle/head position information of the user.
- the selection module 728 is further configured to select the encoded versions of the determined scene portions based on the available data rates to support streaming of content.
- the streaming control module 730 is configured to control streaming of image content, e.g., multiple portions of a 360 degree stereoscopic scene, at various supported data rates in accordance with the features of the invention.
- the streaming control module 730 is configured to control the streaming of N portions of a 360 degree stereoscopic scene to the playback device requesting content to initialize scene memory in the playback device.
- the streaming control module 730 is configured to send the selected encoded versions of the determined scene portions periodically, e.g., at a determined rate.
- the streaming control module 730 is further configured to send 360 degree scene updates to the playback device in accordance with a time interval, e.g., once every minute.
- sending 360 degree scene update includes sending N scene portions or N-X scene portions of the full 360 degree stereoscopic scene, where N is the total number of portions into which the full 360 degree stereoscopic scene has been partitioned and X represents the selected scene portions recently sent to the playback device.
- the streaming control module 730 waits for a predetermined time after initially sending N scene portions for initialization before sending the 360 degree scene update.
- the timing information to control sending of the 360 degree scene update is included in the timing information 736 .
- the streaming control module 730 is further configured identify scene portions which have not been transmitted to the playback device during a refresh interval; and transmit an updated version of the identified scene portions which were not transmitted to the playback device during the refresh interval.
- the streaming control module 730 is configured to communicate at least a sufficient number of the N portions to the playback device on a periodic basis to allow the playback device to fully refresh a 360 degree version of said scene at least once during each refresh period.
- FIG. 8 illustrates a computer system/playback device 800 implemented in accordance with the present invention which can be used to receive, decode, store and display imaging content received from a content delivery system such as the one shown in FIGS. 1 and 7 .
- the playback device may be used with a 3D head mounted display such as the OCULUS RIFTTM VR (virtual reality) headset which may be the head mounted display 805 .
- the device 800 includes the ability to decode the received encoded image data and generate 3D image content for display to the customer.
- the playback device in some embodiments is located at a customer premise location such as a home or office but may be located at an image capture site as well.
- the device 800 can perform signal reception, decoding, display and/or other operations in accordance with the invention.
- the device 800 includes a display 802 , a display device interface 803 , input device 804 , a decoder 864 , input/output (I/O) interface 806 , a processor 808 , network interface 810 and a memory 812 .
- the various components of the playback device 800 are coupled together via bus 809 which allows for data to be communicated between the components of the system 800 .
- display 802 is included as an optional element as illustrated using the dashed box
- an external display device 805 e.g., a head mounted stereoscopic display device, can be coupled to the playback device via the display device interface 803 .
- the network interface 810 includes a receiver 860 and a transmitter 862 .
- the memory 812 includes various modules, e.g., routines, which when executed by the processor 808 control the playback device 800 to perform decoding and output operations in accordance with the invention.
- the memory 812 includes control routines 814 , a request for content generation module 816 , a head position and/or viewing angle determination module 818 , a decoder module 820 , a stereoscopic image rendering module 822 also referred to as a 3D image generation module, and data/information including received encoded image content 824 , decoded image content 826 , a 360 degree decoded scene buffer 828 , and generated stereoscopic content 830 .
- the control routines 814 include device control routines and communications routines to control the operation of the device 800 .
- the request generation module 816 is configured to generate a request for content to send to a content delivery system for providing content.
- the request for content is sent in various embodiments via the network interface 810 .
- the head position and/or viewing angle determination module 818 is configured to determine a current viewing angle and/or a current head position of the user, e.g., position of the head mounted display, and report the determined position and/or viewing angle information to the content delivery system 700 .
- the playback device 800 periodically sends current head position information to the system 700 .
- the decoder module 820 is configured to decode encoded image content 824 received from the content delivery system 700 to produce decoded image data 826 .
- the decoded image data 826 may include decoded stereoscopic scene and/or decoded scene portions.
- the 3D image rendering module 822 generates 3D images in accordance with the features of the invention, e.g., using the decoded image content 826 , for display to the user on the display 802 and/or the display device 805 .
- the generated stereoscopic image content 830 is the output of the 3D image generation module 822 .
- the rendering module 822 renders the 3D image content 830 to the display.
- the display device 805 may be a 3D display such as an oculus rift.
- the operator of the playback device 800 may control one or more parameters via input device 804 and/or select operations to be performed, e.g., select to display 3D scene.
- FIG. 8 illustrates an exemplary content playback device that can be used to receive, decode and display the content streamed by the system of FIG. 7 .
- the system 800 includes a display interface 803 coupled to a head mounted stereoscopic display 805 , an input device 804 , an optional display 802 and I/O interface.
- the interface 802 coupled the various input/output elements 803 , 802 , 804 to the bus 809 which in turn is coupled to processor 808 , network interface 810 and memory 812 .
- the network interface 810 allows the playback device to receive content from the streaming device 114 and/or communicate information such as view head position and/or position (camera rig) selection indicating selection of particular viewing position at an event.
- the memory 812 includes various data and modules as shown in FIG. 8 .
- the decoder module 820 causes received images to be decoded while 3D image rendering module 822 causes further processing of the images in accordance with the present invention and optionally stitching of images together as part of the presentation process.
- FIG. 9 which comprises a first part FIG. 9A and a second part FIG. 9B illustrates the steps 900 of a method of operating a content playback device.
- different UV maps may be used at different times for mapping a portion of one or more received images to an environmental model, e.g., a mesh model, of an environment.
- an environmental model e.g., a mesh model
- the mapping of pixels of a received image to a segment of the environmental model may change.
- using a first UV map may result in a first number of pixels in a received image mapping to a first portion of an environmental model while use of a second different UV map may result in a different number of pixels in a received image mapping to the same portion of the environmental model.
- the system generating and transmitting the images also in some embodiments communicates the UV maps and/or indicates to the playback device which UV map is to be used when mapping an image or set of images to the environmental model.
- the encoding and transmission device can change the amount of data and/or resolution associated with a particular portion of the environmental model.
- the image content will be scaled and/or otherwise modified as needed as part of the rendering process to cover the segment of the 3D model to which it applies.
- a first UV map maps one pixel to a first segment of the environmental model and a second UV map maps two pixels to the first segment of the environmental model
- the resolution of the displayed first segment will be higher when the second UV map is used than when the first UV map is used for image rendering.
- While the UV map may be changed from image to image or from group of images to group of images thereby allowing the server generating and sending the images and UV map information to the playback device to dynamically alter the allocation of data and/or resolution within a portion of the environment, e.g., front portion, based on the scene areas considered of particular interest, e.g., scene areas where the actors, players, performers are in the environment or where movement is in the environment, the data rate used for transmitting images can be held relatively constant since the number of pixels in the images can remain the same with the UV map controlling the allocation of pixels to portions of the environment.
- the methods allow for the image encoding technique to remain the same at least in some embodiments with the captured image or images being downsampled differently prior to encoding depending on the location of the scene portions considered of particular interest within a captured image and based on knowledge of which UV map will be used to apply the image, e.g., as a texture, to one or more segments of an environmental module.
- the UV map may be changed on a per frame or image basis from one image or frame to the next, in some embodiments the change in UV maps is constrained to occur on I-frame or group of picture boundaries with a UV map being used for multiple frames within a group of pictures or between I-frames. While such a UV map transition constraint is used in some embodiments, it is not necessary or critical to the invention and some embodiments allow the UV map to be changed on a per frame basis.
- the steps of the exemplary method 900 will now be discussed in detail.
- the method 900 starts in step 902 , e.g., with a content playback device being powered on.
- the playback device may be, e.g., a game system connected to a head mounted display or TV or as is the case in various embodiments a cell phone mounted in a head mount with a touch pad or other control and one or more lenses for allowing a user to view left and right eye images on different portions of a cell phone screen which is used as a display device.
- the method 900 may be implemented by any of the content playback devices described in the present application.
- step 903 e.g., in response to user input indicating user selection of content to be played to a user, the content playback device transmits a request for content in step 903 .
- this request is communicated to a content server or content provider system, e.g., a device which receives, processes and encodes images of an environment and supplies them to the playback device along with UV maps and/or information about which UV map to be used at a given time.
- the server may also provide an environmental model or a default model may be used.
- a model of an environment e.g., a 3D mesh model is received, e.g., from the content server.
- the model may be and sometimes is a model of an environment where an event such as a play or sporting event is ongoing.
- the model may be a complete 360 degree model of the environment or a model of the portion of the environment to which image content is to be mapped, e.g., a front portion of the environment.
- the features relating to using different UV maps to map images to a portion of the environment may be used for a full 360 degree environment, a portion of an environment, with stereo images and/or with non-stereoscopic images, e.g., panoramic images where the same image is displayed to both left and right eyes of a viewer.
- step 906 model of the environment received in step 903 is stored for future use, e.g., in rendering and displaying images mapped onto the model in accordance with one of the UV maps, e.g., texture maps, which are received in step 908 .
- the texture maps may be and sometimes are received from the same server which provides the environmental model.
- the UV map indicates how a 2d image should be segmented with the segments then being applied to corresponding segments of the environmental model, e.g., as a texture or textures.
- an initial texture e.g., initial UV map
- a set of maps are received and stored with the different UV maps indicating different mappings between an image and a portion of the environmental model.
- Each map may be identified by a texture map identifier.
- the content server providing the images can indicate which texture map to use with which set of images.
- a new texture map may be streamed with or before the images to which the new texture map is to be applied. Storing of a set of texture maps in the playback device can provide efficient transmission since the maps can be reused without transmitting the UV/texture maps multiple times to the playback device.
- step 910 the received set of texture maps is stored for future use.
- operation proceeds to step 914 in which image content is received.
- step 904 in addition to image content an indicator identifying the texture map to be used to map the received image onto the model of the environment is received or the texture map to be used is received.
- an indicator identifying the texture map to be used to map the received image onto the model of the environment is received or the texture map to be used is received.
- an indicator it identify the texture map in the stored set of texture maps which is to be used.
- An indicated texture map may remain in effect until a new texture map is specified and/or provided.
- a single texture map may be used for a sequence of images, e.g., a group of pictures.
- the texture map may be changed by the sever when, e.g., motion is detected indicating a different area of the environment is an area of higher priority than an area to which high resolution was previously allocated.
- resolution allocation can be changed and the UV map corresponding to the current resolution allocation may be used in place of a previous UV map corresponding to a different resolution allocation.
- Step 914 includes, in some embodiments steps 916 , 918 , 920 , 926 and 928 .
- step 916 a first encoded image is received.
- step 918 which is optional, a second encoded image is received.
- step 920 which is an alternative to steps 916 , 918 an encoded frame including one or both images is received.
- the second encoded image may be a second image of a stereoscopic image pair with the first and second images being left and right eye images to be displayed to a user of the playback device.
- odd lines of a frame may provide the first image and even lines of the encoded frame may provide the second encoded image.
- a top half of an encoded frame may provide the first image and the bottom half the second image.
- Other ways of including the first and second images in a single frame are also possible.
- step 914 in addition to receiving image content which can be mapped to the environmental model, in step 926 a first indicator indicating which of a plurality of texture maps corresponding to different resolution allocation is to be used with the received first and/or second encoded images is also received. If a new texture map indicator is not received in step 914 , and a new texture map is not received, the playback device will continue to use the last UV map which was being used. Rather than receive a texture map indicator, a new texture map may be received in step 928 which is to be used in rendering the received images.
- step 930 the received image or images are decoded.
- the first encoded image is decoded to recover a first image.
- the second encoded image is decoded to recover a second image.
- the first and second images may be left and right eye views.
- the first and second images are included in a single encoded frame decoding of the received frame and separation of the left and second images may be used in step 930 to produce left and right eye images which may be and sometimes are applied separately to the environmental map to generate separate, potentially different, left and right eye views.
- the images communicate a complete 360 degree environment or panoramic view.
- the first and second images may correspond to a portion of the environment, e.g., a front portion or a 360 degree middle panoramic portion but not the sky and ground.
- step 936 other images which are sent with the encoded first and second image or in a separate stream may be decoded to obtain textures for portions of the environment which are not provided by the first and/or second images.
- a sky or ground image is obtained by decoding a received encoded image or frame.
- step 938 image content is rendered using the received, e.g., decoded image or images, the UV map which was to be used in rendering the received images, and the environmental model.
- Step 938 involves applying the first image to the environmental model in accordance with UV map to be used.
- the first image is used as a texture which is applied to segments of the environmental model in accordance with the applicable UV map, e.g., a first UV map.
- the rendering may be performed separately for left and right eye views.
- step 938 includes step 940 .
- the first image is rendered by using the first texture map (UV map) corresponding to a first resolution allocation to apply at least a portion of the first image to a surface of a first portion, e.g., first segment, of the model of the environment.
- a first set of pixels of the first image may be mapped to the first segment of the mesh model of the environment based on the first texture map.
- the second image is rendered by using the first texture map (UV map) corresponding to a first resolution allocation to apply at least a portion of the second image to a surface of the first portion, e.g., the first segment, of the model of the environment.
- a first set of pixels of the second image may be mapped to the first segment of the mesh model of the environment based on the first texture map.
- images of portions of the environment not included in the first image e.g., the sky or ground portions, are rendered, e.g., applied to the environmental model in accordance with a UV map relevant to these portions. It should be appreciated in some embodiments separate sky and ground portions are not communicated with such portions being part of the first and second images in some embodiments.
- step 946 which is performed for each eye view, rendered images corresponding to different portions of a 360 degree simulated environment are combined to the extent need to provide a contiguous viewing area to a user.
- Step 946 is performed separately for the left and right eye images since while the ground and sky portions may be the same for these images when they are presented in non-stereo format, the other portions of the left and right eye images may include differences which may result in the perception of depth when the left and right eye images are viewed by different eyes of a user of the playback device.
- step 950 the rendered image content is displayed to a user of the playback device, e.g., on a display screen.
- a first rendered image or a combined image generated from recovered first image content is displayed for viewing by one of a users left and right eyes or both eyes if stereoscopic display is not supported.
- step 954 which is performed in the case of stereoscopic display, a second rendered image is displayed to a second one of a users left and right eyes.
- the displayed rendered second image is an image that was generated from recovered second image data or a combination of recovered, e.g., decoded second image data and data from another image, e.g., a sky or ground image portion.
- step 956 content corresponding to another image or pair of images is received and processed.
- the image or images received in step 956 may be and sometimes do correspond to a second group of pictures and corresponds to a different point in time than the first image.
- the players actors or an area of motion may have shifted position from where the activity was at the time the first image was captured.
- the players on a field may have moved left triggering the sever providing the third image to use a resolution allocation giving more resolution to the left portion of the front field of view than a center or right portion where the action was at the time the first image was captured.
- the different resolution allocation e.g., a second resolution allocation by the server or encoding device, will correspond to specification that the playback device should use a different UV map, e.g., a second texture map, for rendering the third image than the first image.
- the second UV map may specify using fewer pixels from the third image to map to the first segment than were used to map from the first image to the first segment of the environmental map and to use more pixels from third image to map to a second segment located in the left side of the forward field of view in the environmental model where the action is now located at the time of capture of the third image than were used to map to the second segment of the environmental map from the first frame.
- Step 956 includes, in some embodiments, steps 958 , 960 , 962 , 926 and/or 968 .
- step 958 a third encoded image is received.
- step 960 which is optional, a fourth encoded image is received.
- step 962 which is an alternative to steps 958 , 960 an encoded frame including one or both of the third and fourth encoded images is received.
- the third encoded image may be a first image of a second stereoscopic image pair with the third and fourth images being left and right eye images to be displayed to a user of the playback device.
- step 956 in addition to receiving image content which can be mapped to the environmental model, in step 968 a second indicator indicating which of a plurality of texture maps corresponding to different resolution allocation is to be used with the received third and/or fourth encoded images is also received. If a new texture map indicator is not received, in step 968 and a new texture map is not received, the playback device will continue to use the last UV map which was being used. Rather than receive a texture map indicator, a new texture map may be received in step 970 which is to be used in rendering the received third and/or fourth images.
- step 970 the received third and/or fourth image or images are decoded.
- the third encoded image is decoded to recover a third image.
- the fourth encoded image is decoded to recover a fourth image.
- the third and fourth images may be left and right eye views.
- the third and fourth images are included in a single encoded frame decoding of the received frame and separation of the third and fourth images may be performed in step 972 to produce left and right eye images which may be and sometimes are applied separately to the environmental map to generate separate, potentially different, left and right eye views.
- the third and/or fourth images communicate a complete 360 degree environment or panoramic view.
- the third and fourth images may correspond to a portion of the environment, e.g., a front portion or a 360 degree middle panoramic portion but not the sky and ground.
- step 978 other images which are sent with the encoded third and fourth images or in a separate stream may be decoded to obtain textures for portions of the environment which are not provided by the third and/or fourth images.
- a sky or ground image is obtained by decoding a received encoded image or frame.
- step 980 image content is rendered using the received, e.g., decoded image or images, the UV map which was to be used in rendering the received images, e.g., the second UV map, and the environmental model.
- Step 980 involves applying the third image to the environmental model in accordance with second UV map to be used which results in a different allocation of pixels from the received image to the model of the environment than occurred when using the first UV map. Thu, as part of the rendering the third image is used as a texture which is applied to segments of the environmental model in accordance with the applicable UV map, e.g., the second UV map.
- the rendering may be performed separately for left and right eye views.
- step 980 includes step 982 .
- the third image is rendered by using the second texture map (UV map) corresponding to a second resolution allocation to apply at least a portion of the third image to a surface of the first portion, e.g., first segment, of the model of the environment.
- a first set of pixels of the third image may be mapped to the first segment of the mesh model of the environment based on the second texture map where the first set of pixels includes fewer pixels than the first set which were mapped when the first UV map was used.
- a second set of pixels may be mapped to a second segment of the model where the second set of pixels includes more pixels than were mapped to the second segment when the first UV map was used.
- step 978 which may be performed in the case of stereo image playback, the fourth image is rendered by using the second texture map (UV map) corresponding to the second resolution allocation to apply at least a portion of the fourth image to a surface of the first portion, e.g., the first segment, of the model of the environment.
- the second UV map is used to control mapping of pixels from the fourth image to the second segment of the environmental model.
- images of portions of the environment not included in the first image are rendered, e.g., applied to the environmental model in accordance with a UV map relevant to these portions. It should be appreciated in some embodiments separate sky and ground portions are not communicated with such portions being part of the first and second images in some embodiments.
- step 988 which is performed for each eye view, rendered images corresponding to different portions of a 360 degree simulated environment are combined to the extent need to provide a contiguous viewing area to a user.
- Step 988 is performed separately for the left and right eye images since while the ground and sky portions may be the same for these images when they are presented in non-stereo format, the other portions of the left and right eye images may include differences which may result in the perception of depth when the left and right eye images are viewed by different eyes of a user of the playback device.
- step 990 the rendered image content is displayed to a user of the playback device, e.g., on a display screen.
- step 992 a third rendered image or a combined image generated from recovered third image content is displayed for viewing by one of a users left and right eyes or both eyes if stereoscopic display is not supported.
- step 994 which is performed in the case of stereoscopic display, a fourth rendered image is displayed to a second one of a user's left and right eyes.
- the displayed rendered fourth image is an image that was generated from recovered fourth image data or a combination of recovered, e.g., decoded fourth image data and data from another image, e.g., a sky or ground image portion.
- the process of receiving and decoding images and rendering images using the UV map provided or specified by the server providing the images occurs on an ongoing basis as represented with operation proceeding from step 990 back to step 914 via connecting node B 996 allowing for additional images to be received and processed, e.g., a new first and second images.
- the images correspond to a live sporting event with the server providing the images specifying different UV maps to be used during different portions of the sporting event based on where the action is occurring on the sports field with the generation of the images to be transmitted in encoded form taking into consideration the UV map which will be used to render the images.
- the server provides the images specifying different UV maps to be used during different portions of the sporting event based on where the action is occurring on the sports field with the generation of the images to be transmitted in encoded form taking into consideration the UV map which will be used to render the images.
- FIG. 11 which illustrates an image capture and content streaming method in accordance with an exemplary embodiment.
- the method 1100 shown in FIG. 11 starts in step 1102 when it is time to capture images, e.g., images corresponding to an event such as a sporting event or music performance.
- From start step 1102 operation proceeds along a plurality of paths, the paths bringing with steps 1114 , 1104 , 1106 , 1108 , 1110 , 1112 , which may be performed in parallel and, optionally, asynchronously.
- the camera rig 1300 can be used as the rig 102 of the FIG. 1 system and includes a plurality of stereoscopic pairs each corresponding to a different one of three sectors.
- the first camera pair 1301 includes a left eye camera 1302 and a right camera 1304 intended to capture images corresponding to those which would be seen by a left and right eye of a person.
- Second sector camera pair 1305 includes left and right cameras 1306 , 1308 while the third sector camera pair 1309 includes left and right cameras 1310 , 1312 .
- Each camera is mounted in a fixed position in the support structure 1318 .
- An upward facing camera 1314 is also included.
- a downward facing camera which is not visible in FIG. 13 may be included below camera 1314 .
- Stereoscopic camera pairs are used in some embodiments to capture pairs of upward and downward images however in other embodiments a single upward camera and a single downward camera are used.
- a downward image is captured prior to rig placement and used as a still ground image for the duration of an event. Such an approach tends to be satisfactory for many applications given that the ground view tends not to change significantly during an event.
- the output of the cameras of the rig 1300 are captured and processed by the method of FIG. 11 which will now be discussed further.
- Image capture steps shown in FIG. 11 are normally performed by operating a camera of the camera rig 102 to capture an image while encoding of images is performed by encoder 112 with responses to streaming requests and streaming of content being preformed by the streaming server 114 .
- step 1114 an image is captured of the ground, e.g., beneath rig 102 . This may happen prior to rig placement or during the event if the rig includes a downward facing camera. From step 1114 operation proceeds to steps 1144 where the captured image is cropped prior to encoding in step 1145 . The encoded ground image is then stored pending a request for content which may be responded to by supplying one or more encoded images in step 1146 to a requesting device.
- step 1104 monitor for request for content occurs, e g., by content server 114 .
- step 1128 a request for content is received from a playback device, e.g. device 122 located at customer premise 106 .
- the playback device In response to the content request the playback device is provided with information including one or UV maps corresponding to different resolution allocations which may be used.
- step 1128 is performed in cases where an environmental map was generated and/or other environmental information which may be different from a predetermined default setting or environmental is supplied to the playback device to be used in rendering images as part of an environmental simulation.
- step 1132 a playback device requesting content is provided the information need to model the environment and/or with other information which may be needed to render images onto the model.
- step 1132 may optionally include communication of a set of UV maps to the playback device requesting content for future use, e.g., with some different UV maps corresponding to different resolution allocations but the same area of a model in some embodiments.
- each of the sectors corresponds to a known 120 degree viewing area with respect to the camera rig position, with the captured images from different sector pairs being seamed together based on the images known mapping to the simulated 3D environment. While a 120 degree portion of each image captured by a sector camera is normally used, the cameras capture a wider image corresponding to approximately a 180 degree viewing area. Accordingly, captured images may be subject to masking in the playback device as part of the 3D environmental simulation or cropping prior to encoding.
- FIG. 14 is a composite diagram 1400 showing how a 3D spherical environment can be simulated using environmental mesh portions which correspond to different camera pairs of the rig 102 . Note that one mesh portion is shown for each of the sectors of the rig 102 with a sky mesh being used with regard to the top camera view and the ground mesh being used for the ground image captured by the downward facing camera.
- the overall meshes corresponding to different cameras results in a spherical mesh 1500 as shown in FIG. 15 .
- the mesh 1500 is shown for a single eye image but that it is used for both the left and right eye images in the case of stereoscopic image pairs being captured.
- Mesh information of the type shown in FIG. 14 can and sometimes is communicated to the playback device in step 1132 .
- the communicated information will vary depending on the rig configuration. For example if a larger number of sectors were used masks corresponding to each of the sectors would correspond to a small viewing area than 120 degrees with more than 3 environmental grids being required to cover the diameter of the sphere.
- Environmental map information is shown being optionally transmitted in step 1132 to the playback device. It should be appreciated that the environmental map information is optional in that the environment may be assumed to be a default size sphere in the event such information is not communicated having a predetermined number of segments arranged in a known mesh. In cases where multiple different default size spheres are supported an indication as to what size sphere is to be used may be and sometimes is communicated in step 1132 to the playback device.
- Step 1132 Operation proceeds from step 1132 to streaming step 1146 .
- Image capture operations may be performed on an ongoing basis during an event particularly with regard to each of the 3 sectors which can be captured by the camera rig 102 . Accordingly, processing paths starting with steps 1106 , 1108 and 1110 which correspond to first, second and third sectors of the camera rig are similar in terms of their content.
- the first sector pair of cameras is operated to capture images, e.g., a left eye image in step 1116 and a right eye image in step 1118 .
- FIG. 16 shows an exemplary image pair 1600 that may be captured in step 1106 .
- the captured images are then cropped in step 1134 , e.g., to remove undesired image portions such as image portions captured by another camera pair.
- a resolution allocation to be used for the captured left and right eye image is determined, e.g., selected. The selection may be based in information about which portion of the environment and thus captured images was important at the time of the capture of the images.
- the importance information may be based on detection of where individuals at the event being videoed are looking at the time of image capture, system controller input and/or the location of motion in the environment at the time of image capture.
- a resolution reduction operation is performed on the captured images in step 1146 based on the determined, e.g., selected, resolution allocation.
- the selected resolution allocation may be one of a plurality of supported resolution allocations corresponding to different supported UV maps corresponding to the portion of the environment captured by the first sector camera pair.
- the reduced resolution images generated in step 1146 are encoded.
- Information indicating the UV map to be used for rendering the reduced resolution images generated in step 1146 is generated in step 1149 and will, in some embodiments be associated with and transmitted with the encoded images generated in step 1146 so that the playback device can determine which UV map to use when rendering images recovered by decoding the encoded images generated in step 1146 .
- FIG. 17A shows an exemplary mesh model 1700 of an environment in accordance with the invention.
- FIG. 17B shows a UV map 1702 which can be used to map portions of a 2D image onto surfaces of the mesh model shown in FIG. 17A .
- FIG. 18 shows an exemplary result 2000 of cropping the left and right eye view images of FIG. 16 as may occur in step 1134 .
- the cropping of image pair shown in FIG. 18 may be performed prior to encoding and transmission to one or more playback devices.
- the image capture, cropping and encoding is repeated on an ongoing basis at the desired frame rate as indicate by the arrow from step 1149 back to step 1106 .
- step 1172 the encoded images generated from the captured images are streamed to a playback device along with the information indicating the UV maps to be used in rendering the encoded images being streamed.
- the UV maps are embedded in the content stream used to deliver the encoded images to requesting playback device or devices.
- FIG. 12 illustrates a method 1200 of operating a playback device or system, which can be used in the system of FIG. 1 , in accordance with one exemplary embodiment.
- the method 1200 beings in start step 1202 .
- the playback device transmits a request for content, e.g., to the streaming server of FIG. 1 .
- the playback device receives in step 1206 various information which may be used for rendering images.
- environmental model information may be received in step 1206 as well as one or more UV maps corresponding to different resolution allocations for one or more regions of the environment.
- the playback device may receive environmental model and/or UV map information corresponding to different resolution allocations.
- the information received in step 1206 is stored in memory for use on an as needed basis.
- step 1208 Operation proceeds from step 1206 to step 1208 in which one or more images are received, e.g., image captured of an environment to be simulated while an event was ongoing in the environment.
- step 1210 information indicating which UV maps are to be used for rendering the one or more received images is indicated.
- the information indicates which UV map in a set of UV maps corresponding to different resolution allocation which may have been used for a portion of the environment is to be used for rendering left and right eye images of a frame pair, e.g., corresponding to a front portion of an environment.
- step 1212 one or more of the received images are decoded.
- Operation proceeds from step 1212 to steps 214 in which the decoded images corresponding to surfaces of the environmental model are applied using one or more UV maps corresponding to the indicated resolution allocation that was used to generate the decoded image or images.
- Operation proceeds from steps 1214 to step 1218 in which image areas corresponding to different portions of the 360 degree simulated environment are combined to the extent needed to generate a contiguous image of a viewing area to be displayed.
- step 1220 the images are output to a display device with, in the case of stereoscopic image content, different images being displayed to a user's left and right eyes. Operation process from step 1220 back to step 1204 with content being requested, received and processed on an ongoing basis.
- FIG. 19 is a drawing 2100 that illustrates mapping of an image portion corresponding to a first sector to the corresponding 120 degree portion of the sphere representing the 3D viewing environment.
- step 1216 images corresponding to different portions of the 360 degree environment are combined the extent needed to provide a contiguous viewing area to the viewer, e.g., depending on head position. For example, ins step 1218 if the viewer is looking at the intersection of two 120 degree sectors portions of the image corresponding to each sector will be seemed and presented together to the viewer based on the know angle and position of each image in the overall 3D environment being simulated.
- FIG. 20 is an illustration 2200 showing the result of applying textures to mesh models to form a complete 360 degree view of an environment which may be presented to a user viewing the environment from the perspective of being located in the center of the illustrated environment and with the images applied to the inside of the spherical environment.
- the result of the simulation and display is a complete world effect in which a user can turn and look in any direction.
- the mapped images are output to a display device in step 1220 for viewing by a user.
- the images which are displayed will change over time based on the received images and/or because of changes in head position or the user selected viewer position.
- the images corresponding to the environment which are communicated to a playback device exceed the area a user can view at a given time so that content is available in the event the user changes his/her viewing angle by, for example, moving his/her head.
- the playback device has enough information to provide images should the user's viewing angle change without the playback device having to wait for new images or other content corresponding to a portion of the environment which the user was not previously viewing.
- the environment is represented using a mesh.
- Images are captured and encoded into frames, e.g., frames intended for viewing by a left eye and frames intended to be viewed by a right eye. While the techniques are described in the context of 3D stereoscopic applications, the methods can be used for stereoscopic viewing as well with a single stream of frames being communicated rather than a stream of frame pairs.
- the techniques are used to communicate images corresponding to a 360 degree viewing area.
- the techniques may be used for communicating images corresponding to less than a 360 degree viewing area, e.g., with a single frame communicating image content corresponding to the 360 degree viewing area.
- the methods and apparatus of the present invention are particularly well suited for streaming of stereoscopic and/or other image content where data transmission constraints may make delivery of 360 degrees of content difficult to deliver at the maximum supported quality level, e.g., using best quality coding and the highest supported frame rate.
- the methods are not limited to stereoscopic content.
- images corresponding to a 360 degree or other area are captured and combined to form an image of the area.
- the different portions of the image content of the area e.g., a 360 degree environment
- a frame which is to be encoded and transmitted. Separate frames may be generated and transmitted for each of the left and right eye views.
- the mapping of the captured images to the frame may, and in some embodiments is, different for different areas of the environment. For example, the front view portion of the environment may be preserved at full or near full resolution, with the sides and back being incorporated into the frame at lower resolutions.
- Images corresponding to the top and bottom of a 360 degree environment may be incorporated into the frame at a different, e.g., lower, resolution than the front and/or side views.
- images corresponding to the top and bottom of an environment are sent separately and, in many cases, as static images or at a different rate than images corresponding to the other portions of the environment.
- a frame communicating an environment may use different numbers of pixels to represent the same size area of a physical environment. For example, a larger number of pixels may be used to represent a forward viewing area with a lower number of pixels being used to represent a rear viewing area. This represents selective downsampling at the time of generate a frame representing the multiple image areas.
- a decoder a the image is mapped or wrapped onto a 3D model of the environment as part of the display process in some embodiments.
- the map is sometimes referred to as a UV map since UV coordinates are used in mapping the two dimensional frame that is communicated to XYZ space of a 3D model of the environment.
- the grid (UV map) used to map the transmitted frame to the 3D model takes into consideration the reduced resolution used in some embodiments for the back and side portions of the environment.
- the map used to wrap a communicated frame unto the model of the environment may change to reflect the different allocations of resolution to different portions of the environment. For example, portions of the environment having high motion may be allocated more resolution at points in time when there is high motion and less resolution at other times.
- Information on how the transmitted frame should be processed by the decoder to take into consideration the allocation of different amounts of resources, e.g., pixels, to different image areas at different points in time is communicated to the playback device and used to interpret the communicated frame and how it should be applied to the 3D environment.
- the method used in various embodiments may be referred to as use of selective resolution allocation in a Panoramic Image map.
- This approach allows the encoder and playback device to use a UV map to optimize the resolution in a equi rectangular projection so that more of the limited number of pixels available in a communicated frame are used for the more important image element(s) and pixels aren't wasted on image areas of low importance.
- the methods and apparatus are particularly well suited for devices with limited pixel buffers, such as phones where every pixel is precious because of the phones fairly limited pixel buffer which is available for decoding images.
- FIG. 21 shows a 3D mesh model 2300 of an environment onto which captured images are to be wrapped as part of the process of rendering an image on a display device.
- the 3D model 2300 includes a sky mesh 2300 , a 360 degree panoramic mesh 2308 and a ground mesh 2310 .
- a frame representing an image of the sky is transmitted.
- a map is used to determine which parts of the transmitted frame are applied to which segments of the sky mesh.
- the sky map includes one segment for each segment of the sky mesh and provides a method of determining which portion of a frame representing an image in what is sometimes referred to as UV space will map to the segments of the sky mesh 2306 .
- the frame representing the image of the sky is sent once and is thus static or sent at a low rate much less frequently than images to be mapped to the 360 degree panoramic mesh portion of the model 2300 .
- a frame representing an image of the ground is transmitted.
- a ground map is used to determine which parts of the transmitted frame are applied to which segments of the ground mesh.
- the ground map includes one segment for each segment of the ground mesh 2310 and provides a method of determining which portion of a frame representing an image in what is sometimes referred to as UV space will map to the segments of the ground mesh 2310 .
- the frame representing the image of the ground is sent once and is thus static or sent at a low rate much less frequently than images to be mapped to the 360 degree panoramic mesh portion of the model 2300 .
- frames corresponding to the 360 degree mesh portion since this includes the areas of the environment which tend to be most frequently viewed. While the image of this environmental area may be captured at a consistent resolution as represented by the uniform segments of the uncompressed panoramic image map 2302 , different areas to which the panoramic image and panoramic mesh correspond may be of different amounts of importance at different times. For example, frontal areas were the main action is ongoing and/or areas with high motion may be important to represent in detail while other environmental areas may be less important.
- the uniform allocation of limited resources in terms of pixels of a frame to different areas of an environment is wasteful when the importance of the different image areas is taken into consideration along with the fact that the pixels of the frame are a limited resource.
- a map may be used to allocate different numbers of pixels to different portions of the 360 degree mesh.
- some portions of the mesh 2308 may be coded using more pixels than other portions.
- a panoramic image map 2304 with non-uniform segments sizes may be used. While in the case of the FIG. 21 map each segment of the map will be used to map pixels of a received frame to a corresponding segment of the panoramic mesh 2308 , some segments will use more pixels from the transmitted image than other segments. For example, mode pixels will be allocated to the middle portions of the panoramic mesh in the FIG.
- the map 2304 is used to map portions of a received frame to the mesh 2308
- one or more segments of the uncompressed image of representing the panoramic environment will be downsampled taking into consideration the panoramic image map. For example, portions of an uncompressed image representing the top and bottom portions of the environment will be downsampled to reflect the small number of pixels allocated in the panoramic image map for representing such image portions while other portions may be subject to lower or no downsampling.
- the panoramic image map is generated in some embodiments based on scene analysis and/or taking into consideration a user viewing position.
- the panoramic image map may be and in some embodiments is changed over time as the location of the main action changes, e.g., ball position during a sporting event in a stadium environment changes. The change is normally limited to occur on a group of pictures boundary within a video stream and/or upon a scene change boundary such as a boundary associated with the start or end of a commercial break in a video sequence.
- the new map to be used for interpreting frames may be transmitted at a playback device with or prior to a frame which is constructed taking into consideration the new map. Alternatively the playback device may so a variety of predetermined maps which may be used for mapping received frames to the mesh model of the environment and the video stream may include information indicating which of the plurality of maps is to be used for a particular set of communicated frames.
- the selective allocation and varying of the image map to take into consideration content and/or user viewing position can be applied to a full 306 degree area or some small portion of the environment. Accordingly, while shown with an example that maps a frame to a 360 degree environmental area the same method may be applied to a map that is used to map a frame to a 180 degree image area or some other portion of a 360 degree environment. While the map used to map an image to corresponding segments of a 3D model may change, the model need not change. However, changes in the model may be made to reflect changes in the environment, e.g., when a stage is moved and/or other changes in the environment are detected. Thus, both map and model changes are possible.
- FIG. 21 shows how selective resolution can be used with regard to an image, e.g., frame, which maps to an environmental grid corresponding to, for example, a 360 spherical panoramic mesh. Separate images may be communicated for applying as textures to the sky and ground mesh portions of the world model shown in FIG. 21 .
- image e.g., frame
- Separate images may be communicated for applying as textures to the sky and ground mesh portions of the world model shown in FIG. 21 .
- the panoramic image 2302 prior to compression, corresponding to the 360 degree panoramic mesh 2308 includes image content at a generally uniform resolution in the example.
- image content at a generally uniform resolution in the example.
- the use of a fisheye lens may introduce some distortions and thus differences in resolution due to lens issues.
- image capture results in an image with a uniform resolution.
- the grid applied to the panoramic image 2302 is uniform and if used as a UV map would result in uniform resolution allocation to the segments of the 360 degree panoramic portion of the mesh model 2308 .
- the upper and lower portions are subject to a resolution reduction operation and the UV map to be used during playback is adjusted accordingly.
- mesh 2304 which represents a UV map to be used to render a resolution adjusted image corresponding to the 360 panoramic area of the mesh model
- the grid sizes are smaller.
- the playback device will use the UV mesh which reflects the resolution reduction applied to an image prior to transmission to the playback device when rendering the received image, e.g., applying the received image as a texture, onto the surface of the environmental model, e.g., mesh model of the environment.
- a static UV map reflecting a resolution reduction operation may be and is used in some embodiments, it may be desirable in at least some embodiments where the portion of the environment with the highest priority may change to support the dynamic selection of a resolution allocation approach to use and to use a UV map corresponding to the selected resolution allocation. In such a way, resolution allocation may be changed to reflect which portion of the environment is given priority in terms of resolution at a given time.
- FIG. 22 represented by reference number 2400 , shows a first captured image 2402 of a first portion of an environment. Each large dot represents a pixel.
- the image 2402 is of uniform resolution as represented by the 4 pixels in each square grid area. Small dots are used to indicate that the image continues and extends toward the other illustrated portions of the image 2402 .
- a first resolution allocation e.g., a resolution which gives priority to the middle portion of the image 2402
- resolution will be preserved during the middle portion of the image 2402 but reduced for the left and right portions.
- Such a resolution allocation may be desirable where, for example, the image 2402 is of a sports field and the action is at the center portion of the sports field when image 2402 is captured.
- Arrows extending from image 2402 towards reduced resolution image 2404 represent the application of a first selective resolution reduction operation to image 2402 to generate image 2404 .
- the first resolution reduction operation may involve a downsampling applied to the left and right portions of image 2402 but not the middle portion.
- the grid shown as being applied to image 2404 represents the resolution allocation used to generate image 2404 from image 2402 .
- the first resolution adjusted image 2404 includes half as many pixel in the two left and right most rows of the image as did image 2402 but the same number of pixels for segments towards the center portion of the image 2404 .
- Grid 2406 represents a first UV map corresponding to the first resolution allocation which is suitable for mapping segments of the image 2404 to segments of the model of the environment.
- FIG. 23 represented by reference number 2500 , shows a first captured image 2502 of the first portion of the environment.
- each large dot represents a pixel.
- the image 2502 is of uniform resolution as represented by the 4 pixels in each square grid area. Small dots are used to indicate that the image continues and extends toward the other illustrated portions of the image 2502 .
- a second resolution allocation e.g., a resolution which gives priority to the left and middle portions of the image 2502
- resolution will be preserved in the left and middle portions of the image 2502 but reduced for the right portion.
- Such a resolution allocation may be desirable where, for example, the image 2502 is of a sports field and the action is at the left portion of the sports field when image 2502 is captured.
- Arrows extending from image 2502 towards reduced resolution image 2504 represent the application of a second selective resolution reduction operation to image to generate image 2504 .
- the second resolution reduction operation may involve a downsampling applied to the right portion of image 2502 but not the left or middle portions. Note that the area to which the downsampling is applied is of a size equal to the area to which downsampling was applied in FIG. 22 .
- images 2404 and 2504 will have the same total number of pixels but with the resolution being different in different areas of the images 2404 , 2504 .
- the grid shown as being applied to image 2504 represents the resolution allocation used to generate image 2504 from image 2502 .
- the second resolution adjusted image 2504 includes half as many pixel in the four right most rows of the image as did image 2502 but the same number of pixels for segments towards the left and center portions.
- Grid 2506 represents a first UV map corresponding to the first resolution allocation which is suitable for mapping segments of the image 2504 to segments of the model of the environment.
- FIG. 24 represented by reference number 2600 , shows a first captured image 2602 of the first portion of the environment.
- each large dot represents a pixel.
- the image 2602 is of uniform resolution as represented by the 4 pixels in each square grid area. Small dots are used to indicate that the image continues and extends toward the other illustrated portions of the image 2602 .
- a third resolution allocation e.g., a resolution which gives priority to the middle and right portions of the image 2602 , resolution will be preserved in the middle and right portions of the image 2602 but reduced for the right portion.
- Such a resolution allocation may be desirable where, for example, the image 2602 is of a sports field and the action is at the right portion of the sports field when image 2602 is captured.
- Arrows extending from image 2602 towards reduced resolution image 2604 represent the application of a third selective resolution reduction operation to image to generate image 2604 .
- the third resolution reduction operation may involve a downsampling applied to the left portion of image 2602 but not the right or middle portions. Note that the area to which the downsampling is applied is of a size equal to the area to which downsampling was applied in FIGS. 22 and 24 .
- images 2604 will have the same total number of pixels as images 2404 , 2504 but with the resolution being allocated differently in terms of the portion of the environment to which higher resolution is allocated.
- the grid shown as being applied to image 2604 represents the resolution allocation used to generate image 2604 from image 2602 .
- the third resolution adjusted image 2604 includes half as many pixel in the four left most rows of the image as did image 2602 but the same number of pixels for segments towards the right and center portions.
- Grid 2606 represents a first UV map corresponding to the first resolution allocation which is suitable for mapping segments of the image 2604 to segments of the model of the environment.
- UV map 2406 is communicated to a playback device for use with an image generated suing the first resolution allocation.
- UV map 2406 is communicated to a playback device for use in rendering an image generated using the second resolution allocation and UV map 2606 is communicated to the playback device for use in rendering an image generated using the third resolution allocation.
- the streaming system and the playback system both store the set of UV maps 2406 , 2506 , 2606 with the streaming system indicating which UV map should be applied to which image and the rendering device, e.g., playback device, using the indicated UV map associated with a received image.
- FIG. 25 which comprises FIGS. 25A and 25B illustrates an exemplary method 2900 of operating a content processing and delivery system in accordance with an exemplary embodiment.
- FIG. 25A shows the first part of method 2900 .
- FIG. 25B shows the second part of method 2900 .
- the method 2900 shown in FIG. 25 starts in step 2902 with the content processing and delivery system being initialized to process and deliver content, e.g., image content and/or information used to render images.
- the method of flowchart 2900 is performed using the content delivery and processing system 700 of FIG. 7 .
- step 2904 operation proceeds to steps 2904 and 2906 , which may be performed in parallel and, optionally, asynchronously.
- customer rendering and playback devices are provided with information that can be used in rendering of image content and/or providing 3D playback experience to the viewers. In some embodiments this includes providing environmental model and/or other environmental information to the customer devices to be used in rendering images as part of an environmental simulation.
- step 2904 a 3D environmental model and/or information that can be used to model is communicated to one or more customer devices.
- the model is a mesh model of the environment from which one or more images are captured.
- UV maps are also communicated to the customer devices, e.g., content playback devices, in step 2905 .
- the UV maps correspond to different resolution allocations with different UV maps, also referred to as texture maps, providing different mappings of pixels of transmitted images to segments of the environmental model. If the UV maps are communicated in step 2905 they can later be identified when they are to be used to map a transmitted image and need not be retransmitted multiple time to the playback device. However, in some embodiments a set of UV maps is not communicated in step 2905 and an applicable UV map is transmitted with or prior to communication of an image to which the UV map is to be applied and used.
- the information in steps 2904 and 2905 is communicated once, e.g., prior to communicating actual image content to the customer devices. While environmental map information and/or environmental models may be communicated to the playback device in some embodiments where such information is generated and/or available at the server side, in some other embodiments the environment may be assumed to be a default size and shape, e.g., a sphere or half sphere and in such a case the default environmental module and/or UV maps may be preloaded in the playback device and need not be transmitted by the server.
- step 2906 The processing of image content begins in step 2906 which can be performed in parallel with steps 2904 , 2905 .
- image content is received by the processing system, e.g., content delivery system 700 shown in FIG. 7 .
- the image content received in step 2906 may be from an image capturing device such as the ones discussed in the application such as the one shown in FIG. 13 .
- the step 2906 of receiving image content includes step 2908 wherein a first image corresponding to a first portion of an environment, e.g., environment of interest where images are captured, is received.
- the first image is one image of an image pair that also includes a second image, with the first image being one of a left and right eye image pair, the second image being a second one of a left and right eye image pair.
- the first and second images are received as part of the image pair in step 2906 .
- step 2906 further includes step 2910 of receiving the second image.
- step 2912 the system selects a first resolution allocation to be used for at least one image corresponding to a first portion of the environment.
- This selection may be and sometimes is based on detection of motion in the received image content, the location of particular objects such as a sports jersey, and/or human input indicating which portion of the captured image is to be given priority and preserved at a higher resolution during encoding. For example, detection of player's jerseys or uniforms may indicate areas to be preserved at high resolution in which case a resolution allocation which preserves the areas where the uniforms are detected may and in some embodiments will be selected.
- Other portions of the image may be and sometimes are subject to down sampling. Each resolution may correspond to a particular UV map which is intended to be used for mapping images produced by using a particular corresponding resolution allocation.
- step 2914 in which it is determined if the selected first resolution allocation is different from a previously selected resolution allocation, e.g, indicative of a change in down sampling and UV map.
- the selected first resolution allocation may be one of a plurality of supported resolution allocations corresponding to different supported UV maps corresponding to the first portion of the environment captured in the first image.
- a resolution allocation may be selected at a given time to process a current image and/or group of images.
- step 2914 it is determined that the selected first resolution allocation is different than the previously selected resolution allocation the operation proceeds from step 2914 to step 2916 where new downsampling and/or filtering information corresponding to the newly selected resolution allocation used to control resolution reduction is loaded and then operation proceeds to step 2918 . If in step 2914 it is determined that the selected first resolution allocation is the same as the previously selected resolution allocation (or is the same as a default allocation if no previous selection was made) then there is no need for new down sampling and/or filtering information to be loaded and thus the operation proceeds directly to step 2918 .
- the selected resolution allocation for an image indicates how down sampling is to be applied to an image which is to be encoded and transmitted to the playback device.
- step 2918 a resolution reduction operation, e.g., downsampling, is performed on the first image of the first portion of the environment in accordance with the selected first resolution allocation to generate a first reduced resolution image 2919 .
- the first reduced resolution image 2919 which is the output of step 2918 includes at least some image portions having different resolutions.
- step 2920 Operation proceeds from step 2916 to step 2920 in embodiments where pairs of images are processed, e.g., stereoscopic image pairs including left and right eye views.
- a resolution reduction operation is performed on the second image of the first portion of the environment, e.g., the second image in stereoscopic image pair, in accordance with the selected first resolution allocation to generate a second reduced resolution image 2921 .
- the second reduced resolution image 2921 which is the output of step 2918 includes at least some image portions having different resolutions.
- step 2920 is shown as being performed after step 2918 it may be performed in parallel with step 2918 simultaneously.
- the data output of steps 2918 and 2920 e.g., the generated first and second reduced resolution images 2919 and 2921 , serve as inputs to the next step 2922 .
- the next step 2922 In the case of non-stereo image content, a single image will be processed and the second image will not be present.
- step 2922 the reduced resolution image 2919 and/or reduced resolution image 2921 are encoded.
- step 2924 the first reduced resolution image is encoded.
- step 2926 the second reduced resolution image, when present, is encoded.
- step 2928 the encoded reduced resolution images are stored in memory, e.g., for subsequent communication, e.g., streaming to a content playback device, e.g., located at a customer premises such as a house or home. Operation proceeds from step 2928 to step 2930 via connecting node B 2929 .
- step 2930 the encoded reduced resolution image(s) are communicated to a playback device. This may involve transmitting, e.g., streaming, the images to the playback device over a wired network, cable network or wireless network or some other type of network.
- Step 2930 includes steps 2932 and step 2934 .
- step 2932 the first reduced resolution image is communicated to the customer playback device, e.g., in encoded form and in step 2934 in the second reduced resolution image is communicated to the playback device, e.g., in encoded form.
- Step 2934 is performed when a stereo pair of images is communicated, e.g., in a single frame or pair of frames.
- step 2936 Operation is shown proceeding from step 2930 to step 2936 .
- step 2936 may precede step 2930 .
- a texture map e.g., first texture map
- the identification of the first texture map may be sufficient where the first texture map, e.g., UV map, was already loaded into the playback device e.g., as part of step 2905 .
- the playback device Based on the communicated information and/or map, the playback device knows that it is to use the first UV map with the first and second images which were produced using the first resolution allocation to which the first UV map corresponds.
- the first UV map may be used by the playback device to render other images which are also produced in accordance with the first resolution allocation.
- a resolution allocation is maintained for a group of pictures and thus the same UV map may be used for multiple consecutive images in such embodiments.
- step 2936 the playback device is provided information about what texture map to use while rendering the first and second images to step 2938 which relates to processing of an additional image or images, e.g., a third image and/or forth image.
- the third and/or fourth image may be and in some embodiments are left and right images of a second stereoscopic image pair or some other image or images of the environment captured after the first image.
- a second resolution allocation is selected to be used for the received images, e.g., third and/or fourth images.
- the resolution allocation may be determined using the same techniques used to determine the first resolution allocation, e.g., identifying an area or areas of importance based on motion, presence of an object such as sports jersey, ball, etc.
- the second resolution allocation is selected from the set of resolution allocations, e.g., each corresponding to a different UV map, operation proceeds to step 2942 .
- a check is made to determine if the second resolution allocation is different from the first resolution allocation.
- the second resolution allocation may be different, e.g., because the ball or players may have moved to a different portion of the field since the first image was captured.
- step 2944 the new downsampling and/or other resolution allocation information is loaded so that it can be used in the resolution reduction step 2946 . If in step 2942 it is determined that the second resolution allocation is the same as the first, the processor of the system implementing the method 2900 already knows the downsampling to be preformed since it was used process the first image and need not load new downsampling information and operation proceeds to step 2946 .
- step 2946 a resolution reduction operation, e.g., downsampling, is performed on the received third and/or fourth image to produce reduced resolution versions of the third and/or fourth images 2947 . Operation proceeds from step 2946 to step 2948 in which the reduced resolution third and/or fourth images are encoded prior to being communicated, e..g, transmitted, to the playback device in step 2950 .
- step 2948 the reduced resolution third and/or fourth images are encoded prior to being communicated, e..g, transmitted, to the playback device in step 2950 .
- step 2952 which is shown being performed after step 2950 but which may and sometimes does precede step 2950 or occur in parallel with step 2950 , the information indicating the UV map to be used for rendering the third and fourth images is communicated to the playback device. This may involve sending the UV map to be used to the playback device or simply identifying a previously stored UV map. Since the third and fourth images were generated using the second resolution allocation the information will identify the UV map corresponding to the second UV allocation. Operation proceeds from step 2952 via connecting node 2954 to step 2906 where additional image content is received, e.g., from a camera device, and treated as new first and second images.
- additional image content is received, e.g., from a camera device, and treated as new first and second images.
- a sequence of images representing view may be received and processed with the resolution allocation used at a given time depending on the received image content and/or user input.
- the content playback device Over time as different resolution allocations are used, the content playback device will be signaled to use different corresponding UV maps.
- the playback device will be instructed to use a second different UV map to render images generated in accordance with the second resolution allocation which is different from a first UV map used to render images generated in accordance with the first resolution allocation.
- a large number of different resolution allocations can be used in combination with corresponding UV maps allowing for a wide variety of different resolution allocations to be supported.
- FIG. 26 illustrates an exemplary embodiment of a content playback method 2700 which may be, and in some embodiments is, implemented on exemplary computer system/content playback device 800 .
- the method 2700 may be used by a device which receives content encoded and transmitted in accordance with the method of FIG. 25 .
- the exemplary content playback method 2700 will be explained in connection with the playback device 800 shown in FIG. 8 . It should be appreciated that the method 2700 can be implemented on other apparatus.
- the exemplary playback method 2700 begins in start step 2702 from which operation proceeds to step 2704 .
- step 2704 the receiver 860 of the network interface 810 of the content playback device 800 receives a mesh model of an environment. Operation proceeds from step 2704 to step 2706 .
- step 2706 the receiver 860 of the network interface 810 of the content playback device 800 receives one or more image maps, e.g., one or more image UV maps, indicating a mapping between an image and the mesh model of an environment.
- step 2706 includes sub-step 2708 and/or sub-step 2710 .
- sub-step 2708 the receiver 860 of the network interface 810 of the content playback device 800 receives a first image map.
- sub-step 2710 the receiver 860 of the network interface 810 of the content playback device 800 receives a second image map. Operation proceeds from step 2706 to step 2712 .
- the content playback device 800 stores the received image map or maps in a storage device, e.g., memory 812 .
- a storage device e.g., memory 812 .
- UV MAP 1 836 and UV MAP 2 836 are stored in memory 812 .
- the received image maps are stored in a storage device coupled to the content playback device 800 . Operation proceeds from step 2712 to step 2714 .
- step 2714 the receiver 860 of the network interface 810 receives an encoded image. Operation proceeds from step 2714 to step 2716 .
- step 2716 the decoder 864 of the playback device 800 , decodes the received encoded image.
- a hardware decoder module decodes the received encoded images.
- the processor 808 executing instructions from decoder module 820 decodes the received encoded image. Operation proceeds from step 2716 to step 2718 .
- the decoded image is mapped to the mesh model of the environment in accordance with the first image map to produce a first rendered image.
- the first image map mapping different numbers of pixels of the decoded image to different segments of the mesh model of the environment. While the mapping of the different numbers of pixels of the decoded image to different segments of the mesh model of the environment may occur in a variety of different ways, in some embodiments, the different numbers of pixels are mapped to environmental regions of the same size but located at different locations in the environment. In some such embodiments, segments in the environment corresponding to action are allocated more pixels than segments in which less or no action is detected. In some embodiments, at least some segments corresponding to a front viewing area are allocated more pixels per segment than segments corresponding to a rear viewing area. This mapping may be, and in some embodiments is, performed by the processor 808 of the playback device 800 . Operation proceeds from step 2718 to step 2719 .
- step 2719 the first rendered image is displayed for example on display 802 of content display device 800 .
- operation proceeds from step 2719 to step 2720 .
- the receiver 860 of the network device 810 of the playback device 800 receives a signal indicating that a second image map should be used to map portions of received frames to the environmental mesh model.
- the decoded image is a frame.
- Operation proceeds from step 2720 to optional step 2722 .
- step 2722 in response to receiving the signal indicating that a second image map should be used to map portions of received frames to the environmental mesh model, a second image map is used to map portions of received frames to the environmental mesh model to produce one or more additional rendered images, e.g., a second rendered image.
- the second image map is the second image map received in step 2710 .
- the first image map allocates a first number of pixels of a frame to a first segment of said environmental mesh model wherein the decoded image is a frame and said second image map allocates a second number of pixels of the frame to the first segment of said environmental mesh model, the first and second number of pixels being different.
- the mapping of step 2722 may be, and in some embodiments is, performed by the processor 808 of the playback device 800 . Operation proceeds from optional step 2722 to optional step 2724 .
- step 2724 the additional rendered image(s), e.g., the second rendered image, is displayed for example on display 802 of content display device 800 . Operation proceeds from step 2724 to step 2704 where the method continues as previously described.
- the received mesh model of an environment is stored in a storage device, e.g., 3D environmental mesh model 832 stored in memory 812 of the playback device 800 .
- the received encoded image data which may be, and in some embodiments are encoded scene portions, is stored in a storage device, e.g., received encoded data 824 stored in memory 812 of the playback device 800 .
- the decoded image data is stored in a storage device, e.g., decoded data 826 stored in memory 812 of the playback device 800 .
- the one or more of the rendered images are stored in a storage device, e.g., memory 812 of the playback device 800 .
- the first and second images are rendered by the processor 808 executing instructions contained in the image generation module 822 .
- a hardware, e.g., circuits, image generation module performs the operation of rendering the one or more images, e.g., the first and/or second rendered images.
- the exemplary embodiment of method 2800 of communicating information to be used to represent an environment will now be described in connection with FIG. 10 .
- the exemplary method 2800 may be, and in some embodiments is, implemented by a content delivery system such as for example content delivery system 700 illustrated in FIG. 7 .
- Operation of the method 2800 begins in start step 2802 . Operation proceeds from step 2802 to step 2804 .
- a first image map to be used to map portions of a frame to segments of an environmental model are communicated, e.g., to a content playback device such as for example content playback device 800 illustrated in FIG. 8 .
- the first image map allocates different size portions of the frame to different segments of the environmental model thereby allocating different numbers of pixels to different segments of the environmental model.
- the network interface 710 of the content delivery system 700 performs this operation.
- the network interface 710 includes a transmitter 711 which performs this function. Operation proceeds from step 2804 to step 2806 .
- a first frame including at least a portion of a first image to be mapped to the environmental model using the first image map is communicated, e.g., to the content playback device 800 .
- the network interface 710 of the content delivery system 700 performs this operation.
- the network interface 710 includes a transmitter 711 which performs this operation. Operation proceeds from step 2806 to step 2808 .
- a second image map to be used to map portions of a frame to segments of the environmental mode is communicated, e.g., to the content playback device such as for example content playback device 800 .
- the second image map allocates different size portions of the frame to different segments of the environmental model thereby allocating different numbers of pixels to different segments of said model.
- the second image map allocates a different number of pixels to a first segment of the environmental model than are allocated by the first image map.
- the network interface 710 of the content delivery system performs this operation.
- the network interface 710 includes a transmitter 711 which performs this operation. Operation proceeds from step 2808 to step 2810 .
- a second frame including at least a portion of a second image to be mapped to the environmental model using the second image map is communicated e.g., to the content playback device such as for example content playback device 800 .
- the first and second image maps map different numbers of pixels to an area corresponding to the same portion of an environment thereby providing different resolution allocations for the same portion of the environment based on which of the first and second image maps are used.
- the network interface 710 of the content delivery system performs this operation.
- the network interface 710 includes a transmitter 711 which performs this operation. Operation proceeds from step 2810 to step 2804 where operation proceeds as previously described.
- FIGS. 27, 28 and 29 show how a playback device, such as the playback device or devices shown in any of the other figures, can perform image rendering using a UV map corresponding to the resolution allocation that was used to generate the image to be rendered.
- FIG. 27 shows how a reduced resolution image 2404 can be rendered using the UV map 2406 and an environmental module 3002 with environmental segments in the model corresponding to segments of the UV map.
- the top portion of FIG. 27 shows the relationship between segments of the UV map 2406 and the segments of the environmental model 3002 .
- a first segment of the UV map 2406 corresponds to a first environmental module segment (EMS 1) of environmental model 3002 , as represented by the solid arrow extending from the first segment of the UV map 2406 and EMS 1.
- a second environmental module segment (EMS 2) of environmental model 3002 corresponds to the second segment of the UV map 2406 as indicated by the dashed arrow extending from the second segment of the UV map 2406 and EMS 2.
- a third environmental module segment (EMS 3) of environmental model 3002 corresponds to the third segment of the UV map 2406 as indicated as represented by the dashed arrow extending from the second segment of the UV map 2406 and EMS 3.
- the UV map 2406 is used to determine how to apply portions of an image generated in accordance with the first resolution allocation to portions of the environmental model 3002 , as a texture.
- UV map 2404 is applied to the communicated image 2404 to determine how to segment the image 2404 into sets of pixels to be applied to the corresponding segments of the environmental model 3002 .
- the pixels in the segments of the image 2404 corresponding to a segment of the UV map 2406 are then applied to the corresponding segment of the environmental model 3002 , e.g., as a texture, with scaling and reshaping being used as necessary to cover the surface of the segment of the environmental model 3002 .
- the portion of the image applied to the corresponding segment of the environmental model 3002 is scaled and/or adjusted in shape as necessary to fully occupy the corresponding segment of the environmental model 3002 in some embodiments.
- two pixels of the communicated image corresponding to the first segment of the UV map 2406 are scaled to fully occupy the first segment EMS1 of the environmental model 3002 to which they are applied.
- the two pixels of the image 2404 being rendered, corresponding to the second segment of the UV map 2406 are scaled to fully occupy the second segment EMS2 of the environmental model 3002 to which they are applied as a texture.
- the third segment of the UV map corresponds to four pixels of the image 2404 to be rendered.
- the four pixels are applied as a texture to the third segment EMS 3 of the environmental model 3002 as a texture during the rendering process.
- the third segment of the environmental model 3002 is the same size as the first and second segments of the environmental model, the third segment will be of higher resolution than the first and second segments and correspond to more pixels in the received image 2404 than either of the first and second segments.
- the segments of the UV map 2406 corresponding to portions of an image which were subject to resolution reduction prior to encoding may correspond to the same size area of the environmental model 3002 of another segment which does not correspond to a resolution reduction operation.
- the segment corresponding to the area where resolution reduction was not performed will be displayed in the generated image of the simulated environment at a higher resolution than the portion to which resolution reduction was performed prior to encoding.
- UV maps 2406 , 2506 , 2606 may be used with the same environmental model 3002 .
- FIG. 28 shows the application of UV map 2506 to an image 2504 , generated using the second selective resolution reduction operation, which allocates less resolution to the right portion of an image corresponding to a portion of an environment than the left and middle portions.
- FIG. 28 shows how a reduced resolution image 2504 can be rendered using the UV map 2506 and the environmental model 3002 with environmental segments corresponding to segments of the UV map.
- the top portion of FIG. 28 shows the relationship between segments of the UV map 2506 and the segments of the environmental model 3002 .
- a first segment of the UV map 2506 corresponds to the first environmental module segment (EMS 1) of environmental model 3002 , as represented by the solid arrow extending from the first segment of the UV map 2506 and EMS 1.
- EMS 1 environmental module segment
- a second environmental module segment (EMS 2) of environmental model 3002 corresponds to the second segment of the UV map 2506 as indicated by the dashed arrow extending from the second segment of the UV map 2506 and EMS 2.
- a third environmental module segment (EMS 3) of environmental model 3002 corresponds to the third segment of the UV map 2506 as indicated by the dashed arrow extending from the second segment of the UV map 2506 and EMS 3.
- the UV map 2506 is used to determine how to apply an image to be rendered to the environmental model 3002 .
- FIG. 28 shows how the communicated image 2504 and the pixels in the segments of the image corresponding to a segment of the UV map are applied to the corresponding segment of the environmental model 3002 .
- the portion of the image 2504 applied to the corresponding segment of the UV map is scaled and/or adjusted in shape as necessary to fully occupy the corresponding segment of the UV map.
- four pixels of the communicated image corresponding to the first segment of the UV map 2504 are scaled to fully occupy the first segment EMS 1 of the environmental model to which they are applied.
- FIG. 28 shows how the communicated image 2504 and the pixels in the segments of the image corresponding to a segment of the UV map are applied to the corresponding segment of the environmental model 3002 .
- the portion of the image 2504 applied to the corresponding segment of the UV map is scaled and/or adjusted in shape as necessary to fully occupy the corresponding segment of the UV map.
- the four pixels of the image being rendered, corresponding to the second segment of the UV map are scaled to fully occupy the second segment EMS 2 of the environmental model 3002 to which they are applied as a texture.
- the third segment of the UV map also corresponds to four pixels of the image to be rendered. The four pixels are applied as a texture to the third segment of the environmental model as a texture during the rendering process.
- the third segment of the environmental model is the same size as the first and second segments of the environmental model
- the third segment will be of the same resolution as the first and second segments.
- resolution reduction is not applied to the left and middle portions of the image but resolution reduction is performed with regard to the right side of the image.
- segments corresponding to the right side of the image and thus the right side of the environmental model 3002 will be of lower resolution.
- FIG. 29 shows the application of UV map 2606 to an image 2604 , generated using the third selective resolution reduction operation, which allocates less resolution to the left portion of an image corresponding to a portion of an environment than the middle and right portions.
- FIG. 28 shows how a reduced resolution image 2604 can be rendered using the UV map 2606 and the environmental model 3002 with environmental segments corresponding to segments of the UV map 2606 .
- the top portion of FIG. 29 shows the relationship between segments of the UV map 2606 and the segments of the environmental model 3002 .
- a first segment of the UV map 2606 corresponds to the first environmental module segment (EMS 1) of environmental model 3002 , as represented by the solid arrow extending from the first segment of the UV map 2606 and EMS 1.
- EMS 1 environmental module segment
- a second environmental model segment (EMS 2) of environmental model 3002 corresponds to the second segment of the UV map 2506 as indicated by the dashed arrow extending from the second segment of the UV map 2606 and EMS 2.
- a third environmental module segment (EMS 3 ) of environmental model 3002 corresponds to the third segment of the UV map 2606 as indicated by the dashed arrow extending from the second segment of the UV map 2606 and EMS 3.
- the UV map 2606 is used to determine how to apply an image to be rendered to the environmental model 3002 .
- FIG. 29 shows how the communicated image 2604 and the pixels in the segments of the image corresponding to a segments of the UV map are applied to the corresponding segments of the environmental model 3002 .
- the portion of the image 2604 corresponding to a segment of the environmental model 3002 as indicated by the UV map 2606 is scaled and/or adjusted in shape as necessary to fully occupy the corresponding segment of the environmental model 3002 .
- two pixels of the communicated image 2604 corresponding to the first segment of the UV map 2606 are scaled to fully occupy the first segment EMS1 of the environmental model to which they are applied.
- FIG. 29 shows how the communicated image 2604 and the pixels in the segments of the image corresponding to a segments of the UV map are applied to the corresponding segments of the environmental model 3002 .
- the portion of the image 2604 corresponding to a segment of the environmental model 3002 as indicated by the UV map 2606 is scaled and
- the two pixels of the image being rendered, corresponding to the second segment of the UV map 2606 are scaled to fully occupy the second segment EMS2 of the environmental model 3002 to which they are applied as a texture.
- the third segment of the UV map also corresponds to two pixels of the image to be rendered. The two pixels are applied as a texture to the third segment of the environmental model 3002 as a texture during the rendering process.
- the third segment of the environmental model 3002 is the same size as the first and second segments of the environmental model 3002
- the third segment will be of the same resolution as the first and second segments.
- resolution reduction is not applied to the middle and right portions of the transmitted image but resolution reduction is performed with regard to the left side of the image.
- first, second and third segments of the rendered image will be of the same resolution in the FIG. 29 example, segments corresponding to the middle and side of the image and right side of the environmental model 3002 will be of higher resolution.
- the apparatus includes a receiver for receiving signals, a mesh model of an environment, one or more image maps, e.g., UV map(s), indicating a mapping between an image and the mesh model of an environment, and one or more encoded images.
- the receiver of the apparatus is configured to receive a mesh model of an environment, a first image map, a second image map, and an encoded image.
- the apparatus also includes or is coupled to a storage device such as a memory for storing received signals, mesh models, image maps, and images such as encoded, decoded and produced images.
- the apparatus further includes a decoder for decoding received encoded images and a processor configured to map a decoded image to a mesh model of an environment in accordance with a first image map to produce a first rendered image.
- the first image map mapping different numbers of pixels of the decoded image to different segments of said mesh model of the environment.
- the apparatus is configured so that the different numbers of pixels are mapped to environmental regions of the same size but located at different locations in the environment.
- the segments in the environment corresponding to action are allocated more pixels than segments in which less or no action is detected.
- the apparatus is configured so that at least some segments corresponding to a front viewing area are allocated more pixels per segment than segments corresponding to a rear viewing area.
- the apparatus includes or is coupled to a display device on which images produced by the apparatus are displayed.
- the processor of the apparatus may be , and typically is, configured to operate the apparatus to store received signals, mesh models, image maps, and images such as encoded, decoded and produced images in a storage device included in or coupled to the apparatus.
- the receiver of the apparatus is configured to receive a signal indicating that a second image map should be used to map portions of received frames to said environmental mesh model.
- the processor of the apparatus is further configured to operate the apparatus in response to the received signal indicating that a second image map should be used to map portions of received frames to the environmental mesh model to use a second image map, typically the second received image map, to map portions of received frames to the environmental mesh model to produce a second rendered image.
- the decoded image is a frame and the first image map allocates a first number of pixels of the frame to a first segment of the environmental mesh model and the second image map allocates a second number of pixels of the frame to the first segment of the environmental mesh model, the first and second number of pixels being different.
- the processor the apparatus is typically configured to display the second rendered image to a display which may be either included as part of the apparatus or coupled to the apparatus.
- the exemplary apparatus includes a processor configured to operate said apparatus to: (i) communicate a first image map to be used to map portions of a frame to segments of an environmental model, the first image map allocating different size portions of the frame to different segments of the environmental model thereby allocating different numbers of pixels to different segments of the model, and (ii) communicate a first frame including at least a portion of a first image to be mapped to said environmental model using said first image map.
- the processor of the apparatus is further configured to operate the apparatus to: (i) communicate a second image map to be used to map portions of a frame to segments of the environmental model, said second image map allocating different size portions of the frame to different segments of the environmental model thereby allocating different numbers of pixels to different segments of said model, the second image map allocating a different number of pixels to a first segment of said model than are allocated by said first image map, e.g., UV map, and (ii) communicate a second frame including at least a portion of a second image to be mapped to said environmental model using the second image map.
- the first and second image maps map different numbers of pixels to an area corresponding to the same portion of an environment thereby providing different resolution allocations for said same portion of the environment based on which of the first and second image maps are used.
- the apparatus is a server providing a real time content stream.
- the apparatus is a real time content delivery system including an environmental mesh generation module, a map generation module, e.g., UV map generation module, and an I/O interface and/or an network interface for communicating information including signals, models, maps and images.
- the modules include software instructions which when executed cause the processor to perform various routines.
- the modules are hardware modules, e.g., circuitry. In some embodiments, the modules are a combination of hardware and software modules.
- An exemplary content processing and delivery system e.g., system 700 , implemented in accordance with one exemplary embodiment comprises: a processor (e.g., processor 708 ) configured to: i) select a first resolution allocation to be used for at least one image corresponding to a first portion of an environment; and ii) perform a resolution reduction operation on a first image of the first portion of the environment in accordance with the selected first resolution allocation to generate a first reduced resolution image; and a transmitter (e.g., a transmitter 713 of interface 710 ) configured to communicate the first reduced resolution image to a playback device.
- a processor e.g., processor 708
- a transmitter e.g., a transmitter 713 of interface 710
- selection of a resolution allocation is performed based on a region of importance in the first portion of the environment.
- the region of importance corresponds to an area of motion in the first portion of the environment.
- the region of importance is a region indicated by a system operator.
- the region of importance is a region determined by detecting which portion of the environment included in the first image one or more individuals is looking at prior to or at the time the first image is captured.
- the transmitter is further configured to: communicate to the playback device a first texture map (UV map) to be used to map portions of the images generated in accordance with the first resolution allocation to a surface of a model of the environment.
- a first texture map (UV map) to be used to map portions of the images generated in accordance with the first resolution allocation to a surface of a model of the environment.
- the size of a first segment in the first texture map is a function of the amount of resolution reduction applied to a corresponding first area of the first image to generate a first segment of the first reduced resolution image.
- the first texture map includes a second segment corresponding to a portion of the first image which was not subject to a resolution reduction operation, the size of the second segment in the second texture map being the same as the size of the segment in the first image.
- the size of the first segment in the texture map is reduced from the size of the source of the corresponding area in the first image by an amount which is based on the amount of resolution reduction applied to the corresponding first area of the first image.
- the transmitter is further configured to communicate to the playback device an environmental model.
- the first texture map corresponds to a portion of the environmental model, the first texture map providing information indicating how to map portions of images subject to the first resolution allocation to a portion of the environmental model.
- the first image is one image of an image pair the first image and a second image, the first image being one of a left and right eye image pair, the second image being a second one of a left and right eye image pair.
- the processor is further configured to perform a resolution reduction operation on the second image in accordance with the selected first resolution allocation to generate a second reduced resolution image
- the transmitter is further configured to communicate the second reduced resolution image to the playback device as part of a first stereoscopic image pair.
- the processor is further configured to: select a second resolution allocation to be used for another image corresponding to a first portion of the environment, the another image being a third image; and perform a resolution reduction operation on the third image in accordance with the selected second resolution allocation to generate a third reduced resolution image.
- the transmitter is further configured to communicate the third reduced resolution image to a playback device.
- the transmitter is further configured to communicate to the playback device a second texture map (UV map) to be used to map portions of images generated in accordance with the second resolution allocation to the surface of the model of the environment.
- a second texture map (UV map) to be used to map portions of images generated in accordance with the second resolution allocation to the surface of the model of the environment.
- the size of a first segment in the second texture map is a function of the amount of resolution reduction applied to a corresponding first area of the third image to generate a first segment of the third reduced resolution image.
- the second texture map includes a third segment corresponding to a portion of the third image which was not subject to a resolution reduction operation, the size of the third segment in the second texture map being the same as the size of the segment in the third image.
- the size of the first segment in the second texture map is reduced from the size of the source of the corresponding area in the third image by an amount which is based on the amount of resolution reduction applied to the corresponding first area of the third image.
- the second texture map corresponds to the same portion of the environmental model as the first texture map, the second texture map providing information indicating how to map portions of images subject to the second resolution allocation to a corresponding portion of the environmental model.
- the methods and apparatus can be used for rendering stereoscopic images, e.g., pairs of images to be displayed to a users left and right eyes, or mono-scopic images.
- stereoscopic images e.g., pairs of images to be displayed to a users left and right eyes, or mono-scopic images.
- the methods are well suited for use in simulating 3D environments they are also well suited for use in communicating panoramic images which may correspond to an area less than a full 360 degree environment and which may not be stereoscopic in nature.
- Some embodiments are directed a non-transitory computer readable medium embodying a set of software instructions, e.g., computer executable instructions, for controlling a computer or other device to encode and compresses stereoscopic video.
- Other embodiments are embodiments are directed a computer readable medium embodying a set of software instructions, e.g., computer executable instructions, for controlling a computer or other device to decode and decompresses video on the player end.
- encoding and compression are mentioned as possible separate operations, it should be appreciated that encoding may be used to perform compression and thus encoding may, in some include compression. Similarly, decoding may involve decompression.
- Various embodiments may be implemented using software, hardware and/or a combination of software and hardware.
- Various embodiments are directed to apparatus, e.g., a image data processing system.
- Various embodiments are also directed to methods, e.g., a method of processing image data.
- one or more of the method steps is implemented using a processor.
- Various embodiments are also directed to a non-transitory machine, e.g., computer, readable medium, e.g., ROM, RAM, CDs, hard discs, etc., which include machine readable instructions for controlling a machine to implement one or more steps of a method.
- modules may, and in some embodiments are, implemented as software modules. In other embodiments the modules are implemented in hardware. In still other embodiments the modules are implemented using a combination of software and hardware. In some embodiments the modules are implemented as individual circuits with each module being implemented as a circuit for performing the function to which the module corresponds. A wide variety of embodiments are contemplated including some embodiments where different modules are implemented differently, e.g., some in hardware, some in software, and some using a combination of hardware and software. It should also be noted that routines and/or subroutines, or some of the steps performed by such routines, may be implemented in dedicated hardware as opposed to software executed on a general purpose processor. Such embodiments remain within the scope of the present invention.
- machine executable instructions such as software
- a machine readable medium such as a memory device, e.g., RAM, floppy disk, etc.
- a machine e.g., general purpose computer with or without additional hardware
- the present invention is directed to a machine-readable medium including machine executable instructions for causing a machine, e.g., processor and associated hardware, to perform one or more of the steps of the above-described method(s).
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Graphics (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Geometry (AREA)
- Human Computer Interaction (AREA)
- Databases & Information Systems (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Image Generation (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/046,314 US20160241838A1 (en) | 2015-02-17 | 2016-02-17 | Methods and apparatus for generating and using reduced resolution images and/or communicating such images to a playback or content distribution device |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562117427P | 2015-02-17 | 2015-02-17 | |
US201562262374P | 2015-12-02 | 2015-12-02 | |
US201662296065P | 2016-02-16 | 2016-02-16 | |
US15/046,314 US20160241838A1 (en) | 2015-02-17 | 2016-02-17 | Methods and apparatus for generating and using reduced resolution images and/or communicating such images to a playback or content distribution device |
Publications (1)
Publication Number | Publication Date |
---|---|
US20160241838A1 true US20160241838A1 (en) | 2016-08-18 |
Family
ID=56621708
Family Applications (11)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/046,331 Active US10075701B2 (en) | 2015-02-17 | 2016-02-17 | Methods and apparatus for mapping at least one received image to a surface of a model in a manner that efficiently uses the image content as a texture |
US15/046,311 Active US9832450B2 (en) | 2015-02-17 | 2016-02-17 | Methods and apparatus for generating and using reduced resolution images and/or communicating such images to a playback or content distribution device |
US15/046,327 Active US9729850B2 (en) | 2015-02-17 | 2016-02-17 | Methods and apparatus for receiving and/or using reduced resolution images |
US15/046,314 Abandoned US20160241838A1 (en) | 2015-02-17 | 2016-02-17 | Methods and apparatus for generating and using reduced resolution images and/or communicating such images to a playback or content distribution device |
US15/672,236 Active US10027944B2 (en) | 2015-02-17 | 2017-08-08 | Methods and apparatus for receiving and/or using reduced resolution images |
US15/806,251 Abandoned US20180220121A1 (en) | 2015-02-17 | 2017-11-07 | Methods and apparatus for generating and using reduced resolution images and/or communicating such images to a playback or content distribution device |
US16/035,459 Active US10506215B2 (en) | 2015-02-17 | 2018-07-13 | Methods and apparatus for receiving and/or using reduced resolution images |
US16/037,273 Active 2036-09-22 US11388384B2 (en) | 2015-02-17 | 2018-07-17 | Methods and apparatus for receiving and/or using reduced resolution images |
US16/708,420 Active US10805593B2 (en) | 2015-02-17 | 2019-12-09 | Methods and apparatus for receiving and/or using reduced resolution images |
US17/027,140 Active 2036-03-10 US11381801B2 (en) | 2015-02-17 | 2020-09-21 | Methods and apparatus for receiving and/or using reduced resolution images |
US17/654,667 Active 2036-04-18 US11924394B2 (en) | 2015-02-17 | 2022-03-14 | Methods and apparatus for receiving and/or using reduced resolution images |
Family Applications Before (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/046,331 Active US10075701B2 (en) | 2015-02-17 | 2016-02-17 | Methods and apparatus for mapping at least one received image to a surface of a model in a manner that efficiently uses the image content as a texture |
US15/046,311 Active US9832450B2 (en) | 2015-02-17 | 2016-02-17 | Methods and apparatus for generating and using reduced resolution images and/or communicating such images to a playback or content distribution device |
US15/046,327 Active US9729850B2 (en) | 2015-02-17 | 2016-02-17 | Methods and apparatus for receiving and/or using reduced resolution images |
Family Applications After (7)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/672,236 Active US10027944B2 (en) | 2015-02-17 | 2017-08-08 | Methods and apparatus for receiving and/or using reduced resolution images |
US15/806,251 Abandoned US20180220121A1 (en) | 2015-02-17 | 2017-11-07 | Methods and apparatus for generating and using reduced resolution images and/or communicating such images to a playback or content distribution device |
US16/035,459 Active US10506215B2 (en) | 2015-02-17 | 2018-07-13 | Methods and apparatus for receiving and/or using reduced resolution images |
US16/037,273 Active 2036-09-22 US11388384B2 (en) | 2015-02-17 | 2018-07-17 | Methods and apparatus for receiving and/or using reduced resolution images |
US16/708,420 Active US10805593B2 (en) | 2015-02-17 | 2019-12-09 | Methods and apparatus for receiving and/or using reduced resolution images |
US17/027,140 Active 2036-03-10 US11381801B2 (en) | 2015-02-17 | 2020-09-21 | Methods and apparatus for receiving and/or using reduced resolution images |
US17/654,667 Active 2036-04-18 US11924394B2 (en) | 2015-02-17 | 2022-03-14 | Methods and apparatus for receiving and/or using reduced resolution images |
Country Status (7)
Country | Link |
---|---|
US (11) | US10075701B2 (ja) |
EP (1) | EP3259739A4 (ja) |
JP (1) | JP2018511971A (ja) |
KR (2) | KR102493754B1 (ja) |
CN (2) | CN116962659A (ja) |
CA (1) | CA2977051C (ja) |
WO (1) | WO2016134048A1 (ja) |
Cited By (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170154462A1 (en) * | 2015-11-30 | 2017-06-01 | Photopotech LLC | Systems and Methods for Processing Image Information |
US9729850B2 (en) | 2015-02-17 | 2017-08-08 | Nextvr Inc. | Methods and apparatus for receiving and/or using reduced resolution images |
US10114467B2 (en) | 2015-11-30 | 2018-10-30 | Photopotech LLC | Systems and methods for processing image information |
US10200672B2 (en) | 2016-08-17 | 2019-02-05 | Nextvr Inc. | Methods and apparatus for capturing images of an environment |
US10298819B2 (en) | 2016-07-20 | 2019-05-21 | Nextvr Inc. | Camera apparatus and methods which allow for filters to be used |
US10306156B2 (en) | 2015-11-30 | 2019-05-28 | Photopotech LLC | Image-capture device |
US10362290B2 (en) | 2015-02-17 | 2019-07-23 | Nextvr Inc. | Methods and apparatus for processing content based on viewing information and/or communicating content |
US10397538B2 (en) | 2015-03-01 | 2019-08-27 | Nextvr Inc. | Methods and apparatus for supporting content generation, transmission and/or playback |
US10412382B2 (en) | 2015-02-24 | 2019-09-10 | Nextvr Inc. | Methods and apparatus related to capturing and/or rendering images |
US10432910B2 (en) | 2015-01-30 | 2019-10-01 | Nextvr Inc. | Methods and apparatus for controlling a viewing position |
US10438369B2 (en) | 2015-08-25 | 2019-10-08 | Nextvr Inc. | Methods and apparatus for detecting objects in proximity to a viewer and presenting visual representations of objects in a simulated environment |
US10447994B2 (en) | 2014-05-20 | 2019-10-15 | Nextvr Inc. | Methods and apparatus including or for use with one or more cameras |
CN110462678A (zh) * | 2017-03-30 | 2019-11-15 | 富士胶片株式会社 | 图像处理装置及图像处理方法 |
US10531071B2 (en) | 2015-01-21 | 2020-01-07 | Nextvr Inc. | Methods and apparatus for environmental measurements and/or stereoscopic image capture |
US10567733B2 (en) | 2017-03-06 | 2020-02-18 | Nextvr Inc. | Methods and apparatus for communicating and/or using frames including a captured image and/or including additional image content |
US10735709B2 (en) | 2018-04-04 | 2020-08-04 | Nextvr Inc. | Methods and apparatus for capturing, processing and/or communicating images |
US10742964B2 (en) | 2017-04-04 | 2020-08-11 | Nextvr Inc. | Methods and apparatus for displaying images |
US10778877B2 (en) | 2015-11-30 | 2020-09-15 | Photopotech LLC | Image-capture device |
WO2021072376A3 (en) * | 2019-10-10 | 2021-05-27 | Hover, Inc. | Interactive path tracing on the web |
US11217009B2 (en) | 2015-11-30 | 2022-01-04 | Photopotech LLC | Methods for collecting and processing image information to produce digital assets |
US11252391B2 (en) | 2017-03-06 | 2022-02-15 | Nevermind Capital Llc | Methods and apparatus for packing images into a frame and/or including additional content or graphics |
Families Citing this family (82)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2526674B1 (en) * | 2010-01-18 | 2017-03-15 | Telefonaktiebolaget LM Ericsson (publ) | Method and arrangement for supporting playout of content |
US9973692B2 (en) * | 2013-10-03 | 2018-05-15 | Flir Systems, Inc. | Situational awareness by compressed display of panoramic views |
US20150228106A1 (en) * | 2014-02-13 | 2015-08-13 | Vixs Systems Inc. | Low latency video texture mapping via tight integration of codec engine with 3d graphics engine |
GB2528446B (en) * | 2014-07-21 | 2021-08-04 | Tobii Tech Ab | Method and apparatus for detecting and following an eye and/or the gaze direction thereof |
KR101946019B1 (ko) * | 2014-08-18 | 2019-04-22 | 삼성전자주식회사 | 파노라믹 동영상을 생성하는 동영상 처리 장치 및 그 방법 |
US20170006219A1 (en) | 2015-06-30 | 2017-01-05 | Gopro, Inc. | Image stitching in a multi-camera array |
US9992502B2 (en) | 2016-01-29 | 2018-06-05 | Gopro, Inc. | Apparatus and methods for video compression using multi-resolution scalable coding |
US10291910B2 (en) | 2016-02-12 | 2019-05-14 | Gopro, Inc. | Systems and methods for spatially adaptive video encoding |
US10484621B2 (en) | 2016-02-29 | 2019-11-19 | Gopro, Inc. | Systems and methods for compressing video content |
US10645362B2 (en) * | 2016-04-11 | 2020-05-05 | Gopro, Inc. | Systems, methods and apparatus for compressing video content |
US10887573B1 (en) | 2016-05-03 | 2021-01-05 | Erin Sanaa Kenrick | System for processing 2D content for 3D viewing |
US10404963B1 (en) * | 2016-05-03 | 2019-09-03 | David Gerald Kenrick | System for processing 2D content for 3D viewing |
US11228754B2 (en) * | 2016-05-06 | 2022-01-18 | Qualcomm Incorporated | Hybrid graphics and pixel domain architecture for 360 degree video |
US10163030B2 (en) | 2016-05-20 | 2018-12-25 | Gopro, Inc. | On-camera image processing based on image activity data |
US10462466B2 (en) | 2016-06-20 | 2019-10-29 | Gopro, Inc. | Systems and methods for spatially selective video coding |
EP3458942B1 (en) * | 2016-06-22 | 2021-06-02 | Siemens Aktiengesellschaft | Display of three-dimensional model information in virtual reality |
US20180040164A1 (en) * | 2016-08-05 | 2018-02-08 | Gopro, Inc. | Apparatus and methods for selective coding of images |
WO2018045098A1 (en) * | 2016-08-30 | 2018-03-08 | Sonic Ip, Inc. | Systems and methods foe encoding and playing back 360 view video content |
CN107844190B (zh) * | 2016-09-20 | 2020-11-06 | 腾讯科技(深圳)有限公司 | 基于虚拟现实vr设备的图像展示方法及装置 |
CN106548446B (zh) * | 2016-09-29 | 2019-08-09 | 北京奇艺世纪科技有限公司 | 一种在球面全景图像上贴图的方法及装置 |
CN113891096B (zh) | 2016-10-04 | 2022-12-06 | 有限公司B1影像技术研究所 | 图像数据编码/解码方法和装置 |
US10244215B2 (en) | 2016-11-29 | 2019-03-26 | Microsoft Technology Licensing, Llc | Re-projecting flat projections of pictures of panoramic video for rendering by application |
US10244200B2 (en) | 2016-11-29 | 2019-03-26 | Microsoft Technology Licensing, Llc | View-dependent operations during playback of panoramic video |
US10595069B2 (en) * | 2016-12-05 | 2020-03-17 | Adobe Inc. | Prioritizing tile-based virtual reality video streaming using adaptive rate allocation |
GB2558206A (en) * | 2016-12-16 | 2018-07-11 | Nokia Technologies Oy | Video streaming |
US10242714B2 (en) | 2016-12-19 | 2019-03-26 | Microsoft Technology Licensing, Llc | Interface for application-specified playback of panoramic video |
US10999602B2 (en) | 2016-12-23 | 2021-05-04 | Apple Inc. | Sphere projected motion estimation/compensation and mode decision |
US10198862B2 (en) | 2017-01-23 | 2019-02-05 | Gopro, Inc. | Methods and apparatus for providing rotated spherical viewpoints |
US10691220B2 (en) * | 2017-02-14 | 2020-06-23 | Samsung Electronics Co., Ltd. | Method for display of information from real world environment on a virtual reality (VR) device and VR device thereof |
US11259046B2 (en) | 2017-02-15 | 2022-02-22 | Apple Inc. | Processing of equirectangular object data to compensate for distortion by spherical projections |
US10924747B2 (en) | 2017-02-27 | 2021-02-16 | Apple Inc. | Video coding techniques for multi-view video |
US10586379B2 (en) | 2017-03-08 | 2020-03-10 | Ebay Inc. | Integration of 3D models |
US10979663B2 (en) * | 2017-03-30 | 2021-04-13 | Yerba Buena Vr, Inc. | Methods and apparatuses for image processing to optimize image resolution and for optimizing video streaming bandwidth for VR videos |
US10587800B2 (en) * | 2017-04-10 | 2020-03-10 | Intel Corporation | Technology to encode 360 degree video content |
US11093752B2 (en) | 2017-06-02 | 2021-08-17 | Apple Inc. | Object tracking in multi-view video |
US9998664B1 (en) | 2017-06-20 | 2018-06-12 | Sliver VR Technologies, Inc. | Methods and systems for non-concentric spherical projection for multi-resolution view |
US20190005709A1 (en) * | 2017-06-30 | 2019-01-03 | Apple Inc. | Techniques for Correction of Visual Artifacts in Multi-View Images |
US10754242B2 (en) | 2017-06-30 | 2020-08-25 | Apple Inc. | Adaptive resolution and projection format in multi-direction video |
WO2019009324A1 (en) * | 2017-07-07 | 2019-01-10 | Sharp Kabushiki Kaisha | SYSTEMS AND METHODS FOR SIGNALING A PROJECTED REGION FOR VIRTUAL REALITY APPLICATIONS |
TWI687088B (zh) * | 2017-11-16 | 2020-03-01 | 宏達國際電子股份有限公司 | 適應性交錯影像捲繞的方法、系統以及記錄媒體 |
CN109803163B (zh) * | 2017-11-16 | 2021-07-09 | 腾讯科技(深圳)有限公司 | 图像展示方法及其装置、存储介质 |
US11145116B2 (en) | 2017-11-21 | 2021-10-12 | Faro Technologies, Inc. | System and method of scanning an environment and generating two dimensional images of the environment |
US11012676B2 (en) * | 2017-12-13 | 2021-05-18 | Google Llc | Methods, systems, and media for generating and rendering immersive video content |
EP3724858A4 (en) * | 2017-12-14 | 2021-01-13 | Samsung Electronics Co., Ltd. | PROCEDURE AND DEVICE FOR MANAGING IMMERSIVE DATA |
US10878536B1 (en) * | 2017-12-29 | 2020-12-29 | Gopro, Inc. | Apparatus and methods for non-uniform downsampling of captured panoramic images |
US10721510B2 (en) | 2018-05-17 | 2020-07-21 | At&T Intellectual Property I, L.P. | Directing user focus in 360 video consumption |
US10482653B1 (en) | 2018-05-22 | 2019-11-19 | At&T Intellectual Property I, L.P. | System for active-focus prediction in 360 video |
US10666863B2 (en) | 2018-05-25 | 2020-05-26 | Microsoft Technology Licensing, Llc | Adaptive panoramic video streaming using overlapping partitioned sections |
US10764494B2 (en) | 2018-05-25 | 2020-09-01 | Microsoft Technology Licensing, Llc | Adaptive panoramic video streaming using composite pictures |
US10827225B2 (en) | 2018-06-01 | 2020-11-03 | AT&T Intellectual Propety I, L.P. | Navigation for 360-degree video streaming |
US11727656B2 (en) | 2018-06-12 | 2023-08-15 | Ebay Inc. | Reconstruction of 3D model with immersive experience |
US10740957B1 (en) * | 2018-06-14 | 2020-08-11 | Kilburn Live, Llc | Dynamic split screen |
US10573060B1 (en) * | 2018-06-14 | 2020-02-25 | Kilburn Live, Llc | Controller binding in virtual domes |
US11108841B2 (en) | 2018-06-19 | 2021-08-31 | At&T Intellectual Property I, L.P. | Apparatus, storage medium and method for heterogeneous segmentation of video streaming |
EP3595319A1 (en) * | 2018-07-12 | 2020-01-15 | InterDigital VC Holdings, Inc. | Methods and apparatus for volumetric video transport |
US20200022632A1 (en) * | 2018-07-17 | 2020-01-23 | Limbix Health, Inc. | Digital content processing and generation for a virtual environment |
US10679668B2 (en) * | 2018-08-31 | 2020-06-09 | Gopro, Inc. | Systems and methods for editing videos |
CN110972202B (zh) | 2018-09-28 | 2023-09-01 | 苹果公司 | 基于无线通信信道带宽条件的移动设备内容提供调节 |
CN116980967A (zh) * | 2018-09-28 | 2023-10-31 | 苹果公司 | 基于无线通信信道带宽条件的电子设备内容提供调节 |
CN110832539A (zh) * | 2018-09-29 | 2020-02-21 | 深圳市大疆创新科技有限公司 | 调整图传分辨率的方法、图像拍摄设备、机器可读存储介质 |
US10778938B2 (en) * | 2018-12-20 | 2020-09-15 | Hulu, LLC | Video chunk combination optimization |
GB2582251B (en) * | 2019-01-31 | 2023-04-19 | Wacey Adam | Volumetric communication system |
CN110290385B (zh) * | 2019-06-11 | 2021-12-14 | 观博云标(北京)文化科技有限公司 | 高时空分辨率的天珠视频处理方法及装置 |
KR102582407B1 (ko) * | 2019-07-28 | 2023-09-26 | 구글 엘엘씨 | 포비에이티드 메시들로 몰입형 비디오 콘텐츠를 렌더링하기 위한 방법들, 시스템들, 및 매체들 |
JP7368699B2 (ja) * | 2019-08-26 | 2023-10-25 | 株式会社リコー | 画像処理装置、画像通信システム、画像処理方法、及びプログラム |
CN110619669B (zh) * | 2019-09-19 | 2023-03-28 | 深圳市富视康智能股份有限公司 | 一种支持多种图形样式的鱼眼图像渲染系统及方法 |
US11694314B2 (en) | 2019-09-25 | 2023-07-04 | The Regents Of The University Of Michigan | Digital foveation for machine vision |
KR20220081986A (ko) * | 2019-10-16 | 2022-06-16 | 가부시키가이샤 네지로 | 화상 정보 취득형 체결 수단 및 감시 시스템 |
US11284141B2 (en) | 2019-12-18 | 2022-03-22 | Yerba Buena Vr, Inc. | Methods and apparatuses for producing and consuming synchronized, immersive interactive video-centric experiences |
WO2021136876A1 (en) * | 2020-01-02 | 2021-07-08 | Nokia Technologies Oy | An apparatus, a method and a computer program for volumetric video |
EP4094425A1 (en) * | 2020-01-23 | 2022-11-30 | Volvo Truck Corporation | A method for adapting to a driver position an image displayed on a monitor in a vehicle cab |
US11740465B2 (en) * | 2020-03-27 | 2023-08-29 | Apple Inc. | Optical systems with authentication and privacy capabilities |
CN112218132B (zh) * | 2020-09-07 | 2022-06-10 | 聚好看科技股份有限公司 | 一种全景视频图像显示方法及显示设备 |
US11875462B2 (en) * | 2020-11-18 | 2024-01-16 | Adobe Inc. | Systems for augmented reality authoring of remote environments |
CN114938418A (zh) * | 2021-02-04 | 2022-08-23 | 佳能株式会社 | 具有视线检测功能的取景器单元、摄像设备和附接配件 |
US11978149B2 (en) | 2021-06-21 | 2024-05-07 | The Weather Company, Llc | UV map using weight painting |
US11853473B2 (en) * | 2021-08-18 | 2023-12-26 | Meta Platforms Technologies, Llc | Differential illumination for corneal glint detection |
CN113573142B (zh) * | 2021-09-22 | 2021-12-24 | 北京海誉动想科技股份有限公司 | 分辨率调整方法与装置 |
US20230176377A1 (en) * | 2021-12-06 | 2023-06-08 | Facebook Technologies, Llc | Directional illuminator and display apparatus with switchable diffuser |
CN115546019B (zh) * | 2022-01-10 | 2023-07-11 | 荣耀终端有限公司 | 一种渲染数据的采集方法及电子设备 |
US12002290B2 (en) * | 2022-02-25 | 2024-06-04 | Eyetech Digital Systems, Inc. | Systems and methods for hybrid edge/cloud processing of eye-tracking image data |
US11912429B2 (en) * | 2022-04-05 | 2024-02-27 | Gulfstream Aerospace Corporation | System and methodology to provide an augmented view of an environment below an obstructing structure of an aircraft |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080206720A1 (en) * | 2007-02-28 | 2008-08-28 | Nelson Stephen E | Immersive video projection system and associated video image rendering system for a virtual reality simulator |
US20140098186A1 (en) * | 2011-05-27 | 2014-04-10 | Ron Igra | System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view |
US20140327736A1 (en) * | 2013-05-01 | 2014-11-06 | Legend3D, Inc. | External depth map transformation method for conversion of two-dimensional images to stereoscopic images |
US20150054913A1 (en) * | 2013-08-21 | 2015-02-26 | Jaunt Inc. | Image stitching |
US20150138314A1 (en) * | 2013-11-20 | 2015-05-21 | Google Inc. | Generating Panoramic Images |
Family Cites Families (79)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6044168A (en) * | 1996-11-25 | 2000-03-28 | Texas Instruments Incorporated | Model based faced coding and decoding using feature detection and eigenface coding |
US6208347B1 (en) | 1997-06-23 | 2001-03-27 | Real-Time Geometry Corporation | System and method for computer modeling of 3D objects and 2D images by mesh constructions that incorporate non-spatial data such as color or texture |
US6246422B1 (en) * | 1998-09-01 | 2001-06-12 | Sun Microsystems, Inc. | Efficient method for storing texture maps in multi-bank memory |
US6400370B1 (en) * | 1999-09-10 | 2002-06-04 | Intel Corporation | Stochastic sampling with constant density in object space for anisotropic texture mapping |
US6788333B1 (en) * | 2000-07-07 | 2004-09-07 | Microsoft Corporation | Panoramic video |
US6774869B2 (en) * | 2000-12-22 | 2004-08-10 | Board Of Trustees Operating Michigan State University | Teleportal face-to-face system |
US7146036B2 (en) * | 2003-02-03 | 2006-12-05 | Hewlett-Packard Development Company, L.P. | Multiframe correspondence estimation |
US20040249617A1 (en) * | 2003-06-03 | 2004-12-09 | Pccw-Hkt Datacom Services Limited | Selective, pregressive transmission of 3D geometry models with the pre-ordered hierarchical meshes |
US8351761B2 (en) | 2003-06-06 | 2013-01-08 | Harris Technology, Llc | Commercial skipping standalone device |
EP1494175A1 (en) * | 2003-07-01 | 2005-01-05 | Koninklijke Philips Electronics N.V. | Selection of a mipmap level |
DE602005003575T2 (de) * | 2004-06-16 | 2008-10-23 | Nxp B.V. | Vorwärts-texturabbildungs-3d-graphiksystem |
US8130260B2 (en) * | 2005-11-09 | 2012-03-06 | Johns Hopkins University | System and method for 3-dimensional display of image data |
GB2436921A (en) * | 2006-04-06 | 2007-10-10 | British Broadcasting Corp | Methods and apparatus providing central, primary displays with surrounding display regions |
US20080031325A1 (en) * | 2006-08-03 | 2008-02-07 | Yingyong Qi | Mesh-based video compression with domain transformation |
JP4689639B2 (ja) * | 2007-04-25 | 2011-05-25 | キヤノン株式会社 | 画像処理システム |
US7701354B2 (en) * | 2007-06-28 | 2010-04-20 | Standard Telecommunications Ltd. | Portable alarm transmitter for compliance monitoring |
EP2181433B1 (en) * | 2007-07-19 | 2016-05-18 | Disney Enterprises, Inc. | Methods and apparatus for multiple texture map storage and filtering |
US7850306B2 (en) * | 2008-08-28 | 2010-12-14 | Nokia Corporation | Visual cognition aware display and visual data transmission architecture |
US8451320B1 (en) | 2009-01-23 | 2013-05-28 | Next3D, Inc. | Methods and apparatus for stereoscopic video compression, encoding, transmission, decoding and/or decompression |
US9648346B2 (en) | 2009-06-25 | 2017-05-09 | Microsoft Technology Licensing, Llc | Multi-view video compression and streaming based on viewpoints of remote viewer |
US8531473B2 (en) | 2009-07-28 | 2013-09-10 | Technion R&D Foundation Ltd. | Photogrammetric texture mapping using casual images |
US8610757B2 (en) | 2009-08-24 | 2013-12-17 | Next3D, Inc. | Stereoscopic video encoding and decoding methods and apparatus |
JP5233926B2 (ja) | 2009-09-10 | 2013-07-10 | 大日本印刷株式会社 | 魚眼監視システム |
US20130278631A1 (en) | 2010-02-28 | 2013-10-24 | Osterhout Group, Inc. | 3d positioning of augmented reality information |
JP5430456B2 (ja) * | 2010-03-16 | 2014-02-26 | キヤノン株式会社 | 幾何特徴抽出装置、幾何特徴抽出方法、及びプログラム、三次元計測装置、物体認識装置 |
US8405680B1 (en) * | 2010-04-19 | 2013-03-26 | YDreams S.A., A Public Limited Liability Company | Various methods and apparatuses for achieving augmented reality |
US20110280311A1 (en) * | 2010-05-13 | 2011-11-17 | Qualcomm Incorporated | One-stream coding for asymmetric stereo video |
KR101829345B1 (ko) * | 2010-06-28 | 2018-02-19 | 톰슨 라이센싱 | 입체 컨텐츠의 3-차원적인 효과들을 요구에 맞추기 위한 방법 및 장치 |
US20130125155A1 (en) | 2010-07-26 | 2013-05-16 | Thomson Licensing | Dynamic adaptation of displayed video quality based on viewers' context |
WO2012037157A2 (en) * | 2010-09-13 | 2012-03-22 | Alt Software (Us) Llc | System and method for displaying data having spatial coordinates |
GB2485532A (en) | 2010-11-12 | 2012-05-23 | Sony Corp | Three dimensional (3D) image duration-related metadata encoding of apparent minimum observer distances (disparity) |
US8907968B2 (en) * | 2011-03-31 | 2014-12-09 | Panasonic Corporation | Image rendering device, image rendering method, and image rendering program for rendering stereoscopic panoramic images |
US9407902B1 (en) | 2011-04-10 | 2016-08-02 | Nextvr Inc. | 3D video encoding and decoding methods and apparatus |
US9485494B1 (en) | 2011-04-10 | 2016-11-01 | Nextvr Inc. | 3D video encoding and decoding methods and apparatus |
JPWO2012147621A1 (ja) * | 2011-04-28 | 2014-07-28 | ソニー株式会社 | 符号化装置および符号化方法、並びに、復号装置および復号方法 |
US9161024B2 (en) | 2011-07-07 | 2015-10-13 | Nec Corporation | Display device, display control method, and program |
US20130030074A1 (en) * | 2011-07-26 | 2013-01-31 | Basf Se | Process for the continuous production of polyetherols |
EP2756682A4 (en) * | 2011-09-12 | 2015-08-19 | Intel Corp | NETWORK CAPTURE AND 3D DISPLAY OF LOCALIZED AND SEGMENTED IMAGES |
US20130141526A1 (en) | 2011-12-02 | 2013-06-06 | Stealth HD Corp. | Apparatus and Method for Video Image Stitching |
US9204127B1 (en) | 2012-01-17 | 2015-12-01 | Nextvr Inc. | Stereoscopic image processing methods and apparatus |
US20150245063A1 (en) | 2012-10-09 | 2015-08-27 | Nokia Technologies Oy | Method and apparatus for video coding |
ITTO20121073A1 (it) * | 2012-12-13 | 2014-06-14 | Rai Radiotelevisione Italiana | Apparato e metodo per la generazione e la ricostruzione di un flusso video |
US9538160B1 (en) | 2013-04-11 | 2017-01-03 | Nextvr Inc. | Immersive stereoscopic video acquisition, encoding and virtual reality playback methods and apparatus |
US9930363B2 (en) | 2013-04-12 | 2018-03-27 | Nokia Technologies Oy | Harmonized inter-view and view synthesis prediction for 3D video coding |
CN105264876B (zh) * | 2013-05-26 | 2019-04-23 | 皮克索洛特公司 | 低成本电视制作的方法及系统 |
US10067634B2 (en) | 2013-09-17 | 2018-09-04 | Amazon Technologies, Inc. | Approaches for three-dimensional object display |
JP2017505037A (ja) | 2013-12-26 | 2017-02-09 | サムスン エレクトロニクス カンパニー リミテッド | デプス符号化方法及びその装置、デプス復号化方法及びその装置 |
EP2902973B1 (en) | 2014-02-03 | 2018-06-27 | Thomson Licensing | Method and device for processing a geometry image of a 3d scene |
CN106031180A (zh) | 2014-02-18 | 2016-10-12 | Lg电子株式会社 | 收发用于全景服务的广播信号的方法和设备 |
US9699437B2 (en) | 2014-03-03 | 2017-07-04 | Nextvr Inc. | Methods and apparatus for streaming content |
US10264211B2 (en) | 2014-03-14 | 2019-04-16 | Comcast Cable Communications, Llc | Adaptive resolution in software applications based on dynamic eye tracking |
WO2015179574A1 (en) | 2014-05-20 | 2015-11-26 | Nextvr Inc. | Methods and apparatus including or for use with one or more cameras |
US20150346812A1 (en) | 2014-05-29 | 2015-12-03 | Nextvr Inc. | Methods and apparatus for receiving content and/or playing back content |
EP4113991A1 (en) | 2014-09-03 | 2023-01-04 | Nevermind Capital LLC | Methods and apparatus for capturing, streaming and/or playing back content |
US10484697B2 (en) * | 2014-09-09 | 2019-11-19 | Qualcomm Incorporated | Simultaneous localization and mapping for video coding |
GB201419438D0 (en) | 2014-10-31 | 2014-12-17 | Microsoft Corp | Modifying video call data |
US10531071B2 (en) | 2015-01-21 | 2020-01-07 | Nextvr Inc. | Methods and apparatus for environmental measurements and/or stereoscopic image capture |
US11218682B2 (en) | 2015-01-21 | 2022-01-04 | Nevermind Capital Llc | Methods and apparatus for processing and or encoding images with negative parallax |
WO2016123364A2 (en) | 2015-01-28 | 2016-08-04 | Nextvr Inc. | Zoom related methods and apparatus |
US9832449B2 (en) | 2015-01-30 | 2017-11-28 | Nextvr Inc. | Methods and apparatus for controlling a viewing position |
JP2018506797A (ja) | 2015-02-12 | 2018-03-08 | ネクストブイアール・インコーポレイテッド | 環境測定を行うための、及び/又は、このような測定を使用するための方法及び装置 |
US10362290B2 (en) | 2015-02-17 | 2019-07-23 | Nextvr Inc. | Methods and apparatus for processing content based on viewing information and/or communicating content |
US10075701B2 (en) | 2015-02-17 | 2018-09-11 | Nextvr Inc. | Methods and apparatus for mapping at least one received image to a surface of a model in a manner that efficiently uses the image content as a texture |
US9865055B2 (en) | 2015-02-24 | 2018-01-09 | Nextvr Inc. | Calibration for immersive content systems |
US9894350B2 (en) | 2015-02-24 | 2018-02-13 | Nextvr Inc. | Methods and apparatus related to capturing and/or rendering images |
CA2977113A1 (en) | 2015-03-01 | 2016-09-09 | Nextvr Inc. | Methods and apparatus for making environmental measurements and/or using such measurements in 3d image rendering |
CN105141970B (zh) * | 2015-07-03 | 2019-02-12 | 哈尔滨工业大学深圳研究生院 | 一种基于三维模型几何信息的纹理图像压缩方法 |
US9821920B2 (en) | 2015-08-18 | 2017-11-21 | Nextvr Inc. | Methods and apparatus for compensating for vehicular motion |
US9836845B2 (en) | 2015-08-25 | 2017-12-05 | Nextvr Inc. | Methods and apparatus for detecting objects in proximity to a viewer and presenting visual representations of objects in a simulated environment |
US10715788B2 (en) | 2016-07-15 | 2020-07-14 | Nextvr Inc. | Camera alignment and/or calibration methods and apparatus |
US10298819B2 (en) | 2016-07-20 | 2019-05-21 | Nextvr Inc. | Camera apparatus and methods which allow for filters to be used |
US10200672B2 (en) | 2016-08-17 | 2019-02-05 | Nextvr Inc. | Methods and apparatus for capturing images of an environment |
US20180225537A1 (en) | 2017-02-08 | 2018-08-09 | Nextvr Inc. | Methods and apparatus relating to camera switching and/or making a decision to switch between cameras |
US20180249189A1 (en) | 2017-02-08 | 2018-08-30 | Nextvr Inc. | Methods and apparatus for use in a system or device where switching between cameras may occur |
US10567733B2 (en) | 2017-03-06 | 2020-02-18 | Nextvr Inc. | Methods and apparatus for communicating and/or using frames including a captured image and/or including additional image content |
US11252391B2 (en) | 2017-03-06 | 2022-02-15 | Nevermind Capital Llc | Methods and apparatus for packing images into a frame and/or including additional content or graphics |
US20180280656A1 (en) | 2017-03-28 | 2018-10-04 | Nextvr Inc. | Virtual reality methods and apparatus which use or include brain activity sensors |
US10742964B2 (en) | 2017-04-04 | 2020-08-11 | Nextvr Inc. | Methods and apparatus for displaying images |
US10477186B2 (en) | 2018-01-17 | 2019-11-12 | Nextvr Inc. | Methods and apparatus for calibrating and/or adjusting the arrangement of cameras in a camera pair |
-
2016
- 2016-02-17 US US15/046,331 patent/US10075701B2/en active Active
- 2016-02-17 US US15/046,311 patent/US9832450B2/en active Active
- 2016-02-17 EP EP16753000.5A patent/EP3259739A4/en not_active Withdrawn
- 2016-02-17 JP JP2017543784A patent/JP2018511971A/ja active Pending
- 2016-02-17 CA CA2977051A patent/CA2977051C/en active Active
- 2016-02-17 CN CN202310914642.1A patent/CN116962659A/zh active Pending
- 2016-02-17 CN CN201680021644.6A patent/CN107615338B/zh active Active
- 2016-02-17 WO PCT/US2016/018315 patent/WO2016134048A1/en active Application Filing
- 2016-02-17 KR KR1020177026175A patent/KR102493754B1/ko active IP Right Grant
- 2016-02-17 KR KR1020237002994A patent/KR102533555B1/ko active IP Right Grant
- 2016-02-17 US US15/046,327 patent/US9729850B2/en active Active
- 2016-02-17 US US15/046,314 patent/US20160241838A1/en not_active Abandoned
-
2017
- 2017-08-08 US US15/672,236 patent/US10027944B2/en active Active
- 2017-11-07 US US15/806,251 patent/US20180220121A1/en not_active Abandoned
-
2018
- 2018-07-13 US US16/035,459 patent/US10506215B2/en active Active
- 2018-07-17 US US16/037,273 patent/US11388384B2/en active Active
-
2019
- 2019-12-09 US US16/708,420 patent/US10805593B2/en active Active
-
2020
- 2020-09-21 US US17/027,140 patent/US11381801B2/en active Active
-
2022
- 2022-03-14 US US17/654,667 patent/US11924394B2/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080206720A1 (en) * | 2007-02-28 | 2008-08-28 | Nelson Stephen E | Immersive video projection system and associated video image rendering system for a virtual reality simulator |
US20140098186A1 (en) * | 2011-05-27 | 2014-04-10 | Ron Igra | System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view |
US20140327736A1 (en) * | 2013-05-01 | 2014-11-06 | Legend3D, Inc. | External depth map transformation method for conversion of two-dimensional images to stereoscopic images |
US20150054913A1 (en) * | 2013-08-21 | 2015-02-26 | Jaunt Inc. | Image stitching |
US20150055937A1 (en) * | 2013-08-21 | 2015-02-26 | Jaunt Inc. | Aggregating images and audio data to generate virtual reality content |
US20150138314A1 (en) * | 2013-11-20 | 2015-05-21 | Google Inc. | Generating Panoramic Images |
Cited By (41)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10447994B2 (en) | 2014-05-20 | 2019-10-15 | Nextvr Inc. | Methods and apparatus including or for use with one or more cameras |
US11245891B2 (en) | 2015-01-21 | 2022-02-08 | Nevermind Capital Llc | Methods and apparatus for environmental measurements and/or stereoscopic image capture |
US10531071B2 (en) | 2015-01-21 | 2020-01-07 | Nextvr Inc. | Methods and apparatus for environmental measurements and/or stereoscopic image capture |
US10432910B2 (en) | 2015-01-30 | 2019-10-01 | Nextvr Inc. | Methods and apparatus for controlling a viewing position |
US10506215B2 (en) | 2015-02-17 | 2019-12-10 | Nextvr Inc. | Methods and apparatus for receiving and/or using reduced resolution images |
US11336881B2 (en) | 2015-02-17 | 2022-05-17 | Nevermind Capital Llc | Methods and apparatus for processing content based on viewing information and/or communicating content |
US10721454B2 (en) | 2015-02-17 | 2020-07-21 | Nextvr Inc. | Methods and apparatus for processing content based on viewing information and/or communicating content |
US11388384B2 (en) | 2015-02-17 | 2022-07-12 | Nevermind Capital Llc | Methods and apparatus for receiving and/or using reduced resolution images |
US10362290B2 (en) | 2015-02-17 | 2019-07-23 | Nextvr Inc. | Methods and apparatus for processing content based on viewing information and/or communicating content |
US9729850B2 (en) | 2015-02-17 | 2017-08-08 | Nextvr Inc. | Methods and apparatus for receiving and/or using reduced resolution images |
US11902493B2 (en) | 2015-02-17 | 2024-02-13 | Nevermind Capital Llc | Methods and apparatus for processing content based on viewing information and/or communicating content |
US11924394B2 (en) | 2015-02-17 | 2024-03-05 | Nevermind Capital Llc | Methods and apparatus for receiving and/or using reduced resolution images |
US9832450B2 (en) | 2015-02-17 | 2017-11-28 | Nextvr Inc. | Methods and apparatus for generating and using reduced resolution images and/or communicating such images to a playback or content distribution device |
US10075701B2 (en) | 2015-02-17 | 2018-09-11 | Nextvr Inc. | Methods and apparatus for mapping at least one received image to a surface of a model in a manner that efficiently uses the image content as a texture |
US11381801B2 (en) | 2015-02-17 | 2022-07-05 | Nevermind Capital Llc | Methods and apparatus for receiving and/or using reduced resolution images |
US10412382B2 (en) | 2015-02-24 | 2019-09-10 | Nextvr Inc. | Methods and apparatus related to capturing and/or rendering images |
US10742948B2 (en) | 2015-03-01 | 2020-08-11 | Nextvr Inc. | Methods and apparatus for requesting, receiving and/or playing back content corresponding to an environment |
US10574962B2 (en) | 2015-03-01 | 2020-02-25 | Nextvr Inc. | Methods and apparatus for requesting, receiving and/or playing back content corresponding to an environment |
US10701331B2 (en) | 2015-03-01 | 2020-06-30 | Nextvr Inc. | Methods and apparatus for supporting content generation, transmission and/or playback |
US10397538B2 (en) | 2015-03-01 | 2019-08-27 | Nextvr Inc. | Methods and apparatus for supporting content generation, transmission and/or playback |
US11870967B2 (en) | 2015-03-01 | 2024-01-09 | Nevermind Capital Llc | Methods and apparatus for supporting content generation, transmission and/or playback |
US10438369B2 (en) | 2015-08-25 | 2019-10-08 | Nextvr Inc. | Methods and apparatus for detecting objects in proximity to a viewer and presenting visual representations of objects in a simulated environment |
US10706621B2 (en) * | 2015-11-30 | 2020-07-07 | Photopotech LLC | Systems and methods for processing image information |
US10114467B2 (en) | 2015-11-30 | 2018-10-30 | Photopotech LLC | Systems and methods for processing image information |
US10778877B2 (en) | 2015-11-30 | 2020-09-15 | Photopotech LLC | Image-capture device |
US11217009B2 (en) | 2015-11-30 | 2022-01-04 | Photopotech LLC | Methods for collecting and processing image information to produce digital assets |
US20170154462A1 (en) * | 2015-11-30 | 2017-06-01 | Photopotech LLC | Systems and Methods for Processing Image Information |
US11699243B2 (en) | 2015-11-30 | 2023-07-11 | Photopotech LLC | Methods for collecting and processing image information to produce digital assets |
US10306156B2 (en) | 2015-11-30 | 2019-05-28 | Photopotech LLC | Image-capture device |
US10979607B2 (en) | 2016-07-20 | 2021-04-13 | Apple Inc. | Camera apparatus and methods |
US10298819B2 (en) | 2016-07-20 | 2019-05-21 | Nextvr Inc. | Camera apparatus and methods which allow for filters to be used |
US11375090B2 (en) | 2016-07-20 | 2022-06-28 | Nevermind Capital Llc | Camera apparatus and methods |
US11381802B2 (en) | 2016-08-17 | 2022-07-05 | Nevermind Capital Llc | Methods and apparatus for capturing images of an environment |
US10200672B2 (en) | 2016-08-17 | 2019-02-05 | Nextvr Inc. | Methods and apparatus for capturing images of an environment |
US10567733B2 (en) | 2017-03-06 | 2020-02-18 | Nextvr Inc. | Methods and apparatus for communicating and/or using frames including a captured image and/or including additional image content |
US11252391B2 (en) | 2017-03-06 | 2022-02-15 | Nevermind Capital Llc | Methods and apparatus for packing images into a frame and/or including additional content or graphics |
CN110462678A (zh) * | 2017-03-30 | 2019-11-15 | 富士胶片株式会社 | 图像处理装置及图像处理方法 |
US10742964B2 (en) | 2017-04-04 | 2020-08-11 | Nextvr Inc. | Methods and apparatus for displaying images |
US10735709B2 (en) | 2018-04-04 | 2020-08-04 | Nextvr Inc. | Methods and apparatus for capturing, processing and/or communicating images |
WO2021072376A3 (en) * | 2019-10-10 | 2021-05-27 | Hover, Inc. | Interactive path tracing on the web |
US11954169B2 (en) | 2019-10-10 | 2024-04-09 | Hover Inc. | Interactive path tracing on the web |
Also Published As
Publication number | Publication date |
---|---|
CN107615338A (zh) | 2018-01-19 |
US10027944B2 (en) | 2018-07-17 |
US20220210391A1 (en) | 2022-06-30 |
US20160241836A1 (en) | 2016-08-18 |
US20180220121A1 (en) | 2018-08-02 |
US11381801B2 (en) | 2022-07-05 |
CN107615338B (zh) | 2023-08-04 |
KR20170120631A (ko) | 2017-10-31 |
WO2016134048A1 (en) | 2016-08-25 |
KR102533555B1 (ko) | 2023-05-18 |
US20190110041A1 (en) | 2019-04-11 |
US20200186779A1 (en) | 2020-06-11 |
CA2977051C (en) | 2023-02-07 |
EP3259739A4 (en) | 2018-08-29 |
US11388384B2 (en) | 2022-07-12 |
US9832450B2 (en) | 2017-11-28 |
US20160241837A1 (en) | 2016-08-18 |
KR20230020574A (ko) | 2023-02-10 |
US20190082161A1 (en) | 2019-03-14 |
US10075701B2 (en) | 2018-09-11 |
US9729850B2 (en) | 2017-08-08 |
US20210006770A1 (en) | 2021-01-07 |
US10805593B2 (en) | 2020-10-13 |
CN116962659A (zh) | 2023-10-27 |
EP3259739A1 (en) | 2017-12-27 |
US10506215B2 (en) | 2019-12-10 |
US20160241892A1 (en) | 2016-08-18 |
JP2018511971A (ja) | 2018-04-26 |
CA2977051A1 (en) | 2016-08-25 |
US20170359564A1 (en) | 2017-12-14 |
US11924394B2 (en) | 2024-03-05 |
KR102493754B1 (ko) | 2023-02-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11924394B2 (en) | Methods and apparatus for receiving and/or using reduced resolution images | |
US11902493B2 (en) | Methods and apparatus for processing content based on viewing information and/or communicating content | |
EP3114844A1 (en) | Methods and apparatus for streaming content | |
US11582384B2 (en) | Methods and apparatus for encoding, communicating and/or using images | |
US11348252B1 (en) | Method and apparatus for supporting augmented and/or virtual reality playback using tracked objects | |
US11250612B1 (en) | Methods and apparatus rendering images using point clouds representing one or more objects | |
WO2018035383A1 (en) | Methods and apparatus for processing content based on viewing information and/or communicating content |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: NEXTVR INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:COLE, DAVID;MOSS, ALAN MCKAY;MEDINA, HECTOR M.;REEL/FRAME:038017/0990 Effective date: 20160314 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |
|
AS | Assignment |
Owner name: NEVERMIND CAPITAL LLC, DELAWARE Free format text: NUNC PRO TUNC ASSIGNMENT;ASSIGNOR:NEXTVR INC.;REEL/FRAME:053139/0942 Effective date: 20200513 |