KR102611448B1 - Methods and apparatus for delivering content and/or playing back content - Google Patents

Abstract

Content delivery and playback methods and devices are described. The methods and apparatus are suitable for delivery and playback of content in response to a 360-degree environment, streaming of content, e.g., 3D content, in response to an event such as a sports game, while the event is in progress or after the event has ended. and/or may be used to support real-time delivery. Portions of the environment are captured by cameras located at different locations. Content captured from different locations is encoded and made available for delivery. The playback device selects content to be received based on the position of the user's head. Streams may be prioritized and selected for delivery based on the user's current field of view and/or direction of head rotation. Still images or composited images can be used and combined with content from one or more streams, for example for background, sky and/or ground locations.

Description

Methods and devices for delivering content and/or playing content {METHODS AND APPARATUS FOR DELIVERING CONTENT AND/OR PLAYING BACK CONTENT}

The present invention relates to content delivery and/or playback, for example, playback of stereoscopic image content.

Display devices intended to provide an immersive experience typically allow a user to turn his head to experience corresponding changes in the displayed scene. Head-mounted displays sometimes support 360-degree viewing, where the user can turn their body while wearing the head-mounted display, so the displayed scene changes as the user's head position changes.

These devices provide the user with a scene captured in front of the camera position when looking forward and a scene captured behind the camera position when the user turns fully around. While the user turns his head backwards, the user's field of view at any given time is typically limited to less than 120 degrees due to the nature of a person's ability to perceive a limited field of view at any given time.

To support a 360-degree view, a 360-degree scene can be captured using multiple cameras together, and the images are combined to create a 360-degree scene that is made available for viewing.

A 360-degree view is much more than a simple front view that is typically captured and encoded for regular television and many other video applications where the user does not have the opportunity to change the viewing angle used to determine which image will be displayed at a particular point in time. It should be understood that it includes image data.

Given transmission limitations, such as network data limitations associated with streamed content, it may not be possible to stream a full 360-degree view in full high-definition video to all customers who wish to receive and interact with the content. . This is especially the case when the content is stereoscopic video content containing image content intended to correspond to left and right eye views to allow a 3D viewing effect.

In light of the above discussion, a method for supporting streaming and/or playback of content in a manner that allows an individual user to change his viewing position and view a desired portion of the environment, for example by turning the user's head. It should be understood that a need exists for fields and devices. It would be desirable if the user were given the option to change his or her head position and viewing direction accordingly, while staying within data streaming limitations that may be imposed by bandwidth or other delivery related limitations. Although not necessarily for all embodiments, at least some embodiments allow multiple users in different locations to receive streams at the same time and create the environment they want, regardless of which part or portions are visible to other users. It is desirable to be able to see any separate parts.

Methods and apparatus for supporting delivery, e.g., streaming, of video or other content corresponding to a 360-degree viewing area are described. The methods and apparatus of the present invention provide for stereoscopic imaging, where data transmission limitations may make it difficult to deliver 360 degrees of content, e.g., at the maximum supported quality level using best quality coding and the highest supported frame rate. and/or other image content. However, the methods are not limited to stereoscopic video content.

In various embodiments, a 3D model of the environment in which video content is acquired and/or corresponding 3D dimensional information is created and/or accessed. Camera positions in the environment are recorded. Multiple distinct camera locations may be provided within the environment. For example, separate final target camera positions and one or more mid-field camera positions can be supported and used to capture real-time camera feeds.

The 3D model and/or other 3D information is stored on an image capture device or server used to stream video to one or more users.

The 3D model is provided to a user playback device, such as a customer premises device with image rendering and compositing capabilities. The customer premises device creates a 3D representation of the environment that is displayed to a user of the customer premises device, for example, via a head mounted display.

In various embodiments, less than a full 360 degree environment is streamed to an individual customer premises device at any given time. The customer premises device indicates that the camera feed is streamed based on user input. A user may select compartments and/or camera locations through an input device attached to or on part of a customer premises device.

In some embodiments, the 180-degree video stream is transmitted from the servers and/or video cameras responsible for streaming the content to the customer playback device, for example, as a live, real-time, or near-real-time stream. The playback device monitors the user's head position, so that the playback device recognizes the viewing area in which the user of the playback device is looking within the 3D environment created by the playback device. The customer premises device provides video when available for the portion of the 3D environment being viewed, and this video content is displayed or replaced as an alternative to the simulated 3D environment that would be presented in the absence of the video content. When the user of the playback device turns his head, portions of the environment presented to the user may be fed, e.g., streamed, from the video content to the playback device, and other portions may be compositely created from 3D models and/or video content. This is previously supplied image content that was captured at a different time.

Thus, the playback device can display video, for example supplied via streaming, while a game, music concert, or other event is still in progress, corresponding to a frontal 180-degree camera view, for example, in a 3D environment. The rear and/or side portions are created either fully compositely or from image content of the side or rear areas of the environment at different times.

Although the user can select camera positions by signaling a change in position to the server providing the streaming content, the server providing the streaming content can provide useful information for creating a composite environment for parts of the 3D environment that are not streamed. You can.

For example, in some embodiments, multiple back and side views are captured at different times, such as before streaming a portion of the content or from an earlier point in time. Images are buffered on the playback device. The server providing the content may, and in some embodiments does, signal to the playback device which of the non-real-time scenes or sets of images are used for compositing parts of the environment that are not supplied in the video stream. For example, an image of seated concert participants and another image of concert participants standing behind the camera position may be supplied and stored to the playback device. The server may signal which of the set of stored image data should be used at a particular point in time. Thus, when the audience is standing, the server can signal that the image corresponding to the standing audience should be used for the background 180 degree view during image compositing, whereas when the audience is sitting, the server can signal that the image corresponding to the standing audience is to be used for the background 180 degree view during image compositing, while when the audience is sitting, the server can signal that the image corresponding to the standing audience is to be used for the background 180 degree view during image compositing. Alternatively, it may be indicated to the customer premises device that the image or image composition information corresponding to the seated audience member should be used when compositing the rear portions.

In at least some embodiments, the orientation of the cameras at each of one or more locations in the 3D environment is tracked during image capture. Markers and/or identification points in the environment facilitate alignment and/or other mapping of captured images, e.g., live images, to a previously modeled and/or mapped 3D environment to be simulated by a customer premises device. It can be used to do this.

A mix of synthetic environment parts and real-time (streaming video) provides an immersive video experience. Environments can be measured or modeled using 3D photometry to generate 3D information used to simulate the environment, or sometimes when video is not available, for example, if the environment has not previously been modeled. Measured or modeled.

The use of fiducial markers in real-world space at determined locations aids in calibration and alignment of the video with the previously created 3D model.

Position tracking of each camera is performed when video is captured. Camera position information is about location mapping in X, Y, Z, and angles (so we know where each camera is pointed). This allows for easy detection of which part of the environment the captured image corresponds to and, when passed along with the captured video to the playback device, automatically matches our video capture to a composite environment created by the playback device during image display. Playback to overlay, for example, allow the user to playback. Streamed content may be limited to a view of less than 360 degrees, for example, a captured 180 degree view of the area in front of the camera position. When the viewer looks around, when turning backwards, the viewer will see a simulated background (not a black cavity), and when turning forward, the viewer will see video.

Synthetic environments can interact, and in some environments they interact. In some embodiments, multiple real viewers, such as users of different customer premises devices, are included in the simulated environment so that the user can watch the game with his friends in a virtual 3D environment, as if the users were actually at the stadium. Make it visible.

Images of users may be captured by cameras included with or attached to customer premises devices, or, in some embodiments, captured and fed to a server, or, in some embodiments, for use in creating a simulated environment. Provided to users, for example, members of a group. User images need not be, but may be, real-time images.

The methods may be used to encode and provide content in real time or near real time, but are not limited to such real time applications. Given the ability to support real-time and near-real-time encoding and streaming to multiple users, the methods and devices described herein allow individuals to view an event, observe a stage or field, as well as interact with the environment, e.g. It's ideal for streaming scenes from sporting events, concerts, and/or other venues that can be recognized by turning to the stadium or audience views. By supporting 360-degree viewing and 3D, the methods and apparatus of the present invention allow the user to view the scene from different viewing angles when the user is present in the environment and turns the user's head to the left, right, or backwards. It is suitable for use with head-mounted displays intended to provide a 3D immersive experience through the freedom to rotate and observe.

Methods and apparatus for delivering image content, e.g., content corresponding to a 360 degree field of view, are described. In various embodiments, the field of view corresponds to different portions of the environment, for example, a frontal portion, at least one posterior portion, a top, and a bottom. In some embodiments, the left and right rear, eg, rear portions of the environment are created and/or delivered separately. The playback device monitors the position of the user's head and generates images, for example stereoscopic images, corresponding to the part of the environment the user is viewing at a given time, which are then displayed to the user. For stereoscopic video playback, separate left and right eye images are created. The generated images may correspond to, and in some embodiments correspond to, one or more scenes, eg, parts of the environment.

At the start of playback, the user's forward-looking head level position is set by default to correspond to the forward scene portion. As the user turns his head and/or raises or lowers his head, different parts of the environment may come into the user's field of view.

Bandwidth and image decoding capabilities for many playback devices are limited by the bandwidth and/or processing capabilities of the device to receive image content. In some embodiments, the playback device determines which part of the environment corresponds to the user's primary field of view. The device then selects as the main stream that part to be received in full resolution at a higher rate, for example by the stream specified from a priority perspective. Content from one or more other streams providing content corresponding to other parts of the environment is also received, but generally at lower data rates. Content delivery for a particular stream may be initiated by a playback device, for example, by transmitting a signal used to trigger content delivery. The signal may be used to provide content corresponding to a portion of the environment or to connect to a multicast group to initiate delivery of switched digital broadcasts. For broadcast content that does not request a request such as a multicast group connection signal or other signal, the device can initiate reception by tuning to the channel on which the content is available.

Assuming that users are primarily interested in the forward view portion of the environment, especially when the content corresponds to a sporting event, a rock concert, a fashion show, or a number of different events, since the main action typically takes place, in some embodiments: The forward view portion of the environment is given data transmission priority. In at least some embodiments, images corresponding to the front view location are streamed at a higher rate than one or more other portions of the 360-degree environment. Images corresponding to different parts of the environment are transmitted at a lower data rate or as still images. For example, one or more still images of an upper, eg, sky and a lower, eg, ground, image may be transmitted.

In some embodiments, multiple still captured images are transmitted of one or more portions of the environment, for example rear view portion(s) or portions of the sky. In some embodiments, control information is transmitted indicating which of the still images for a portion of the environment should be used at a given time. When still images of a portion of the environment are to be transmitted, they may be transmitted in encoded form and then stored in memory in decoded form for use in combination with other image content. In this way, decoding resources required during an event can be reduced because multiple streams do not need to be decoded simultaneously at the same frame rate. Still images may be transmitted prior to streaming the content of the main event. Alternatively, several images of different parts of the environment can be stored and transmitted if they are needed during playback given the change in user's head position from the front viewing position. Still or rare images may be encoded and transmitted as part of a content stream providing content for the primary, eg, front, viewing direction, or they may be transmitted as a separate content stream.

Still images corresponding to the rear may be, and sometimes are, images captured before the event, and content corresponding to the front portion of the environment includes content captured and streamed while the event is taking place, for example in real time. It can, and in many cases, includes him.

For example, consider the case where two different rear view scenes are delivered and stored to a playback device. One scene may correspond to an audience member in a standing position, while another image may correspond to an audience member in a seated position. The control information may indicate, or in some embodiments, indicates whether either an image of the audience in a sitting or standing position is being used at a given time, to which the user should turn his or her head to a position where the rear portion of the environment is visible. .

Similarly, multiple images of the sky can be delivered to a playback device and stored on the playback device in encoded or decoded form. In some embodiments, control information is conveyed as to which image of the portion of the sky will be used at a given time. In other embodiments, which scene of the sky is used is automatically determined based on the brightness of one or more images corresponding to a front scene area that has a portion of the sky that matches or is close to the selected front environment scene portion, for example , bright front scene areas can be detected and used to control the selection of bright sky images with fewer clouds. Similarly, in some embodiments detection of a dark front environmental area will result in a dark cloudy sky image being used.

In cases where an image corresponding to a portion of the environment of view is not available, the scene portion may be synthesized, for example, from content or information from other parts of the environment that are available. For example, if a portion of the rear image is not available, content forming the left and/or right side of the front scene area can be copied and used to fill in the missing rear portions of the environment. In addition to content duplication, blurring and/or other image processing operations may be used to fill in missing parts of the environment in some embodiments. Alternatively, in some embodiments, composition information is provided in the content stream and the playback device creates composite images complete with missing portions. Such content, such as video game content, may be realistic in nature and may include a wide variety of image effects and content generated from authoring and/or other image generation rules stored on the playback device.

According to some embodiments, an example method of operating a playback system includes determining a viewer's head position corresponding to the current field of view; Receiving a first content stream providing content corresponding to a first portion of the environment; At least some received content included in the first content stream and one corresponding to the current field of view based on i) stored content corresponding to a second portion of the environment or ii) a composite image simulating the second portion of the environment generating the above output images; and outputting or displaying a first output image, one of one or more generated output images. According to some embodiments, an example content playback system includes: a viewer head position determination module configured to determine a viewer's head position, corresponding to the current field of view; a content stream receiving module configured to receive a first content stream providing content corresponding to a first portion of the environment; At least some received content included in the first content stream and one corresponding to the current field of view based on i) stored content corresponding to a second portion of the environment or ii) a composite image simulating the second portion of the environment an output image content stream-based generation module configured to generate the above output images; and at least one of an output module configured to output the first output image or a display module configured to display the first output image.

Numerous variations and embodiments are possible and are discussed in the detailed description that follows.

The present invention provides improved methods and devices for delivering and/or playing content.

1 illustrates an example system implemented in accordance with some embodiments of the invention that can be used to capture and stream content for subsequent display by one or more users with one or more composite portions of an environment. A drawing showing.
FIG. 2A illustrates an exemplary stereoscopic scene, e.g., an unsegmented full 360-degree stereoscopic scene.
Fig. 2B illustrates an example stereoscopic video scene divided into three example scenes according to an example embodiment.
Fig. 2C is a diagram illustrating an example stereoscopic video scene divided into four scenes according to an example embodiment.
FIG. 3 illustrates an example process for encoding an example 360-degree stereoscopic video scene according to an example embodiment.
Figure 4 illustrates an example showing how an input image portion is encoded using various encoders to produce different encoded versions of the same input image portion.
Figure 5 shows stored encoded parts of an input stereoscopic image scene divided into three parts;
FIG. 6 is a flow chart illustrating the steps of an example method of streaming content according to an example embodiment implemented using the system of FIG. 1.
FIG. 7 illustrates an example content delivery system including encoding capabilities that may be used to encode and stream content in accordance with aspects of the present invention.
FIG. 8 illustrates an example content playback system that may be used to receive, decode, and display content streamed by the system of FIG. 7.
FIG. 9 shows an example camera rig having three camera pairs mounted in three different mounting positions along with a calibration target that can be used to calibrate the camera rig.
Figure 10 shows a more focused view of the camera rig with three camera pairs mounted on the camera rig.
11 is a detailed diagram of an example camera rig implemented in accordance with an example embodiment.
12 illustrates an example 360 scene environment, e.g., a 360 scene area that can be divided into different viewing areas/portions corresponding to different camera positions of each camera capturing different portions of the 360-degree scene. A drawing showing.
FIG. 13 shows three different portions of the example 360 scene area of FIG. 12 that can be captured by different cameras positioned to correspond to and/or include viewing areas/portions of the example 360 scene area. A drawing comprising different drawings.
FIG. 14A illustrates a first portion of a flowchart illustrating steps of an example method of operating a playback device in accordance with an example embodiment of the present invention.
FIG. 14B illustrates a second portion of a flowchart illustrating steps of an example method of operating a playback device in accordance with an example embodiment of the present invention.
Figure 14 is a view including a combination of Figures 14A and 14B.
Figure 15 is a flowchart showing steps of a stream selection subroutine according to an example embodiment.
Figure 16 is a flow chart illustrating steps of a stream prioritization subroutine according to one example embodiment.
Fig. 17 is a flowchart showing steps for rendering a subroutine according to an example embodiment.
Figure 18 is an example table containing stream information corresponding to a plurality of content streams.
Figure 19 illustrates an exemplary playback system implemented in accordance with the present invention.
FIG. 20A illustrates a first portion of a flowchart of an example method of operating a content playback system according to an example embodiment.
FIG. 20B illustrates a second portion of a flowchart of an example method of operating a content playback system according to an example embodiment.
FIG. 20C illustrates a third portion of a flowchart of an example method of operating a content playback system according to an example embodiment.
FIG. 20D illustrates a fourth portion of a flowchart of an example method of operating a content playback system according to an example embodiment.
FIG. 20E illustrates a fifth portion of a flowchart of an example method of operating a content playback system according to an example embodiment.
Figure 20 is a view including a combination of Figures 20A, 20B, 20C, 20D, and 20E.
21 is a diagram of an example content playback system, eg, a content playback device or computer system coupled for display, according to an example embodiment.
FIG. 22 is a diagram of an example assembly of modules that may be included in the example content playback system of FIG. 21.
FIG. 23 illustrates an example stream selection module that may be used in the playback system of FIG. 19 in accordance with some embodiments.
FIG. 24 illustrates an example stream prioritization module that may be implemented as part of the stream selection module of FIG. 23 or as a separate module.

1 shows an example system 100 implemented in accordance with some embodiments of the present invention. System 100 supports content delivery, eg, imaging content delivery, to one or more customer devices, eg, playback devices/content players located at customer premises. System 100 includes an exemplary image capturing device 102, a content delivery system 104, a communications network 105, and a plurality of customer premises 106,..., 110. The image capturing device 102 supports capturing of stereoscopic images. Image capturing device 102 captures and processes imaging content in accordance with the features of the present invention. Communications network 105 may be, for example, a hybrid fiber-coaxial (HFC) network, a satellite network, and/or the Internet.

Content delivery system 104 includes an encoding device 112 and a content streaming device/server 114. Encoding device 112 may include, and in some embodiments includes, one or a plurality of encoders for encoding image data in accordance with the present invention. Encoders may be used simultaneously to encode different portions of the scene and/or to encode a given portion of the scene to produce encoded versions with different data rates. Using multiple encoders simultaneously can be especially useful when real-time or near-real-time streaming is supported.

Content streaming device 114 is configured to stream, e.g., transmit, encoded content to deliver the encoded image content to one or more customer devices, e.g., via communications network 105. Via network 105, content delivery system 104 transmits information and/or is located at customer premises 106, 110, shown in the figure, by link 120 traversing communication network 105. Information can be exchanged with devices.

Although encoding device 112 and content delivery server 114 are shown as separate physical devices in the FIG. 1 example, in some embodiments they are implemented as a single device that encodes and streams content. The encoding process may be a 3D, e.g. stereoscopic, image encoding process in which information corresponding to the left and right eye views of a scene portion is encoded and included in encoded image data so that 3D image viewing can be supported. . The specific encoding method used is not critical to the present application, and various encoders may be used as or to implement encoding device 112.

Each customer premises 106, 110 may include a plurality of devices/players, e.g., playback systems, used to decode and play/display imaging content streamed by content streaming device 114. there is. Customer Premise 1 (106) includes a decoding device/playback device (122) coupled to a display device (124) and Customer Premise N (110) includes a decoding device/playback device (126) coupled to a display device (128). Includes. In some embodiments, display devices 124, 128 are head mounted stereoscopic display devices. In some embodiments, playback devices 122/126 and head mounted devices 124/128 together form a playback system.

In various embodiments, decoding devices 122 and 126 present imaging content on corresponding display devices 124 and 128. Decoding devices/players 122, 126 may decode imaging content received from content delivery system 104, which generates imaging content using the decoded content and displays imaging content on display devices 124, 128. , for example, may be devices capable of rendering 3D image content. Either of the decoding device/playback devices 122, 126 may be used as the decoding device/playback device 800 shown in FIG. 8. A system/playback device such as the one shown in FIGS. 8 and 19 may be used as either the decoding device/playback devices 122 and 126.

FIG. 2A illustrates an example unsegmented stereoscopic scene 200, e.g., a full 360-degree stereoscopic scene. A stereoscopic scene can, and usually is, the result of combining image data captured from multiple cameras, for example, cameras, often mounted on a single video capture platform or camera mount.

2B shows a segmented version 250 of an example stereoscopic scene 200, where the scene is divided into three (N=3) example portions, e.g., a frontal 180 degree angle according to one example embodiment. It is divided into a left posterior 90-degree portion and a right posterior 90-degree portion.

FIG. 2C shows another segmented version 280 of an example stereoscopic scene 200 divided into four (N=4) portions according to one example embodiment.

2B and 2C show two example portions, but it should be understood that other divisions are possible. For example, scene 200 may be divided into 12 (n=12) 30 degree portions. In one such embodiment, rather than encoding each portion individually, multiple portions are grouped together and encoded as a group. Different groups of parts correspond to different parts of the image that can be streamed depending on the viewing angle, which is equal to the total angles of the scene but is measured on the user's head position, e.g., in the range of 0 degrees to 360 degrees. It can be encoded with the size of each group and streamed to the user.

3 illustrates an example process for encoding an example 360-degree stereoscopic video scene, according to an example embodiment. Input to the method 300 shown in FIG. 3 includes, for example, 360-degree stereoscopic image data captured by a plurality of cameras configured to capture a 360-degree view of a scene. Stereoscopic image data, for example stereoscopic video, may be in any of a variety of known formats and, in most embodiments, includes left and right eye image data used to allow for a 3D experience. Although the methods are particularly suitable for stereoscopic video, the techniques and methods described herein can also be applied to 2D images of, for example, 360 degrees or small scene areas.

At step 304, the scene data 302 is divided into data corresponding to different scene areas, eg, N scene areas corresponding to different viewing directions. For example, in one embodiment, as shown in Figure 2B, the 360-degree scene area is divided into three parts: a left rear portion, a front 180-degree portion, and a right rear 90-degree portion, corresponding to the 90-degree portion. . Different parts may have been captured by different cameras, but this is not necessarily the case, and in practice the 360-degree scene is compiled from data captured from multiple cameras before being divided into the N scene regions shown in Figures 2b and 2c. It can be.

At step 306, data corresponding to different scene parts are encoded according to the invention. In some embodiments, each scene portion is encoded independently by multiple encoders to support multiple possible bit rate streams for each portion. At step 308, the encoded scene portions are stored in the content delivery server 114 of the content delivery system 104 for streaming, for example, to customer playback devices.

4 illustrates an example 400 showing how an input image portion, e.g., a 180 degree frontal portion of a scene, is encoded using various encoders to produce different encoded versions of the same input image portion. am.

As shown in diagram 400, an input scene portion 402, e.g., a 180 degree frontal portion of the scene, is fed to a plurality of encoders for encoding. In the above example, there are K different encoders that encode the input data with different resolutions and using different encoding techniques to produce encoded data to support different data rate streams of image content. The plurality of K encoders includes high definition (HD) encoder 1 (404), standard definition (SD) encoder 2 (406), reduced frame rate SD encoder 3 (408), ..., and high compression reduced frame rate SD encoder. Includes K (410).

HD encoder 1 (404) is configured to perform full high definition (HD) encoding to generate a high bit rate HD encoded image (412). SD Encoder2 (406) is configured to perform low-resolution standard definition encoding to produce SD encoded version2 (414) of the input image. Reduced frame rate SD encoder3 408 is configured to perform reduced frame rate low resolution SD encoding to produce a reduced rate SD encoded version3 416 of the input image. The reduced frame rate may be, for example, half the frame rate used by SD Encoder2 406 for encoding. High compression reduced frame rate SD encoder K 410 is configured to perform reduced frame rate low resolution SD encoding with high compression to produce a highly compressed reduced rate SD encoded version K 420 of the input image.

Accordingly, control of spatial and/or temporal resolution can be used to generate data streams of different data rates, and control of other encoder settings, such as the level of data compression, can also be used alone or at one or more preferred data rates. It should be understood that may be used in addition to control of spatial and/or temporal resolution to generate data streams corresponding to scene parts.

Figure 5 shows stored encoded portions 500 of an input stereoscopic video scene divided into three example portions. The stored encoded portions may be stored in content delivery system 104, for example, as data/information in memory. The stored encoded portions 500 of a stereoscopic image scene include three different sets of encoded portions, each portion corresponding to a different scene portion, and each set a plurality of different encoded portions of the corresponding scene portion. Includes versions. Each encoded version is a version of encoded video data and therefore represents a number of coded frames. Each encoded version 510, 512, 516 of video corresponds to multiple time periods, and when streaming, a portion, e.g., frames corresponding to the time period being played, is used for transmission purposes. It must be understood.

As shown and discussed above with respect to FIG. 4, each scene portion, e.g., front, rear scene portions, may be encoded using a plurality of different encoders to generate K different versions of the same scene portion. there is. The outputs of each encoder corresponding to the provided input scene are grouped and stored together as a set. The first set of encoded scene portions 502 corresponds to a frontal 180 degree scene portion, encoded version 1 of the frontal 180 degree scene 510, encoded version 2 of the frontal 180 degree scene 512,... , and an encoded version K 516 of the frontal 180-degree scene. A second set of encoded scene portions 504 corresponds to scene portion 2, e.g., 90 degree left rear scene portion, and encoded version 1 520 of 90 degree left rear scene portion, 90 degree left rear. Includes encoded versions 2 (522) of the scene portion, ..., and encoded versions K (526) of the 90 degree left rear scene portion. A third set of similarly encoded scene portions 506 correspond to scene portion 3, e.g., a 90 degree right rear scene portion, and an encoded version 1 530 of the 90 degree right rear scene portion, 90 degrees It includes encoded versions 2 532 of the right rear scene portion, ..., and encoded versions K 536 of the 90 degree right rear scene portion.

A variety of different stored encoded portions of a 360-degree scene can be used to generate a variety of different bit rate streams for transmission to customer playback devices.

FIG. 6 is a flowchart 600 illustrating steps of an example method of providing image content, according to an example embodiment. The method of flowchart 600 is implemented in some embodiments using the capturing system shown in FIG. 1.

The method begins at step 602, for example, with the delivery system being powered and initialized. The method proceeds from start step 602 to steps 604. At step 604, content delivery system 104, e.g., server 11 within system 104, delivers a request for content, e.g., a previously encoded program, or in some cases, real-time or Receive requests for live events to be encoded and streamed in near real-time, for example, while the event is still in progress.

In response to the request, at step 606, server 114 determines the data rate available for delivery. The data rate may be determined from information included in the request indicating supported data rates and/or from other information, such as network information indicating the maximum bandwidth available to deliver content to the requesting device. As will be appreciated, the available data rate may vary depending on network load and may vary over the period of time the content is streamed. Changes are reported by the user device or packets are dropped or the network has difficulty supporting the data rate being used and the desired data rate is lower than the original data rate determined to be available for use. It can be detected from messages or signals indicating a delay beyond an amount of time.

Operation proceeds from step 606 to step 608, where the current head position of the user device at which the request for content is initiated, e.g., the current head position at the time of the request, is the 0 degree position. The 0 degree or forward-facing position may in some embodiments be reinitialized by the user with the playback device signaling that reinitialization is to occur. For example, with respect to the original head position, the user's head position and/or changes in the user's head position over time are reported to the content delivery system 104 and the update location makes content delivery decisions, as discussed below. It is used to perform.

Operation proceeds from step 608 to step 610, where portions of the 360-degree scene corresponding to the requested content are sent to initialize the playback device. In at least some embodiments, initialization includes transmitting a full 360-degree set of scene data, e.g., N portions, with the 360-degree scene divided into N portions.

As a result of the initialization in step 610, the playback device will have scene data corresponding to each of the different parts of the 360-degree viewing area. Accordingly, if the user of the playback device suddenly turns backwards, at least some data will be available for display to the user, even if it may not be as current as the portion the user was looking at before turning his head.

Operation proceeds from step 610 to steps 612 and 622. Step 622 corresponds to the full scene update path used to ensure that the playback device receives an updated version of the full 360-degree scene at least once every full update period. Once initiated at step 610, the entire update process is delayed at wait step 622 for a predetermined period of time. In a later step 624, a 360-degree scene update is performed. Dashed arrow 613 indicates the transfer of information that scene portions are delivered to the playback device during the support period corresponding to step 622. At step 624, the full 360 degree scene may be transmitted. However, in some embodiments, not all portions are transmitted in step 624. Portions of the scene that are updated during the waiting period 622 are, in some embodiments, performed at step 624 because they are already refreshed during regular streaming processes that transmit at least some portions of the scene based on the user's head position. omitted from updated updates.

Operation proceeds from step 624 back to wait step 622, where waiting is performed before the next full update. It should be understood that different overall refresh rates may be supported by adjusting the waiting period used in step 622. In some embodiments, the content server selects a waiting period based on the type of scene content being provided, and an overall reference period is provided accordingly. In the case of sporting events where the main action is in the front-facing area and one of the reasons for refreshing is possible changes in outdoor lighting conditions, the waiting period can be relatively long, for example on the order of a minute or several days. It can be minutes. For rock concerts, where audience movements and activities may change frequently as different songs are played, users may want to turn their bodies to see the audience's reactions and see what is happening in the audience in addition to what is happening in the front stage viewing area. Because you may want to get a feel for it, the overall refresh rate can be, and sometimes is, higher than that of sporting events.

In some embodiments, the overall reference period changes as a function of the portion of the representation that is streamed. For example, during the game portion of a sporting event, the overall refresh rate may be relatively low, but late touches exist, with a person on the field or watching the game on a playback device more likely to turn his or her head away from the front key area. During down moments or during timeouts or interruptions, the overall refresh rate may be increased, in some embodiments, by decreasing the wait, e.g., refresh period control used in step 622.

Although the entire refresh process has been described with reference to steps 622 and 624, a regular supply of portions of the scene are described. As will be understood, regular refresh of a scene or scene portion will occur, for at least one portion, at a data rate allowed by the supported video frame rate. Accordingly, for at least one portion of the frame, for example the portion where the user's head is shown to be facing, it will be fed at full video streaming frame rate assuming that the available data rate is sufficient.

At step 612, scene portions are selected to be presented based on the user's indicated head position, e.g., viewing angle. The selected portions are periodically transmitted, eg streamed, to a playback device. The rate at which data corresponding to portions is streamed depends on the video frame rate in some embodiments. For example, at least one selected portion will be streamed at the full supported frame rate. At least one scene portion is selected in step 612, but usually multiple scene portions are selected, for example, the next closest scene portion as well as the scene portion that the user is facing. If the available data rate is sufficient to support delivery of multiple frame portions, additional scene portions may also be selected and supplied.

After the scene portions to be streamed are selected at step 612, operation proceeds to step 614 where encoded versions of the selected stream portions are selected, e.g., based on the user's viewing position and the available data rate. do. For example, a full-rate, high-resolution version of the portion of the scene that the user is facing, as indicated by the currently reported head position, can and typically will be streamed. One or more portions of the scene to the left and/or right of the current head position at lower resolution, at a lower time rate, or using another encoding device that reduces the amount of bandwidth required to transmit scene regions that are not currently visible. may be selected to be streamed. The selection of encoded versions of adjacent scene portions will depend on the amount of bandwidth reaming after a high-quality version of the currently viewed scene portion is transmitted. Portions of the scene that are not currently visible may be transmitted as a lower resolution encoded version or as an encoded version with a greater temporal distance between frames, but if there is sufficient bandwidth available, a full resolution high quality version may be transmitted periodically or Sent frequently.

At step 616, the selected encoded versions of the selected scene portions are transmitted to the playback device that requested the content. Accordingly, at step 616, encoded content corresponding to one or more portions, e.g., stereoscopic video content corresponding to a number of consecutive frames, is streamed to a playback device.

Operation proceeds from step 616 to step 618 and information indicating the user's current head position is received. This information may be transmitted from the playback device periodically and/or in response to detecting a change in head position. In addition to changes in head position, changes in available data rates can affect how content is streamed. Operation proceeds from step 618 to step 620, where a determination of the current data rate that can be used for content delivery to the playback device is performed. Accordingly, the content delivery system can detect changes in the amount of available bandwidth to support streaming to the requesting device.

Operation proceeds from step 620 to step 612 until the content is completely delivered, e.g., a program or event ends, or a signal from the playback device that requested the content indicates that the session has ended. Streaming continues until either is received, or a failure to receive an expected signal from the playback device, such as a head position update, is detected, indicating that the playback device is no longer in communication with the content server 114.

From the scene data delivered in the manner described above, the playback device will have at least some data corresponding to each scene portion available for displaying instances where the user quickly turns his head. It should be understood that users rarely turn their heads completely in a very short period of time, as this is an inconvenient change in viewing position for many people. Accordingly, the full 360-degree scene may not be transmitted at all times, but high quality versions of the scene portion(s) most likely to be viewed at any given time may be streamed and made available to the user.

Because the encoding process allows the N portions of the scene to be transmitted and processed differently to different users rather than having to encode the content separately for each individual user, the content delivery system 104 supports multiple concurrent users. You can apply. Accordingly, multiple parallel encoders may be used to support real-time encoding to allow real-time or near real-time streaming of sporting events or other events, but the number of encoders used tends to be much smaller than the number of playback devices from which the content is streamed. there is.

Although portions of content are described as portions corresponding to a 360-degree view, it should be understood that scenes can, and in some embodiments, represent a flattened version of space that also has a vertical dimension. The playback device may use a model of the 3D environment, e.g. space, to map portions of the scene and adjust for vertical viewing positions. Accordingly, 360 degrees as discussed in this application refers to the head position relative to the horizontal as the user changes his viewing angle to the left or right while maintaining his gaze level.

7 illustrates an example content delivery system 700 with encoding capabilities that can be used to encode and stream content in accordance with features of the present invention.

The system may be used to encode, store, and transmit and/or output content in accordance with the features of the present invention. In some embodiments, system 700 or elements therein perform operations corresponding to the process depicted in FIG. 6. Content delivery system 700 may be used as system 104 of FIG. 1 . Although the illustrated system shown in FIG. 7 is used for encoding, processing, and streaming of content, it is understood that system 700 may also include the ability to decode and display processed and encoded image data to an operator. It must be understood.

System 700 includes display 702, input device 704, input/output (I/O) interface 706, processor 708, network interface 710, and memory 712. The various components of system 700 are coupled together via bus 709, which allows data to be transferred between components of system 700.

Memory 712, when executed by various modules, e.g., processor 708, controls system 700 to implement segmentation, encoding, storage, and streaming/transmission and/or output operations in accordance with the present invention. Includes routines that

Memory 712 may, when executed by various modules, e.g., processor 708, implement immersive stereoscopic video acquisition, encoding, storage, and transmission and/or output methods in accordance with the present invention (computer system ( 700) and routines that control it. Memory 712 includes control routines 714, segmentation module 716, encoder(s) 718, streaming controller 720, and received input images 732, e.g., 360-degree stereoscopic images of a scene. Includes visual video, encoded scene portions 734, and timing information 736. In some embodiments, modules are implemented as software modules. In other embodiments, modules are implemented in hardware, for example, as individual circuits with each module implemented as a circuit for performing a corresponding function. In still other embodiments, modules are implemented using a combination of software and hardware.

Control routines 714 include device control routines and communication routines to control the operation of system 700. The segmentation module 716 is configured to segment the received stereoscopic 360-degree version of the scene into N scene parts according to the features of the present invention.

Encoder(s) 718 may include a plurality of encoders configured to encode received image content, e.g., a 360-degree version of a scene and/or one or more scene portions, in accordance with features of the invention, and may include several In embodiments, they are included. In some embodiments, the encoder(s) includes multiple encoders, each encoder configured to encode stereoscopic video scenes and/or segmented scene portions to support a given bit rate stream. Accordingly, in some embodiments, each scene portion may be encoded using multiple encoders to support multiple different bit rate streams for each scene. The output of encoder(s) 718 are encoded scene portions 734 that are stored in memory for streaming to customer devices, e.g., playback devices. Encoded content may be streamed via network interface 710 to one or multiple different devices.

Streaming controller 720 is configured to control streaming of encoded content to deliver the encoded image content to one or more customer devices, for example, via communications network 105. In various embodiments, multiple steps of flowchart 600 are implemented by elements of streaming controller 720. Streaming controller 720 includes a request processing module 722, a data rate determination module 724, a current head position determination module 726, a selection module 728, and a streaming control module 730. Request processing module 722 is configured to process received requests for imaging content from customer playback devices. A request for content is received via a receiver at network interface 710 in various embodiments. In some embodiments, a request for content includes information indicating the identity of the requesting playback device. In some embodiments, the request for content may include the data rate supported by the customer playback device, the user's current head position, for example, the location of the head mounted display. Request processing module 722 processes received requests and provides the retrieved information to other elements of streaming controller 720 to take other actions. The request for content may include data rate information and current head position information, but in various embodiments, the data rate supported by the playback device may vary depending on network checks and network information that differs between system 700 and the playback device. It can be determined from exchange.

Data rate determination module 724 is configured to determine available data rates that can be used to stream imaging content to customer devices, e.g., as multiple encoded scene portions are supported, in a content delivery system ( 700) may support streaming content at multiple data rates to a customer device. Data rate determination module 724 is also configured to determine the data rate supported by a playback device requesting content from system 700. In some embodiments, data rate determination module 724 is configured to determine an available data rate for delivery of image content based on network measurements.

The current head position determination module 726 is configured to determine the current viewing angle and/or the current head position of the user, for example, the position of the head mounted display, from information received from the playback device. In some embodiments, the playback device periodically transmits current head position information to system 700, and current head position determination module 726 receives and receives the information to determine the current viewing angle and/or current head position. Process it.

The selection module 728 is configured to determine which portions of the 360-degree scene to stream to the playback device based on the user's current viewing angle/head position information. Selection module 728 is also configured to select encoded versions of determined scene portions based on available data rates to support streaming of content.

Streaming control module 730 is configured to control streaming of multiple portions of image content, e.g., a 360-degree stereoscopic video scene, at various supported data rates in accordance with the features of the present invention. In some embodiments, streaming control module 730 is configured to control streaming of N portions of a 360-degree stereoscopic video scene to a playback device requesting the content to initialize a scene memory in the playback device. In various embodiments, streaming control module 730 is configured to periodically transmit selected encoded versions of determined scene portions, for example, at a determined rate. In some embodiments, streaming control module 730 is also configured to transmit 360-degree scene updates to the playback device at time intervals, such as once per minute. In some embodiments, transmitting a 360-degree scene update includes transmitting N scene portions or N-X scene portions of a full 360-degree stereoscopic video scene, where N is a segment of the full 360-degree stereoscopic video scene. is the total number of parts, and X represents the selected scene parts currently transmitted to the playback device. In some embodiments, streaming control module 730 waits a predetermined amount of time after initially transmitting the N scene portions for initialization before transmitting a 360-degree scene update. In some embodiments, timing information for controlling transmission of 360-degree scene updates is included in timing information 736. In some embodiments, streaming control module 730 also identifies portions of the scene that were not transmitted to the playback device during the refresh interval; and transmit updated versions of identified scene portions that were not transmitted to the playback device during the refresh interval.

In various embodiments, the streaming control module 730 is configured to periodically deliver at least a sufficient number of the N portions to the playback device to cause the playback device to fully refresh the 360-degree version of the scene at least once during each refresh period. It is composed.

8 illustrates a playback system 800 implemented in accordance with the present invention that can be used to receive, decode, store, and display imaging content received from a content delivery system such as that shown in FIGS. 1 and 7. A system 800 that may be implemented includes a single playback device 800 comprising as a combination of elements such as a display 802, or an external display, e.g., a head mounted display 805 coupled to a computer system 800'. ')am.

Playback system 800, in at least some embodiments, includes a 3D head mounted display. The head mounted display may be implemented using an OCULUS RIFT ^™ VR (virtual reality) headset, which may include a head mounted display 805. Other head mounted displays may also be used. In some embodiments, one or more display screens in a head mounted helmet or other head mounting device are used to display content to the user's left and right eyes. By displaying different images for the left and right eyes on a single screen by a head mount configured to expose different portions of the screen to different eyes, a single display can be perceived separately by the left and right eyes of viewers. Can be used to display eye images. In some embodiments, a mobile phone screen is used as the display of a head mounted display device. In at least some of these embodiments, a mobile phone is inserted into a head mounting device, and the mobile phone is used to display images.

The playback system 800 decodes the received encoded image data, e.g. left and right eye images and/or mono (single images) corresponding to different parts of the environment or screen, for example: It has the ability to generate 3D image content for display to a customer by rendering and displaying different left and right eye views that are perceived by the user as a 3D image. Playback system 800 is, in some embodiments, located at a customer premises location, such as a home or office, but may also be located at an image capture site. System 800 may receive, decode, display, and/or perform other operations in accordance with the present invention.

System 800 includes a display 802, a display device interface 803, an input device 804, an input/output (I/O) interface 806, a processor 808, a network interface 810, and memory ( 812). The various components of system 800 are coupled together via a bus 809 that allows data to be transferred between components of system 800 and/or by other connections or via a wireless interface. In some embodiments, display 802 is included as an optional element, shown using a dash box, but in some embodiments, external display device 805, e.g., a head mounted stereoscopic display device, is a display device. It can be coupled to a playback device via interface 803.

For example, if a mobile phone processor is used as processor 808 and the mobile phone generates and displays images on a head mount, the system may include processor 808, display 802, and memory (802) as part of the head mounted device. 812) may be included. Processor 808, display 802, and memory 812 may all be part of the mobile phone. In other embodiments of system 800, processor 808 may be part of a gaming system, such as an XBOX or PS4, that has a display 805 mounted on a head mounting device and coupled to the gaming system. It does not matter whether the processor 808 or the memory 812 located on the device is headgeared or not, and as will be appreciated, in some cases, the processor 808 may be attached to the headgear from a power, heat, and weight perspective. Although it may be convenient to position, in at least some cases, it may be desirable to have the processor 808 and memory 812 coupled to a headgear that includes a display.

Although various embodiments contemplate a head mounted display 805 or 802, the method and apparatus may also be used with non-head mounted displays that can support 3D images. Accordingly, in many embodiments, system 800 includes a head mounted display, but may also be implemented by a non-head mounted display.

Memory 812 includes various modules, such as routines that, when executed by processor 808, control playback device 800 to perform decoding and output operations in accordance with the present invention. Memory 812 includes control routines 814, requests to content creation module 816, head position and/or viewing angle determination module 818, decoder module 820, stereoscopic image generation module, also called 3D image generation module. Rendering module 822, and data/information including received encoded image content 824, decoded image content 826, 360-degree decoded scene buffer 828, and generated stereoscopic image content 830. Includes.

Control routines 814 include device control routines and communication routines to control the operation of device 800. The request generation module 816 is configured to generate a request for content to be transmitted to a content delivery system for providing the content. Requests for content are transmitted via network interface 810 in various embodiments. The head position and/or viewing angle determination module 818 determines the user's current viewing angle and/or current head position, for example, the position of the head mounted display, and sends the determined position and/or viewing angle information to the content delivery system. It is configured to report to (700). In some embodiments, playback device 800 periodically transmits current head position information to system 700.

Decoder module 820 is configured to decode encoded image content 824 received from content delivery system 700 to generate decoded image data 826. Decoded image data 826 may include a decoded stereoscopic image scene and/or decoded scene portions.

3D image rendering module 822 uses decoded image content 826 to display 3D images, e.g., to a user on display 802 and/or display device 805, e.g. For example, features of the present invention create left and right eye images that are displayed in a way that is recognized as a 3D image. The generated stereoscopic image content 830 is the output of the 3D image generation module 822. Accordingly, the rendering module 822 renders the 3D image content 830 on the display. In some embodiments, display device 805 may be part of a 3D display device, such as an Oculus Rift. An operator of playback device 800 may control one or more parameters via input device 804 and/or select operations to be performed, for example, to display a 3D scene.

9 is sometimes referred to as a camera rig or camera array having three camera pairs 902, 904, 906 mounted in three different mounting positions and a calibration target 915 that can be used to calibrate the camera assembly 800. A diagram illustrating an exemplary camera assembly 900, referred to as FIG. Camera rig 900 is used to capture image content according to some embodiments of the invention. In some embodiments, camera rig 900 is used as image capture device 102 of FIG. 1 . The camera rig 900 has three pairs (902, 904) of stereoscopic cameras (901, 903), (905. 907), (909, 911) for a total of six cameras, cameras at the indicated locations. , 906) and a support structure (shown in FIG. 11). The support structure supports the camera and includes a base 1120, also referred to herein as a mounting plate (see element 1120 shown in FIG. 11) to which the plates on which the cameras are mounted can be fixed. The support structure may be made of plastic, metal or synthetic materials such as graphite or fiberglass and is represented by lines forming triangles that are also used to view the spacing and relationship between the cameras. The center point where the dotted lines intersect represents the central node around which camera pairs 902, 904, 906 may be rotated in some, but not necessarily all, embodiments. The central node corresponds to, for example, a triangular-based steel rod or threaded central mount, around which the camera support frame 912, represented by triangular lines in some embodiments, can be rotated. The support frame can be a plastic housing or a triangular structure on which the cameras are mounted.

In Figure 9, each pair of cameras 902, 904, 906 corresponds to a different camera pair location. The first camera pair 902 corresponds to 0 degrees forward relative to the front-facing position. These locations correspond to key scene areas of interest, for example, the field where a sporting event is being played, the stage, or some other area where key action is likely to occur. The second camera pair 904 corresponds to a 120 degree camera position and is used to capture the right rear viewing area. The third camera pair 906 corresponds to a 240 degree position (relative to the 0 degree position) and the left rear viewing area. Note that the three camera positions are 120 degrees apart. Each camera viewing position includes one camera pair in the FIG. 9 embodiment, and each camera pair includes a left camera and a right camera used to capture images. The left camera captures what are sometimes called left eye images, and the right camera captures what are sometimes called right eye images. The images may be part of a view sequence or still images captured more than once. Typically at least the front camera position corresponding to camera pair 902 will be occupied by high quality video cameras. Other camera positions may be occupied by high quality video cameras, low quality video cameras, or a single camera used to capture still or mono images. In some embodiments, the second and third camera embodiments are left unoccupied and equipped with cameras that allow the first camera pair 902 to capture images corresponding to all three camera positions at different times. The supported support plate is rotated. In some such embodiments, left and right rear images are captured first, and then video of the front camera position is captured during the event. Captured images may be encoded and streamed to one or more playback devices in real time, for example, while the event is still in progress.

The first camera pair shown in Figure 9 includes a left camera 901 and a right camera 903. The left camera 901 has a first lens assembly 920 fixed to the first camera, and the right camera 903 has a second lens assembly 920' fixed to the right camera 903. Lens assemblies 920, 920' include lenses that allow a wide angle field of view to be captured. In some embodiments, each lens assembly 920, 920' includes a fisheye lens. Accordingly, each of cameras 902, 903 can capture a 180 degree field of view or nearly 180 degrees. In some embodiments, less than 180 degrees are captured, but in some embodiments there is also at least some overlap in images captured from adjacent camera pairs. 9 In the embodiment, a pair of cameras is positioned at each of the first (0 degrees), second (120 degrees), and third (240 degrees) camera mounting positions, with each pair viewing at least 120 degrees of the environment. However, in many cases, each camera pair captures 180 degrees or nearly 180 degrees of the environment.

The second (904) and third (906) camera pairs are the same or similar to the first camera pair (902), but are positioned at 120 and 240 degree camera mounting positions relative to the frontal 0 degree position. The second camera pair 904 includes a left camera 905 and a left lens assembly 922, and a right camera 907 and a right camera lens assembly 922'. The third camera pair 906 includes a left camera 909 and a left lens assembly 924, and a right camera 911 and a right camera lens assembly 924'.

In Figure 9, D represents the distance between axes of the first stereoscopic image pair of cameras 901 and 903. In the Figure 9 example, D is 117 mm, which is equal to or similar to the distance between the pupils of the average human left and right eyes. Dashed line 950 in Figure 9 shows the distance from the center point of the panoramic array to the entrance pupil of the right camera lens 920' (also known as the node offset). In one embodiment corresponding to the example in Figure 9, the distance indicated by reference numeral 950 is 315 mm, but other distances are possible.

In one particular embodiment, the footprint of camera rig 900 is relatively small, with a horizontal area of less than 640 mm2. This small size allows the camera rig to be placed in the audience, for example in a seating position where fans or attendees are normally positioned or placed. Accordingly, in some embodiments, the camera rig is placed in the audience area allowing the viewer to detect members of the audience desiring this effect. In some embodiments the footprint corresponds to the size of the base encompassed by the support structure on which, in some embodiments, the central support rod is mounted or the support tower is located. As will be appreciated, in some embodiments the camera rigs may rotate about an underlying center point that corresponds to the center point between the three pairs of cameras. In other embodiments, the cameras are fixed and do not rotate around the center of the camera array.

The camera rig can capture relatively close objects as well as distant objects. In one particular embodiment, the minimum imaging distance of the camera array is 649 mm, but other distances are possible and this distance is by no means critical.

The distance from the center of the camera assembly to the intersection 951 of the views of the second and third camera portions represents one example calibration distance that can be used to calibrate images captured by the first and second camera pairs. Note that the target 915 can be placed at a known distance from the camera pair, positioned in the area of maximum distortion or slightly beyond it. A calibration target contains a known fixed calibration pattern. A calibration target can be and is used to calibrate the size of images captured by the cameras of a camera pair. This calibration is possible because the size and location of the calibration target are known relative to the camera that captures the image of the calibration target 915.

FIG. 10 is a diagram 1000 of the camera array shown in FIG. 9 in more detail. The camera rig is also shown with six cameras, but in some embodiments the camera rig is occupied by only two cameras, for example camera pair 902. As shown, there is a 120 degree gap between the mounting positions of each camera pair. For example, if the centroid between each pair of cameras corresponds to the direction of the camera mounting positions, in this case the first camera mounting position corresponds to 0 degrees, the second camera mounting position corresponds to 120 degrees, and Consider that the third camera mounting position corresponds to 240 degrees. Therefore, each camera mounting position is separated by 120 degrees. This can be seen when the angle between the lines and the center line extending through the center of each camera pair 902, 904, 906 is measured.

In the FIG. 10 example, a pair of cameras 902, 904, 906 can rotate around a central point of the camera rig allowing different views to be captured at different times without having to change the camera base position, in some embodiments. In, it rotates around. That is, the cameras can be rotated around the central support of the rig and allowed to capture different scenes at different times allowing for 360 degree scene capture using the rig shown in Figure 10, which requires only two cameras. occupied by fields. This configuration is particularly desirable from a cost perspective given the cost of stereoscopic cameras, and it is desirable to show the background captured at the same time but at a different time than when a frontal scene containing key action may occur during a sporting event or other event. It is suitable for many applications. For example, consider that during an event objects that would otherwise be desirable to be invisible during the main event may be placed behind the camera. In such a scenario, rear images may be captured before the main event and made available along with images captured in real time of the main event to provide a 360 degree set of image data, and may be captured and available from time to time.

Figure 11 shows a detailed view of an example camera rig 1100 implemented according to an example embodiment. As can be appreciated from FIG. 11 , camera rig 1100 includes three pairs of cameras 1102, 1104, 1106, which in some but not all embodiments are stereoscopic cameras. Each camera pair includes two cameras in some embodiments. Camera pairs 1102, 1104, 1106 are the same or similar to camera pairs 902, 904, 906 discussed above with respect to FIGS. 9 and 10. In some embodiments, camera pairs 1102, 1104, and 1106 are mounted on a support structure 1120 of camera rig 1100. In some embodiments, three pairs (six cameras) of cameras 1102, 1104, 1106 are mounted on support structure 1120 via respective camera pair mounting plates. The support structure 1120 includes three mounting positions for mounting pairs of stereoscopic cameras, each mounting position corresponding to a different 120 degree viewing area. 11 , the first pair of stereoscopic imaging cameras 1102 is mounted in a first of the three mounting positions, e.g., the frontal position, corresponding to a 120 degree frontal viewing area. . The second pair of stereoscopic imaging cameras 1104 is mounted in a second of three mounting positions, e.g., a background position rotated 120 degrees clockwise with respect to the frontal position, and positioned in a different 120 degree viewing area. respond. The third pair of stereoscopic imaging cameras 1106 is mounted in a third of the three mounting positions, e.g., a background position rotated 240 degrees clockwise with respect to the frontal position, and in the other 120 degree viewing area. respond. The three camera mounting positions on camera rig 1100 are offset by 120 degrees relative to each other, although in some embodiments, each camera mounted on the camera rig has a field of view of approximately 180 degrees. In some embodiments, this expanded field of view is achieved by using a fisheye lens on the camera devices.

Although not all mounting plates are visible in the drawing, camera mounting plate 1110 is shown as used to mount camera pair 1102. Mounting plates for the cameras have slots for screws passing through threaded holes in the mounting plate from the bottom past the slots in the support base 1120. This allows the mounting plate for the pair to be adjusted by loosening screws accessible from the bottom and then tightening the screws to secure the camera pair mounting plate to the support structure. Individual camera positions can also be adjusted and fixed after subsequent adjustments. In some embodiments, individual cameras can be adjusted/secured with a mounting plate from the top and camera mounting plates can be adjusted/secured from the bottom.

In many embodiments, camera rig 1100 includes a base 1122 on which a support structure 1120 is rotatably mounted. Accordingly, in various embodiments, the camera assembly on support structure 1120 may be rotated 360 degrees about an axis passing through the center of the base. In some embodiments, base 1122 may be part of a tripod or other mounting device. The support structure may be made of plastic, metal, or synthetic materials such as graphite or fiberglass. Camera pairs may, in some embodiments, be rotated about a central point, sometimes called a central node.

In addition to the above elements, in some embodiments, camera rig 1100 additionally includes two simulated ears 1130 and 1132. These simulated ears 1130, 1132 mimic human ears and, in some embodiments, are comprised of silicone molded into the shape of a human ear. Simulated ears 1130, 1132 include microphones where the two ears are spaced from each other at or about the same distance as the spacing between the average human's human ears. The microphones mounted on the simulated ears 1130 and 1132 are mounted on a front-facing camera pair 1102, but could alternatively be mounted on a support structure, for example, platform 1120. Simulated ears 1130, 1132 are positioned perpendicular to the front of the camera pair 1102 in a similar way that human ears are positioned perpendicular to the front of the eyes on a human head. The holes on the sides of the simulated ears 1130, 1132 serve as audio input holes on the sides of the simulated ears, and the simulated ears and holes simulate how the human ear directs audio sounds to the eardrum contained in the human ear. It operates in combination to direct audio to a microphone mounted on each of the ears. Microphones in the left and right simulated ears 1130, 1132 provide stereo sound capture similar to what a human would perceive through the left and right ears of the camera rig 1100 if the human were positioned at the camera rig's location. .

11 shows one configuration of an example camera rig 1100 with three stereoscopic camera pairs, but it should be understood that other variations are possible within the scope. For example, in one implementation, camera rig 1100 may be a single camera pair, e.g., a pair of stereoscopic cameras that can rotate around a center point of the camera rig such that different portions of a 360-degree scene are captured at different times. Includes video cameras. Thus, a single camera pair can be mounted on a support structure and rotated around the central support of the rig, allowing them to capture different scenes at different times allowing for 360 degree scene capture.

12 illustrates an example 360 scene environment 1200, which may be divided into different viewing areas/portions corresponding to different camera positions of each camera, e.g., capturing different portions of the 360-degree scene. Shows a 360-degree scene area. In the example shown, the 360 scene area 1200 includes three different cameras, such as, for example, the cameras mounted and positioned on camera rig 1100 shown in FIGS. 9, 10, and 11. It is divided into three parts corresponding to three 180 degree zones captured by the camera pairs. The 0 degree marking on the 360 degree scene 1200 may be considered to be the center of the scene. In some embodiments where a fisheye lens is not used, the field of view of each camera is approximately 120 degrees, thus allowing the camera to capture approximately a 120 degree scene area. In these embodiments, the boundaries of different 120-degree scene portions are shown in the drawing using solid black lines that divide the 360-degree scene into three portions of 120 degrees each. In embodiments where the cameras are equipped with fisheye lenses, the field of view of each camera extends to approximately 180 degrees (+-5 degrees), thus allowing the camera to capture nearly 180 degrees (+-5 degrees) of the scene area.

A first region comprising 90 degrees to the left and right of the 0 degree mark (region 1 corresponding to a 180 degree frontal scene area from 270 to 90) is defined by a first camera, e.g. a camera device, of approximately 180 degrees. It may be captured by a pair of cameras 1102 positioned to capture the frontal scene area, equipped with a fisheye lens to provide a field of view. The second zone (zone 2) is a 180 degree angle from 30 to 210 that can be captured by a second camera, for example camera pair 1104 positioned to capture the rear right scene area with a fisheye lens. Corresponding to the rear right scene area, a third zone (zone 3) is to be captured by a third camera, for example a camera pair 1106 positioned to capture the rear left scene area with a fisheye lens. It corresponds to a 180 degree rear left scene area ranging from 150 to 330 degrees. Legend 1250 includes information identifying different line patterns used to indicate different zone boundaries, for example, marking the beginning and end of scene areas included under the different zones. It can be understood from the drawing that there is substantial overlap between the scene areas under the three different zones covered by the different cameras. In the illustrated example of Figure 12, the overlap between zones 1 and 2 is 60 degrees, i.e. the scene area below 30 to 90 degrees, and the overlap between zones 2 and 3 is also 60 degrees. , that is, the scene area under 150 to 210 degrees, and the overlap between zone 3 and zone 1 is 60 degrees, i.e., the scene area under 270 to 330 degrees. In the example shown, the overlap is 60 degrees, but it should be understood that many different overlaps are possible. In some embodiments, the overlap between two different scene coverage areas is between 30 and 60 degrees.

In some embodiments, a content provider publishes, eg, multicasts, a content stream containing content corresponding to different portions of 360-degree scene areas captured by different cameras to customer playback devices. In some embodiments, multiple versions of content corresponding to different scene regions, encoded differently, are multicast by the content provider, and a playback device that supports and prefers a particular version can select the appropriate content stream to decode and play. there is. According to an aspect of some embodiments, a playback device tracks the current head position representing the user's current field of view and provides available content, including content corresponding to a portion of the 360-degree scene area, to select to receive for use in playback. Determine one or more of the streams. For example, if the user's head position is such that the user is looking forward, the playback device will decode a stream that conveys a 180-degree frontal portion of a 360-degree scene, but when it detects that the user's head position and viewing angle has changed, the playback device will decode a stream that conveys a 180-degree frontal portion of the 360-degree scene. The device decodes the stream corresponding to the portion of the scene that matches the user's current viewing angle, e.g., back right, back left, and behind the 360-degree scene area. In some embodiments, the stream containing content corresponding to the first portion of the 360-degree scene area, e.g., the frontal 180 degrees, is configured to include a pair of front-facing cameras used to capture the frontal portion of the 360-degree scene. Contains the scene area captured by the left and right cameras.

FIG. 13 is an example of FIG. 12 that may be captured by different cameras, for example, on an example camera rig 1100 that may be positioned to include viewing areas/portions of an example 360 scene area. An example showing different parts of a 360 scene area is shown. The FIG. 13 example includes diagrams 1300, 1320, and 1350 depicting different portions of an example 360 scene area. However, it can be understood from the drawings that in some embodiments at least some of each of the different parts of the scene captured by different cameras overlap. Different shading patterns were used in each of the figures 1300, 1320, and 1350 to show portions of the scene area corresponding to different camera positions. In some embodiments, different scene portions shown in figures 1300, 1320, and 1350 are delivered via different content streams that provide content corresponding to different viewing directions. Although the scene portions in the FIG. 13 example are shown as encompassing approximately a 180 degree viewing area, in some embodiments the scene portions may encompass between 120 and 180 degrees.

Drawing 1300 depicts a first example scene portion 1305 of a 360 degree scene 1200. The first example scene portion 1305 corresponds to the front viewing direction and includes a 180 degree or approximately 180 degree viewing area of the 360 degree scene environment. The first scene portion 1305 may be captured by a first camera pair 902 or 1102, for example positioned at a 0 degree camera position. Areas on the first example scene portion 1305 are depicted in diagram 1300 using a diagonal pattern. The first example scene portion 1305 may be conveyed by a first stream carrying frames of content corresponding to a first, eg frontal, viewing direction.

Diagram 1320 illustrates a second example scene portion 1307 (including portions 1307' and 1307" shown in diagram 1320) of 360-degree scene 1200. Second example scene Portion 1307 corresponds to the rear right viewing direction and includes a 180 degree or approximately 180 degree viewing area of the 360 degree scene environment extending from 30 to 210. Second scene portion 1307 is shown in Figures 9 and 10. It may be captured by, for example, a second camera pair 904, positioned at the 120 degree camera position shown. The area under the second example scene portion 1307 is shown in diagram 1300 using a horizontal line pattern. is shown. Consider the drawings 1300 and 1302. Note the overlapping portion 1308 of the scene area between the first and second scene portions 1305 and 1307. The overlapping portion 1308 is the scene area ( illustrates a portion of the captured scene area that is common to both first and second scene portions 1305, 1307 of 1200. In some embodiments, first and second scene portions 1305, 1307 The overlap varies between +-2 and 3 degrees and is between 30 and 60 degrees.In the illustrated example of Figure 13, the overlap portion 1308 is in the region of 60 degrees, e.g., 30 to 90 degrees. Accordingly, in some embodiments, at least a portion of the scene area corresponding to different viewing directions provided by different content streams and/or captured by different cameras overlaps. In some other embodiments, the different viewing directions overlap. There is no overlap between scene areas corresponding to different viewing directions captured by the cameras.In some embodiments, the second example scene portion 1307 includes frames of content corresponding to the second viewing direction. It may be transmitted by a second stream that transmits.

Diagram 1350 illustrates a third exemplary scene portion 1309 (including portions 1309' and 1309" shown in diagram 1350) of 360-degree scene 1200. Third exemplary scene portion 1309 Scene portion 1309 corresponds to the rear left viewing direction and includes a 180 degree or approximately 180 degree viewing area of the 360 degree scene environment extending from 150 to 330. Third scene portion 1309 is shown in Figures 9 and 10. may be captured by, for example, a third camera pair 906 positioned at the 240 degree camera position shown in. The area under the third example scene portion 1309 is represented in diagram 1300 using a vertical line pattern. Consider the drawings 1320 and 1350. Note the overlapping portions 1310 and 1310', which are formed by combining portions of the overlapping area between the second and third scene portions 1307 and 1309. The common area under overlapping portions 1310, 1310' shows the portion of the captured scene area that is common to both the second and third scene portions 1307, 1309 of scene area 1200. Some Embodiments , the overlap between the second and third scene parts 1307, 1309 lies between 30 and 60 degrees with a variation of ±2-3 degrees. In the illustrated example of Figure 13, the overlap parts 1310, 1310') together comprise an area of approximately 60 degrees, e.g., from 150 to 210. Consider further the figures 1300, 1350, first and third scene portions 1305, 1309. Note the overlap portion 1312, which represents a portion of the overlapping area between In some embodiments, the third exemplary scene portion 1309 is connected to a third stream carrying frames of content corresponding to the third viewing direction. can be transmitted by

The drawings shown as part of the example in FIG. 13 illustrate an example to facilitate understanding of some aspects of the invention; however, it should be understood that other variations are possible within the scope of the disclosure.

Figure 14, which includes the combination of Figures 14A and 14B, is a flowchart 1400 illustrating the steps of an example method of operating a playback system in accordance with embodiments of the invention. The system may be the playback system 800 shown in Figure 8 or may be a playback system of any of the other figures shown in this application.

The example method begins at step 1402, where a playback device, such as playback device 1900 of Figure 19 or any of the other figures, is turned on and initiated. For purposes of discussion, a head mounted display device 1905 wherein the playback system includes a display on which image content is presented, e.g., in the case of stereoscopic content, to the left and right eyes of a user on which different images are presented. Consider including a computer system 1900' coupled to . Although computer system 1900' is shown as external to a head mounted device that includes a display, computer system 1900' may be integrated into a head mounted display rather than external to it.

Operation proceeds from steps 1402 to step 1404. At step 1404, playback system 1900 receives information regarding a plurality of content streams and/or initialization data, for example, as part of a program guide. The information received may be in the form shown in Figure 18, with information that can be used to receive the content streams, such as a multicast group identifier or other identifier that can be used to request content or tune to content. Contains information indicating what is available or available. For example, a multicast address associated with a content stream can be included in received information or a program identifier that can be used to request content when the content is provided over switched digital video. For broadcast content, the received information may, and sometimes includes, tuning information indicating the channel and/or frequency to which the playback device must be tuned to receive a particular content stream.

Information received at step 1404 may include information about one or more programs. For a given program, eg, a sporting event, concert, etc., different streams may be available to present the content corresponding to different viewing directions with respect to the camera position in the environment to which the content corresponds. The camera position corresponds to the viewing position during playback. Accordingly, the user's viewing angles during playback are correlated with the portions of the environment represented in the content that can be received. Different parts of the environment may be delivered to different streams. For each portion of the environment, eg, a portion of a 360-degree environment, one or more streams corresponding to different data rates may be listed in the information provided. Upper and lower portions of the environment may also be provided. In some embodiments the content of the individual streams is stereoscopic video content with different information provided for the left and right eye images, thereby allowing the different images to be displayed to the user to provide the desired 3D effect. In some embodiments, the upper and lower portions of the spherical environment are presented as mono images where the left and right eye views are the same, thus requiring only one image to be presented rather than two images.

Information about the program, the portion of the environment, and the data rate of the stream may be used by the playback system to prioritize which streams are received given the limited bandwidth available for streaming the content, and in some embodiments, It is used. Prioritization and selection of streams to receive at a given time may be based, in some embodiments, on the user's head position and/or the direction of the user's current or past head rotation.

For streaming of stereoscopic content, given limited bandwidth and/or data limitations, selection of the appropriate stream to receive can be important for a satisfactory and high quality viewing experience. Information received at step 1404, which may be the same or similar to the information shown in FIG. 18, e.g., stream information 1405, is stored in memory and, e.g., fed to a playback device to which the desired stream is delivered. By indicating to the network device which digital video channel will be switched, by associating a multicast group corresponding to the selected stream that tunes to the selected stream and/or a channel providing the requested delivery of the stream, It is used to receive at a point in time and initiate delivery of the selected stream.

Operation proceeds from step 1404 to step 1406. At step 1406, during initialization, the user's current head position is detected. The user, aware that the head position detected during the initialization step will be assumed to be a front-facing position, usually maintains his head at a comfortable level of front-facing position during step 1406.

Operation proceeds from step 1406 to step 1408. At step 1408, the user's head position 1407 detected at step 1406 is captured and subsequently encoded and included in the content stream corresponding to the specific portion of the environment in which the user held his head in a 0 degree environmental position. The displayed front (0 degree) environment is treated as the viewing position and view when in this position, which will correspond to the front position captured by the cameras used in the images. In the case of a sporting event, this location will usually correspond to the main area of action in the environment, for example the stream or streams if they correspond to a concert, the stage and the center of the field if the streams correspond to a sporting event. Accordingly, in step 1408, the user's viewing position will be set to be interpreted as a 0 degree viewing position, eg, a front/frontal position of the scene area. Note that parts of the 360 degree view correspond to horizontal viewing positions with different parts visible when the user rotates his head. By moving the user's head up or down, the user can view the sky portion and/or the ground portion alone or in combination with one or more other portions. Since the main scene area is divided into parts based on a 360 degree rotation assuming the level head position, the streams corresponding to these parts are usually allocated more bandwidth, while the upper/lower scene parts are still images or rarely change. It can be expressed using:

Operation proceeds from step 1408 to step 1406, where an environment depth map is received. The depth map defines the surface of the 3D environment onto which images in the content stream will be mapped. In the absence of reception of a depth map, the default shape of the environment is assumed to have the surface interior of a sphere, which is the surface to which images of the environment are mapped during rendering prior to display. By providing and using a depth map, a more realistic experience is achieved because the shapes and surfaces of the environment in which the images in the content stream will be simulated will be mapped to more realistic surfaces. Accordingly, the depth map received at step 1410 corresponds to an environment to which the content selected by the user to be received also corresponds. The environment map received at step 1410, or the default map if no map is received, is stored as environment map 1411 for subsequent use when rendering images.

Operation proceeds from step 1410 to step 1412. At step 1412, one or more UV maps are received to be used to map the 2D image content onto at least a portion of the 3D surface. In one embodiment, at least one UV map is received for each portion of the environment, which can be represented in different individual images. In some such embodiments, when the images carried by the content stream are captured by different cameras, for example left and right eye cameras, different UV maps may be provided for the different cameras, and sometimes , provided therefor. Thus, in the FIG. 14A example, a first UV map is received at step 1414, corresponding to a first portion of the environment, such as the front portion, and at step 1416, a first UV map corresponding to a second portion of the environment, such as the left rear portion. A second UV map is received, and at step 1417, a third UV map corresponding to a third portion of the environment, such as the right rear portion, is received. UV maps corresponding to the upper and lower portions of the environment are received at steps 1418 and 1420, respectively. If the parts are the same size, the same UV map is used for multiple parts. However, in some embodiments, different UV maps are used for images captured by different cameras.

Accordingly, in one such embodiment, for each portion of the environment for which stereoscopic image data is provided, e.g., left and right eye images, UV maps are used to capture specific left or right eye image content. Separate UV maps can be received and stored for each of the left and right eye images to take into account specific characteristics of the camera assembly.

Each UV map provides a map used to map the two-dimensional image of the content stream to which the map corresponds to a corresponding portion of the surface of the 3D environment. Images captured by a camera in this way can be transmitted as 2D images and then mapped as a texture on a surface or portion of a surface defined by a 3D model.

With the 3D model information and UV maps, images are received that can be used as defaults when other content for the scene part can be received and stored. At step 1422, content, e.g., images corresponding to one or more of the first, second, third, and fifth scene portions, is received and stored at step 1422. In some embodiments, multiple alternative images corresponding to a portion of the environment, such as a background portion or a sky portion, are received and stored at step 1422. Control information may be received indicating which default image is to be used at a given time during an event for a portion of which multiple default images are stored. For example, in some embodiments, a background image of a seated audience member and a background area image of a standing audience member are stored as two default images. Control information is used to indicate which background image is used for a given part of the event. For example, during a portion corresponding to a standing ovation at a game or concert, an image of the crowd standing is signaled as the background image to be displayed with the user facing in the direction of the background. However, during the main part when the audience is normally seated, the control information signals that if the user turns his head towards the background, the default image of the seated audience should be used. Control information may be signaled separately from the content stream or may be included with the content stream for a portion of the environment that is different from the portion to which one or more of the default images are related. For example, a content stream corresponding to the front direction may include images corresponding to the front direction, e.g., left and right eye images, as well as default images of the sky, the ground, the right background portion, and the like at various times during the event. You can provide control information indicating what should be used for the left background part. Alternatively, the playback device may determine whether to use the background or the sky portion based on the luminance of similarity to one or more features of the foreground image at a particular point in time. For example, when the foreground images are dark, this can be detected and the automatically detected cloudy sky image, whereas when the foreground images are brighter, this can also be detected and the available default sky images received and stored. This may be a less cloudy, brighter sky image automatically selected from .

At step 1422, default images corresponding to different parts of the environment are received and stored, usually in encoded form. At step 1424, one or more of the received images are decoded, and the decoded content is then stored in one or more image buffers at step 1426. In this way, default images can be decoded and stored in decoded form so that they do not have to be decoded again later when needed for rendering during playback. Because the default image may be used multiple times, decoding and storage can reduce decoding requirements otherwise required to decode the image at rendering or only before rendering. Assuming that processing resources are capable of providing pre-decoding and storage of default images in decoded form, images are decoded immediately before display, and thereafter, when they are no longer needed, the decoded image is deleted, e.g. in memory. Improves use of processor resources compared to embodiments removed from.

The same default decoded image may be used multiple times, for example the sky, etc., but combined with other received image content so that it more closely matches other images of the environment with which it is combined to create the viewed portion of the environment. It can be processed before it happens. For example, in some embodiments, the decoded default images undergo a luminance adjustment based on the portion of the image in which they are combined or blurred, at least along edges where the default image will be combined with images corresponding to other parts of the environment. Accordingly, during use in at least some embodiments, image brightness and/or color features are filtered or modified to make them more similar to the same features of the ground image with which they are combined.

With initialization data and default images stored for future use, operation proceeds to step 1428, where a set of one or more streams to be received, e.g., a currently selected set of streams, is initialized. Each stream can provide stereoscopic image or mono image data. Optionally corresponding audio may also be received in selected streams but is more common in one or more individual audio streams. The technique will focus on the reception of video streams, with the understanding that the audio stream will usually also be received and decoded by the playback device, and that the audio may include stereo audio.

In an example step 1428, the currently selected set of streams is set equal to the first stream carrying content corresponding to the front/front portion of the environment. This is because the initial position on startup is set to be the front viewing position and therefore as a result of initialization the user will expect to see the front scene area on startup.

Operation proceeds from step 1428 to step 1429. At step 1429, resource allocation information is received. Resource allocation information may be in the form of bandwidth and/or data rate allocation control information. The information received at step 1429 includes, in some embodiments, information regarding how much bandwidth or data delivery capacity should be allocated to one or more delivery streams corresponding to different parts of the environment. Information can be expressed as bandwidth or data rate with the understanding that data rate usually corresponds to bandwidth. For example, the amount of data that can be received may be a function of the amount of bandwidth, taking into account the form of data coding used to convey across the bandwidth.

The information received at step 1429 may indicate the relative maximum amount of available transmission capacity to allocate to the reception of images corresponding to a particular portion of the environment. This means, for example, that a maximum of 80% of the bandwidth or supportable data rate should be allocated to the primary data stream, e.g. that the forward data stream with the remaining 20% of the bandwidth should be allocated to one or more other streams. It can be expressed. The allocation of resources for different directions may be, and in some embodiments is a function of, the image content in the corresponding part of the environment and/or detected spectator feedback. For example, in some embodiments, during intermissions that occur during events to which content corresponds, the information received at step 1429 means that an increased amount of resources may It can indicate that it should be assigned to receive This is because, during intermissions, there is a greater chance that users will start to turn their heads and look away from the main field or stage, as some videos provide for the back sections where it appears that there is ongoing motion in the crowd during intermissions. It may be desirable to be For example, there may be images of people buying hot dogs or changing seats at a baseball game, and in some embodiments, the background may be transmitted to appear live during intermissions but static at other times. Similarly, images of billboards in the background may be changed during intermissions for advertising and/or entertainment purposes. Accordingly, it may be desirable to trigger the playback device to allocate more resources for receiving background portions during intermissions than during other portions of the game. The control information received at step 1429 can, and sometimes is, different during major parts of the game and during other precautionary portions or intermissions of the game. In at least some embodiments during primary competition, the control information received at step 1429 causes more bandwidth and/or data rate to be allocated to primary, eg, front, areas of the environment rather than back portions. However, during intermissions or other individual parts, the data rate assigned to one or both of the rear parts may be forced to increase.

Separation between the resources allocated to a given direction, the content presented in that part of the environment, the measure of audience attention, and/or the part of the ongoing event, such as the main part, the intermission part, and the end of the show part. It can be based on In some embodiments the control information provided at step 1429 may include, for example, the maximum amount of bandwidth or data rate to be allocated to the reception of images corresponding to one or more portions of the environment while the event is in progress, and/or Specify the minimum amount. In some embodiments, the information indicates that bandwidth or data may not be allocated to receiving ground-to-sky image portions during an event, and thus these portions are to be filled in when it is necessary to use still images in such cases.

Bandwidth/data rate allocation control information may change over time, and different information is received at different times. The control information may be embedded into the forward content stream and/or transmitted separately, for example as a separate set of control information.

Operation proceeds from step 1429 to step 1430, where content delivery of stream(s) in the selected set of streams that have not already been received is initiated. This may include connecting to the multicast group to which the selected stream corresponds, sending a message to a network device requesting delivery of the selected stream, and/or tuning to the broadcast channel on which the selected stream or streams are transmitted. In the case of a first pass through step 1430, this will involve initiating delivery of the content stream corresponding to the forward portion of the environment since this has been established as the initial viewing portion to be selected for delivery. However, as the user's head position changes, for example as the user turns his head left or right, the set of streams that are selected can and will usually change. For example, if the user turns his head to the left so that the front area and a portion of the right rear area are visible, the set of selected streams will change such that content corresponding to the front area and the left rear area is received. If the left rear portion has not been received and is selected, delivery of content in the stream corresponding to the left rear portion will commence at step 1430. In case the maximum data rate streams in both directions may not be supported, the lower data rate front stream is selected, and thus the initiation of both the lower data rate front stream and the left rear content stream will be initiated. Streams outside the selected set are terminated prior to receipt of streams in the newly selected set. The termination and initiation of streams can be characterized by dramatic changes in the received content and/or temporal gaps, with blurring and/or filtering used to significantly reduce changes in the quality or source of images when stream switching occurs. This is done in a flexible manner to minimize For example, blurring may be performed over portions of images that are stitched together as part of a rendering or display process.

Operation proceeds from step 1430 to step 1432 with content delivery initiated for the current set of selected streams. At step 1432, content is received from streams in the selected set of content streams. This may include receiving content corresponding to the highest priority stream, and in various embodiments includes receiving content, for example, where the stream is content corresponding to a majority of the field of view, and For example, the portions provide content corresponding to one or more different parts of the environment, providing a small portion of image content corresponding to the field of view. Streams that provide a small portion of the content for the current field of view may be described as secondary streams. In one embodiment, where a single stream provides content for the entire field of view, less than 20% of the available bandwidth/supportable received data rate is reserved and out-of-field content is stored, for example, when the user turns away from the field-of-view. It is used to receive one or more secondary or lower priority streams, providing. If the field of view is approximately evenly separated between areas corresponding to two different content streams, each stream is distributed in an approximately equal manner and the user does not see areas of the environment outside the area for which the two streams provide images. Since there is no possibility of changing position quickly, approximately half of the available/received data rate can be allocated.

With initiated content reception, at step 1432, content, e.g., images, is received from a selected set of streams. In the case of primary, eg, highest priority content, the stream will usually be stereoscopic video content with both left and right eye image content received in the stream. For low priority streams and/or low data rate assigned streams mono images may be received with a single image being received for display in both left and right eye images. The front scene portion is usually received as stereoscopic video content, but one or more rear portions may be provided as mono image streams.

Encoded image content is usually received as streams at step 1432. In step 1434, the received content is decoded and then, upon reaching via connection node A 1436, in step 1438, the decoded content is stored in one or more image buffers. In some embodiments, a decoded image buffer is maintained for each portion of the environment. The entire received frame is usually decoded and buffered, although only one portion of the received image may be ultimately displayed. The decoded buffered image may be maintained in memory until it is replaced by a more recent image for the same portion of the scene, and in some embodiments remains so. Accordingly, at any given time, a decoded image is available for each portion of the environment to use to render the final output image required based on the current field of view. Because the decoded image content is maintained in memory until it is replaced with a more recent decoded image, decoding of the images corresponding to each of the parts of the 360-degree environment need not occur during each frame time. Therefore, a frame rate of 30 frames per second may be supported, but 150 frames need to be decoded in each frame period, e.g., for each of the top, bottom, front, left rear, right rear portions. of images from previously decoded still image portions or previously decoded image portions that are updated at a lower rate than the corresponding portion of the main field of view, but rather a smaller number of frames equal to or slightly higher than the frame rate to be supported. It can be decoded with several parts.

Accordingly, after updating at step 1438, the currently decoded frames are available for use in rendering an image based on the user's current field of view.

Operation proceeds from step 1438 to step 1440. At step 1440, the content is converted into a 2D to model of the 3D surface and an environment map defining the surface to which one or more image portions are applied, e.g., as a texture, using the decoded content available from the image buffer(s). The decoded images are rendered to display UV map(s) that provide information on how to apply them. The 3D mesh model of the environment may be in the form of a grid with points on the UV map corresponding to vertices of the mesh model defining the surface on which images of the environment are applied.

As part of rendering, portions of the image corresponding to content received from different streams create an image of the environment corresponding to the user's field of view when content from a single stream cannot provide a portion that completely occupies the user's field of view. will be combined to do this. Filtering or blurring may, and in some embodiments is applied, across the images being concatenated to form a composite image corresponding to the user's field of view. This tends to reduce the seam significantly for the user. Additionally, the luminance of the image portions that provide a portion of the composite image may, in some embodiments, be determined by the luminance of the rear, top, or side portions when making luminance adjustments to the image portions that are combined to form the composite image. adjusted to reduce luminance differences between image parts that are combined to form a composite image with the luminance values of the front view starting to be given greater priority than the values.

The rendered image, which may be a composite of image portions of one or more of the images received from different content streams, is stored, displayed, or transmitted at step 1442. This represents the generation and display of one or more frames of content based on the current set of selected content streams.

Due to changes in communication channel conditions or network issues over time, the maximum supportable data rate may vary. This can be detected at step 1443 and taken into account when selecting which content stream or streams should be received and processed. At step 1443, the maximum supportable data rate and/or bandwidth is determined as indicated by data 1444 with respect to that used in subsequent steps. At step 1445, the user's current head position is detected. This can be done by using a position sensor on headgear that includes a head mounted display.

Operation proceeds from step 1445 to step 1446, where a determination is made as to whether the user's head position has changed. If the user's head position has not changed, operation proceeds to step 1447 and determines whether the currently available maximum bandwidth or maximum supportable data rate has changed, e.g., since the last time a stream selection was made. An inspection is performed. If no change in maximum supportable data rate or head position is detected, the previous stream selection remains in effect and no changes are made to the set of selected content streams. Accordingly, the playback system will continue to receive content corresponding to the user's current field of view, which remains unchanged. If no change is detected at step 1447, operation proceeds back to step 1429 via connection node B 1456.

However, if a change is detected in either step 1446 or 1447, operation proceeds to the stream selection step 1448, which includes a call to the stream selection subroutine. In this way, detected changes in head position and/or supportable data rates can be taken into account and selection of streams can be used to receive the user's head position and/or data, e.g. image content. Reconsidered taking into account possible data rates.

Once the streams have been selected by the stream selection subroutine, operation proceeds to step 1450 and a check is made as to whether the selected streams are different from the current set of selected streams. If the newly selected set is the same as the currently selected set of streams in use, no changes need to be made regarding the streams being received, and the currently selected set of streams at step 1452 is transferred to step 1429 via connection NodeB 1456. ) is left unchanged by the proceeding operation. However, if the currently selected set of streams is different from the current set of selected streams, the currently selected set of streams is updated at step 1454 to reflect changes in the selected set of streams, e.g. It is the same set as the newly selected set of streams selected by the selection subroutine.

Operation proceeds from step 1454 to step 1455, where reception of streams that are not present in the updated currently selected set of streams ends. This is a multicast group corresponding to a stream that is no longer being received or taking any other action to receive a stream that is terminated for another purpose, e.g. to use resources such as a newly selected stream or a tuner used for the reception of streams. It may include a replay system that signals that it no longer wishes to be a member of .

Operation proceeds from step 1455 to step 1429 via connection note B 1456. Then, at step 1430, reception of any newly selected streams will begin and the received content will be used to render one or more images. Accordingly, the selected streams may also change over time as the user changes his head position and/or changes the data rate that can be supported.

FIG. 15 is a flow chart illustrating the steps of a stream selection subroutine according to one example embodiment that may be used as the stream selection subroutine called in step 1448 of the FIG. 14 method.

The stream selection subroutine 1500 begins at step 1502 when the subroutine is called by step 1448 in FIG. 14. Operation proceeds from the start step 1502 to the stream selection step 1504. Inputs to the stream selection step 1504 include bandwidth and/or data rate limitations 1503. These may contain information about the total received data rate or the maximum portion of available bandwidth that should be allocated to one or more scene parts, streams and/or stream directions. Limitations may also include minimum bandwidth or data allocations for scene portions, streams and/or viewing directions. Accordingly, restrictions may limit the maximum amount of resources allocated to the reception of content corresponding to a particular direction and/or when action occurs in a portion of the environment, such as a rear area, during an event intermission, the portion corresponding to the direction may at least rarely Alternatively, you can specify a minimum amount of resources allocated for a direction to be updated at specific times.

The maximum supportable data rate and/or maximum available bandwidth 1444 are other inputs to the selection step 1504. This information 1444 indicates the maximum bandwidth available for receiving content and/or the maximum supportable data rate that can be used to support reception of one or a combination of content streams. The user's current head position 1407 and information about available streams 1405, for example stream guide information as shown in Figure 18, are also inputs to step 1504.

In a stream selection step 1504, one or more content streams are selected that correspond to a program or event, for example, in some cases an ongoing real-time event. Selection of one or more streams is based on the user's current head position, stream information such as stream bandwidth or stream data rate requirements, and/or information regarding the maximum supportable data rate or maximum supportable bandwidth. Bandwidth and/or data rate limitations 1503, which may be on a viewing direction or stream priority basis, may also be, and sometimes are, considered and used in making stream selections in step 1504.

An example stream selection step 1504 includes step 1506, where content streams are prioritized based on the user's current and/or past head position. This may include a call to a stream prioritization subroutine such as the one shown in Figure 16.

Streams corresponding to the same environmental direction are assigned the same priority in some embodiments. Accordingly, multiple streams corresponding to content corresponding to the same part of the environment and/or viewing direction can, and sometimes are, assigned the same priority. The streams correspond to the same viewing direction, but in some cases the streams provide mono image content and/or low resolution images and/or support low frame (image) rates in some cases high resolution stereoscopic images at high frame rates. Some streams provide lower resolution streams with lower resolution content and have different data rates. Accordingly, although a particular direction is considered a high priority and all streams providing content corresponding to the high priority direction are equal, in some embodiments later the amount of bandwidth available for receiving content corresponding to a particular direction may vary. A choice is made about

After prioritizing content stream operations, operation proceeds from step 1506 to step 1508. At step 1508, the maximum bandwidth and/or data rate to be used for the highest priority stream is determined. This decision may be performed based on bandwidth or other constraints 1503, which may indicate the maximum, minimum, or portion of available receive resources to be allocated for the highest priority stream. In some embodiments, the minimum bandwidth/data rate allocation for the highest priority stream is 50% or greater, but other allocations are possible.

At step 1510, the maximum bandwidth and/or data rate to be used for each stream with lower priority is determined. In some embodiments, at least 20% of the data rate or bandwidth is used for secondary or lower priority streams.

The data rates to be used for the streams of different priorities are determined in steps 1508, 1510, and operation proceeds to step 1512, where the highest priority stream has the maximum bandwidth for the highest priority stream and/or Alternatively, a check is performed to determine if it can be supported based on the data rate. If either of the streams corresponding to the highest priority can be supported, the decision in step 1512 will be affirmative, and operation proceeds to step 1514, where the stream corresponding to the highest priority can be supported. High quality streams will be selected. This usually involves selecting the highest data rate stream from the set of streams assigned the highest priority. For example, if the forward direction streams are assigned the highest priority, the highest data rate forward direction content stream that can be supported for a given data rate available for the highest priority content stream will be selected.

Operation proceeds from step 1514 to step 1516. At step 1516, a determination is made as to whether the second highest priority stream can be supported. This means, in some embodiments, how much bandwidth/data is available after the highest priority stream is selected and how much of that bandwidth/data will be used for the second highest priority stream based on the received restrictions. This includes determining whether it is possible. If restrictions are not placed on the second highest priority stream, the full amount of bandwidth/data receiving capacity can be used for the second highest priority stream. If it is determined at step 1516 whether the second highest priority stream can be supported, then operation proceeds to step 1518 and the second highest priority stream is selected, e.g. is selected from a set of one or more assigned streams. Step 1518 may include, and in some embodiments includes, selecting the highest data rate stream that can be supported with the second highest priority. For example, if the second highest priority corresponds to the right rear portion of the environment, step 1518 includes selecting the highest data rate stream that can be supported that corresponds to the right rear portion of the environment.

In most cases, the highest priority and secondary streams will be supported, but it remains after selection of the two highest priority streams to receive some content that corresponds to other parts of the environment, e.g. unseen. There may be sufficient bandwidth available. Operation proceeds from step 1518 to step 1520, or, if the second highest priority stream cannot be supported, from step 1516 directly to step 1520.

At step 1520, whether a third highest priority stream can be supported using available bandwidth/data reception resources after the first and/or second priority streams have been selected, for example. An inspection is performed. If at step 1520 it is determined that the third highest priority stream can be supported, then operation proceeds to step 1522, using the remaining bandwidth/data rates that can be supported, for example, The third highest priority stream is selected. Operation proceeds from step 1522 to step 1524, or directly if the third highest priority stream cannot be supported given the available bandwidth and/or received data and/or bandwidth allocation constraints, step ( Proceeds from step 1520 to step 1524.

At step 1524, after other stream selections, a check is performed to determine whether there is any bandwidth, e.g., capacity to receive remaining data, that can be used to receive additional content. If additional bandwidth remains, operation proceeds to step 1526 and one or more lower priority streams are selected to use the remaining available bandwidth/data rate. From step 1526, operation proceeds to step 1530. Operation proceeds from step 1524 to return step 1530 if no additional bandwidth is available.

Return step 1530 causes processing to return to the point where stream selection subroutine 1500 is called, for example, with the newly selected set of streams determined by routine 1500.

Figure 16 is a flow chart illustrating steps of a stream prioritization subroutine according to an example embodiment. 1600 in the figure illustrates an example stream prioritization routine that may be used, for example, may be called one or more of the other routines or subroutines described herein. For example, step 1506 of Figure 15 may be implemented by a call to subroutine 1600. Stream prioritization subroutine 1600 begins at step 1602 when the routine is called. Operation proceeds from start step 1602 to step 1604. At step 1604, the user's current field of view is determined based on the user's detected head position. Given that the user's field of view is less than 180 degrees, the current field of view may correspond to a portion of the environment captured from a single camera position, for example a forward-looking camera position. However, when the user moves his head, for example, turns his head left or right and/or looks up or down, the user's field of view is captured by cameras positioned at different camera positions. Can respond to parts of the environment. For example, when viewing a portion of the environment that corresponds to a point where images captured by different cameras meet or overlap, the user's field of view may correspond to content to be delivered in two different streams. The stream providing images corresponding to the largest portion of the field of view will usually be given the highest priority in some embodiments.

With the user's field of view determined in step 1604, operation proceeds to step 1605 and streams carrying content, e.g., a mono image comprising left and right eye images corresponding to the user's current field of view. Groups or pairs of stereoscopic images are identified. Operation then proceeds to step 1606, where the size of the portion or portions, eg, image portions of the scene area, correspond to the user's current field of view available from the identified streams. Accordingly, at the end of step 1606, information about which streams provide image content corresponding to the current field of view and the relative sizes of the portions is available and can be used for ranking, e.g., prioritizing the streams. there is.

Operation proceeds from steps 1606 to step 1608. At step 1608, priorities are assigned to one or more streams providing image content corresponding to the user's current field of view. Prioritization, eg, grading, is based on the size of the portion or portions of the user's field of view that the stream provides. For example, a stream that provides image content that covers 80% of the user's field of view will be ranked higher than a stream that provides image content that covers 15% of the user's field of view, and the remaining 5%, e.g. , the stream providing the upper or lower portion will be assigned a third priority lower than the highest priority assigned to the stream providing the image(s) corresponding to the 80% portion of the field of view.

Step 1608 may include, and in some embodiments includes, step 1610 and/or step 1612. At step 1610, the highest priority is assigned to the stream providing the largest portion of the field of view. Step 1610 may include designating the stream providing the largest portion of the field of view as the primary stream. Step 1612 includes assigning the next highest priority or priorities to the streams providing portions of the field of view that are outside the portion provided by the highest priority stream. These streams are prioritized in step 1612 based on the size of the portion they can present in the current field of view, with streams capable of providing smaller portions being assigned lower priorities.

For example, assuming that multiple streams with different data rates can provide the same portion of the field of view, multiple streams can be assigned the same priority, although potentially at different resolutions or frame rates. and, for example, streams corresponding to the front view may be assigned the same priority, and streams providing the left rear view may have the same priority, for example, different priorities assigned to the front streams. may be assigned, and the streams providing the right rear view may be given the same priority, for example a different priority than that assigned to the front or left rear view streams.

Accordingly, in step 1608, the streams providing for view will be ranked, i.e., prioritized. Prioritization can be expressed by listing the streams in a ranked list, with the primary stream(s) assigned the highest priority and other streams assigned lower priorities.

Not all streams can correspond to the field of view. For example, the top or two portions of the scene may be out of view and therefore the streams providing these views may not have been prioritized in step 1608. Operation proceeds from step 1608 to step 1614. At step 1614, a determination is made as to whether there are any remaining streams to be prioritized. If there are no remaining streams to be prioritized, for example because they all correspond to the user's current field of view, the operation proceeds to the return step 1630. However, if at step 1614 it is determined that one or more streams remain to be prioritized, then operation proceeds to step 1616.

At step 1616, priorities are assigned to one or more additional streams carrying content, e.g., streams outside the current field of view. The prioritization performed at step 161 may, in some embodiments, be based on the proximity of the content presented by the prioritized stream to the content visible in the current field of view and/or based on the user's current or past head rotations. Based on direction. For example, if a stream provides image content corresponding to a portion of the environment immediately adjacent to the user's current field of view, in some embodiments it may have a higher image content than a stream providing content corresponding to a portion of the image distant from the user's current field of view. Priorities will be assigned. Similarly, a stream that provides image content in the direction of the user's head rotation can be viewed by the user under the assumption that content in the direction of head rotation is more likely to come into the user's field of view quickly than content in the opposite direction away from the detected direction of head rotation. may be given higher priority than content in directions of rotation away from the detected head rotation.

In at least one embodiment, step 1616 includes step 1618 and determines whether the change in head position indicates a head rotation, e.g., left or right rotation as opposed to tilting the head up or down. Testing is performed to: If head position is not detected at step 1618, operation proceeds to step 1620 and the streams are, in some embodiments, prioritized based on the portion of image data they provide relative to the user's current field of view. Streams serving the upper and/or lower portions may by default be assigned a lower priority than the streams serving the left or right rear portions as well as the left or right rear portions when the upper and lower portions are out of view. Operation proceeds from step 1620 to return step 1630.

If it is assumed that user head rotation is detected at step 1618, then operation proceeds to step 1622. At step 1622, the direction of head rotation is determined, for example, whether the user's head is turned left or right. This allows the direction of head rotation to be taken into account, since it is often more likely that the next part of the environment that will come into the user's view will be in the direction of the head rotation rather than away from it.

Operation proceeds from step 1622 to step 1624, where priorities are assigned to one or more content streams, e.g., out-of-view streams based on the direction of head rotation. In at least one embodiment, step 1624 includes step 1626, where the next lowest unused priority is assigned to the stream providing content corresponding to the portion of the environment in the direction of head rotation. For example, if a user looking at the front part of the environment turns his head to the right, a stream providing right rear content that is outside of the field of view will be assigned a higher priority than a stream providing left rear content that is also outside of the field of view. will be. Operation proceeds from step 1624 to step 1628, where the remaining streams that have not been previously prioritized are assigned lower priorities indicating that they are of lower importance than the streams to which priorities have already been assigned.

Operation proceeds from step 1628 to step 1630. By the time the return step 1630 is reached, the content streams will have been prioritized, eg ranked or ordered, according to priority.

Figure 17 is a flow chart illustrating steps 1700 of a rendering subroutine according to one example embodiment.

Rendering subroutine 1700 may be called by one or more routines when image rendering is required. In the case of stereoscopic content, separate images are rendered for each of the user's left and right eyes. In the case of mono content, a single image is rendered and used for each of the user's left and right eyes. Rendering often involves the combination of image content from one or multiple streams. Accordingly, some portions of the environment may be presented as mono content, while other portions may be presented as stereoscopic content, in which case the different left and right eye images are such that some of the content is stereoscopic and other content is mono. , a single image generated for each of the left and right eye images may be rendered when at least a portion of the environment is presented as stereoscopic image content.

Rendering routine 1700 begins at start step 1702 and proceeds to render step 1706. The input to the rendering step 1706 is an environment map 1411, decoded image content 1703 corresponding to one or more views, and one or more decoded images or images on the surface defined by the environment map 1411. Includes one or more UV maps 1704 used to map portions. As discussed above, environment map 1411 may default to a sphere with images mapped into the surface of the sphere if a more complex geometry is not provided.

In some embodiments, rendering step 1706 may be performed on the user's current field of view and one or more content streams corresponding to the environment map as well as at least one UV map to generate at least one image corresponding to the user's current field of view. Step 1708 includes using content generated by decoding the included images. For stereoscopic, eg, 3D image content, rendering will result in left and right eye images being generated in a format suitable for display. Differences in the rendered left and right eye images, in some embodiments, allow the user to perceive the images as 3D.

Operation proceeds from step 1706 to step 1710, a return step that causes the stored and/or output rendered images to be returned to a program or routine to be supplied to the display device.

Rendering subroutine 1700 may be called whenever an updated version or frame of view is to be displayed. Therefore, rendering usually occurs at a rate that matches the image, e.g. frame rate, display rate.

FIG. 18 shows an example table 1800 containing stream information corresponding to a plurality of content streams. In some embodiments the stream information included in example table 1800 is received as part of guide information, for example, a program guide that provides information about content streams that a playback system can select to receive. Legend 1840 includes information indicating the meaning of various letters used as abbreviations in the information included in table 1800.

Information included in table 1800 can be used to access content streams to which the information corresponds. As discussed in some embodiments, the stream information includes, for a plurality of available content streams, a multicast address of a multicast group that can be connected to receive a provided corresponding content stream, providing a provided content stream. either information that may be used to request access to the switched digital video channel used for the broadcast, or channel tuning information that may be used to control the tuner of the playback system to tune to the broadcast channel on which the provided content stream is broadcast. Includes.

In table 1800, each row corresponds to a respective content stream carrying content, and the content stream to which the row corresponds is identified by the stream identifier shown in the corresponding input of column 1812. Each entry in column 1804 identifies program content carried by the respective content stream to which the row corresponds. As can be appreciated from table 1800, first group of rows 1820 corresponds to program content “football,” which may represent the program/event title shown in the corresponding inputs in column 1804. There are a number of these groups corresponding to a variety of different programs/events. Each group contains content streams each corresponding to a viewing direction and supporting the provided data rate discussed. For simplicity, only two groups are shown in the figure, and the rows corresponding to the second group 1822 are only partially shown to illustrate the concept. The second group of rows 1822 corresponds to the program content “HI” indicated by the corresponding inputs in column 1804.

Each input in column 1806 represents a portion of the scene area, e.g., a region of the 360-degree scene area 1200 conveyed by the corresponding content stream. Accordingly, the first three rows of group 1820, each corresponding to a different content stream, represent a frontal scene portion (e.g., region 1 shown in FIG. 12 comprising a 270° to 90° viewing area) ) is mapped to . The next three rows of group 1820, each corresponding to a different content stream, map to the back right scene portion, e.g., region 1 shown in Figure 12, comprising the 30° to 210° viewing area. conveys). The last three rows of group 1820, each corresponding to a different content stream, map to a rear left scene portion (e.g., region 3 shown in FIG. 12 comprising a 150° to 330° viewing area) conveys).

Each input in column 1808 indicates the data rate supported by the corresponding content stream. Each input of column 1810 represents a multicast group identifier/address of a multicast group that can be connected to receive the corresponding content stream identified by the stream identifier shown in the corresponding input of column 1812. Each input in column 1814 includes a stream descriptor for the corresponding content stream identified by the stream identifier shown in the corresponding input in column 1812. Each input of column 1816 includes access information, such as tuning parameters and/or other access parameters that can be used to access or request access to the corresponding content stream.

As can be understood from the example table 1800, in the example shown, there are multiple, e.g., three, different versions of each content stream corresponding to a given viewing direction available for use during playback. There is, and each version of the content stream supports different data rates. Accordingly, the playback system may, in accordance with the features of the invention discussed in detail with respect to Figures 14-17, perform playback based on one or more factors, such as, for example, supported bandwidth, data rate, user head position, etc. You can select one or more streams to use.

To more clearly understand how the information in table 1800 may be used by a playback system to select and/or access one or more content streams, the first row and columns 1804, 1806 of group 1820 are presented. Let us consider each of the first inputs (1808, 1810, 1812, 1814, 1816). The first input in column 1804 indicates that the event/program “Football” carried by the first content stream identified by stream identifier S ₁ D ₁ is included in column 1812 . The corresponding input in column 1806 indicates that the first stream carries content corresponding to a forward scene portion (e.g., 270° to 90° viewing area). This viewing area information is used by the playback system to identify one or more streams carrying content corresponding to the user/viewer's current head position corresponding to the current field of view. Continuing with the above example, the corresponding first input of column 1808 indicates that the first content stream supports and/or requires data rate D1. The corresponding input in column 1810 indicates that the first content stream can be accessed by concatenating a multicast group M1, with M1 indicating a multicast group address and/or identifier mapping to an address. The corresponding inputs of column 1814 include the camera viewing angle (V ₁ ) to which the first stream corresponds, the codec type (C ₁ ), the supported data rate (D ₁ ), and the frame rate (F) to which the first stream corresponds. ₁ ) and a stream descriptor “V ₁ C ₁ D ₁ F ₁ ” corresponding to the first content stream. The corresponding input of the last column 1816 represents access tuning parameters and/or other access parameters (shown as A123) that can be used to access or request the first content stream.

With this information discussed above regarding the available content streams that can be used, a playback system, such as playback system 1900, can select and access one or more content streams for use in playback in accordance with the features of the present invention. You can. For a better understanding, let's consider one simple example where the playback system determines that the user's head position indicates that the user is looking at the frontal portion of a 360-degree scene. In this case, in one embodiment the playback system selects at least one content stream that carries a portion of the frontal scene. Depending on various other factors discussed with respect to Figures 14-17, such as available bandwidth, supportable data rate, stream bandwidth, and/or data rate limitations, the playback system may use three different methods to convey the frontal scene portion. One stream can be selected among the available streams (S ₁ D ₁ , S ₁ D ₂ , S ₁ D ₃ ). If constraints allow, the playback system will select from the plurality of content streams corresponding to the frontal scene portion the highest quality stream of the three, eg stream S ₁ D ₁ . The information provided in table 1800 facilitates selection of an appropriate stream for use in playback, since at least some of the information that can be used to select is provided by stream information 1800. After selecting the stream, the playback system initiates content delivery, e.g., content reception, by connecting a multicast group corresponding to the selected stream, e.g., M1, or by using the access information to obtain the content stream. The stream information 1800 can be used again for this purpose.

19 illustrates a playback system 1900 implemented in accordance with the present invention that can be used to receive, decode, store, and display imaging content received from a content delivery system. System 1900 may be implemented as a single playback device 1900' that includes a display 1902 or as a combination of elements such as an external display, e.g., a head mounted display 1905 coupled to computer system 1900'. It can be.

Playback system 1900, in at least some embodiments, includes a 3D head mounted display. The head mounted display may be implemented using an OCULUS RIFT ^™ VR (virtual reality) headset, which may include a head mounted display 1905. Other head mounted displays may also be used. In some embodiments, a head mounted helmet or other head mounting device in which one or more display screens are used to display content to the user's left and right eyes is used as device 1905. By displaying different images to the left and right eyes on a single screen with a head mount configured to expose different portions of the single screen to the different eyes, a single display can cause left and right eye images to be separately perceived by the viewer's left and right eyes. It can be used to display things. In some embodiments, a mobile phone screen is used as the display of a head mounted display device. In at least some of these embodiments, a mobile phone is inserted into a head mounting device and the mobile phone is used to display images. In some embodiments, display device 1905 may be part of a 3D display device, such as an Oculus Rift.

The playback system 1900 decodes the received encoded image data, e.g., left and right eye images and/or mono (single images) corresponding to different parts of the environment or scene, e.g. It has the ability to generate 3D image content for display to a customer by rendering and displaying different left and right eye views that are perceived by the user as a 3D image. Playback system 1900 is, in some embodiments, located at a customer premises location, such as a home or office, but may also be located at an image capture site. System 1900 may receive, decode, display, and/or perform other operations in accordance with the present invention.

System 1900 includes a display 1902, a display device interface 1903, an input device 1904, an input/output (I/O) interface 1906, a processor 1908, a network interface 1910, and memory 1912. ) includes. The various components of system 1900 are coupled together via a bus 1909 and/or other connections that allow data to be transferred between components of system 1900 or via a wireless interface. In some embodiments, display 1902 is included as an optional element, shown using a dash box, but in some embodiments, an external display device 1905, e.g., a head mounted stereoscopic display device, may be used as a display device. It can be coupled to a playback device via device interface 1903.

For example, if a mobile phone processor is used as the processor and the mobile phone generates and displays images on a head mount, the system includes processor 1908, display 1906 and memory 1912 as part of the head mounted device. can do. Processor 1908, display 1902, and memory 1912 may all be part of the mobile phone. In other embodiments of system 1900, processor 1908 may be part of a gaming system, such as an XBOX or PS4, and display 1905 is mounted on a head mounting device and coupled to the gaming system. The processor 1908 or memory 1912 is located in a device where it does not matter whether it is worn on the head or not, and as will be appreciated, in some cases, the processor 1908 within the headgear from a power, heat, and weight perspective. Although it may be convenient for the processor 1908 to be co-located, it may be desirable in at least some cases to have the memory and processor 1908 coupled to a headgear that includes a display.

Although the various embodiments contemplate a head mounted display 1905 or 1902, the methods and apparatus may also be used with non-head mounted displays that can support 3D images. Accordingly, in many embodiments, system 1900 includes a head mounted display, but it may also be implemented with a non-head mounted display.

An operator of playback system 1900 may control one or more parameters via input device 1904 and/or select operations to be performed, for example, selecting to display a 3D scene. Via I/O interface 1906, system 1900 can be coupled to external devices and/or exchange signals and/or information with other devices. In some embodiments, via I/O interface 1906, system 1900 may receive images captured by various cameras that may be part of a camera rig, such as camera rig 900.

Processor 1908, e.g., CPU, executes routines 1914 and uses various modules to control playback system 1900 to operate in accordance with the present invention. Processor 1908 is responsible for controlling the overall general operation of playback system 1900. In some various embodiments, processor 1908 is configured to perform the functions discussed as being performed by the playback device.

Via network interface 1610, system 1900 can transmit signals and/or information (including, e.g., images and/or video content) to various external devices, e.g., communications network 105 and Transmitted and/or received over the same telecommunication network. The network interface 1910 includes a receiver 1911 and a transmitter 1913 on which receiving and/or transmitting operations are performed. In some embodiments, the system receives one or more selected content streams from a content provider via network interface 1910. In some embodiments, system 1900 receives one or more selected content streams for use for playback via receiver 1911 of interface 1910. The received content stream may be received as encoded data, for example encoded scene portions 1952. Receiver 1911 is also configured to receive stream information 1946 and/or initialization data, for example, as part of a program guide. System 1900 also receives, for example, via receiver 1911, data rate allocation control information 1952 including bandwidth and/or bandwidth limits for different viewing directions, where individual bandwidth limits are individually Specifies the maximum bandwidth to be used to receive one or more content streams providing content corresponding to the viewing direction corresponding to the bandwidth limitation. In some embodiments, receiver 1911 may also include at least one environment map, e.g., a 3D depth map defining a 3D surface, and a and configured to receive one or more UV maps to be used for mapping image content. In some embodiments, receiver 1911 is configured to display a first UV map corresponding to a first portion of the scene environment, a second UV map corresponding to a second portion of the scene environment, a third UV map corresponding to a third portion, A fourth UV map corresponding to the fourth part, and a fifth UV map corresponding to the fifth part of the scene environment are received. In some embodiments, during initialization, system 1900 receives content, e.g., first, second, third, fourth, and fourth elements of a scene, e.g., via a receiver of interface 1910. Receive an image corresponding to one or more of the 5 parts.

Memory 1912 includes various modules, such as routines that, when executed by processor 1908, control playback system 1900 to perform decoding and output operations in accordance with the present invention. Memory 1912 includes control routines 1914, head position determination module 1916, current viewing position initialization module 1918, decoder module 1920, currently selected stream initialization module 1922, and content delivery initiation module 1924. ), image buffer(s) 1926, image buffer update module 1928, image rendering module 1930, also called image generation module, available bandwidth and/or supported data rate determination module 1932, head position change. Determination module 1934, Change available bandwidth and/or supported data rate Determination module 1936, Stream selection module 1938, Change selected stream set determination module 1940, Update selected stream set module 1942, Stream (s) Termination module 1944, data/information comprising received stream information 1946, received bandwidth and/or data rate allocation information 1948, determined currently maximum available bandwidth and/or supportable data rate; (1950), received encoded image content (1952), received environment map (1954), received UV map(s) (1956), decoded image content (1958), and generated 3D content (1960). Includes.

Control routines 1914 include device control routines and communication routines for controlling the operation of system 1900. Head position determination module 1916 is configured to determine the user's current head position, e.g., the position of a head mounted display. Head positioning module 1916 may integrate with and/or operate in conjunction with a position sensor, which may be present, for example, on headgear that includes a head mounted display. The current viewing position initialization module 1918 sets the detected current head position of the user to be in a forward (0 degree) viewing position, such that during the initialization phase, the user's current viewing position is set to a forward (0 degree) environment viewing position. It is configured to be initialized to be in position.

Decoder module 1920 is configured to decode encoded image content 1952 received from content delivery system 700 to generate decoded image data 1958. Decoded image data 1958 may include a decoded stereoscopic image scene and/or decoded scene portions. Decoded content is stored in one or more image buffer(s) 1926 in some embodiments. The currently selected stream initialization module 1922 is configured to initialize the current set of one or more selected content streams to be received. The currently selected stream initialization module 1922 is configured to set the currently selected set of streams to be the first stream carrying content corresponding to the front/front portion of the environment/scene.

The content delivery initiation module 1924 is configured to initiate delivery of the selected content stream(s). In some embodiments, content delivery initiation module 1924 initiates delivery of content streams in the selected set that have not already been received. In some embodiments, the content delivery initiation module 1924 may configure a multicast group corresponding to a selected content stream, e.g., a multicast group corresponding to content streams delivering content corresponding to the currently selected set of streams ( ) is configured to transmit a request signal to connect. In some other embodiments, the content delivery initiation module 1924 is configured to generate and transmit a request to a device in the network requesting delivery of the switched digital channel over which the selected content stream is delivered.

Image buffer update module 1928 is configured to update image buffer(s) 1926 with updated content, for example, when updated content delivered by a selected set of content streams is received and decoded.

Image rendering module 1930 uses decoded image content 1958 to display a 3D image, e.g., on display 1902 and/or display device 1905, to a user, incorporating features of the invention. Create, for example, left and right eye images, which are displayed in a way that is recognized as a 3D image. In some embodiments, image rendering module 1930 is configured to render content for display using decoded image content 1958, environment map 1954, and UV map corresponding to the user's current viewing area. . Accordingly, in some embodiments, image rendering module 1930 is configured to perform the functions discussed with respect to the steps shown in FIG. 17. The generated image content 1960 is the output of the 3D image generation module 1930. Accordingly, the rendering module 1930 renders the 3D image content 1960 on the display. In some embodiments, image rendering module 1930 is configured to output one or more generated images, for example, to a display device or to another device. The generated images may be output through the network interface 1910 and/or the display device interface 1903.

The available bandwidth and/or supported data rate determination module 1932 may, for example, determine the currently maximum available bandwidth and/or currently maximum supportable data rate that may be available at a given time for receiving content streams. It is configured to do so. Because the available bandwidth and/or supportable data rate may vary over time due to changes in communication channel conditions or network issues, in some embodiments, decision module 1932 determines the available bandwidth and/or supported data rate. Continuously monitors and/or makes decisions to detect changes in data rate. The determined current maximum supportable data rate and/or bandwidth 1950 is the output of decision module 1932 and may be updated as needed.

Head position change determination module 1934 is configured to determine whether there has been a change in the user's head position, for example, by examining and comparing changes in the output of head position determination module 1916. Change available bandwidth and/or supported data rate determination module 1936 determines the available bandwidth and/or supported data rate by comparing the available bandwidth and/or supported data rate to the current maximum available bandwidth and/or current maximum supportable data rate as determined by decision module 1932. and configured to detect whether there is any change in data rate.

The stream selection module 1938 is configured to select, from among a plurality of content streams, a stream to receive for use in playback at a given time based on the user's current head position. Based on changes in the user's current head position and/or other factors, stream selection module 1938 may select different stream(s) at different times. The currently selected set of streams 1961 is the output of the stream selection module 1938 and represents the set of content streams currently selected for reception. In some embodiments, stream selection module 1938 includes a plurality of sub-modules configured to perform various functions as part of a stream selection operation. Figure 23 shows the stream selection module 1938 and the various modules included therein in greater detail, which will be discussed later.

The selected stream set change determination module 1940 determines the current set of streams 1961, for example, by the selection module selecting one or more additional content streams and/or by one or more streams received being terminated/paused. It is configured to determine whether a change existed in the selected set. The selected stream set update module 1942 updates any selected set of streams when there are changes to the selected set of streams, for example, by adding or terminating content streams to the selected set of streams 1961. and update the currently selected stream set 1961 to reflect changes. The stream(s) termination module 1944 may have previously been received, but is updated, for example, by the currently selected set of streams 1961 being updated due to changes in the selection of streams. configured to terminate/stop receiving one or more content streams that no longer exist.

Stream information 1946 includes information regarding a plurality of content streams that may be available for reception and use during playback. The information included in stream information 1946 is the same or similar to that shown in Figure 18 and previously discussed. The received bandwidth and/or data rate allocation control information 1948 may include bandwidth limitations for different viewing directions and/or limitations on data rates corresponding to content streams providing content for a variety of different viewing directions. Contains information representing them. Determined current maximum supportable data rate and/or bandwidth 1950 represents the maximum supportable data rate and/or bandwidth determined by playback system 1900 at a given time.

The received environment map 1954 includes a 3D depth map of the environment defining a 3D surface. One or more such depth maps corresponding to an environment of interest may be received by playback system 1900 in some embodiments. The received UV map(s) 1956 includes one or more UV maps corresponding to portions of the environment/scene of interest. Decoded data 1958 includes data decoded according to the present invention by decoder 1920. Decoded data 1958 includes content comprising a scene or scene portions of the environment conveyed by the selected set of streams.

In some embodiments, the various modules discussed above are implemented as software modules. In other embodiments, the modules are implemented in hardware, for example as individual circuits, and each module is implemented as a circuit for the module to perform its corresponding function. In still other embodiments, modules are implemented using a combination of software and hardware.

Although shown as included in memory 1912 in the FIG. 19 example, modules shown as included in playback device 1900 may be fully implemented in hardware, for example, as separate circuits within processor 1908, in some embodiments. This is implemented that way in the examples. The modules may be fully implemented in hardware, for example as individual circuits corresponding to different modules, and in some embodiments are so implemented. In other embodiments, some of the modules are implemented as circuits, for example, within the processor 1908 with other modules implemented as circuits external to and coupled thereto. As can be appreciated, the level of integration of modules on the processor and/or with some modules external to the processor may be one of design choices. Alternatively, rather than being implemented as circuits, all or part of the modules may be implemented in software and stored in memory 1912 of system 1900, where the modules may be connected to a processor, e.g., processor 1908. ) controls the operation of the system 1900 to perform functions corresponding to the modules. In still other embodiments, multiple modules are implemented as a combination of hardware and software, such as other circuitry external to the processor, which then operates under software control to perform some of the functions of the modules. ) provides input to .

Figure 23 shows in more detail the stream selection module 1938 and the various modules included therein used in the playback system 1900. The stream selection module is configured to select one or more content streams according to the methods of the invention discussed in detail with respect to Figures 14-16. In some embodiments, the stream selection module is configured to select which of the plurality of content streams to receive based on the user's head position, stream information 1946, and/or maximum supportable data rate. In some embodiments, stream selection module 1938 includes a stream prioritization module 2306 configured to prioritize content streams based on the user's head position. The output of stream prioritization module 2306 is, for example, a prioritized list of content streams with assigned priorities. Stream prioritization module 2306 is discussed in more detail with respect to Figure 24 below.

The stream selection module 1938 is configured to determine the maximum bandwidth and/or data rate to be used for the highest priority stream, e.g., based on bandwidth and/or data rate limitations. A maximum bandwidth and/or data rate determination module 2308, and a lower priority stream(s) maximum bandwidth and/or data rate determination module configured to determine the maximum bandwidth and/or data rate to be used for each stream having a lower priority. Additionally includes module 2310. Decision modules 2308 and 2310, in some embodiments, use bandwidth control information 1948 and the output of stream prioritization module 2306 to make their respective decisions. Accordingly, stream selection module 1938 includes one or more stream bandwidth determination modules configured to determine the bandwidth for at least one content stream based on a bandwidth limitation communicated to the playback system, e.g., from a network device/server. can do.

The stream selection module 1938 determines which highest priority stream will be supported based on the determined maximum bandwidth and/or data rate for the highest priority stream(s) and based on available bandwidth and/or supportable data rate. a module 2312 configured to determine whether the stream can be supported, and a module 2314 configured to select the highest priority, highest data rate stream that can be supported. In some embodiments, selection module 2314 is configured to select one content stream from a plurality of highest priority assigned content streams, wherein each highest priority content stream is in the same viewing direction. Provided as a part configured to select content corresponding to from a plurality of content streams having the same priority. In some embodiments, module 2314 is configured to select from a plurality of content streams with the same priority, for example, the highest priority based on a determined amount of available bandwidth. Accordingly, in some embodiments, when multiple streams with the same priority are available, e.g., some have higher data rate requirements compared to others with lower data rate requirements, the selection module 2314 selects the highest quality stream, e.g., a high data rate stream, if available bandwidth and/or supportable data rate and bandwidth limitations allow for such selection.

The stream selection module 1938 selects the available bandwidth (e.g., fully available or remaining available) and/or based on the maximum bandwidth and/or data rate determined for the second highest priority stream(s). a module 2316 configured to determine whether a second highest priority stream can be supported based on a supportable data rate, a module configured to select a second highest priority highest data rate stream that can be supported ( 2318), based on the determined maximum bandwidth and/or data rate for the third highest priority stream(s) and available bandwidth (e.g., fully available or remaining available) and/or supportable data rate. a module 2320 configured to determine whether a third highest priority stream can be supported based on 2322, and a module 2322 configured to select a third highest priority highest data rate stream that can be supported Includes additional Accordingly, in some embodiments, stream selection module 1938 is configured to select one or more content streams that have been assigned the highest priorities, for example, by prioritization module 2306.

Stream selection module 1938 may be configured to determine whether any remaining or additional available bandwidth exists to receive additional content streams, for example, after one or more higher priority streams have been selected for reception. /Additionally includes a bandwidth availability determination module 2324. In some embodiments, the stream selection module 1938 determines the maximum determined maximum bandwidth and/or data rate for one or more lower priority streams to be supported based on the available bandwidth and/or supportable data rate. It further includes a module 2326 configured to select one or more lower priority streams.

24 illustrates a stream prioritization module 2306, which may be implemented as part of the stream selection module 1938, for example, as a submodule thereof, or as a separate module. Stream prioritization module 2306 is configured to prioritize content streams based on the user's head position. Once the content streams are prioritized, stream selection module 1938 can perform stream selection from the prioritized content streams. In some embodiments, the stream prioritization module 2306 includes a current field of view identification module 2404 configured to identify the user's current field of view, which represents the portion of the scene area the user is viewing based on the user's current head position; and a current field of view stream(s) identification module 2404 configured to identify stream(s) carrying content corresponding to a portion of the scene area corresponding to the user's current field of view. The output of the current field of view stream(s) identification module 2404 is a list of identified streams that in some embodiments can be stored in memory 1912, and the list can be updated as the user's head position and thus field of view changes. . Accordingly, in various embodiments, to prioritize the various available content streams, the user's current field of view is first identified corresponding to head position, and streams carrying content corresponding to the field of view are identified.

In some embodiments, the stream prioritization module 2306 includes a module 2406 configured to determine the size of portions of the scene area corresponding to the user's current field of view available from the identified stream(s), and It further includes a prioritization/assignment module 2408 configured to assign priorities to one or more streams providing portions of the scene area corresponding to the user's current field of view based on the size of the portions provided. In some embodiments, prioritization/assignment module 2408 assigns the highest priority to the stream providing the largest portion of the field of view, e.g., providing the largest portion of the scene corresponding to the current field of view. and a module 2410 configured to designate a stream as a primary stream. In some embodiments, prioritization/assignment module 2408 may, based on the size of the portion of the field of view provided by each of the remaining streams, e.g., determine which of the remaining streams provides a smaller portion of the scene corresponding to the current field of view. Assign the next highest priority(s) to the stream that provides a larger portion of the field of view given a higher priority and assignment than the stream, and assign the remaining streams (e.g., as secondary, tertiary, etc.) It further includes a configured module 2412.

In some embodiments, the stream prioritization module 2306 may be configured to determine whether there are remaining streams to be prioritized, e.g., streams providing content corresponding to a region of the scene that is currently out of view. Additionally includes.

In some embodiments, stream prioritization module 2306 may configure one or more additional streams that provide content outside the user's current field of view based on either the proximity of the image content to the current field of view or the orientation of the head position. It further includes a module 2416 configured for prioritization, e.g., assigning priorities. In some embodiments, module 2416 is configured to prioritize one or more additional stream delivered content corresponding to portions outside the current field of view based on the proximity of the delivered image content to the current field of view, Content streams carrying image content close to the area are assigned higher priority than content streams that are out of and further away from the current field of view.

In some embodiments, module 2416 includes a head rotation determination module 2418 configured to determine whether a rotation of the user's head is detected, for example, as part of a change in the position of the user's head. In some, but not all, embodiments, when a user looks up toward the sky or roof or down toward the ground, there is a change in head position, but this head movement is not considered a head rotation. In some embodiments, module 2416 is configured to prioritize one or more additional content streams based on the direction of the user's head rotation, including a content stream that provides image content that is currently out of view but in the direction of head rotation; is assigned a higher priority than other content streams that provide image content outside the current field of view in a direction away from the direction of head rotation. In some such embodiments, module 2416 assigns the next lower priority (e.g., after higher priorities have been assigned to streams providing content corresponding to the field of view) and stream designation, e.g. , further comprising a module 2420 configured to assign the third order to a stream providing content corresponding to a portion of the scene outside of the current field of view, for example, the top or bottom of the scene environment. In some embodiments, when it is determined that no head rotation is present, the head rotation determination module provides control input to module 2420 to assign priorities to additional streams.

In some embodiments, module 2416 further includes a head rotation direction determination module 2422 configured to determine the direction of the user's head rotation, e.g., left or right with respect to a previous head position. In some embodiments, module 2416 further includes a module 2424 configured to assign priorities to one or more additional streams carrying content corresponding to portions outside of the current field of view, taking into account the direction of head rotation. Includes. In some embodiments, module 2424 determines the next lower priority (e.g., next available priority from the higher priority) and designates, for example, a tertiary stream, of the scene in the direction of head rotation. and a module 2426 configured to assign content corresponding to a portion to a stream that provides it. Accordingly, it should be understood that when head rotation is detected, assignment of priorities to the streams is in some embodiments performed based on the direction of head rotation. In some embodiments, module 2416 further includes an additional module 2428 configured to assign lower priorities to any of the remaining streams considered.

Although shown in the FIG. 19 example as a single processor, such as a computer, it should be understood that processor 1908 could be implemented as one or more processors, such as computers. When implemented in software, modules include code that, when executed by processor 1908, configures processor 1908 to perform the functions corresponding to the module. In embodiments in which the various modules shown in FIGS. 19, 23, and 24 are stored in memory 1912, memory 1912 is configured to store the corresponding modules in at least one computer, e.g., processor 1908. It is a computer program product that includes a computer-readable medium containing codes for performing functions, for example, individual codes for each module.

Completely hardware-based or completely software-based modules may be used. However, it should be understood that any combination of software and hardware, such as circuit implemented modules, may be used to implement the functions. As will be appreciated, the modules shown in FIGS. 19, 23, and 24 perform the functions of the corresponding steps shown and/or described in the method of flowchart 1400 of FIG. 14 and shown in FIGS. 15-17. Control and/or configure the system 1900 or elements therein, such as the processor 1908, to perform the functions of the corresponding steps.

FIG. 20, which includes a combination of FIGS. 20A, 20B, 20C, 20D, and 20E, is a flowchart 2000 of an example method of operating a content playback system in accordance with various example embodiments. A content playback system is, for example, a content playback device or computer system coupled to a display according to various embodiments.

Operation of the example method begins at step 2002, where the content playback system is powered on or initiated. Operation proceeds from step 2002 to step 2004, where the content playback system receives a first image corresponding to a first rear view portion of the environment. Operation proceeds from step 2004 to step 2006, where the content playback system stores the received first image corresponding to the first rear view portion of the environment. Operation proceeds from step 2006 to step 2008, wherein the content playback system displays at least a second image corresponding to the first rear view portion of the environment, including at least a second image corresponding to the first rear view portion of the environment. Receive one or more additional images. Operation proceeds from step 2008 to step 2010, where the content playback system stores the received one or more additional images corresponding to the first rear view portion of the environment. Operation proceeds from step 2010 to step 2012.

At step 2012, the content playback system receives a first image corresponding to a second rear view portion of the environment. Operation proceeds from step 2012 to step 2014, where the content playback system stores the received first image corresponding to the second rear view portion of the environment. Operation proceeds from step 2014 to step 2016, wherein the content playback system displays a second rear view portion of the environment that includes at least one second image corresponding to the second rear view portion of the environment. One or more corresponding additional images are received. Operation proceeds from step 2016 to step 2018, where the content playback system stores the received one or more additional images corresponding to the second rear view portion of the environment. Operation proceeds from step 2018 to step 2020.

At step 2020, the content playback system receives one or more images corresponding to a sky view portion of the environment. Operation proceeds from step 2020 to step 2022, where the content playback system stores the received one or more images corresponding to the sky view portion of the environment. Operation proceeds from step 2022 to step 2024, where the content playback system receives one or more images corresponding to a ground view portion of the environment. Operation proceeds from step 2024 to step 2026, where the content playback system stores the received one or more images corresponding to the ground view portion of the environment. In some embodiments, the sky view and the ground view refer to upward and downward directions with respect to the viewer's nominal headward perspective and are used to apply to indoor environments as well as outdoor environments.

In some embodiments, images may be received corresponding to some, but not necessarily all, of a first rear view portion, a second rear view portion, a sky view portion, and a ground view portion, for example, depending on a particular embodiment. You can.

Operation proceeds from step 2026 via connection node A 2028 to step 2030, to step 2034, and through connection node B 2036 to steps 2028, 2040, 2042, 2044, 2046. , 2048, 2050, 2052). Returning to step 2030, at step 2030, the content playback system determines the viewer's head position, which corresponds to the current field of view. Operation proceeds from step 2030 to step 2032, where the content playback system determines the current field of view for the viewer based on the determined head position. Operation proceeds from step 2032 to step 2030. Steps 2032 and 2030 are performed iteratively, eg, periodically, such that the current view is updated, eg, refreshed. The determined current field of view is available for use in generating output images.

At step 2034, the content playback system receives content corresponding to the first portion, e.g., a first content stream providing a front portion view of the environment.

At step 2038, the content playback system determines the playback time of which of the plurality of previously delivered images corresponding to the first rear view portion of the environment is measured relative to the playback time represented in the first content stream. Receives control information indicating whether it should be displayed for a while. At step 2040, the content playback system receives image selection information indicating which of the plurality of images corresponding to the first rear view portion of the environment should be used during the portion of the event.

At step 2042, the content playback system displays which of the plurality of previously delivered images corresponds to the second rear view portion of the environment for a playback time measured relative to the playback time represented in the first content stream. Receive control information indicating what should be done. At step 2044, the content playback system receives image selection information indicating which of the plurality of images corresponding to the second rear view portion of the environment should be used during the portion of the event.

At step 2046, the content playback device determines which of a plurality of previously delivered images corresponding to the sky view portion of the environment should be displayed for a playback time measured relative to the playback time indicated in the first content stream. Receives the control information indicated. At step 2048, the content playback system receives image selection information indicating which of a plurality of images corresponding to the sky view portion of the environment should be used during the portion of the event.

At step 2050, the content playback system determines which of the plurality of previously delivered images corresponding to the ground view portion of the environment should be displayed for a playback time measured relative to the playback time indicated in the first content stream. Receive control information indicating whether to do so. At step 2052, the content playback device receives image selection information indicating which of a plurality of images corresponding to the ground view portion of the environment should be used during a portion of the event.

The operations are connected from step 2032, step 2034, and steps 2038, 2040, 2042, 2044, 2046, 2048, 2050, 2052, which may be performed simultaneously, via connection node C 2054. Through node D 2056, proceed to step 2058.

In some embodiments, control information from steps 2038, 2040, 2042, 2044, 2046, 2048, 2050, 2052 is used, for example, piecemeal, to the corresponding portion of step 2034 for which the control information is used. It is transmitted slightly before the first stream content. In some other embodiments, the block of control information is received prior to or concurrently with the start of reception of the first content stream.

At step 2058, the content playback system displays a first partial view of the environment, e.g., content received from the first content stream corresponding to the front partial view, a stored received image corresponding to the first rear view portion of the environment, a stored received image corresponding to a second rear view portion of the environment, a stored received image corresponding to a sky view portion of the environment, a stored received image corresponding to a ground view portion of the environment, or a portion of the current field of view for which no image is available. One or more output images corresponding to the current field of view are generated based on at least one of the corresponding synthesized images. Step 2058 includes steps 2060, 2062, 2064, 2066, 2068, 2076, and 2078.

At step 2060, the content playback system determines, based on the current field of view, a set of view portions for which there is available data to be used to generate one or more output images, for example. Some example determined sets include, for example: {}, {front view portion}, {first rear view portion}, {second rear view portion}, {sky view portion}, {ground view portion}, {front view portion} view portion, sky view portion}, {front view portion, ground view portion}, {front view portion, first rear view portion}, {front view portion, second rear view portion}, {front view portion, first rear view portion view portion, sky view portion}, {front view portion, second rear view portion, sky view portion}, {front view portion, first rear view portion, ground view portion}, {front view portion, second rear view portion , ground view portion}, {first rear view portion, sky view portion}, {first rear view portion, ground view portion}, {first rear view portion, second rear view portion}, {first rear view portion , second rear view portion, sky view portion}, {first rear view portion, second rear view portion, ground view portion}, {second rear view portion, sky view portion}, and {second rear view portion, Includes a ground view portion}.

Operation proceeds from step 2060 to step 2062. At step 2062, the content playback system determines whether the following two conditions are met: (i) the determined set of view portions from step 2060 includes only the first view portion, and (ii) There are no parts of the current field of view outside the first view part. If it is determined that the determined set includes only the first view portion and that there are no portions of the current field of view outside the first view portion, then operation proceeds from step 2062 to step 2064; Otherwise, operation proceeds from step 2062 to step 2066.

At step 2064, the content playback system generates one or more output images corresponding to the current field of view based on content received from the first content stream.

At step 2066, the content playback system determines whether there are any portions of the current field of view for which images are not available. If the content playback system determines that there is at least a portion of the current field of view for which an image is not available, then operation proceeds from step 2066 to step 2076; Otherwise, operation proceeds from step 2066 to step 2078.

At step 2068, the content playback system generates one or more output images corresponding to the current field of view based on the determined set of view portions to be used in generating the one or more output images. Step 2068 may include, and sometimes includes, step 2070, wherein the current field of view is determined based on at least some received content included in the first content stream and stored content corresponding to the second portion of the environment. Generate one or more output images corresponding to . In some embodiments, step 2070 includes one or both of steps 2072 and 2074. At step 2072, the content playback system selects an image corresponding to the second partial view of the environment based on the received image selection information. Operation proceeds from step 2072 to step 2074. At step 2074, the content playback system combines content obtained from the first content stream captured at a second viewpoint with a first image corresponding to the first viewpoint, wherein the first and second viewpoints are different. .

In some embodiments, the first image is a first image of a second portion of the environment, and the second portion is one of a first rear view portion of the environment and a second rear view portion of the environment. In some such embodiments, the first time point corresponds to a time preceding the second time point. In some such embodiments, the first time point precedes the time of a real-time event during which images of the first content streams are captured.

Returning to step 2076, step 2076 is performed for each portion for which an image is not available. At step 2076, the content playback system composites an image for the portion of the current field of view for which an image is not available. Operation proceeds from step 2076 to step 2078. At step 2078, the content playback system generates one or more output images corresponding to the current field of view based on the determined set of view portions of the environment to be used in generating one or more output images and/or one or more composited images. . The output images produced by step 2078 are: a fully composited image; an image including content from a composite image and content from a first content stream; an image including content from a composited image, content from a first content stream, and content from a stored image; and an image including content from a synthesized image and content from a stored image. In various embodiments, step 2078 may include, and sometimes includes, one or both of steps 2080 and 2082.

At step 2080, the content playback system simulates one or more output images corresponding to the current field of view and a portion, e.g., a second portion, of the environment based on at least some received content included in the first content stream. Create a composite image that At step 2080, the content playback device combines the composited image with at least a portion of the received image to generate an image corresponding to the current field of view.

It should be understood that current visibility can and usually does change over time. In response to changes in the current field of view, different sets of view portions to be used to generate the output image may be determined at step 2060 and different images may be required to exist, e.g. Corresponding to different parts of the field of view that are not present, the composite is performed at step 2076. Additionally, at different times based on the control information received, different stored images may be identified for use in generating a combined output image at different times.

Operation proceeds from step 2058, via connection node E 2084, to step 2086, where the content playback system outputs and/or displays the one or more output images generated. Step 2086 includes a step 2088 wherein the content playback system outputs and/or displays a first output image, the first output image being one of one or more generated output images.

In some embodiments, the generated output image corresponding to the current field of view, e.g., generated in step 2070, includes information from the first portion of the environment, the second portion of the environment, and the third portion of the environment. May include, and sometimes includes him. In some embodiments, the first portion of the environment corresponding to the first content stream is a front view portion, the second portion of the environment is a first rear view portion of the environment, e.g., a right rear view portion, and The second rear view portion is, for example, one of the left rear view portions, and the third portion of the environment is one of the sky view portion and the ground view portion of the environment. In some such embodiments, the content corresponding to the first portion includes real-time content captured and streamed to the playback system while an event is in progress, and the content corresponding to the second and third portions includes non-real-time images. admit.

In various embodiments, the combination of content to produce an output image corresponding to the current field of view may include images obtained from the first content stream in one or more boundary regions, e.g., corresponding to a frontal view portion of the environment, the environment a stored image corresponding to a first rear view portion of the environment, a stored image corresponding to a second rear view portion of the environment, a stored image corresponding to a sky view portion of the environment, a stored image corresponding to a ground view portion of the environment, and an image. and performing filtering, blurring, luminance fluctuations, and/or color fluctuations in a boundary region between any two of the composited images corresponding to a region in the current field of view that does not exist.

In some embodiments, there are some parts of the environment that are not covered by the first, eg frontal view part corresponding to the first content stream, and one or more additional parts in which the received images are stored. In various embodiments, images are composited for these unincluded portions of the environment. For example, in one embodiment, when a stored image corresponding to a portion of the sky view may not exist and the current field of view includes a portion of the sky view, the image is composited. In another example, a dead spot may exist, for example an unincluded area between the first rear view portion and the second rear view portion. In some embodiments, compositing an image includes repeating a portion of an image, a portion of an image obtained from a received first content stream that corresponds to an adjacent region in the environment, or a portion of a stored image.

In some embodiments, the first image content received at step 2034 is stereoscopic image content including a left eye image and a right eye image, for example, with pairs of left and right eyes being received. In some such embodiments, the received and stored images corresponding to the first rear view portion, second rear view portion, sky view portion, and ground view portions include left eye images and right eye images, e.g. Includes pairs. Thus, for example, in step 2070, when generating one or more output images corresponding to the current field of view, a left eye image from the first content stream corresponding to a first view portion, e.g., a frontal view portion. are combined with stored left eye images corresponding to one or more other portions of the environment, and right eye images from the first content stream corresponding to the first view portion, e.g., a frontal view portion, are combined with stored left eye images corresponding to one or more other portions of the environment. The parts are combined with the stored right images corresponding to them.

In some other embodiments, the received and stored images corresponding to the first rear view portion, the second rear view portion, the sky view portion, and the ground view portion are either a left eye image or a right eye image from the original pair of images. Contains one or more single (mono) images, for example from a single camera operating alone to capture a portion of the view. Accordingly, in one such embodiment, for example, at step 2070, when generating one or more output images corresponding to the current field of view, the left eye image from the first content stream and corresponding to, for example, the frontal portion. wherein both corresponding right eye images from the first content stream are combined with the same stored image from the other view portion.

In still other embodiments, some of the stored images include both left and right eye image pairs, while others include single stored images. For example, the stored received images corresponding to the first rear view portion may include pairs of left and right eye images, while the stored received images corresponding to the sky view portion may include single images, e.g. For example, it may contain mono images rather than pairs.

In some embodiments, the first portion of the environment, e.g., corresponding to the received first content stream, is a front view portion, and the second portion of the environment is a rear view portion, e.g., corresponding to the right rear of the viewer. is the first rear view portion or the second rear view portion corresponding to the left rear of the viewer, and the third portion of the environment is the sky view portion or the ground view portion. In some such embodiments, images are received at different rates corresponding to the first, second, and third portions, and more images are received for an event corresponding to the first portion than to the second portion. do. In some such embodiments, more images are received corresponding to the second portion than to the third portion.

According to some embodiments, one example method of operating a content playback system includes: determining a viewer's head position corresponding to the current field of view; Receiving a first content stream providing content corresponding to a first portion of the environment; At least some received content included in the first content stream and one corresponding to the current field of view based on i) stored content corresponding to a second portion of the environment or ii) a composite image simulating the second portion of the environment generating the above output images; and outputting or displaying a first output image, wherein the first output image is one of one or more generated output images. In some embodiments, the content playback system is a content playback device. In some embodiments, the content playback system is a computer system coupled to the display.

In various embodiments, the method includes: receiving a first image corresponding to a second portion of the environment; and storing the first image corresponding to the second portion of the environment.

In some embodiments, the first image of the second portion of the environment corresponds to a first viewpoint; Generating one or more output images corresponding to the current field of view includes combining content obtained from the first content stream captured at a second viewpoint with the first image corresponding to the first viewpoint, and and the second time points are different. In some such embodiments, the first time point corresponds to a time preceding the second time point. In some such embodiments, the first point in time precedes the time of a real-time event during which images included in the first content stream were captured.

In various embodiments, the method further includes receiving one or more additional images corresponding to the second portion of the environment, wherein the one or more additional images corresponding to the second portion of the environment are: Contains at least a second image.

In some embodiments, the method is configured to indicate which of a plurality of previously delivered images corresponding to the second portion of the environment should be displayed for a playback time measured relative to the playback time indicated in the first content stream. and receiving control information.

In some embodiments, the second portion of the environment is one of a first rear view portion, a second rear view portion, a sky view portion, or a ground view portion. In some such embodiments, the method further includes receiving one or more images corresponding to the third portion of the environment.

In various embodiments, the first portion of the environment is a front view portion; The third portion is either a sky view or a ground view portion, and images are received at different rates corresponding to the first, second, and third portions, with more images than for the second portion. An event corresponding to the first part is received.

In various embodiments, the content corresponding to the first portion includes real-time content captured and streamed to the playback device while an event is in progress; Content corresponding to the images corresponding to the second and third portions are non-real-time images. In some such embodiments, receive image selection information indicating which of a plurality of images corresponding to the second portion of the environment should be used during the portion of the event, and select a current field of view based at least in part on the received content. Generating one or more output images corresponding to includes selecting an image corresponding to the second portion of the environment based on received image selection information.

In various embodiments, an example method includes determining that an image is not available for a portion of the current field of view; compositing an image to be used for the portion of the current field of view for which an image is not available; and combining the synthesized image with at least a portion of the received image to generate an image corresponding to the current field of view.

In various embodiments, the first image content is stereoscopic image content including a left eye image and a right eye image.

21 illustrates an exemplary content playback system implemented in accordance with the present invention that can be used to receive, decode, store, process, and display imaging content received from a content delivery system such as that shown in FIGS. 1 and 7. 2100). System 2100 may be implemented as a single playback device 2100' that includes a display 2102 or as a combination of elements such as an external display, e.g., a head mounted display 2105 coupled to computer system 2100'. It can be.

Content playback system 2100 is, in at least some embodiments, a 3D head mounted display. The head mounted display may be implemented using an OCULUS RIFT ^™ VR (virtual reality) headset, which may include a head mounted display 2105. In various embodiments, head mounted display 2105 is the same as head mounted display 805. Other head mounted displays may also be used. In some embodiments, one or more display screens of a head mounted helmet or other head mounting device are used to display content to the user's left and right eyes. By displaying different images to the left and right eyes on a single screen by a head mount configured to expose different portions of the single screen to the different eyes, a single display can be perceived separately by the left and right eyes of the viewer. Can be used to display images. In some embodiments, a mobile phone screen is used as the display of a head mounted display device. In at least some of these embodiments, a mobile phone is inserted into a head mounting device, and the mobile phone is used to display images.

Content playback system 2100 decodes the received encoded image data, for example left and right eye images and/or mono (single images) corresponding to different parts of the environment or scene, for example , has the ability to generate a 3D image for display to a customer by rendering and displaying the different left and right eye views perceived by the user as a 3D image. Content playback system 2100 is located at a customer premises location, such as a home or office, in some embodiments, but may also be located at an image capture site. Content playback system 2100 may receive, decode, process, generate, display, and/or perform other operations in accordance with the present invention.

System 2100 includes a display 2102, a display device interface 2013, an input device 2104, an input/output (I/O) interface 2106, a processor 2108, a network interface 2110, and memory ( 2112). Memory 2112 includes an assembly of modules 2114, eg, software modules, and data/information 2116. In some embodiments, system 2100 includes an assembly of modules 2115, e.g., an assembly of hardware modules, e.g., circuits. The various components of system 2100 are coupled together via a bus 2109 and/or other connections that allow data to be transferred between components of system 2100 or via a wireless interface. In some embodiments, display 2102 is included as an optional element shown using a dash box, while in some embodiments, an external display device 2105, e.g., a head mounted stereoscopic display device, is included as an optional element shown using a dash box. It can be coupled to a playback device via device interface 2103.

For example, if a mobile phone processor is used as processor 2108 and the mobile phone generates and displays images on a head mount, the system may include processor 2108, display 2102, and memory (2102) as part of the head mounted device. 2112). Processor 2108, display 2102, and memory 2112 may all be part of the mobile phone. In other embodiments of system 2100, processor 2108 may be part of a gaming system, such as an XBOX or PS4, that has a display 2105 mounted on a head mounting device and coupled to the gaming system. As will be appreciated, it is not critical whether the processor 2108 and/or memory 2112 are located in the headgear or not, whereas in some cases they may be placed on the headgear from a power, heat, and weight perspective. It may be convenient to co-locate the processor 2108, and in at least some cases, it may be desirable to have the processor 2108 and memory 2112 coupled to a headgear that includes a display.

Although various embodiments contemplate a head mounted display 2105 or 2102, the method and apparatus may also be used with non-head mounted displays that can support 3D images. Accordingly, in many embodiments, system 2100 includes a head mounted display, but it may also be implemented by a non-head mounted display.

Memory 21112 includes various modules, such as routines that, when executed by processor 2108, control content playback system 2100 to perform operations in accordance with the present invention. Memory 2112 includes an assembly of modules 2114, eg, software modules, and data/information 2116.

Data information 2116 includes: received images corresponding to first rear view portion 2118, received images corresponding to second rear view portion 2120, received images corresponding to sky view portion 2112. Includes one or more of both images, and received images corresponding to ground view portion 2124. Exemplary received images corresponding to rear view portion 2118 or 2120 include, for example, images of a standing audience or audience member, images of a seated audience member or audience member, images with different visible advertisements, images of a cheering audience member, etc. Includes fields, etc. Exemplary received images corresponding to sky views include, for example, clear sky, different cloud patterns, different degrees of darkness corresponding to different times, etc. Data information 2116 includes: control information corresponding to the first rear view portion 2130, received control information corresponding to the second rear view portion 2132, and received control information corresponding to the sky view portion 2134. , and received control information corresponding to the ground view portion 2136, all or one or more of them. Data information 2116 includes a received first content stream comprising pairs of right eye images and left eye images corresponding to a determined current viewer head position 2126, a determined current field of view 2128, e.g., a frontal view portion ( 2128), e.g., output images 2138 corresponding to the current field of view for which there is at least some received content available for combining the composited image(s) 2140 and generated output images 2142. It further contains a determined set of view parts to be used when creating the . Generated output images 2142. The generated output images include combined image content from a first content stream corresponding to a first, e.g., front view portion, and a stored image received from, e.g., a rear view portion, sky portion, or ground portion. and/or may include a composited image or a generated output image that includes a portion of the composited image, and sometimes includes the generated output image.

FIG. 22 is a diagram of an assembly of modules 2200 that may be included in system 800 of FIG. 8 or an example content playback system 2100 of FIG. 21 according to an example embodiment. In the assembly of modules 2200, the modules may be, and in some embodiments are, fully implemented as individual circuits, for example, in hardware within processor 2108. The modules in the assembly of modules 2200 may be, and in some embodiments are, fully implemented in hardware within the assembly of modules 2115, for example, as individual circuits corresponding to different modules. In other embodiments, some of the modules are implemented as circuits, e.g., within the processor 2108 with other modules implemented as circuits external to the processor 2108 and within an assembly of modules 2115 coupled thereto. . As will be appreciated, the level of integration with modules on the processor and/or with some modules external to the processor may be one of design choices.

Alternatively, rather than being implemented as circuits, all or portions of the modules may be implemented in software and stored in memory 2112 of system 2100, where the modules may be connected to a processor, e.g., processor 2108. When executed by, the modules control the operation of system 2100 to perform functions corresponding to the modules. In some such embodiments, assembly of modules 2200 is included in memory 2112 as assembly of modules 2114. In still other embodiments, the various modules of assembly of modules 2200 are implemented as a combination of hardware and software, e.g., other circuitry external to the processor operative to perform a portion of the functionality of the modules under software control. Provides input to processor 2108. Although shown in the FIG. 21 embodiment as a single processor, such as a computer, it should be understood that processor 2108 may be implemented as one or more processors, such as computers.

When implemented in software, modules include code that, when executed by processor 2108, configures processor 2108 to perform the functions corresponding to the module. In embodiments where the assembly 2200 of modules is stored in memory 2112, memory 2112 contains code for causing at least one computer, e.g., processor 2108, to implement the corresponding functions of the modules. , for example, is a computer program product that includes a computer-readable medium containing individual code for each module.

Completely hardware-based or completely software-based modules may be used. However, it should be understood that any combination of software and hardware, such as circuit implemented modules, may be used to implement the functions. As will be appreciated, the modules shown in FIG. 22 may be used by system 2100 or its internal components, such as processor 2108, to perform the functions of the corresponding steps shown and/or described in the method of flowchart 2000 of FIG. 20. Control and/or configure elements. Accordingly, assembly of modules 2200 includes various modules that perform functions corresponding to one or more steps of FIG. 20.

The assembly of modules 2200 includes a viewer head position determination module 2202, a current field of view determination module 2204, a content stream selection module 2206, a content stream reception module 2208, an image reception module 2210, and a received image It includes a storage module 2212, a control information reception module 2214, an output image generation module 2216, an output module 2242, a display module 2244, and control routines 2246.

The viewer head position determination module 2202 is configured to determine the viewer's head position, where the head position corresponds to the current field of view. Current field of view determination module 2204 is configured to determine the viewer's current field of view based on the determined head position.

Content stream selection module 2206 is configured to select a content stream from a plurality of alternative content streams based on input from a user, such as a viewer. Different content streams may correspond to different events. In various embodiments, different content streams corresponding to the same event correspond, for example, to different cameras pointing in different directions to provide the viewer with alternative frontal view perspectives. In some embodiments, at least some of the image streams that may be selected include stereoscopic image content including left eye image and right eye image pairs.

Content stream receiving module 2208 is configured to receive a first content stream providing content corresponding to a first location, eg, a front partial view, of one embodiment. In various embodiments, the first image content received in the first content stream is stereoscopic image content including pairs of a left eye image and a right eye image.

The image receiving module 2210 is configured to receive images corresponding to one or more different view portions, for example, a first rear view portion, a second rear view portion, a sky view portion, and a ground view portion. Image receiving module 2210 is configured to receive a first image corresponding to a second portion of the environment. In some such embodiments, the image receiving module is further configured to receive one or more additional images corresponding to the second portion of the environment, the one or more additional images comprising at least a second image. corresponds to In some embodiments, the second portion of the environment is one of a first rear view portion, a second rear view portion, a sky view portion, or a ground view portion. In some embodiments, the second portion of the environment is either a first rear view portion or a second rear view portion. In some embodiments, image reception module 2210 is configured to receive one or more images corresponding to a third portion of the environment. In some embodiments, the first portion of the environment is a frontal view portion and the third portion of the environment is either a sky view or a ground view portion.

In some embodiments, image receiving module 2210 is configured to receive a first image corresponding to a first rear view portion of the environment and one or more additional images corresponding to the first rear view portion of the environment. and configured to receive, wherein the one or more additional images corresponding to the first rear view portion include at least one second image corresponding to the first rear view portion. In some embodiments, image receiving module 2210 is configured to receive a first image corresponding to a second rear view portion of the environment and to receive one or more additional images corresponding to the second rear view portion of the environment. and wherein the one or more additional images corresponding to the second rear view portion include at least one second image corresponding to the second rear view portion. In some embodiments, image reception module 2210 is configured to receive one or more images corresponding to a sky view portion of the environment. In some embodiments, image reception module 2210 is configured to receive one or more images corresponding to a ground view portion of the environment.

The received image storage module 2212 is configured to store images received by the received image storage module 2212 . The received image storage module 2212 is configured to store the first image corresponding to the second portion of the environment. Received image storage module 2212 is configured to store one or more additional images corresponding to the second portion of the environment. Received image storage module 2212 is configured to store one or more received images corresponding to the third portion of the environment. In various embodiments, received image storage module 2212 stores a first image corresponding to the first rear view portion of the environment, and one or more additional images corresponding to the first rear view portion of the environment. It is configured to do so. In some embodiments, received image storage module 2212 is configured to store one or more additional images corresponding to the sky view portion of the environment. In some embodiments, received image storage module 2212 is configured to store one or more images corresponding to the ground view portion of the environment.

The control information receiving module 2214 controls to indicate which of the plurality of previously delivered images corresponding to the second portion of the environment should be displayed for a playback time measured relative to the playback times indicated in the first content stream. Configured to receive information. In various embodiments, the control information receiving module 2214 is configured to determine the playback time of which of the plurality of previously delivered images corresponding to the third portion of the environment is measured relative to the playback times represented in the first content stream. It is also configured to receive control information indicating whether it should be displayed for a while. In some embodiments, control information receiving module 2214 is configured to receive image selection information indicating which of the plurality of images corresponding to the second portion of the environment should be used during a portion of the event. In some embodiments, control information receiving module 2214 is configured to receive image selection information indicating which of the plurality of images corresponding to the third portion of the environment should be used during a portion of the event.

In some embodiments, the control information receiving module 2214 determines which of a plurality of previously delivered images corresponding to the first rear view portion of the environment is to be played in terms of playback time in the first content stream. and is configured to receive control information indicating whether the display should be displayed for a period of time. In some embodiments, control information receiving module 2214 receives image selection information indicating which of a plurality of previously delivered images corresponding to the first rear view portion of the environment should be used during a portion of an event. It is configured to do so. In some embodiments, the control information receiving module 2214 determines which of a plurality of previously delivered images corresponding to the second rear view portion of the environment is to be played in terms of playback time in the first content stream. and is configured to receive control information indicating whether the display should be displayed for a period of time. In some embodiments, control information receiving module 2214 receives image selection information indicating which of a plurality of previously delivered images corresponding to the second rear view portion of the environment should be used during a portion of an event. It is configured to do so.

In some embodiments, the control information receiving module 2214 determines which of a plurality of previously delivered images corresponding to the sky view portion of the environment is measured relative to the playback time in the first content stream. and is configured to receive control information indicating whether it should be displayed. In some embodiments, control information receiving module 2214 is configured to receive image selection information indicating which of a plurality of images corresponding to the sky view portion of the environment should be used during a portion of an event. In some embodiments, the control information receiving module 2214 determines which of a plurality of previously delivered images corresponds to the ground view portion of the environment for which playback time is measured in the first content stream. and is configured to receive control information indicating whether it should be displayed. In some embodiments, control information receiving module 2214 is configured to receive image selection information indicating which of a plurality of images corresponding to the ground view portion of the environment should be used during a portion of an event.

Output image generation module 2216 may generate: content received from a first content stream corresponding to a first partial view of the environment, a stored received image corresponding to a first rear view portion of the environment, a second rear view portion of the environment; a corresponding stored received image, a stored received image corresponding to a sky view portion of the environment, a stored received image corresponding to a ground view portion of the environment, or a composited image corresponding to a portion of the current field of view for which no image is available. and generate one or more output images corresponding to the current field of view based on at least one of the images. The output image generation module 2216 includes a view portion set determination module 2218, a content stream only determination module 2220, a missing portion determination module 2222, an image compositor module 2224, and a content stream output image generation module 2226. ), a composite output image generation module 2228, an output image content stream-based generation module 2230, and an output image non-stream-based generation module 2236.

View portion set determination module 2218 is configured to determine, based on the current field of view, a set of view portions in which there is at least some image content available of the environment to be used, e.g., when generating one or more output images. It is composed. Some example determined sets include, for example: {}, {front view portion}, {first rear view portion}, {second rear view portion}, {sky view portion}, {ground view portion}, {front view portion} view portion, sky view portion}, {front view portion, ground view portion}, {front view portion, first rear view portion}, {front view portion, second rear view portion}, {front view portion, first rear view portion view portion, sky view portion}, {front view portion, second rear view portion, sky view portion}, {front view portion, first rear view portion, ground view portion}, {front view portion, second rear view portion , ground view portion}, {first rear view portion, sky view portion}, {first rear view portion, ground view portion}, {first rear view portion, second rear view portion}, {first rear view portion , second rear view portion, sky view portion}, {first rear view portion, second rear view portion, ground view portion}, {second rear view portion, sky view portion}, and {second rear view portion, Includes a ground view portion}.

The content stream dedicated determination module 2220 may determine whether one or more output images are received images stored from different partial views, e.g., a first rear view portion, a second rear view portion, a sky view portion, or a ground view portion, or and determine whether the image is generated based on content in the first content stream without having to rely on the synthesized image. Content stream only determination module 2220 checks whether the determined set includes a single element that is a first, e.g., front-view portion, corresponding to a first content stream, and where the field of view is a first, e.g., front-view portion. It is configured to check whether it is within.

Missing portion determination module 2222 may determine if an image is not currently available for a portion of the field of view, e.g., if an image corresponds to a frontal view of the environment or from a first content stream that corresponds to another portion of the environment. and determine from the received stored image that an image is not available. Image compositer module 2224 is configured to composite images to be used for those portions of the current field of view for which images are not available. In various embodiments, image compositer module 2224 generates a composite image that is slightly larger than needed to fill in missing portions, for example, to allow for a full portion of the boundary.

Content stream output image generation module 2226 determines that the determined set includes only a first view portion, e.g., only a front view portion, e.g., a first view portion, as determined by content stream only determination module 2220. and generate one or more output images corresponding to the current field of view based solely on content received from the first content stream when there are no portions of the current field of view outside the view portion. In some embodiments, the content stream output image generation module 2226 performs a cropping operation on an image obtained from the first content stream.

The output image content stream based generation module 2230 is configured to: at least some received content included in the first content stream, e.g. corresponding to a front view, and i) stored content corresponding to a second part of the environment, e.g. For example, a stored image corresponding to a first rear view, a second rear view, a sky view or a ground view, or ii) a composite image simulating a second portion of the environment, wherein the image represents a portion of the field of view for which no image is available. and configured to generate one or more output images corresponding to a current field of view based on the synthesized image, which simulates. The stored content corresponding to the second part of the environment is, for example, a stored image stored by the received image storage module 2212. The composite image is, for example, an image generated by module 2224. In some embodiments, the output image content stream based generation module 2230 is configured to generate one or more output images corresponding to the current field of view and an image corresponding to the second portion of the environment based on received image selection information. It is configured to be selected as a part.

The output image content stream based generation module 2230 includes a composited image integration module 2232 configured to integrate one or more composited images into the generated output image. The synthesized image integration module 2232 receives from the received image, e.g., a first, e.g., frontal view portion obtained from the received first content stream, to generate an image corresponding to the current field of view. and combine the composited image with at least a portion of the received image or the stored received image corresponding to one of the first rear view portion, the second rear view portion, the sky view portion, and the ground view portion. Stored image integration module 2134 is configured to integrate portions of one or more stored images into a generated output image. As part of generating the output image, module 2230, module 2232, and/or module 2234 performs blending in boundary regions. In various embodiments, the mixing step includes filtering, blurring, luminance variations and/or color variations.

The output image non-stream based generation module 2236 may generate: i) stored content corresponding to a portion of the environment, e.g., a stored image corresponding to a first rear view, a second rear view, a sky view, or a ground view; or ii) to generate one or more output images corresponding to the current field of view based on at least one of a composite image simulating a portion of the environment, for example a composite image simulating a portion of the field of view for which no image is available. It is composed. Module 2236 generates one or more images when the current field of view is outside a region corresponding to a first, eg, frontal view portion corresponding to the first content stream. In some embodiments, output image non-content stream based generation module 2236 is configured to generate an image corresponding to the second portion of the environment based on received image selection information and one or more output images corresponding to the current field of view. It is configured to be selected as a constituent part.

The output image non-stream based generation module 2236 includes a composite image integration module 2238 configured to integrate one or more composite images into a generated output image. Composite image integration module 2238 combines received images, e.g., from a first rear view portion, a second rear view portion, a sky view portion, or a ground view portion, to generate an image corresponding to the current field of view. It is configured to combine at least a portion of the image and the synthesized image. The stored image integration module 2240 is configured to integrate portions of one or more stored images into a generated output image. As part of generating the output image, module 2236, module 2232, and/or module 2234 performs blending in border regions. In various embodiments, mixing includes filtering, blurring, luminance variations and/or color variations.

Output module 2242 may, for example, generate one of the output images produced by content stream-based generation module 2130, content stream output image generation module 2130, and output image non-content stream-based generation module 2236. and configured to output one or more generated output images, wherein the one or more output images include a first output image. The output module is configured to output the first output image, for example via the network interface 2110 and/or the display device interface 2103.

The display module 2244 may, for example, display one or more images generated by the output image content stream based generation module 2230, the content stream output image generation module 2230, and the output image non-content stream based generation module 2136. configured to display generated output images, wherein the one or more output images include a first output image. Display module 2244 is configured to display the first output image, for example, via display 2102 and/or display 2105.

Control routines 2246 include device control routines and communication routines for controlling the operation of system 2100.

One exemplary content playback system, e.g., system 2100 of FIG. 21, in accordance with the present invention, includes: a viewer head positioning module 2202 configured to determine a head position of a viewer, wherein the head position is in the current field of view; Correspondingly, the viewer head positioning module 2202; a content stream receiving module configured to receive a first content stream providing content corresponding to a first portion of the environment; at least some received content included in the first content stream and i) stored content corresponding to a second portion of the environment, or ii) corresponding to a current field of view based on a composite image simulating a second portion of the environment an output image content stream based generation module 2330 configured to generate one or more output images; and at least one of an output module 2242 configured to output a first output image or a display module 2244 configured to display a first output image, wherein the first output image is one of one or more generated output images. , and includes at least one of the output module 2242 or the display module 2244.

In some embodiments, the content playback system 2100 is a content device 2100'. In some embodiments, the content playback system 2100 is a computer system 2100' coupled to a display 2105.

In some embodiments, the system includes: an image receiving module 2210 configured to receive a first image corresponding to the second portion of the environment; and a received image storage module 2212 configured to store the first image corresponding to the second portion of the environment.

In various embodiments, the first image of the second portion of the environment corresponds to a first viewpoint, and the output image content stream-based generation module 2239 generates from the first content stream captured at the second viewpoint. and combine acquired content with the first image corresponding to the first viewpoint, wherein the first and second viewpoints are different. In some such embodiments, the first time point corresponds to a time preceding the second time point. In various embodiments, the first point in time precedes the time of a real-time event during which the first content stream includes images being captured.

In some embodiments, the image receiving module 2210 is further configured to receive one or more additional images corresponding to the second portion of the environment, the one or more additional images corresponding to the second portion of the environment includes at least a second image.

In various embodiments, the system is configured to: determine which of the plurality of previously delivered images corresponding to the second portion of the environment should be displayed for a playback time measured relative to the playback times indicated in the first content stream; It further includes a control information receiving module 2214 configured to receive control information indicating whether to do so.

In some embodiments, the second portion of the environment is one of a first rear view portion, a second rear view portion, a sky view portion, or a ground view portion. In some such embodiments, the image receiving module 2210 is also configured to receive one or more images corresponding to a third portion of the environment.

In some embodiments, the first portion of the environment is a front view portion; The third portion is either a sky view or a ground view portion, images are received at different rates corresponding to the first, second, and third portions, and more images are in the first than in the second portion. Events corresponding to part 1 are received.

In some embodiments, the content corresponding to the first portion includes real-time content captured and streamed to the playback device while an event is in progress; The content corresponding to the images corresponding to the second and third portions are non-real-time images. In some such embodiments, the control information receiving module 2214 is also configured to receive image selection information indicating which of the plurality of images corresponding to the second portion of the environment should be used during the portion of the event. become; The output image content stream based generation module 2230 is configured to select an image corresponding to the second portion of the environment based on the received image selection information as a portion configured to generate one or more output images corresponding to the current field of view. .

In various embodiments, the system includes a missing portion determination module 2222 configured to determine that an image is not available for a portion of the field of view; an image compositer module 2224 configured to composite an image to be used for the portion of the field of view for which an image is not available; and a composite image integration module 2232 configured to combine the composite image with at least a portion of the received image to generate an image corresponding to the current field of view.

In various embodiments, the first image content is stereoscopic image content including a left eye image and a right eye image.

In some embodiments, assembly of modules 2200 is included in content playback system 800 of FIG. 8, for example, in addition to other modules shown in FIG. 8. Modules in assembly of modules 2200 may be included in memory 812, in processor 808, and/or as hardware modules external to processor 808 coupled to bus 809 in system 800.

FIG. 23 is a diagram 2300 illustrating an example stream selection module 1938 that may be used in playback system 1900 in accordance with some embodiments.

FIG. 24 is a diagram 2400 illustrating an example stream prioritization module 2306 that may be implemented as part of the stream selection module of FIG. 23 or as a separate module.

Some embodiments relate to a non-transitory computer-readable medium embodying a set of software instructions, e.g., computer-executable instructions, for controlling a computer or other device to encode and compress stereoscopic video. Other embodiments relate to a computer-readable medium embodying a set of software instructions, e.g., computer-executable instructions, for controlling a computer or other device to decode and decompress video on a player end. Although encoding and compression are referred to as possible separate operations, it should be understood that encoding can be used to perform compression and thus encoding can include compression in some part. Similarly, decoding may include decompression.

The techniques of various embodiments may be implemented using software, hardware, and/or a combination of software and hardware. Various embodiments relate to devices, such as image data processing systems. Various embodiments also relate to methods, such as methods of processing image data. Various embodiments also include a non-transitory machine, e.g., a computer, a readable medium, e.g., ROM, RAM, CD, containing machine-readable instructions for controlling the machine to implement one or more steps of the method. fields, hard disks, etc.

Various features of the invention are implemented using modules. These modules may be, and in some embodiments are, implemented as software modules. In other embodiments, modules are implemented in hardware. In still other embodiments, modules are implemented using a combination of software and hardware. In some embodiments, the modules are implemented as individual circuits, and each module is implemented as a circuit for the module to perform its corresponding function. A wide variety of embodiments are contemplated, including some embodiments where different modules are implemented differently, for example, some in hardware, some in software, and some using a combination of hardware and software. It should also be noted that routines and/or subroutines, or some of the steps performed by such routines, may be executed in dedicated hardware as opposed to software executed on a general-purpose processor. These embodiments are within the scope of the present invention. Many of the above-described methods or method steps may be used to implement all or portions of the above-described methods, for controlling a machine, e.g., a general-purpose computer, with or without additional hardware, such as a memory device, e.g. For example, it may be implemented using machine-executable instructions such as software included in a machine-readable medium such as RAM, floppy disk, etc. Accordingly, among other things, the present invention relates to a machine-readable medium containing machine-executable instructions for causing a machine, e.g., a processor and associated hardware, to perform one or more of the steps of the method(s) described above. will be.

Many additional variations to the methods and apparatus of the various embodiments described above will be apparent to those skilled in the art upon consideration of the above description. These variations are considered within scope.

700: Content delivery system 702: Display
704: Input device 706: I/O interface
708: Processor 710: Network Interface
712: Memory 714: Control routines
716: Split module 718: Encoder(s)
720: Streaming controller 722: Request processing module
724: Data rate determination module 726: Current head position determination module
728: selection module 730: streaming control module
1900': Computer system/content playback device
1902: Display
1903: Display device interface
1904: Input device
1905: Displays, e.g. head-mounted stereoscopic displays
1906: I/O interface 1908: Processor
1910: Network Interface 1911: Receiver
1912: Memory 1913: Transmitter
1914: Control routines 1916: Head positioning module
1918: Current viewing position initialization module
1920: Decoder module
1922: Currently selected stream initialization module
1924: Content delivery initialization module 1926: Image buffer(s)
1928: Image buffer update module
1930: Image rendering module, e.g. image generation module
1932: Available bandwidth and/or supportable data rate determination module
1934: Head position change detection module
1936: Available bandwidth and/or supportable data rate change determination module
1938: Stream Selection Module
1940: Selected stream set change decision module
1942: Selected Stream Set Update Module
1944: Stream(s) determination module 1946: Received stream information
1948: Received bandwidth and/or data rate allocation control information
1950: Current maximum available bandwidth and/or supportable data rate determined.
1952: Received encoded data, e.g. encoded scene parts
1954: Received environment map 1956: Received UV map(s)
1958: Decoded data, e.g. decoded scene parts
1960 : Generated 3D content 1961 : Currently selected stream set
2200: Assembly of modules 2202: Viewer head positioning module
2204: Current view determination module 2206: Content stream selection module
2208: Content stream reception module
2210: Image reception module 2212: Received image storage module
2214: Control information reception module 2216: Output image generation module
2218: View partial set determination module
2220: Content stream-specific decision module
2222: Missing part determination module 2224: Image synthesizer module
2226: Content stream output image generation module
2228: Composite output image generation module
2230: Output image content stream-based generation module
2232: synthesized image integration module 2234: stored image integration module
2236: Output image non-stream based generation module
2238: synthesized image integration module 2240: stored image integration module
2242: output module 2244: display module
2246: Control routines

Claims (25)

In a method of operating a content playback system,
determining a viewer's head position, corresponding to the current field of view;
Based on the viewer's head position, selecting a first content stream from a plurality of content streams providing different fields of view, wherein the first content stream corresponds to the viewer's current field of view. selected - ;
Receiving the first content stream providing content corresponding to a first portion of an environment from one or more content streaming devices;
corresponding to the current field of view based on at least some image content included in the first content stream and i) stored image content corresponding to a second portion of the environment or ii) a composite image simulating a second portion of the environment generating one or more output images; and
A method of operating a content playback system, comprising using a first output image, which is one of the one or more generated output images, when playing. According to claim 1,
receiving a first image corresponding to the second portion of the environment; and
A method of operating a content playback system, further comprising storing the first image corresponding to the second portion of the environment. According to claim 2,
the first image of the second portion of the environment corresponds to a first viewpoint,
Generating one or more output images corresponding to the current field of view includes combining image content obtained from the first content stream captured at a second viewpoint with the first image corresponding to the first viewpoint; , wherein the first and second viewpoints are different. According to claim 3,
The method of operating a content playback system, wherein the first time point corresponds to a time preceding the second time point. According to claim 4,
Wherein the first time point precedes the time of a real-time event at which images included in the first content stream are captured. According to claim 3,
further comprising receiving one or more additional images corresponding to the second portion of the environment,
The one or more additional images corresponding to the second portion of the environment include at least one second image. According to claim 6,
Receiving control information indicating which of a plurality of previously delivered images corresponding to the second portion of the environment should be displayed for a playback time measured relative to the playback times indicated in the first content stream. A method of operating a content playback system further comprising, wherein the second portion of the environment is one of a first rear view portion, a second rear view portion, a sky view portion, or a ground view portion. According to claim 7,
A method of operating a content playback system, further comprising receiving one or more images corresponding to a third portion of the environment. According to claim 8,
the first portion of the environment is a frontal view portion,
the third portion is either a sky view or a ground view portion,
Content playback, wherein images are received at different rates corresponding to the first, second and third parts, and more images are received for an event corresponding to the first part than for the second part. How to operate the system. According to clause 9,
The content corresponding to the first portion includes real-time content captured while an event is in progress and streamed to the content playback system,
The method of operating a content playback system, wherein the content corresponding to the images corresponding to the second and third portions are non-real-time images. According to claim 10,
further comprising receiving image selection information indicating which of the plurality of images corresponding to the second portion of the environment should be used during the portion of the event,
Generating one or more output images corresponding to the current field of view based at least in part on received content includes selecting an image corresponding to the second portion of the environment based on the received image selection information. , a method of operating a content playback system. According to claim 11,
determining that an image is not available for a portion of the current field of view;
compositing an image to be used for the portion of the current field of view for which an image is not available; and
A method of operating a content playback system, further comprising combining the composite image with at least a portion of a received image to generate an image corresponding to the current field of view. In the content playback system,
an interface comprising a receiver configured to receive content streams;
at least one sensor; and
processor
Including,
The processor,
Based on the sensor output, determine the viewer's head position, where the head position corresponds to the current field of view;
Based on the viewer's head position, select a first content stream of the plurality of content streams providing different fields of view, wherein the first content stream is selected based on the first content stream corresponding to the viewer's current field of view. - ;
receive, from one or more content streaming devices, a first content stream providing content corresponding to a first portion of an environment;
corresponding to the current field of view based on at least some received content included in the first content stream and i) stored content corresponding to a second portion of the environment or ii) a composite image simulating a second portion of the environment generate one or more output images;
A content playback system configured to use a first output image when playing, wherein the first output image is one of the one or more generated output images. According to claim 13,
The processor,
receive, via the receiver, a first image corresponding to the second portion of the environment;
The content playback system is further configured to store the first image corresponding to the second portion of the environment. According to claim 14,
the first image of the second portion of the environment corresponds to a first viewpoint,
wherein the processor is configured to combine content obtained from the first content stream captured at a second viewpoint with the first image corresponding to the first viewpoint, wherein the first and second viewpoints are different. . According to claim 15,
The first viewpoint corresponds to a time preceding the second viewpoint. According to claim 16,
The first point in time precedes the time of the real-time event at which the images included in the first content stream are captured. According to claim 15,
The processor is further configured to receive, via the receiver, one or more additional images corresponding to the second portion of the environment, wherein the one or more additional images corresponding to the second portion of the environment include at least one A content playback system comprising a second image. According to claim 18,
The processor,
Control information indicating, via the receiver, which of a plurality of previously delivered images corresponding to the second part of the environment should be displayed for a playback time measured relative to the playback times indicated in the first content stream. and wherein the second portion of the environment is one of a first rear view portion, a second rear view portion, a sky view portion, or a ground view portion. According to claim 19,
The processor,
The content playback system is further configured to receive, via the receiver, one or more images corresponding to the third portion of the environment. According to claim 20,
the first portion of the environment is a frontal view portion,
the third portion is either a sky view or a ground view portion,
Content playback, wherein images are received at different rates corresponding to the first, second and third parts, and more images are received for an event corresponding to the first part than for the second part. system. According to claim 21,
The content corresponding to the first portion includes real-time content captured while an event is in progress and streamed to the content playback system,
The content corresponding to the images corresponding to the second and third portions are non-real-time images. According to claim 22,
the processor is further configured to receive, via the receiver, image selection information indicating which of the plurality of images corresponding to the second portion of the environment should be used during the portion of the event,
wherein the processor is further configured to, as part of generating one or more output images corresponding to the current field of view, select an image corresponding to the second portion of the environment based on the received image selection information. system. According to claim 23,
The processor,
determine that an image is not available for a portion of the current field of view;
composite an image to be used for the portion of the current field of view for which an image is not available;
The content playback system is further configured to combine the composited image with at least a portion of the received image to generate an image corresponding to the current field of view. 1. A non-transitory machine-readable medium containing processor-executable instructions, comprising:
The processor-executable instructions, when executed by a processor of a content playback system, cause the system to:
determining a viewer's head position, corresponding to the current field of view;
Based on the viewer's head position, selecting a first content stream from a plurality of content streams providing different fields of view, wherein the first content stream corresponds to the viewer's current field of view. selected - ;
Receiving the first content stream providing content corresponding to a first portion of an environment from one or more content streaming devices;
corresponding to the current field of view based on at least some received content included in the first content stream and i) stored content corresponding to a second portion of the environment or ii) a composite image simulating a second portion of the environment generating one or more output images; and
A non-transitory machine-readable medium that controls to perform the step of using a first output image, which is one of the one or more generated output images, when playing.

Images