JP7194125B2 - Methods and systems for generating virtualized projections of customized views of real-world scenes for inclusion within virtual reality media content - Google Patents

Description

関連出願 本願は、２０１７年５月３１日に出願され、「ＭＥＴＨＯＤＳ ＡＮＤ ＳＹＳＴＥＭＳ ＦＯＲ ＧＥＮＥＲＡＴＩＮＧ Ａ ＶＩＲＴＵＡＬＩＺＥＤ ＰＲＯＪＥＣＴＩＯＮ ＯＦ Ａ ＣＵＳＴＯＭＩＺＥＤ ＶＩＥＷ ＯＦ Ａ ＲＥＡＬ－ＷＯＲＬＤ ＳＣＥＮＥ ＦＯＲ ＩＮＣＬＵＳＩＯＮ ＷＩＴＨＩＮ ＶＩＲＴＵＡＬ ＲＥＡＬＩＴＹ ＭＥＤＩＡ」と題され、その全体が参照によりPriority is claimed to US patent application Ser. No. 15/610,595, incorporated herein.

The virtual reality media content can be presented to a user (i.e., viewer) of the virtual reality media content, immersing the user in an interactive virtual reality world, wherein the virtual • The reality world can be experienced by the user by directing his or her attention to any variety of objects that are presented at the same time. For example, at any time the virtual reality media content is presented, a user experiencing the virtual reality media content can look around the immersive virtual reality world in any direction, allowing the user The sense is that the user is actually present in the immersive virtual reality world and has a particular location and view within the immersive virtual reality world (e.g. orientation, viewpoint, etc.). ), it feels like experiencing an immersive virtual reality world.

In some instances, the immersive virtual reality world may be desired to be based on real world scenes. For example, some or all of the immersive virtual reality worlds represented within the virtual reality media content model scenery, locations, events, objects, and/or other objects that exist in the real world. , as opposed to existing only in a virtual or virtual world. Thus, a capture device (e.g., an image and/or video capture device (e.g., camera, video camera, etc.)) is used to detect, record, and/or capture data representing a real-world scene. so that the data can be included in virtual reality media content that can generate representations of real-world scenes. Unfortunately, it may be impossible or impractical to position a physical capture device on a real-world scene to capture data from all locations, orientations, views, etc. These things may be desired.

Furthermore, even if a large number of physical capture devices are employed to capture data from a large number of locations, orientations, fields of view, etc., all data captured by these capture devices can be viewed by the user. It may be impractical and/or inefficient to include within the virtual reality media content presented to. For example, data delivery limitations (e.g. network bandwidth, device decoding capabilities, etc.), significant redundancy in captured data, and at the same time real-world scenes with different relevance to different users of virtual reality media content. data describing different details of the impracticality and/or inefficiency of capturing and distributing;

The accompanying drawings illustrate various embodiments and are a part of this specification. The illustrated embodiments are exemplary only and are not intended to limit the scope of the present disclosure. Throughout the drawings, same or similar reference numbers designate same or similar elements.

1 illustrates an exemplary virtualized projection generation system for generating a virtualized projection of a customized view of a real-world scene for inclusion within virtual reality media content in accordance with the principles described herein; It is for generating.

1 illustrates an exemplary configuration, where data representing an exemplary real-world scene are captured from different views of the real-world scene according to principles described herein;

3 illustrates an exemplary capture device that captures color and depth frames for inclusion within the sequence of surface data frames representing the real-world scene of FIG. 2 in accordance with principles described herein;

3B shows an exemplary graphical depiction of color data represented in color frames captured by the capture device of FIG. 3A in accordance with the principles described herein; FIG.

3B shows an exemplary graphical depiction of depth data represented in depth frames captured by the capture device of FIG. 3A in accordance with principles described herein; FIG.

FIG. 2 depicts the real-world scene of FIG. 2 and shows different representations of an exemplary sequence of surface data frames generated by the capture device of FIG. 3A in accordance with the principles described herein; FIG. 2 depicts the real-world scene of FIG. 2 and shows different representations of an exemplary sequence of surface data frames produced by the capture device of FIG. 3A in accordance with the principles described herein;

shows an exemplary arrangement based on the arrangement of FIG. 2, wherein data representing the real-world scene of FIG. 2 is further generated according to principles described herein for customized views of the real-world scene. be done.

6 shows an exemplary virtualized surface data frame sequence, said sequence showing color and color for an exemplary virtualized projection of a customized view of the real-world scene of FIG. 5 in accordance with principles described herein. Contains depth frames.

1 shows a graphical representation of an exemplary transport stream, said stream including an exemplary plurality of surface data frame sequences in accordance with the principles described herein;

8 shows a data structure representation of the exemplary transport stream of FIG. 7, according to principles described herein;

1 shows a graphical representation of an exemplary transport stream, said stream including an exemplary sequence of frames implementing a tile map according to the principles described herein;

10 shows a data structure representation of the exemplary transport stream of FIG. 9 in accordance with principles described herein; FIG.

1 illustrates an exemplary configuration, wherein an exemplary virtual reality media content serving system generates virtual reality media content based on real-world scenes according to principles described herein; and providing virtual reality media content to an exemplary client-side media player device, said device being used by a user, said user experiencing a representation of a real-world scene. .

Various exemplary types of media player devices in accordance with the principles described herein are shown and may be used by users to experience virtual reality media content.

1 illustrates an exemplary virtual reality experience in accordance with the principles described herein, wherein a user is presented with exemplary virtual reality media content based on a real-world scene, said content comprising: Experienced from a dynamically selectable virtual viewpoint, said viewpoint corresponding to an exemplary arbitrary virtual location of a real-world scene.

An exemplary method according to the principles described herein for generating a virtualized projection of a customized view of a real-world scene for inclusion within virtual reality media content. belongs to. An exemplary method according to the principles described herein for generating a virtualized projection of a customized view of a real-world scene for inclusion within virtual reality media content. belongs to.

1 illustrates an exemplary computing device in accordance with the principles described herein;

Methods and systems are described herein for generating virtualized projections of customized views of real-world scenes for inclusion within virtual reality media content. For example, as described in more detail below, a virtualized projection generation system may receive (eg, request, acquire, access, etc.) multiple captured surface data frame sequences. Each surface data frame sequence in the plurality of captured surface data frame sequences may include: a color and depth frame, the color and depth frame capturing a real-world scene; • Rendering according to each set of parameters, said parameters being included in multiple sets of capture parameters associated with different views of the real-world scene. For example, each set of capture parameters associated with each view of a real-world scene can include parameters representing: capture location, orientation, field of view, depth mapping, depth range, quality level, format, source. , dynamic range, and/or other characteristics for each surface data frame sequence to represent a view of the real-world scene. Each surface data frame sequence in the plurality of captured surface data frame sequences may be captured by a different capture device in the plurality of capture devices, the plurality of capture devices It may be placed at different locations with respect to the real-world scene to capture different views of the scene. For example, each different capture device may be associated with one of a different set of capture parameters within the plurality of sets of capture parameters (e.g., according to one of the different sets of capture parameters, the real-world scene ).

In addition to receiving multiple sequences of surface data frames associated with different sets of capture parameters, the virtualization projection generation system can determine which sets of capture parameters are included within the multiple sets of capture parameters. Additional sets of different capture parameters can be specified. An additional set of capture parameters may relate to a customized view of the real-world scene, which may differ from different views of the real-world scene captured by multiple capture devices. good. Based on the surface data frame sequences in the plurality of captured surface data frame sequences and an additional set of capture parameters, the virtualization projection generation system generates a virtual view of the customized view of the real-world scene. Color and depth frames can be rendered for color projection.

The virtualization projection generation system can provide a virtualized surface data frame sequence, the surface data frame sequence being a rendered color for a virtualization projection of a customized view of a real-world scene. and depth frames, which may be provided to one or more other systems (e.g., to one or more media player devices associated with the user, the virtual reality media content, to one or more downstream systems in the provider pipeline, etc.). For example, a virtualization projection generation system can provide a surface data frame sequence to be virtualized to a media player device for inclusion within virtual reality media content (e.g., virtual reality media content). The media content may be configured to be streamed by means of a virtual reality media provider pipeline to a media player device associated with the user experiencing the virtual reality media content) .

The systems and methods described herein for generating virtualized projections of customized views of real-world scenes for inclusion within virtual reality media content provide various advantages and advantages. can be done. As an example, the systems and methods described herein can capture data (e.g., virtualized surface data frame sequences, etc.) representing a virtualized projection of a customized view of a real-world scene from any captured image. Allows generation based on parameters (eg, arbitrary capture location, orientation, field of view, depth mapping, quality level, source, dynamic range, etc.). Thus, a robust view of a real-world scene can be generated that can generate a surface data frame sequence that is virtualized and that can be provided separately from the surface data frame sequence that is captured and that can contribute to a particular implementation. A (robust) set can be covered. For example, trying to place a large number of physical capture devices at various locations about a real-world scene (e.g., providing different levels of detail for different objects with different bit depths, etc.). Instead, the methods and systems described herein can enable a relatively small number of physical capture devices to capture data from which a vast number of virtualized surfaces can be captured. A data frame sequence can be generated to represent a customized view of the real-world scene.

Further, generating capture data (e.g., surface data frame sequences) associated with different sets of capture parameters (e.g., parameters representing different vantage points, different capture resolutions, etc.). By doing so, the systems and methods described herein can facilitate: inclusion within virtual reality media content captured by physical capture devices and provided to end users; Deliver data realistically and efficiently. For example, using high-resolution data captured by eight capture devices positioned around the real-world scene, a sequence of eight high-resolution captured surface data frames depicting the real-world scene can be generated by 8 In addition to generating from each view of one capture device, a relatively large number (e.g., 300) of low-resolution virtualized surface data frame sequences can also be generated, wherein the surface data frames • The sequence may relate to various customized views that are different (eg, ragged) from the capture device's view.

A virtual reality media content delivery system that receives surface data frame sequences to be virtualized can advantageously provide greater flexibility in terms of: which data the system can access to a particular media; • To be provided to a player device at a specific time (ie included within the virtual reality media content provided to said device). Thus, virtual reality media content providers can take advantage of certain enhanced capabilities, said capabilities to optimize the data provided to media player devices (e.g. Not by flooding the media player device with relatively irrelevant data (e.g., based on various aspects of the particular virtual reality experience that the media player device provides to the user) ), and by providing the data using a bit depth that is optimized in the depth representation of the surface to optimize depth accuracy and/or depth resolution. This will be explained in more detail later.

As an example, rather than delivering all of the high-resolution data captured by eight physical capture devices to all media player devices (which would be impractical or impossible due to the large amount of possible), the customized data may be delivered more selectively and more flexibly. Specifically, for example, data customized for the first media player device (e.g., a few virtualizations selected from a robust set of virtualized surface data frame sequences). data representing a surface data frame sequence to be displayed) can be delivered to the first media player device, and a portion of the real world scene associated with the user of the first media player device is rendered at a high level. can be provided in detail while the data customized for the second media player device (eg, data representing a few different virtualized surface data frame sequences) It can be delivered to two media player devices, and another portion of the real world scene relevant to the user of the second media player device can be provided in high level detail. Thus, the virtual reality media content providing system provides both the first and second media player devices with data associated with each user (eg, each real-world scene the user is experiencing). customized and localized data for some). During this time, any media player device (or any distribution channel used to communicate with the media player device) may have excessive amounts of redundant data or You are not overloaded with detailed data. In this way, data delivery can be enhanced, and can be made more efficient and effective, because the data required to be delivered to the client-side media player device can be (even though higher resolution improves the user experience and results in more realistic and immersive content). These improvements are driven by customization of virtual reality media content, which dynamically involves rendering only the most relevant parts of the real-world scene in high quality.

Various embodiments are further described below with reference to the drawings. The disclosed methods and systems may provide one or more of the advantages described above, and/or may provide various additional and/or alternative advantages, which are set forth herein. would be

FIG. 1 illustrates an exemplary virtualized projection generation system 100 (“system 100”), which virtualizes customized views of real-world scenes for inclusion within virtual reality media content. It is for generating projections. As shown, the system 100 can include, but is not limited to: a communication facility 102, a surface data frame sequence management facility 104, a virtualized projection generation facility 106, and a storage facility 108. (selectably and communicatively connected to each other). It should be appreciated that although facilities 102-108 are shown as separate facilities in FIG. or divided into more facilities that can contribute to a particular implementation. In some examples, each of the facilities 102-108 may be distributed among multiple devices and/or among multiple locations as they may contribute to a particular implementation. Additionally, it should be appreciated that in certain implementations of system 100, certain facilities shown in FIG. 1 (and associated functionality associated with such facilities) may be omitted from system 100. Each of the facilities 102-108 is described in greater detail below with reference to other specific figures contained herein.

Communication facility 102 can include: one or more physical computing devices (e.g., hardware and/or software components (e.g., processors, memory, communication interfaces, memory etc.)). They can perform various operations related to sending and receiving data used and/or provided by system 100 . For example, the communication facility 102 may receive (or facilitate receiving) multiple captured surface data frame sequences, each surface data frame sequence may include color and depth frames, A color and depth frame may depict a real-world scene according to each set of capture parameters, which may be included in multiple sets of capture parameters associated with different views of the real-world scene. .

Communication facility 102 may receive multiple captured surface data frame sequences in any manner that may contribute to a particular implementation. For example, in certain embodiments, each surface data frame sequence in the plurality of captured surface data frame sequences is captured (eg, generated) by a different capture device in the plurality of capture devices. Alternatively, the capture devices may be placed at different locations with respect to the real-world scene to capture different views of the real-world scene. Thus, the communication facility 102 can receive data (e.g., captured surface data frame sequences) directly from multiple capture devices, which may be: e.g., from a capture device Requesting and receiving transmitted data or accessing or obtaining data from a capture device. In other examples, one or more other systems (eg, real-world scene capture systems) can mediate between the capture device and system 100 so that communication facility 102 is captured. The surface data frame sequence may be received by means of one or more other systems.

Additionally or alternatively, communication facility 102 provides data (eg, surface data frame sequences to be virtualized or other data received and/or generated by system 100) to: May be: other server-side systems in the virtual reality media content provider pipeline and/or client-side media player devices used by end users. As used herein, "server-side" refers to transactions between a server and a client (e.g., a client-side transaction where a content providing system uses content (e.g., virtual reality media content) for end-user consumption). It may also refer to the server side (e.g., provider side) of a transaction provided to a device. For example, as described in more detail below, a virtual reality media content providing system can provide virtual reality media content to media player devices associated with users. Thus, server-side systems and components can refer to certain systems and components, and said systems and components can be related (e.g., contained, implemented, interoperable, etc.) to content serving systems. etc.), the content providing system may provide data (eg, virtual reality media content) to media player devices (eg, by means of a network). On the one hand, a "client-side" device can relate to a client device (e.g., a media player device), which can be used by a user located on the other side of the network, a "client-side" device A "side" device can include: a device that facilitates the client device receiving data from the content serving system (e.g., a media player device and/or a user device on the user's side of the network). other computer components operated by

Communication facility 102 may be configured to perform the following operations: server-side and/or client-side using any communication interface, protocol, and/or technology that may contribute to a particular implementation • Communicate with side systems. For example, communication facility 102 may be configured to communicate via one or more of the following means: networks (e.g., wired or wireless local area networks, wide area networks, provider networks, the Internet, etc.) ), a wired communication interface (eg, Universal Serial Bus (“USB”)), a wireless communication interface, or any other suitable communication interface, protocol, and/or technology.

The surface data frame sequence management facility 104 can include: one or more physical computing components (e.g., separate from the hardware and/or software components of the communication facility 102; or , shared with communication facility 102). They can perform various operations related to organizing, synchronizing, managing, tracking, and/or managing: surface data frame sequences received or generated by system 100; and each set of capture parameters associated with the surface data frame sequence. For example, the surface data frame sequence management facility 104 may manage captured surface data frame sequences (eg, surface data frame sequences captured by the capture device and received by the communication facility 102 described above). ) and/or the surface data frame sequence management facility 104 can specify (or facilitate specifying): : one or more additional sets of capture parameters that are different from the set of capture parameters associated with the captured surface data frame sequence. For example, the surface data frame sequence management facility 104 can specify one or more sets of capture parameters, the one or more sets representing different views of the real-world scene captured by multiple capture devices. may each be associated with one or more customized views of a different real-world scene. Surface data frame sequence management facility 104 may also perform: other operations described herein and/or other operations that may contribute to a particular implementation of system 100. .

Virtualized projection generation facility 106 may include: one or more physical computing components (e.g., hardware and software components separate from the hardware and/or software components of facilities 102 and/or 104); /or software components, or hardware and/or software components shared with facilities 102 and/or 104). They can perform various operations related to: preparing, shaping, virtualizing projections of views of the real-world scene (e.g., customized views) and/or data associated therewith; render or generate. For example, virtualization projection generation facility 106 can render color and depth frames for virtualization projections of customized views of real-world scenes. More specifically, for example, the virtualization projection generation facility 106 can render color and depth frames based on at least one of the surface data frame sequences received by the communication facility 102. and may be further based on an additional set of capture parameters specified by the surface data frame sequence management facility 104 . The virtualization projection facility 106 may also generate a virtualized surface data frame sequence, which may be based on the rendered color and depth frames for the virtualization projection (e.g., this may include). Once the virtualized surface data frame sequence is generated, the virtualization projection facility 106 associates the surface data frame sequence with the user for inclusion within the virtual reality media content. It can be provided in a media player device. Alternatively, as noted above, the virtualized surface data frame sequence may be provided by communication facility 102 for inclusion within virtual reality media content in certain implementations. Virtualization projection facility 106 may also perform: other operations described herein and/or other operations that may contribute to particular implementations of system 100 .

Storage facility 108 may store and/or manage any suitable data, which in particular implementations may be received, generated, managed, tracked, managed, used, and/or processed by facilities 102-106. or may be sent. For example, as shown, storage facility 108 may include: surface data frame sequence data 110 and/or capture parameter data 112 . They may be received, generated, managed, tracked, administered, used, and/or transmitted (eg, provided to other systems) in any of the ways described herein. Additionally, storage facility 108 may include: other types of data (e.g., instructions (e.g., programming • instructions)) and/or other data used by the equipment 102-106 to perform the operations described herein. Storage facility 108 may be implemented in any of the ways described herein, and storage facility 108 may include hardware and/or hardware for any transient or non-transitory mode of storing data. or software, including but not limited to: random access memory (“RAM”), non-transitory storage (e.g., disk storage, flash memory storage) And so on.

In some examples, system 100 can perform one or more operations described herein in real-time when events occur within a real-world scene. Thus, in implementations where system 100 is used within a virtual reality media content provider pipeline (where other systems also operate in real time), virtual reality media content (e.g., system 100 virtual reality media content, including virtualized surface data frame sequences generated in real-time by the media player device, so that each user of the media player device (user may not be physically located near the real-world scene, and may wish to experience the real-world scene (e.g., events occurring within the real-world scene), each media player of the user The device can be used to virtually experience real-world scenes and events occurring within the real-world scenes in a live format (eg, in real-time as the events occur). Data processing and data delivery can take a finite amount of time such that the user cannot accurately experience the real-world scene when events occur within the real-world scene. On the other hand, as used herein, an operation is performed in "real time" if the operation is performed immediately and without undue delay. Thus, it may be said that the user experiences the real-world scene in real-time even if: the user is in the real-world scene after some delay (e.g. If you experience a particular event a few seconds or minutes after .

As noted above, in certain implementations, system 100 can generate data representing a relatively large number of virtualization projections. Such data can provide flexibility regarding: how to generate virtual reality media content (e.g., virtual reality media content that uses the data), and how clients - Deliver to side media player device? For example, providing details related to the experience of one user to a media player device associated with said user by generating data representing a large number of localized virtual projections of real world scenes. while other users' media player devices with less relevance to said details are not provided.

In one particular implementation of system 100, for example, communication facility 102 can receive multiple captured surface data frame sequences (eg, in real-time as events occur within the real-world scene). , each surface data frame sequence includes a color and depth frame, the color and depth frame depicting a real-world scene according to a respective set of capture parameters, each set of the capture parameters representing a real-world Included within a first plurality of sets of capture parameters associated with different views of the scene. As described above, each surface data frame sequence in the plurality of captured surface data frame sequences may be captured by a different capture device in the plurality of capture devices, wherein the plurality of capture device Devices may be placed at different locations with respect to the real-world scene to capture different views of the real-world scene. In addition to receiving the surface data frame sequence to be captured, the surface data frame sequence management facility 104 can specify a second plurality of sets of capture parameters, said second plurality of The plurality of sets may be different from the set of capture parameters contained within the first plurality of sets of capture parameters. For example, each set of capture parameters in the second plurality of sets of capture parameters may be associated with each customized view of the real-world scene, each customized view associated with a plurality of capture parameters. Different views of the real-world scene captured by the device may differ. For example, the second plurality of sets of capture parameters can include: a relatively large number of sets of capture parameters (e.g., included within the first plurality of sets of capture parameters); set of numbers even greater than ). Such identifying operations may also be performed in real-time as events occur within the real-world scene.

In response to specifying the second plurality of sets of capture parameters, virtualization projection generation facility 106 can render color and depth frames for the virtualization projection of each customized view of the real-world scene. The rendering may be based on a plurality of captured surface data frame sequences and may be based on a second plurality of sets of capture parameters. In some examples, the virtualization projection generation facility 106 may package the rendered color and depth frames to be included within each virtualized surface data frame sequence; The surface data frame sequence to be encoded may be transmitted by means such as one or more transport streams, which will be described in more detail below. Also, such rendering and/or packaging of data may be performed in real-time, as the events occur within the real-world scene. Thus, communication facility 102 can provide a plurality of virtualized surface data frame sequences (eg, in real-time, upon occurrence of the event within the real-world scene), wherein the surface data frames - The sequence may include rendered color and depth frames for each customized view virtual projection of the real-world scene. For example, multiple virtualized surface data frame sequences may be provided to a media player device for inclusion within virtual reality media content (eg, virtual reality media content may be , configured to be streamed in real-time by means of a virtual reality media provider pipeline to a media player device associated with the user experiencing the virtual reality media content).

Data representing a real-world scene (e.g., surface data frame sequences received by system 100) may be arranged in any suitable configuration and/or device in any suitable manner to contribute to a particular implementation. may be captured by For example, as described above, each surface data frame sequence in the plurality of captured surface data frame sequences received by system 100 may be captured by a different capture device in the plurality of capture devices. Well, the plurality of capture devices may be placed at different locations with respect to the real world scene so as to capture different views of the real world scene.

For illustrative purposes, FIG. 2 shows an exemplary configuration 200, where data representing an exemplary real-world scene are captured from different views of the real-world scene. Specifically, as shown in configuration 200, a real-world scene 202 containing a real-world object 204 may be surrounded by multiple views 206 (eg, views 206-1 through 206-8) of the real-world scene 202. good.

Real-world scene 202 can represent: any real-world scene, real-world location, real-world event (e.g., live event, etc.), or can contribute to a particular implementation. Other objects that exist in the real world (eg, as opposed to those that exist in the virtual world or only in the virtual world). As indicated by the circle representing real-world scene 202 in FIG. 2, real-world scene 202 may specifically be a delineated area (eg, a stage, an arena, etc.). Conversely, in other examples, the real-world scene 202 may be less compartmentalized or delineated. For example, real-world scenes 202 can include: any indoor or outdoor real-world location (eg, city streets, museums, scenic views, etc.). In particular examples, real-world scene 202 may relate to real-world events (e.g., sporting events, musical events, play or theater presentations, large celebrations (e.g., in Times Square). New Year's Eve, Mardi Gras, etc.), political events, or any other real world event). In the same or other examples, real-world scene 202 may relate to: a setting for a fictionalized scene (e.g., a live action virtual reality television show or movie set); /or any other scene-related settings in any other indoor or outdoor real-world location that may contribute to a particular implementation.

Thus, real-world object 204 can represent any real-world object, whether animate or inanimate, and said real-world object may be associated with real-world scene 202 (eg, within real-world scene 202). may be detectable (eg, viewable, etc.) from at least one of the views 206 . For purposes of clarity, real-world object 204 is depicted as a relatively simple geometric shape, but it should be understood that: real-world object 204 can have various shapes with varying levels of complexity. can represent any type of object. For example, rather than being geometric shapes, real-world objects 204 can represent: any animated or non-animated object or surface (e.g., a person or another living creature), non-transparent solids, liquids. , or gas, objects lacking discontinuity (eg, walls, ceilings, floors), or any other type of object that can contribute to the implementations described herein or in particular.

Real-world object 204 may include various surfaces, each of which may reflect light (e.g., ambient light in real-world scene 202, structured light emitted by a depth capture device, etc.). Infrared in a pattern), the light may be detected by capture devices positioned at different locations with respect to the real-world scene 202 to capture the real-world scene 202 from a view 206. may Although a relatively simple depiction of the real-world object 204, the depth and/or appearance of the surface of the real-world object 204 may differ based on which real-world scene 202 view 206 the surface is detected from. It may also have an appearance, which will be discussed later. In other words, real-world object 204 may appear differently based on the field of view (eg, location, Vantage point, etc.) in which real-world object 204 is viewed.

As described above, views 206 of real-world scene 202 can provide different views, vantage points, etc., from which real-world scene 202 (eg, including real-world objects 204) can be viewed. Using color and depth data of the real-world scene 202 captured from various different views 206 (e.g., views 206 surrounding the real-world scene 202 for the purpose of capturing the real-world scene 202 from different perspectives), as described below. , system 100 can generate a virtualized projection of any view of real-world scene 202 . In other words, using color and depth data captured from one or more views 206 , system 100 can generate a customized view of real-world scene 202 (eg, from a different location, orientation, etc. than view 206 ). Any view of the world scene 202) can be rendered color and depth data for a virtualized projection.

Views 206 may each be fixed with respect to real-world scene 202 . For example, both real-world scene 202 and view 206 may be static, or real-world scene 202 and view 206 may move together. In some examples, for example, as shown in configuration 200, view 206 views real-world scene 202 along at least two dimensions associated with real-world scene 202 (eg, along a plane (eg, ground)). You can surround it. In particular examples, views 206 may surround real-world scene 202 along three dimensions (eg, by also including views 206 above and below real-world scene 202).

Each view 206 may be associated with a particular location relative to the real-world scene 202 , as indicated by the different locations surrounding the real-world scene 202 in which the views 206 are placed. Additionally, view 206 may further relate to other aspects of how real-world scene 202 is captured. For example, as indicated by the dotted lines emanating from each view 206, the views 206 may be associated with: a particular capture orientation (eg, a particular direction toward which the capture device corresponding to the view 206 is directed); The specific field of view of the capture (e.g., the area of the real-world scene 202 that the capture device captures based on: e.g., how narrow or wide the lens of the capture device is, the zoom level of the capture device, etc.) etc. Each view 206 may also be associated with aspects of capture not explicitly shown in FIG. For example, each view 206 may be associated with: a particular quality level (eg, image resolution, frame rate, etc.) at which data is captured by the capture device associated with view 206; The particular format in which the data captured by is encoded, and/or any other aspect of data capture that may contribute to a particular implementation.

In some examples, as shown in configuration 200, vantage points (eg, orientation, field of view, etc.) associated with each view 206 may be angled inward toward real-world scene 202, which may capture the real-world scene 202 from a sufficient field of view to be able to later recreate the real-world scene 202 from a customized view that may be misaligned with the view 206 . Further, in the same or other examples, one or more vantage points associated with view 206 may be angled outward (i.e., away from real-world scene 202) such that real-world scene 202 may capture data representing objects, etc. surrounding the . For example, a spherical 360-degree capture device (outward facing vantage point) may be placed at a position (not explicitly shown) within the real-world scene 202 to provide an additional field of view of the objects contained within the real-world scene 202. and/or a device external to the real-world scene 202 can be captured. Additionally or alternatively, in certain examples, multiple outward-facing views may enable the capture of panoramic, wide-angle, or 360-degree views of real-world scenes. can.

A different capture device in the plurality of capture devices may be positioned at each different location of the view 206 for the purpose of capturing the real-world scene 202 from the field of view of each view 206 . For illustrative purposes, FIG. 3A shows an exemplary capture device 302 that captures color and depth frames for inclusion within a sequence of surface data frames representing real-world scene 202. do.

As shown in FIG. 3A, capture device 302 may be associated with view 206-1 and thus may be positioned at a location corresponding to view 206-1 with respect to real-world scene 202 and real-world object 204. good. As FIG. 3A illustrates, the capture device 302 can include: a two-dimensional (“2D”) color capture device 304 that provides a real-world scene 202 (e.g., a real-world object 204 and/or or other objects contained within a real-world scene), and depth A capture device 306 , a device configured to capture depth data representing real-world scene 202 .

2D color capture device 304 can be implemented by any suitable 2D color capture device (e.g., camera, video camera, etc.), and in any manner that can contribute to a particular implementation, 2D color data can be captured. In some examples, 2D color capture device 304 may be a separate device from depth capture device 306 . Collectively, such separate devices (e.g., as well as any communication interfaces and/or other hardware or software mechanisms used to functionally blend the devices) are referred to as capture devices (e.g., capture device 302). In another example, as shown in FIG. 3A, 2D color capture device 304 and depth capture device 306 may be integrated into a single device (i.e., capture device 302), said device and depth data can be captured, which is described below.

Whether implemented as a separate device or integrated with the 2D color capture device 304, the depth data capture device 306 converts depth data representing the real-world scene 202 to contribute to a particular implementation. can be captured in any manner possible. For example, depth data capture device 306 can employ one or more depth map capture techniques (e.g., structured light depth map capture technique, stereoscopic depth map capture technique, time-of-flight depth capture technique, another suitable depth map capture technique, or any combination of depth map capture techniques that may contribute to a particular implementation).

Regardless of the type and number of depth map capture techniques used to capture depth data, capture device 302 captures the surface of real-world object 204 and/or other objects contained within real-world scene 202. Both representative color data (eg, color frame) and depth data (eg, depth frame) can be captured from view 206-1. As used herein, color frames and depth frames captured at approximately the same time by capture device 302 may be collectively referred to as "surface data frames" or "color and depth frames"; For this reason, the data contained in these frames represent data describing the surface of the real-world objects contained within the real-world scene (i.e., both the visible appearance of the surface as well as the depth geometry of the surface). because they are

Thus, as used herein, a surface data frame or color and depth frame may refer to a dataset, which represents various types of data associated with the surface of a real-world object. Alternatively, the surface of the real-world object may be visible within the real-world scene at a particular point in time from a particular view of the real-world scene. For example, a surface data frame can include: color data (ie, image data) as well as depth data representing objects visible from a particular view with respect to a real-world scene. Thus, multiple related surface data frames may be sequenced together to produce a video-like representation (representing not only color data but also depth data) of the real-world scene from a particular view. . In certain examples, a surface data frame may also be associated with: other types of data (e.g., audio data, metadata (e.g., capture data describing the view from which the surface data frame is captured) Metadata, including a set of parameters, information about a particular real-world object represented in a surface data frame, etc.)), and/or other types of data that can contribute to a particular implementation. As described and illustrated below, such a sequence of surface data frames may be referred to herein as a "surface data frame sequence."

As used herein, "color data" can broadly include any image data, video data, etc., whether represented in color or represented in grayscale (i.e., "black and white"). Regardless, these data may represent the appearance of objects (e.g., real-world objects contained within a real-world scene) from the field of view of a particular view over a particular point in time or over a particular period of time. . Color data is not limited to: any particular format, file type, frame rate, resolution, quality level, or various definitions and/or image data and/or video data in the art. Other characteristics relevant to the standard being defined. Similarly, as used herein, "depth data" can include: any data representing the position and/or geometry of an object in space. For example, depth data representing a real-world object can include: a coordinate system of different points on the surface of the real-world object (e.g., a coordinate system associated with a particular capture device, a coordinate system associated with a real-world scene, etc.) coordinates with respect to the global coordinate system).

Similar to capture device 302, said device captures color and depth frames from view 206-1, but it should be understood that other capture devices capture other views 206 (e.g. views 206 - 2 through 206 - 8 ), and similarly color and depth frames can be captured from each vantage point associated with the other views 206 . In some examples, the surface data frames may be captured at the same particular point in time with different capture devices associated with different views 206 so that they can be synchronized with each other. As used herein, a surface data frame may be said to be captured "at the same particular point in time" if: the surface data frame is captured at an instant in time ( That is, if captured within a time close enough to be able to effectively represent an object (e.g., a real-world object in a real-world scene), as opposed to representing the object over a range of time ( (even if the surface data frames were not captured at exactly the same instant). For example, depending on how dynamic a particular object is (e.g., how quickly one or more real-world objects move within a real-world scene, etc.), a surface data frame can be: may be considered to be captured at the same particular point in time: for example, if they are captured within several tens or hundreds of milliseconds of each other, or if they contribute to a particular implementation. if captured within another suitable timeframe that can be used (e.g., within microseconds, milliseconds, seconds, etc.). Thus, each surface data frame captures surface color and depth data for real-world objects contained within the real-world scene from each vantage point of view 206 to which each capture device is associated at a particular point in time. It can be expressed as a surface appearance.

3B-3C show exemplary graphical depictions of data captured by capture device 302 and contained within color and depth frames (ie, within surface data frames). Specifically, as shown, the color frame embedded in the surface data frame can include color data 308 (shown in FIG. 3B), while the depth frame embedded in the surface data frame includes depth Data 310 may be included (shown in FIG. 3C).

In FIG. 3B, color data 308 depicts a real-world scene 202 (eg, including real-world object 204), said real-world scene 202 being a 2D color image in capture device 302 from view 206-1. Viewed by capture device 304 . Since color data 308 can represent a single video frame in a sequence of video frames, the depiction of real-world object 204 represented by color data 308 can represent: What appearance 204 (eg, as well as other objects associated with real-world scene 202) would have at a particular point in time from the vantage point of view 206-1. Although shown as an image in FIG. 3B, it should be understood that the color data 308 may be captured, encoded, formatted, transmitted, and represented in any suitable form. For example, color data 308 may be digital data formatted according to standard video encoding protocols, standard image formats, or the like. In some examples, color data 308 may represent a color image of objects in real-world scene 202 (eg, analogous to a color photograph). Alternatively, in another example, color data 308 may be a grayscale image representing the object (eg, similar to a black and white photograph).

In FIG. 3C, depth data 310 (as well as color data 308) also depict real-world scene 202 (including real-world objects 204) from the field of view of view 206-1. However, rather than representing the visible appearance of objects in the real-world scene 202 (e.g., representing in color or grayscale how light interacts with the surface of the real-world object 204), Depth data 310 may represent: for example, the surface of an object (eg, real-world object 204 as well as other objects in real-world scene 202) relative to depth capture device 306 in capture device 302; Depth (i.e. distance or position) of each point above. As with color data 308, depth data 310 may be captured, encoded, formatted, transmitted, and represented in any suitable form. For example, as shown, depth data 310 may be represented by grayscale image data (eg, 6 bits or 8 bits for each pixel captured by depth capture device 306). However, rather than representing how light reflects off the surface of real-world object 204 (i.e., represented by color data 308), the grayscale image of depth data 310 represents: May: For each pixel in the image, how far away from the depth capture device 306 the point represented by that pixel is. For example, points closer to the depth capture device 306 may be represented by values representing darker shades of gray (e.g., 0b111111 where 0b111111 represents black in the case of a 6-bit implementation). binary value close to ). Conversely, points further away from the depth capture device 306 may be represented by values representing lighter shades of gray (e.g., 0b000000 represents white in the case of a 6-bit implementation). in a binary value close to 0b000000).

In certain examples, system 100 (e.g., communication facility 102) communicates with capture device 302 and other capture devices associated with other views 206, one or more networks and/or any other suitable communication interface. , protocols, and techniques. Accordingly, in such examples, communication facility 102 may receive captured surface data frame sequences directly from capture devices by means of one or more networks and/or other communication interfaces, protocols, and technologies. good. In other examples, a real-world scene capture system separate from system 100 may be communicatively connected to each capture device and configured to perform the following actions: Managing the capture of data frames by each capture device and providing the surface data frame sequences to the system 100 (e.g., synchronizing and/or processing the surface data frame sequences). after). Regardless, communication between capture devices, system 100, and/or intermediary real-world scene capture systems may be implemented by the following means: by network means (e.g., wired or wireless local area network, wide area network, provider network, Internet, etc.), by means of a wired communication interface (e.g., Universal Serial Bus (“USB”)), by means of a wireless communication interface, or in certain implementations By means of any other communication interface, protocol, and/or technology that can contribute.

In other examples, multiple capture devices may be integrated within system 100 or included as part of system 100 (e.g., surface data frame sequence management facility 104 of system 100). or as part of another facility). Thus, in such an example, the surface data frame sequence management facility 104 receives the surface data frame sequence by capturing the surface data frame sequence using an integrated capture device. be able to.

4A-4B illustrate an exemplary surface data frame sequence 400-1, which is a real-world scene 202 (eg, view 206-1). Specifically, FIG. 4A shows a detailed graphical view of the surface data frame sequence 400-1, which is a specific view of the surface data frame sequence 400-1 that can be included in the surface data frame sequence 400-1. to depict the data of FIG. 4B, on the other hand, shows a consolidated graphical view of surface data frame sequence 400-1, which specifies many details about the contents of surface data frame sequence 400-1. It is not meant to be descriptive.

As shown in FIG. 4A, surface data frame sequence 400-1 can include various types of data, including color data, depth data, and metadata. Specifically, surface data frame sequence 400-1 is shown to include a color frame sequence 402, a depth frame sequence 404, and a set of capture parameters 406. As shown in FIG. It should be appreciated that surface data frame sequence 400-1 may also include other types of data not explicitly shown in FIG. , other metadata, etc.). Additionally, it should be appreciated that the data contained within surface data frame sequence 400-1 may be arranged or formatted in any suitable manner. For example, as shown, the data contained within surface data frame sequence 400-1 may be arranged as one color frame sequence and one depth frame sequence. In other examples, a single capture device can output multiple color frame sequences and/or multiple depth frame sequences (e.g., for different portions of the field of view of the captured real-world scene). In order to cover). In yet another example, the data in surface data frame sequence 400-1 may be arranged as a sequence of surface data frames to be integrated, each frame representing a particular color frame, a particular depth frame, and may include specific metadata (eg, data representing the set of capture parameters 406), or may be arranged in other ways that may contribute to specific implementations.

The data contained within each chromatic frame of chromatic frame sequence 402 may be similar to chromatic data 308 described above in connection with FIG. However, each chromatic frame in chromatic frame sequence 402 may be captured at a slightly different time, such that chromatic frame sequence 402 is a video-like representation of real-world scene 202 from view 206-1. can be formed. Similarly, the data contained within each depth frame of depth frame sequence 404 may be similar to depth data 310, except that each depth frame within depth frame sequence 404 may be slightly It may be captured at different times (eg, synchronized with the time the color frames of color frame sequence 402 are captured), such that depth frame sequence 404 is a real-world scene from view 206-1. Another video-like representation of 202 may be formed.

The set of capture parameters 406 included within surface data frame sequence 400-1 may include: the view that captures surface data frame sequence 400-1 (ie, the case of view 206-1 in ). For example, the set of capture parameters 406 may include: various parameters regarding where and/or how the surface data frames contained within surface data frame sequence 400-1 are captured; Various parameters indicating aspects. Capture parameters included within set of capture parameters 406 may include: any suitable capture parameters associated with each view of a real-world scene that can contribute to a particular implementation.

For example, set of capture parameters 406 may include capture parameters that represent: a location with respect to real-world scene 202 at which color and depth frames corresponding to view 206-1 of real-world scene 202 are captured; As another example, set of capture parameters 406 may include capture parameters that represent: an orientation in which color and depth frames corresponding to view 206-1 of real-world scene 202 are captured (eg, capture • Capture orientations associated with different angles in different dimensions the device points to). Similarly, as another example, set of capture parameters 406 may include capture parameters that represent: a field of view that captures color and depth frames corresponding to view 206-1 of real-world scene 202; Furthermore, as yet another example, set of capture parameters 406 may include capture parameters representing: an image that captures color and depth frames corresponding to view 206-1 of real-world scene 202; ·quality. In yet another example, the set of capture parameters 406 can include: any representing other ways in which color and depth frames corresponding to view 206-1 of real-world scene 202 can be captured; other appropriate capture parameters for . For example, set of capture parameters 406 may include parameters representing: a depth mapping and/or depth range for capturing depth frames corresponding to view 206-1; parameters describing the particular encoding, format, frame rate, dynamic range, etc., with which to capture the depth frames; identification information about the source of the capture (e.g., the capture device capturing the color and depth frames corresponding to view 206-1); ); or other suitable parameters.

The set of capture parameters 406 may be represented by, and integrated with, other data contained within surface data frame sequence 400-1 in any manner that may contribute to a particular implementation. may For example, in some implementations the capture parameters 406 may be explicitly expressed in data (eg, variables, etc.) representing the capture parameters, while in other implementations the capture parameters 406 may be format, where orientation, location, and/or projection information (e.g., field of view and depth mapping for a frame sequence, left/right/ up/down/myopia/hyperopia, etc.) may be represented. Data representing particular capture parameters 406 may be combined into a single abstract matrix (eg, a 4×4 matrix) that represents a complete transformation from a particular image space to homogeneous coordinates in world space, for example. good too. Thus, in such examples, individual components may not be explicitly specified, but rather may be included within a more general transformation.

Further, in some examples, the set of capture parameters 406 may be integrated into each chroma frame and/or depth frame included in chroma frame sequence 402 and depth frame sequence 404, respectively (e.g., may be repeated for these). In other examples, the set of capture parameters 406 may be integrated into each individual surface data frame (eg, a combination of color and depth frames). In such a manner, the set of capture parameters 406 can flexibly describe the capture parameters for each and every frame (eg, the time period over which the view 206 is represented by the surface data frame sequence 400-1). inside, even if it changes dynamically). In another example, the set of capture parameters 406 may be static over the time period represented by surface data frame sequence 400-1. In these examples, the set of capture parameters 406 may be transmitted separately from the frames of frame sequences 402 and 404 . For example, the set of capture parameters 406 may be transmitted separately from the transmission of the color and depth frames (e.g., before transmission of the color and depth frames, at the start of transmission of the color and depth frames, at the start of transmission of the color and depth frames). and/or at another suitable time).

As mentioned above, FIG. 4B shows an integrated graphical view of the surface data frame sequence 400-1. Specifically, the view of surface data frame sequence 400-1 of FIG. 4B displays real-world scene 202 (ie, real-world objects) in front of blocks viewed from a particular view (ie, view 206-1). 204) is shown as a block of the surface data frame sequence 400-1. This type of surface data frame sequence view will be useful for showing additional surface data frame sequences in later figures. However, it should be appreciated that any surface data frame sequence represented using an integrated graphical view such as that shown in FIG. 4B can include: All of the same types of data shown and/or described in relation to FIG. 4A in any one.

One or more surface data frame sequences 400 (eg, surface data frame sequence 400-1 explicitly shown in FIG. 4, and other similar surface data frame sequences not explicitly shown in FIG. 4 (eg, view 206 -2, surface data frame sequence 400-3 corresponding to view 206-3, and so on), system 100 performs customization of real-world scene 202. Color and depth frames can be rendered for the virtualized projection of the viewed view. For example, system 100 can identify an additional set of capture parameters (eg, a set different from the set of capture parameters associated with surface data frame sequence 400), wherein the additional set is It may also relate to a customized view of world scene 202 (eg, a different view than view 206 shown in FIG. 2). The system 100 then renders color and depth frames for the customized view virtualization projection based on at least one of the surface data frame sequences 400 and based on an additional set of capture parameters. can be rendered.

As used herein, a “customized view” of a real-world scene is any view of the real-world scene that differs from the view associated with a physical capture device that captures data representing the real-world scene. may mean. For example, a customized view may be customized in terms of: a location within the real-world scene near which a particular real-world object is located (e.g., to improve depth resolution or depth accuracy for a real-world object); a location in the real-world scene where the capture device is not located; an orientation different than any view associated with the capture device can provide; any associated with the capture device (e.g., a different zoom level, a field of view associated with a wide-angle or narrow-angle lens, etc.) than the view of the capture device can provide; different level of detail (eg, image resolution, etc.), etc. Thus, as used herein, a "virtualized projection" of a customized view of a real-world scene refers to the projection associated with the customized view (e.g., perspective projection, orthographic projection, etc.). It may mean the data it represents. For example, in certain instances, a virtualized projection may be configured such that when such a capture device is associated with a customized view (i.e., the capture device captures data using a set of capture parameters that define the customized view). If so), it can include a field of view projection that virtually simulates the data that a physical capture device would capture. As another example, virtualization projections can include non-view projections (e.g., orthographic projections, etc.), but said non-view projections are not generated by simulation of the virtual capture device, but rather depth data may be produced by a deep ablation technique to produce a , or other suitable techniques that may be amenable to certain implementations.

As noted above, customized view virtualization projections can provide: new perspectives on aspects of real-world scenes, greater flexibility with respect to improved depth resolution, and, without virtualization projections, Various other benefits not available. However, it should be understood that the virtualization projection may be based on data captured by a physical capture device, and thus the virtualization projection may be based on the data captured by the physical capture device. You do not have to provide any additional data. For example, a virtualized projection may relate to a customized view of a real-world scene in which a physical capture device is not located, while a virtualized projection may relate to a view in which a physical capture device is located. It may not provide any new information that has not yet been obtained.

In certain examples, the customized view of real-world scene 202 can be aligned with the particular capture device used to capture data representing real-world scene 202 . For example, an additional set of capture parameters associated with (eg, defining) a customized view of the real-world scene 202 may include: request data captured by only one capture device ( one or more capture parameters, e.g., requesting a subset of the data captured by the capture device;

For example, additional sets of capture parameters may include capture parameters that represent: a customized field of view associated with a customized view of real-world scene 202, where the customized field of view is the captured • May be narrower than the captured field of view associated with the surface data frame sequence captured by the device. For example, an additional set of capture device parameters may request a cropped portion (ie, zoom in) of data captured by a particular physical capture device.

As another example of capture parameters that require data to be captured by only one capture device, an additional set of capture parameters may include capture parameters that represent: A customized image quality associated with the view, where the customized image quality is lower than the captured image quality associated with the surface data frame sequence captured by the capture device. For example, additional sets of capture parameters may request lower resolution versions of the data captured by a particular physical capture device.

In other examples, the customized view of real-world scene 202 may be unaligned with: multiple capture devices positioned at different locations with respect to real-world scene 202 (i.e. Different views of the real-world scene 202 captured by the associated capture device). Thus, rendering color and depth frames for a virtualized projection for a customized view of the real-world scene 202 can include: rendering color and depth frames into at least two surface data frame sequences 400; Rendering based on Similar to the examples above where the virtualization projection is based on data from a single capture device, additional sets of capture parameters in these examples can include: Capture parameters that require a narrower field of view, lower image quality, etc. However, additional sets of capture parameters in examples where the virtualization projection is based on data from multiple capture devices may further require customized locations, customized orientations, etc., which indicate that the data is May differ from any actual location, orientation, etc. as captured by a physical capture device.

For illustrative purposes, FIG. 5 shows an exemplary configuration 500 based on configuration 200 (ie, including all the same elements shown in relation to configuration 200 and described above), but now with a real-world scene Data representing 202 is additionally generated for customized view 502 of real-world scene 202 . Specifically, as shown in configuration 500 , customized view 502 can be positioned within real-world scene 202 near real-world object 204 with an orientation and field of view toward real-world object 204 . As noted above, a particular customized view may align with one of views 206 (eg, be co-located with one of views 206, provide the same orientation and/or field of view as one of views 206, etc.). However, view 502 in configuration 500 is shown misaligned with view 206 . As used herein, a view can be said to be "ragged" with another view if the views differ from each other in: each location associated with the view, each orientation associated with the view, and /or each field of view associated with the view. In the case of customized view 502 and view 206, for example, customized view 502 is not aligned with all views 206 because customized view 502 is located at a different location than any view 206 ( (i.e., a location inside the real-world scene 202), has a different orientation than any of the views 206 (i.e., an orientation between the orientations of views 206-1 and 206-2), and , has a different field of view than view 206 (ie, a field of view that provides a closer view on real-world object 204).

Thus, based on specifying an additional set of capture parameters that define the customized view 502 and based on the plurality of surface data frame sequences 400 captured by the capture device associated with the view 206 (e.g. , based on surface data frame sequences 400-1 and 400-2), system 100 can render color and depth frames for a virtualized projection of customized view 502 of real-world scene 202, A virtualized surface data frame sequence containing these color and depth frames can then be provided.

In an illustrative sense, FIG. 6 shows an exemplary virtualized surface data frame sequence 600 that may be generated by system 100 for a virtualized projection of customized view 502 . Well, and may be offered. Specifically, a virtualized surface data frame sequence 600 may be generated to include rendered color and depth frames for the virtualized projection 602 of the customized view 502 . As the virtualization projection 602 shows, the color and depth frames contained within the virtualized surface data frame sequence 600 may relate to (e.g., appearance as captured from the perspective of ): a different location, a different orientation, and a different field of view than any color and depth frames contained within the surface data frame sequence 400 .

Specifically, as shown, virtualized projection 602 represents a close-up from a particular orientation on real-world object 204 . This can provide various advantages with respect to downstream systems and devices, which can provide virtual reality media content, which can be viewed in virtual projection 602 can represent the real-world object 204 from a location, orientation, and/or view provided by . For example, processing resources (e.g., video codec solutions), depth quantization issues (e.g., undesirable "layering" of depth data representing relatively large areas) are more It can be mitigated by using depth data representing the localized area. A localized area can have a shorter depth than a larger (eg, less localized) area represented by the available bits. Thus, the finite number of bits available allows depth to be represented with high precision, such as to reduce or eliminate layering effects. Thus, the bit depth representing the surface of the object (eg, a certain number of bits used to represent depth at different distances from the vantage point) can be optimized with respect to the virtualization projection 602, resulting in a virtual A high level of depth accuracy and/or depth resolution from the vantage point of the projection 602 can be provided.

While certain additional sets of capture parameters associated with particular customized views (ie, customized view 502) have been described and illustrated in detail with respect to FIGS. want: a vast number (e.g., significantly greater than the number of physical capture devices employed to capture data representing the real-world scene 202) of the capture parameters associated with the customized views. A vast number of sets can be specified in a particular implementation, potentially generating and providing a large number of virtualized surface data frame sequences for inclusion in virtual reality media content. can be made possible. As noted above, this vast number of virtualized surface data frame sequences can allow for greater flexibility in the generation and delivery of virtual reality media content, allowing different media - Player devices can be presented with different details about the same real-world scene, while burdening any media player device with vast amounts of redundant or relatively irrelevant data. I don't even call.

One or more additional sets of capture parameters (i.e., sets of capture parameters other than the set of capture parameters associated with view 206) may be specified in any manner that can contribute to a particular implementation. can. For example, in certain implementations, an additional set of capture parameters (eg, a set of capture parameters associated with customized view 502) can be specified by: one or more geometries of the real-world scene; analyzing the real-world scene 202 with respect to physical properties; generating multiple additional sets of capture parameters that differ from the set of capture parameters associated with the view 206 (e.g., based on analysis of the real-world scene); and identifying an additional set of capture parameters from the plurality of additional sets of capture parameters. Specifically, for example, system 100 can determine geometric properties of real-world scene 202 (e.g., properties related to: shape of real-world scene 202, division of real-world scene 202 various parts or aspects of the real-world scene 202 when possible, the location and/or trajectory of a particular object within the real-world scene 202, etc.). Based on these properties and/or other properties, system 100 determines that various customized views (eg, including customized view 502) are relevant to generating virtual reality media content. and, as a result, a respective set of capture parameters can be generated for each of these associated customized views.

Once a plurality of captured surface data frame sequences are received for a view of the real world scene associated with the physical capture device, and one or more virtualized surface data frame sequences is generated for a virtualized projection of a customized view of a real-world scene, the system 100 converts one or more captured and/or virtualized surface data frame sequences into a virtual reality • Can be provided for inclusion within media content. For example, as described in more detail below, the system 100 can provide data in captured and/or virtualized surface data frame sequences to: a server-side system ( and/or client-side systems (e.g., virtual reality media content (e.g., data contained within surface data frame sequences); A media player device) associated with a user experiencing virtual reality media content based on).

System 100 uses data (e.g., color and depth frames, as well as other types of data (e.g., audio data, metadata, etc.)) contained within surface data frame sequences to contribute to a particular implementation. can be provided in any manner and to any other system or device. For example, in certain implementations, system 100 can provide color and data frames (as well as audio and metadata, etc.) to an encoding system, which encodes the data into video data, A stream can be generated (eg, a 2D video stream compressed in a standardized format (eg, H.264, H.265), etc.). Thus, for example, data contained within a particular surface data frame sequence may be contained within one or more video data streams (eg, color video data stream, depth video data stream, etc.). good too. Other data contained within the surface data frame sequence (e.g., audio data, metadata, etc.) may also be contained within the color video data stream and/or the depth video data stream. , or may be included in different data streams.

Regardless of which system the surface data frame sequence data is provided to and/or whether the surface data frame sequence data is encoded into one or more video data streams, etc. Sequence data may be packaged and/or multiplexed for transmission over a network. Such data packaging may be performed in any suitable manner and/or using any suitable data structure that can contribute to a particular implementation. As an example, each surface data frame sequence may be packaged into its own unique transport stream. Specifically, for example, the virtualized surface data frame sequence 600 may be packaged such that the rendered color rendering for the virtualized projection 602 of the customized view 502 of the real-world scene 202 is and depth frames may be included within a transport stream, which may not include: additional surface data frames other than the virtualized surface data frame sequence 600; Color and depth frames representing a frame sequence (eg, an additional captured or virtualized surface data frame sequence).

As another example, multiple surface data frame sequences may be packaged together (eg, multiplexed) into a shared transport stream. Specifically, for example, the virtualized surface data frame sequence 600 may be packaged such that the rendered color rendering for the virtualized projection 602 of the customized view 502 of the real-world scene 202 is and depth frames are contained within a transport stream, said transport stream further comprising color and depth frames representing: at least one surface data frame sequence other than the virtualized surface data frame sequence 600; Additional surface data frame sequences (eg, at least one additional captured or virtualized surface data frame sequence).

For illustrative purposes, FIG. 7 shows a graphical representation of an exemplary transport stream 700, said transport stream comprising an exemplary plurality of surface data frame sequences. Specifically, transport stream 700 may include: various captured surface data frame sequences 400 (eg, surface data frame sequence 400-1 shown in FIGS. 4A-4B; and surface data frame sequences 400-2 through 400-8) that are similarly captured by the capture devices associated with views 206-2 through 206-8, respectively, and various virtualized surface data frame sequences 400-2 through 400-8). sequences (eg, virtualized surface data frame sequence 600 shown in FIG. 6 and other virtualized surface data frame sequences 702-1 through 702-N).

As used herein, "transport stream" may mean a data structure used to package data for the purpose of: transferring data from one device or system to another; facilitating transmission (ie, transmission) to a location, rendering or processing or analyzing data, and/or other purposes that may contribute to a particular implementation. In some examples, a transport stream may incorporate one or more data streams (eg, one or more video data streams) and/or other data (eg, metadata, etc.). . A transport stream may be implemented as any type of transport stream that can contribute to a particular implementation. For example, a particular transport stream described herein (eg, transport stream 700) may be implemented by: an MPEG transport stream, an MPEG-2 transport stream, or , another suitable data structure that facilitates the transmission of data (eg, surface data frame sequence, video data stream), etc. A transport stream may be constructed according to any suitable data format, container format, and/or transmission protocol that can contribute to a particular implementation.

Although transport stream 700 is shown as including both captured surface data frame sequences and virtualized surface data frame sequences, it should be understood that: In implementations, transport stream 700 can include: only captured surface data frame sequences or only virtualized surface data frame sequences. Additionally, transport stream 700 may include: any suitable number of surface data frame sequences and any combination of surface data frame sequences that may contribute to a particular implementation. . For example, as described above, in certain examples, transport stream 700 may include a single surface data frame sequence (eg, virtualized surface data frame sequence 600), and Other surface data frame sequences, if at all, may be transmitted by means of other transport streams. It should also be appreciated that FIG. 7 shows a transport stream 700 that includes the surface data frame sequences described and illustrated above (eg, surface data frame sequences 400 and 600). However, these surface data frame sequences may be included in a data structure (e.g., an encoded video data stream (not explicitly shown)), so these surface data frame sequences are , may refer to versions of data that are different from the versions described above that system 100 receives and/or generates (eg, versions of data that are encoded and/or compressed into a video data stream, etc.).

FIG. 8 shows a data structure representation 800 of transport stream 700 . As shown, representation 800 includes sections for different types of data (eg, a section for metadata 802, a section for audio data 804, and a section for video data 806). It should be appreciated that the section shown in representation 800 may be conceptual only, and the data shown in representation 800 may be transported in any way that can contribute to a particular implementation. • may be multiplexed, organized, arranged, transmitted, etc. within the stream 700;

As shown, metadata 802 includes: various sets of capture parameters associated with each surface data frame sequence contained within transport stream 700 (i.e., "capture parameter set 1"); ~"capture parameter set M"). For example, the set of capture parameters contained within metadata 802 may include: the surface data frame sequence to be captured and virtualized shown in FIG. 1 through 400-8, 600, and 702-1 through 702-N) for each set of capture parameters. Metadata 802 is any other metadata that describes the surface data frame sequence (eg, or the video data stream in which the surface data frame sequence is encoded) that can contribute to a particular implementation. can further include

Similarly, audio data 804 may include: audio source data associated with each surface data frame sequence contained within transport stream 700; For example, "Audio Source 1" through "Audio Source M" may relate to surface data frame sequences 400-1 through 400-8, 600, and 702-1 through 702-N, respectively. In other examples, there may be more or less audio sources than there are surface data frame sequences if the audio sources are not associated with the surface data frame sequence (e.g. The number of audio sources is independent of the number of surface data frame sequences).

As further shown in FIG. 8, video data 806 may include: color video data associated with each surface data frame sequence shown in FIG. 7 as included within transport stream 700; data stream and depth video data stream. For example, "color video data stream 1" and "depth video data stream 1" may represent the color and depth frames contained within surface data frame sequence 400-1, and "color video data stream 1" may represent the color and depth frames contained within surface data frame sequence 400-1. Data stream 2" and "depth video data stream 2" may represent color and depth frames contained within surface data frame sequence 400-2, and so on, resulting in , surface data frame sequences 400-1 through 400-8, 600, and 702-1 through 702-N, respectively, both chrominance and depth video data streams in video data 806. corresponds to

As noted above, in certain implementations, it may be useful to: provide a relatively large number of surface data frame sequences to be virtualized to allow different Allowing flexibility in generating different versions of virtual reality media content that can be customized with different details associated with the media player device (i.e., associated with different users with different virtual reality experiences). . For example, in one implementation, eight physical capture devices can generate eight high-resolution captured surface data frame sequences, and system 100 can capture eight captured surface data frames. Based on the data frame sequences, the following can be generated: Hundreds of virtualized surface data frame sequences for hundreds of virtualized projections of hundreds of customized views.

Providing such a large number of virtualized surface data frame sequences may allow significant flexibility in efficiently generating and delivering virtual reality media content. but with available hardware and software resources that may not be equipped to handle too many individual data streams (e.g., video data streams, etc.) can be difficult. As a result, it may be desirable to package multiple color and/or depth frame sequences into a single surface data frame sequence. For example, the virtualized surface data frame sequence 600 may be packaged such that the rendered color and depth frames for the virtualized projection 602 of the customized view 502 of the real-world scene 202 are: each represented as a tile in a video data stream, said video data stream implementing a tile map, said tile map representing a plurality of tiles in each frame of said video data stream do. For example, tile map ping techniques (e.g., texture atlas techniques, sprite sheet techniques, etc.) can be used to pack multiple color and/or depth frames together into a single frame, resulting in A sequence of such frames (e.g., or a video data stream representing these frames) can essentially be treated as a single frame sequence, but can be treated as multiple frame sequences (e.g., for customized views). (representing multiple views, including virtualization projections).

For illustrative purposes, FIG. 9 shows a graphical representation of an exemplary transport stream 900, said graphical representation comprising an exemplary tiled frame sequence 902, said sequence comprising tiled frames. implement a map; The tiled frame sequence 902 is a surface data frame sequence shown herein (eg, surface data frame sequence 400-1 in FIG. 4B, virtualized surface data frame sequence 400-1 in FIG. 6). 600), but it should be understood that the tiled frame sequence 902 contains different data than some of the other surface data frame sequences herein. can be expressed. Specifically, for example, tiled frame sequence 902 may not represent multiple frame sequences (eg, color frame sequence 402 and depth frame sequence 404 (shown in FIG. 4A)). But rather, the tiled frame sequence 902 can include: a plurality of tiles 904 (ie, tiles 904-1-C through 904-9-C) (eg, a tiled frame sequence A single sequence of frames each including the one shown on the front of sequence 902).

The tiles included on each frame of the tiled frame sequence 902 can include: any captured or virtualized surface data frame that can contribute to a particular implementation; Any color or depth frame associated with the sequence. For example, as shown in FIG. 9, each frame of the tiled frame sequence 902 can include: a tile 904- corresponding to the color (“C”) frame captured from view 206-1. 1-C, tiles 904-2-C corresponding to color frames captured from view 206-2, and so on, up to tiles 904-8-C corresponding to color frames captured from view 208-8. . As further shown, each frame of the tiled frame sequence 902 can include: a tile 904-9-C associated with the color frame generated for the virtualized projection 602 of the customized view 502; Although FIG. 9 explicitly shows only nine tiles, it should be appreciated that additional tiles may also be packed into each frame of the tiled frame sequence 902 . For example, tiles corresponding to other virtualization projections, depth frames (eg, depth frames captured from view 206 and/or generated with respect to the virtualization projection), etc. can contribute to a particular implementation. It may also be included within the tile map. Furthermore, although transport stream 900 in FIG. 9 only shows one tiled frame sequence using a tile map, it should be understood that: Stream 900 can be used to package multiple frame sequences, which may be tiled frame sequences (eg, tiled frame sequence 902, etc.), surface data It may contain a frame sequence (eg, frame sequence 400, 600 or 702, etc.) or other data that may contribute to a particular implementation. In some examples, the tiled frame sequence 902 may be sent without being included within a transport stream (eg, transport stream 900).

FIG. 10 shows a data structure representation 1000 of transport stream 900 . As shown, similar to representation 800, representation 1000 includes sections for different types of data (eg, a section for metadata 1002, a section for audio data 1004, and a section for video data 1006). Also, as with representation 800, it should be understood that the sections shown in representation 1000 are conceptual only, and the data shown in representation 1000 may be transported in any manner that may contribute to a particular implementation. - may be multiplexed, organized, arranged, transmitted, etc. within the stream 900;

As shown, metadata 1002 includes: two different types of metadata for each tile (eg, tiles 904-1-C through 904-9-C and/or other tiles not explicitly shown in FIG. 9); . Specifically, for each tile (“Tile 1” through “Tile M”), metadata 1002 includes: tile coordinates representing a section of each frame dedicated to data associated with that particular tile (e.g., “ Tile Coordinate 1” to “Tile Coordinate M”). For example, the tile coordinates for tile 1 may include coordinates indicating the upper left corner of the frame when tile 904-1-C is shown in FIG. 9, and so on. Metadata 1002 also includes, for each tile 1-M, a respective set of capture parameters associated with the tile (ie, "capture parameter set 1" through "capture parameter set M"). For example, a set of capture parameters included within metadata 802 may include: a set of capture parameters for each tile shown in FIG. each set. Metadata 1002 may further include: the tiles of the tiled frame sequence 902, which may contribute to a particular implementation (eg, or the tiles in which the tiled frame sequence 902 is encoded); any other metadata describing the video data stream).

Similar to audio data 804 in representation 800, audio data 1004 may include: for each tile contained within tiled frame sequence 902 (i.e., within transport stream 900) Associated audio source data. For example, "Audio Source 1" through "Audio Source M" may each be associated with tiles 904-1-C through 904-9-C and/or other tiles not explicitly shown in FIG. In other examples, there may be more or less audio sources if the tiles are not specifically associated with the audio sources.

In contrast to representation 800 in FIG. 8, where multiple color and depth video streams are included associated with each surface data frame sequence, representation 1000 shows that video data 1006 is tiled. contains only one video data stream associated with the sequence of frames 902 to be processed. For this reason, as shown in FIG. 9, all images associated with each color and/or depth frame represented by tiled frame sequence 902 are assigned to each frame in tiled frame sequence 902. because they are packed together. In a particular example, transport stream 900 may include: one frame sequence dedicated to color data tiles (eg, tiled frame sequence 902), and one dedicated to depth data tiles. (not explicitly shown in FIG. 9). Thus, in such an example, video data 1006 may include: both a chroma video data stream and a depth video data stream. In yet another example, as discussed above, transport stream 900 can include: tiled frame sequence 902, along with other frame sequences, video streams, etc. (e.g., tile (whether or not it is encrypted). Thus, in such an example, video data 1006 may include: other video data streams not explicitly shown in FIG.

As noted above, in some examples, system 100 and/or other systems (e.g., other server-side systems) and devices described herein can be used to create a virtual - Capable of generating reality media content. For example, in addition to the operations described above, a virtual reality media content serving system (eg, system 100 and/or other devices and systems described herein may be included, or these systems may entity) can further generate and provide virtual reality media content based on data provided by system 100 . The virtual reality media content may represent a real-world scene (eg, real-world scene 202), and the virtual reality media content may be animation corresponding to any virtual location relative to the real-world scene. It may also be presentable to the user for experience from a visually selectable virtual viewpoint. For example, the dynamically selectable virtual viewpoint may be selected (eg, determined, positioned, etc.) by a user of the media player device while the user is using the media player device. We are virtually experiencing a real world scene. In some examples, a virtual viewpoint can be selected to be any location along a 2D or 3D continuum, which can only be selected from a discrete set of viewpoints. Contrast with state. Additionally, the virtual reality media content may be provided to the media player device (eg, by a virtual reality media content providing system that includes or is associated with system 100), and the user , allows a real-world scene to be experienced from dynamically selectable virtual viewpoints corresponding to any virtual location within the real-world scene.

For illustrative purposes, FIG. 11 shows an exemplary configuration 1100, in which an exemplary virtual reality media content providing system 1102 (“provider system 1102”) comprises system 100 and one or more additional , wherein the exemplary virtual reality media content providing system 1102 generates virtual reality media content and the virtual reality media provider pipeline system 1104 of Media content is provided by means of a network 1106 to an exemplary client-side media player device 1108 (“media player device 1108”), said exemplary client-side media player device 1108 , used by the user 1110 to experience the real-world scene 202 .

After the virtualized surface data frame sequence 600 is generated and packaged into a transport stream (eg, transport stream 700, transport stream 900, etc.), as described above, Provider system 1102 may further encode, package, encrypt, or otherwise process one or more transport streams to form virtual reality media content, said content comprising: Media player device 1108 may be configured to render. For example, virtual reality media content may include multiple 2D video data streams (eg, 2D video data streams associated with color and depth data associated with each view and virtualized projection). , or the plurality of 2D video data streams may be rendered by media player device 1108 such that any virtual viewpoint within real-world scene 202 (e.g., any A view of the real-world scene 202 from any view, including a virtual viewpoint that may be of interest to the user 1110, but not aligned with the capture device's view or a customized view, may be presented ( This will be discussed later). The virtual reality media content may then be distributed by means of network 1106 to one or more media player devices (eg, media player device 1108 associated with user 1110). For example, provider system 1102 can provide virtual reality media content to media player device 1108 such that user 1110 can use media player device 1108 to view real-world scene 202 . You can experience it virtually.

In some examples, the user 1110 is restricted to one or more discrete locations within the immersive virtual reality world represented by the virtual reality media content (eg, representing the real world scene 202). may not be desirable. Thus, the provider system 1102 provides sufficient data within the virtual reality media content representing the real-world scene 202 to enable the real-world scene 202 to be represented, wherein the representation is , views 206 and/or 502 , but also from any dynamically selectable virtual viewpoint corresponding to any virtual location within real-world scene 202 . For example, a dynamically selectable virtual viewpoint can be selected by user 1110 while user 1110 is using media player device 1108 to experience real-world scene 202 .

As used herein, "any virtual location" may mean any virtual point in space associated with the representation of a real-world scene. For example, any virtual location may be defined by view 206 and customized view 502, rather than being limited to fixed positions surrounding the real-world scene (eg, fixed positions relative to view 206 and/or customized view 502). All positions between the positions associated with 502 can also be included. In some examples, any such virtual location may correspond to the most desirable virtual viewpoint within the real-world scene 202 . For example, if real-world scene 202 includes a game of basketball, user 1110 can dynamically select a virtual viewpoint to experience the game at any virtual location on the basketball court. For example, the user can dynamically select the user's virtual viewpoint, follow the basketball court up and down, and view the images as if they were standing on the basketball court while the game was in progress. You can experience a game of basketball.

Networks 1106 can include: provider-specific wired or wireless networks (eg, cable or satellite carrier networks, or mobile phone networks), the Internet, wide area networks, content delivery. - A network, or any other suitable network. Data can flow between provider system 1102 and media player device 1108 (as well as other media player devices not explicitly stated), and in so doing any communication that can contribute to a particular implementation. May use technologies, devices, media, and protocols.

Media player device 1108 may be used by user 1110 to access and experience virtual reality media content received from provider system 1102 . For example, media player device 1108 may be configured to operate to generate a 3D virtual representation of real-world scene 202, which may be viewed by user 1110 from any virtual viewpoint (e.g., selected and experienced from a dynamically selectable virtual viewpoint corresponding to any virtual location within the real-world scene 202). To this end, the media player device 1108 may include or be implemented by any device capable of: creating an immersive virtual reality world (e.g., rendering the real world scene 202 and detecting user input from user 1110 as user 1110 experiences the immersive virtual reality world to create an immersive virtual reality world within the field of view. To dynamically update the reality world.

FIG. 12 illustrates various exemplary types of media player devices 1108 that may be used by users 1110 to experience virtual reality media content. Specifically, as shown, media player device 1108 can take one of several different form factors (e.g., head-mounted virtual reality device 1202 (e.g., virtual reality device 1202)). gaming devices) (including head-mounted display screens), personal computing devices 1204 (e.g., desktop computers, laptop computers, etc.), mobile or wireless devices 1206 (e.g., smartphones, tablets, etc.). devices, which can be attached to the head of the user 1110 by means of a head-mounted device) or contribute to certain implementations to facilitate the reception and/or presentation of virtual reality media content. Any other device or configuration of devices that can provide different levels of immersion The user 1110 can be provided with different types of virtual reality experiences having .

FIG. 13 illustrates an exemplary virtual reality experience 1300, where a user 1110 is presented with exemplary virtual reality media content, said content representing a real-world scene, said real-world The scene is experienced from dynamically selectable virtual viewpoints corresponding to any exemplary virtual location relative to the real-world scene. Specifically, virtual reality media content 1302 is presented within field of view 1304, said content showing a real-world scene from a virtual viewpoint, said virtual viewpoint Corresponding to a virtual location, said virtual location is directly below the basketball goal in a representation of the real world scene where the shot is being taken. An immersive virtual reality world 1306 based on a real-world scene may be available to the viewer and user input (e.g., head movements, keyboard input, etc.) (look around and/or move around (i.e., An immersive virtual reality world 1306 can be experienced by providing )) that dynamically selects a virtual viewpoint from the location of the experience.

For example, field of view 1304 can provide a window through which user 1110 can easily and naturally look around immersive virtual reality world 1306 . A field of view 1304 may be presented by media player device 1108 (eg, on a display screen of media player device 1108), and said field of view 1304 may include: immersive virtual reality; A video depicting objects in the user's surroundings in world 1306 . Further, field of view 1304 can dynamically change in response to user input provided by user 1110 as user 1110 experiences immersive virtual reality world 1306 . For example, media player device 1108 can detect user input (eg, move or rotate the display screen on which field of view 1304 is presented). Accordingly, field of view 1304 can present: Views from different virtual viewpoints or virtual locations at locations of different objects and/or objects viewed from previous virtual viewpoints or locations; object.

In FIG. 13, immersive virtual reality world 1306 is shown as a hemisphere, which indicates that user 1110 can look in any direction within immersive virtual reality world 1306. , wherein the directions are substantially forward, backward, leftward, rightward, and/or upward as viewed from a virtual viewpoint at a location under the basketball goal currently selected by the user 1110. is. In another example, the immersive virtual reality world 1306 can include a full 360° with an additional 180° of the sphere so that the user 1110 can also look downwards. Additionally, the user 1110 can move around to other locations within the immersive virtual reality world 1306 (i.e., dynamically select different dynamically selectable virtual viewpoints within the representation of the real-world scene). can do). For example, the user 1110 can select: a virtual viewpoint at half court, a virtual viewpoint from the free throw line facing the basketball goal, a virtual viewpoint stopped above the basketball goal. etc.

FIG. 14 illustrates an exemplary method 1400 for generating a virtualized projection of a customized view of a real-world scene for inclusion within virtual reality media content. Although FIG. 14 shows exemplary operations according to one embodiment, in other embodiments any of the operations shown in FIG. 14 may be omitted, added, reordered, and/or modified. good. One or more of the operations illustrated in FIG. 14 may be performed by: system 100, implementations of the system, and/or associated with system 100 (eg, communicatively coupled, cooperatively operating). ) other systems described above.

In operation 1402, the virtualization projection generation system can receive a plurality of captured surface data frame sequences, each surface data frame sequence can include color and depth frames, and the color and depth frames may depict the real-world scene according to respective sets of capture parameters, which may be included in multiple sets of capture parameters associated with different views of the real-world scene. In some examples, each surface data frame sequence in the plurality of captured surface data frame sequences may be captured by a different capture device in the plurality of capture devices, wherein the plurality of capture • Devices may be placed at different locations with respect to the real-world scene to capture different views of the real-world scene. Operation 1402 may be performed in any of the methods described herein.

In operation 1404, the virtualization projection generation system can identify an additional set of capture parameters, said additional set of capture parameters associated with the surface data frame sequence received and captured in operation 1402. • Different sets of capture parameters contained within multiple sets of parameters. In some examples, the additional set of capture parameters may relate to customized views of the real-world scene, wherein the customized views are different views of the real-world scene captured by multiple capture devices. It can be different from the view. Operation 1404 may be performed in any of the methods described herein.

At operation 1406, the virtualization projection generation system can render color and depth frames for the virtualization projection of the customized view of the real-world scene. For example, the virtualization projection generation system may generate a virtualization projection based on the surface data frame sequence received and captured in operation 1402 and based on the additional set of capture parameters specified in operation 1404. A color and depth frame can be rendered for . Operation 1406 may be performed in any of the methods described herein.

At operation 1408 , the virtualization projection generation system can provide a sequence of surface data frames to be virtualized, said surface data frame sequence being a customized representation of the real-world scene rendered at operation 1406 . Rendered color and depth frames for the virtualized projection of the view. In some examples, the virtualization projection generation system can provide a virtualized sequence of surface data frames to a media player device for inclusion within virtual reality media content. Operation 1408 may be performed in any of the methods described herein.

FIG. 15 illustrates an exemplary method 1500 for generating a virtualized projection of a customized view of a real-world scene for inclusion within virtual reality media content. Although FIG. 15 shows exemplary operations according to one embodiment, in other embodiments any of the operations shown in FIG. 15 may be omitted, added, reordered, and/or modified. good. One or more of the operations illustrated in FIG. 15 may be performed by: system 100, implementations of the system, and/or associated with system 100 (eg, communicatively coupled, cooperatively operating). ) other systems described above.

In operation 1502, the virtualization projection generation system can receive a plurality of captured surface data frame sequences, each surface data frame sequence can include color and depth frames, and the color and A depth frame may depict a real-world scene according to each set of capture parameters, said parameters being included in a first plurality of sets of capture parameters associated with different views of the real-world scene. good too. For example, each surface data frame sequence in the plurality of captured surface data frame sequences may be captured by a different capture device in the plurality of capture devices, the plurality of capture devices comprising: It may be placed at different locations with respect to the real-world scene to capture different views of the real-world scene. In particular examples, operation 1502 may be performed in real-time as events occur within the real-world scene. Operation 1502 may be performed in any of the methods described herein.

In operation 1504, the virtualization projection generation system can identify a second plurality of sets of capture parameters, said second plurality of sets included within the first plurality of sets of capture parameters. may differ from the set of capture parameters provided. For example, each set of capture parameters in the second plurality of sets of capture parameters may relate to: Each customized view of the real-world scene differs from a different view. In some examples, the second plurality of sets of capture parameters may include: a first set of capture parameters associated with the surface data frame sequence received and captured in operation 1502; A set of numbers greater than those contained within the set. Similar to operation 1502, in certain implementations, operation 1504 may be performed in real-time when an event occurs within the real-world scene. Operation 1504 may be performed in any of the methods described herein.

At operation 1506, the virtualization projection generation system may render color and depth frames for the virtualization projection of each customized view of the real-world scene, the customized view being the second of the capture parameters. It may also relate to the set of capture parameters in the plurality of sets identified in operation 1504 . In some examples, the virtualization projection generation system can render color and depth frames based on: a plurality of captured surface data frame sequences received at operation 1502; A second plurality of sets of specified capture parameters. Similar to operations 1502 and 1504, in certain implementations, operation 1506 may be performed in real-time when an event occurs within the real-world scene. Operation 1506 may be performed in any of the methods described herein.

At operation 1508, the virtualization projection generation system can provide a plurality of virtualized surface data frame sequences, the surface data frame sequences representing the real-world scene rendered at operation 1506. Color and depth frames for each customized view virtualization projection can be included. For example, a virtualization projection generation system may provide multiple virtualized surface data frame sequences for inclusion within virtual reality media content for a media player device. Similar to operations 1502-1506, operation 1508 may be performed in real-time when an event occurs within the real-world scene. Operation 1508 may be performed in any of the methods described herein.

In particular embodiments, one or more of the systems, components and/or processes described herein may be implemented and/or executed by one or more suitably configured computing devices. To this end, one or more of the systems and/or components described above may include or be implemented by: any computer hardware and/or one of the components described herein. Computer-implemented instructions (eg, software) recorded on at least one non-transitory computer-readable medium configured to perform the above processes. In particular, system components may be implemented by one physical computing device or may be implemented by multiple physical computing devices. Accordingly, system components may include any number of computing devices and may employ any number of computer operating systems.

In certain embodiments, one or more processes described herein may be implemented, at least in part, as instructions recorded on non-transitory computer-readable media and executable by one or more computing devices. good. Generally, a processor (e.g., microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., memory, etc.) and executes those instructions, thereby implementing one of the methods described herein. One or more processes are executed, including the above processes. Such instructions can be stored and/or transmitted using any of a variety of known computer-readable media.

Computer-readable media (also called processor-readable media) includes any non-transitory medium that contributes to providing data (eg, instructions) that can be read by a computer (eg, by a processor of the computer). Such a medium may take many forms, including but not limited to: non-volatile media and/or volatile media. Non-volatile media can include, for example, optical or magnetic disks and other fixed memory. Volatile media can include, for example, dynamic random access memory (“DRAM”), which typically constitutes the main memory. Common forms of computer-readable media include, for example, discs, hard disks, magnetic tapes, any other magnetic medium, compact disc read-only memory ("CD-ROM"), digital video discs. ("DVD"), any other optical medium, random access memory ("RAM"), programmable read only memory ("PROM"), electrically erasable programmable read only memory (“EPROM”), FLASH-EEPROM, any other memory chip or cartridge, or any other tangible computer-readable medium.

FIG. 16 illustrates an exemplary computing device 1600, which may be configured, among other things, to perform one or more of the processes described herein. As shown in FIG. 16, computing device 1600 can include: a communication interface 1602, a processor 1604, a storage device 1606, and input/output communicatively coupled via a communication infrastructure 1610 ( “I/O”) module 1608 . Although an exemplary computing device 1600 is shown in FIG. 16, the components shown in FIG. 16 are not meant to be limiting. Additional or alternative components may be used in other embodiments. The components of computing device 1600 shown in FIG. 16 are described in further detail below.

Communication interface 1602 may be configured to communicate with one or more computing devices. Examples of communication interface 1602 include, but are not limited to: wired network interface (eg, network interface card), wireless network interface (eg, wireless network interface card), Modems, audio/video connections, and any other suitable interfaces.

Processor 1604 generally represents any type or form of processing unit (e.g., a central processing unit and/or a graphics processing unit) and is capable of processing data or one of the methods described herein. It can interpret, perform and/or direct the execution of any of the above instructions, processes and/or operations. Processor 1604 can direct the execution of operations in accordance with one or more applications 1612 or other computer-readable instructions (eg, which may be stored in storage device 1606 or another computer-readable medium).

Storage device 1606 can include one or more data storage media, devices, or configurations, and employs any type, form, and combination of data storage media and/or devices. can do. For example, storage devices 1606 can include, but are not limited to: hard drives, network drives, flash drives, magnetic disks, optical disks, RAM, dynamic RAM, other non-volatile and/or or volatile data storage units, or combinations or subcombinations thereof. Electronic data, including data described herein, may be temporarily and/or permanently stored in storage device 1606 . For example, data representing one or more executable applications 1612 configured to direct processor 1604 to perform any of the operations described herein may be stored in storage device 1606. may be In some examples, data may be located in one or more databases that reside within storage device 1606 .

I/O modules 1608 may include one or more I/O modules, which may be configured to receive user input and provide user output. One or more I/O modules may be used to receive input for a single virtual reality experience. I/O module 1608 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I/O module 1608 can include hardware and/or software that captures user input and can include, but is not limited to: a keyboard or keypad, a touchscreen component. (eg, touchscreen display), receiver (eg, RF or infrared receiver), motion sensor, and/or one or more input buttons.

I/O module 1608 can include one or more devices for presenting output to a user, including but not limited to: a graphics engine, a display (e.g., display screen), one or more output drivers (eg, display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, I/O module 1608 is configured to provide graphical data to a display for presentation to a user. This graphical data may represent one or more graphical user interfaces and/or any other graphical content that can contribute to a particular implementation.

In some examples, any facility described herein may be implemented by or within one or more components of computing device 1600 . For example, one or more applications 1612 residing within storage device 1606 may be associated with communication facility 102, surface data frame sequence management facility 104, or virtualized projection generation facility 106 of system 100 (see FIG. 1). It may be configured to direct processor 1604 to perform one or more operations or functions. Similarly, storage facility 108 of system 100 may be implemented by or within storage device 1606 .

To the extent the above-described embodiments collect, store, and/or employ personal information provided by individuals, it should be understood that: , can be used. Further, such collection, storage, and use of information may be subject to individual consent to such activities (e.g., through well-known "opt-in" or "opt-out" processes appropriate to the circumstances and type of information). ). Storage and use of personal information may reflect the type of information in an appropriately secure manner, for example through various encryptions and anonymizations for particularly sensitive information.

In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, the following points should be apparent: Various modifications and changes may be made thereto, and further embodiments may be implemented, and such modifications and changes and implementations are subject to the following: possible without departing from the scope of the claimed invention. For example, specific features of one embodiment described herein may be combined or substituted with features of another embodiment described herein. Accordingly, the above description and drawings are to be regarded in an illustrative rather than a restrictive sense.
In one aspect, the present invention includes the following inventions.
(Invention 1)
Methods including:
a virtualization projection generation system receiving a plurality of captured surface data frame sequences, wherein each surface data frame sequence includes color and depth frames, the frames representing a real world scene; , according to each set of capture parameters, said parameters being included in a plurality of sets of capture parameters associated with different views of a real-world scene, said plurality of captured surface data frame sequences. is captured by a different capture device in a plurality of capture devices, the devices positioned at different locations with respect to the real-world scene to capture the different views of the real-world scene. is placed in;
identifying an additional set of capture parameters, wherein the additional set of capture parameters is one of the capture parameters included within the plurality of sets of capture parameters; different from a set and associated with a customized view of the real-world scene, the customized view being different from the different views of the real-world scene captured by the plurality of capture devices;
The virtualization projection generation system rendering color and depth frames for a virtualization projection of the customized view of the real-world scene based on at least one of: the plurality of captured surface data. a sequence of surface data frames within a sequence of frames and an additional set of said capture parameters;
said virtualization projection generating system providing a virtualized surface data frame sequence to a media player device for inclusion within virtual reality media content, wherein said virtualized surface A data frame sequence includes: rendered color and depth frames for the virtual projection of the customized view of the real world scene.
(Invention 2)
The method of Invention 1, characterized by:
the customized view of the real-world scene is misaligned with the different views of the real-world scene captured by the plurality of capture devices positioned at the different locations with respect to the real-world scene; and
The rendering of color and depth frames for the virtualized projection of the customized view of the real-world scene includes rendering color and depth frames, the rendering being based on at least two of: the plurality of The surface data frame sequence in the captured surface data frame sequence of .
(Invention 3)
The method of invention 1, wherein each set of capture parameters included within said plurality of sets of capture parameters associated with said different views of said real-world scene includes at least one of:
a capture parameter representing a location with respect to the real-world scene at which color and depth frames corresponding to a particular view of the real-world scene are captured;
a capture parameter representing an orientation in which the color and depth frames corresponding to the particular view of the real-world scene are captured;
capture parameters representing a field of view in which the color and depth frames corresponding to the particular view of the real-world scene are captured; and
A capture parameter that represents the image quality with which the color and depth frames corresponding to the particular view of the real-world scene are captured.
(Invention 4)
The method of invention 1, wherein said additional set of capture parameters associated with said customized view of said real-world scene includes at least one of:
A capture parameter representing a customized field of view associated with the customized view of the real-world scene, wherein the customized field of view is the basis for the rendering of the color and depth frames for the virtual projection. narrower than the captured field of view associated with at least one of the surface data frame sequences that are
a capture parameter representing a customized image quality associated with the customized view of the real-world scene, wherein the customized image quality is of the color and depth frames for the virtualization projection; lower than the captured image quality associated with at least one of said sequence of surface data frames on which said rendering is based.
(Invention 5)
The method of Invention 1, wherein identifying said additional set of capture parameters includes:
analyzing the real-world scene with respect to one or more geometric properties of the real-world scene;
generating a plurality of additional sets of capture parameters based on said analysis of the real-world scene, wherein said plurality of additional sets of capture parameters are included within said plurality of sets of capture parameters different from and including the additional set of capture parameters; and
Identifying the additional set of capture parameters from the plurality of additional sets of capture parameters.
(Invention 6)
The method of Invention 1, wherein said virtualized surface data frame sequences provided for inclusion within said virtual reality media content are packaged such that said real world Rendered color and depth frames for the virtualized projection of the customized view of a scene are contained within a transport stream, the transport stream other than the virtualized surface data frame sequence. not including color and depth frames representing additional surface data frame sequences of .
(Invention 7)
The method of Invention 1, wherein said virtualized surface data frame sequences provided for inclusion within said virtual reality media content are packaged such that said real world Rendered color and depth frames for the virtualized projection of the customized view of a scene are contained within a transport stream, the transport stream comprising at least one of the virtualized surface data frames. • The method further includes color and depth frames representing additional surface data frame sequences other than the sequence.
(Invention 8)
The method of Invention 1, wherein said virtualized surface data frame sequences provided for inclusion within said virtual reality media content are packaged such that said real world Each color and depth frame rendered for the virtualized projection of the customized view of a scene is represented as a tile in a tiled video data stream, the tiled video data stream being the tiled The method implements a tile map representing multiple tiles in each frame of the video data stream.
(Invention 9)
The method of invention 1, stored as computer readable instructions on at least one non-transitory computer readable medium.
(Invention 10)
Methods including:
A virtualized projection generation system receiving a sequence of a plurality of captured surface data frames in real time as an event occurs within a real world scene, wherein said plurality of captured surface data frames. - each of the sequences includes color and depth frames, said color and depth frames depicting said real-world scene according to respective sets of capture parameters, each set of said capture parameters representing a different real-world scene; included in a first plurality of sets of view-related capture parameters, each surface data frame sequence in said plurality of captured surface data frame sequences being different in a plurality of capture devices; captured by a capture device, said capture device positioned at different locations with respect to said real-world scene to capture different views of said real-world scene;
identifying a second plurality of sets of capture parameters, in real-time, when the event occurs within the real-world scene, wherein the second set of capture parameters comprises: A plurality of sets are different from the set of capture parameters included within the first plurality of sets of capture parameters, and each of the second plurality of sets of capture parameters are different from the real-world scene. wherein each customized view of the real-world scene differs from the different views of the real-world scene captured by the plurality of capture devices according to the first The plurality of sets of 2 includes a larger number than is included in the first plurality of sets of capture parameters;
said virtualization projection generation system rendering color and depth frames for a virtualization projection of each customized view of said real-world scene in real-time and upon said event occurring within said real-world scene; and said rendering is based on said plurality of captured surface data frame sequences and based on said second plurality of sets of capture parameters;
The virtual projection generation system generates a plurality of virtualized surface data in real-time, when the event occurs in the real-world scene, and for inclusion in virtual reality media content. providing a sequence of frames to a media player device, wherein the plurality of virtualized surface data frame sequences are rendered for the virtualized projection of each customized view of the real world scene. Includes color and depth frames.
(Invention 11)
A method of invention 10, characterized by:
A particular customized view of the real-world scene associated with a particular set of capture parameters in the second plurality of sets of capture parameters is located at the different locations with respect to the real-world scene. the different views of the real-world scene captured by the capture device; and
The rendering of the color and depth frames for the virtualized projection of the particular customized view of the real-world scene includes rendering color and depth frames, the rendering comprising at least two of the following: based on: said surface data frame sequence within said plurality of captured surface data frame sequences.
(Invention 12)
11. The method of invention 10, embodied as computer readable instructions on at least one non-transitory computer readable medium.
(Invention 13)
System, including:
At least one physical computing device that:
Receiving a plurality of captured surface data frame sequences, wherein said plurality of captured surface data frame sequences each include a color and depth frame, said color and depth frames representing the real world depicting a scene according to respective sets of capture parameters, said parameters being included in a plurality of sets of capture parameters associated with different views of a real-world scene, and said plurality of captured surface data frames; - each surface data frame sequence in a sequence is captured by a different capture device in a plurality of capture devices, said capture devices capturing different views of said real-world scene; located at different locations with respect to;
identifying an additional set of capture parameters, wherein said additional set of capture parameters is different from said set of capture parameters included within said plurality of sets of capture parameters, and said relating to a customized view of a real-world scene, wherein the real-world scene is different from the different views of the real-world scene captured by the plurality of capture devices;
customizing the real-world scene based on at least one of the surface data frame sequences in the plurality of captured surface data frame sequences and the additional set of capture parameters; rendering color and depth frames for the virtualized projection of the view; and
providing a surface data frame sequence to be virtualized to a media player device for inclusion within virtual reality media content, wherein said surface data frame sequence is a representation of said real world scene; including rendered color and depth frames for the virtualized projection of the customized view.
(Invention 14)
The system of invention 13, comprising the following features:
the customized view of the real-world scene is misaligned with the different views of the real-world scene captured by the plurality of capture devices positioned at the different locations with respect to the real-world scene; and
The rendering of color and depth frames for the virtualized projection of the customized view of the real-world scene includes rendering color and depth frames, the rendering being based on at least two of: the plurality of The surface data frame sequence in the captured surface data frame sequence of .
(Invention 15)
14. The system of invention 13, wherein each set of capture parameters included within said plurality of sets of capture parameters associated with said different views of a real-world scene includes at least one of:
a capture parameter representing a location with respect to the real-world scene at which color and depth frames corresponding to a particular view of the real-world scene are captured;
a capture parameter representing an orientation in which the color and depth frames corresponding to the particular view of the real-world scene are captured;
capture parameters representing a field of view in which the color and depth frames corresponding to the particular view of the real-world scene are captured; and
A capture parameter that represents the image quality with which the color and depth frames corresponding to the particular view of the real-world scene are captured.
(Invention 16)
The system of invention 13, wherein said additional set of capture parameters associated with said customized view of said real-world scene includes at least one of:
A capture parameter representing a customized field of view associated with the customized view of the real-world scene, wherein the customized field of view is the basis for the rendering of the color and depth frames for the virtual projection. narrower than the captured field of view associated with at least one of the surface data frame sequences where
a capture parameter representing a customized image quality associated with the customized view of the real-world scene, wherein the customized image quality is of the color and depth frames for the virtualization projection; lower than the captured image quality associated with at least one of said sequence of surface data frames on which said rendering is based.
(Invention 17)
The system of invention 13, wherein said at least one physical computing device identifies said additional set of capture parameters by:
analyzing the real-world scene with respect to one or more geometric properties of the real-world scene;
generating a plurality of additional sets of capture parameters based on said analysis of the real-world scene, wherein said plurality of additional sets of capture parameters are included within said plurality of sets of capture parameters different from and including the additional set of capture parameters; and
Identifying the additional set of capture parameters from the plurality of additional sets of capture parameters.
(Invention 18)
14. The system of invention 13, wherein said virtualized surface data frame sequences provided for inclusion within said virtual reality media content are packaged such that said real world Rendered color and depth frames for the virtualized projection of the customized view of a scene are contained within a transport stream, the transport stream other than the virtualized surface data frame sequence. system that does not include color and depth frames representing additional surface data frame sequences of
(Invention 19)
14. The system of invention 13, wherein said virtualized surface data frame sequences provided for inclusion within said virtual reality media content are packaged such that said real world Rendered color and depth frames for the virtualized projection of the customized view of a scene are contained within a transport stream, the transport stream comprising at least one of the virtualized surface data frames. • A system further comprising color and depth frames representing additional surface data frame sequences other than the sequence.
(Invention 20)
14. The system of invention 13, wherein said virtualized surface data frame sequences provided for inclusion within said virtual reality media content are packaged such that said real world Each color and depth frame rendered for the virtualized projection of the customized view of a scene is represented as a tile in a tiled video data stream, the tiled video data stream being the tiled A system that implements a tile map that represents multiple tiles in each frame of a video data stream.

Claims (20)

Methods including:
a virtualization projection generation system receiving a plurality of captured surface data frame sequences, wherein each surface data frame sequence includes color and depth frames, the frames representing a real world scene; , according to each set of capture parameters, said parameters being included in a plurality of sets of capture parameters associated with different views of a real-world scene, said plurality of captured surface data frame sequences. is captured by a different capture device in a plurality of capture devices, the devices positioned at different locations with respect to the real-world scene to capture the different views of the real-world scene. is placed in;
identifying an additional set of capture parameters, wherein the additional set of capture parameters is one of the capture parameters included within the plurality of sets of capture parameters; different from a set and related to a customized view of the real-world scene, the customized view different from the different view of the real-world scene captured by the plurality of capture devices ; - identifying said additional set of parameters includes:
analyzing the real-world scene with respect to one or more geometric properties of the real-world scene;
generating a plurality of additional sets of capture parameters based on said analysis of said real-world scene, wherein said plurality of additional sets of capture parameters are within said plurality of sets of capture parameters; different from said set of included capture parameters and including said additional set of capture parameters; and
identifying the additional set of capture parameters from the plurality of additional sets of capture parameters ;
The virtualization projection generation system rendering color and depth frames for a virtualization projection of the customized view of the real-world scene based on at least one of: the plurality of captured surface data. a sequence of surface data frames within a sequence of frames and an additional set of said capture parameters;
said virtualization projection generating system providing a virtualized surface data frame sequence to a media player device for inclusion within virtual reality media content, wherein said virtualized surface A data frame sequence includes: rendered color and depth frames for the virtual projection of the customized view of the real world scene. 2. The method of claim 1, comprising the following features:
the customized view of the real-world scene is misaligned with the different views of the real-world scene captured by the plurality of capture devices positioned at the different locations with respect to the real-world scene; and
The rendering of color and depth frames for the virtualized projection of the customized view of the real-world scene includes rendering color and depth frames, the rendering being based on at least two of: the plurality of The surface data frame sequence in the captured surface data frame sequence of . 2. The method of claim 1, wherein each set of capture parameters included within said plurality of sets of capture parameters associated with said different views of said real-world scene includes at least one of:
a capture parameter representing a location with respect to the real-world scene at which color and depth frames corresponding to a particular view of the real-world scene are captured;
a capture parameter representing an orientation in which the color and depth frames corresponding to the particular view of the real-world scene are captured;
capture parameters representing a field of view in which the color and depth frames corresponding to the particular view of the real-world scene are captured; and
A capture parameter that represents the image quality with which the color and depth frames corresponding to the particular view of the real-world scene are captured. 2. The method of claim 1, wherein the additional set of capture parameters associated with the customized view of the real-world scene includes at least one of:
A capture parameter representing a customized field of view associated with the customized view of the real-world scene, wherein the customized field of view is the basis for the rendering of the color and depth frames for the virtual projection. narrower than the captured field of view associated with at least one of the surface data frame sequences that are
a capture parameter representing a customized image quality associated with the customized view of the real-world scene, wherein the customized image quality is of the color and depth frames for the virtualization projection; lower than the captured image quality associated with at least one of said sequence of surface data frames on which said rendering is based. 2. The method of claim 1 , wherein the virtualized surface data frame sequence comprising the rendered color and depth frames for the virtualized projection of the customized view of the real-world scene is: The step of providing comprises packaging a plurality of surface data frame sequences together into a common transport stream using a tile map ping technique to obtain from each of the plurality of surface data frame sequences: A method comprising packing frames together into a plurality of tiled frames of a single tiled frame sequence, said plurality of surface data frame sequences comprising:
the virtualized surface data frame sequence, the sequence comprising the rendered color and depth frames for the virtualized projection of the customized view of the real world scene; and
At least one additional captured surface data frame sequence or virtualized surface data frame sequence other than said virtualized surface data frame sequence . 2. The method of claim 1, wherein said virtualized surface data frame sequences provided for inclusion within said virtual reality media content are packaged such that said reality Rendered color and depth frames for the virtualized projection of the customized view of a world scene are contained within a transport stream, the transport stream comprising the virtualized surface data frame sequence. not including color and depth frames representing additional surface data frame sequences other than 2. The method of claim 1, wherein said virtualized surface data frame sequences provided for inclusion within said virtual reality media content are packaged such that said reality Rendered color and depth frames for the virtualized projection of the customized view of a world scene are contained within a transport stream, the transport stream comprising at least one of the virtualized surface data images. The method further comprising color and depth frames representing additional surface data frame sequences other than the frame sequence. 2. The method of claim 1, wherein said virtualized surface data frame sequences provided for inclusion within said virtual reality media content are packaged such that said reality Each color and depth frame rendered for the virtualized projection of the customized view of a world scene is represented as a tile in a tiled video data stream, the tiled video data stream comprising the tiles implementing a tile map representing a plurality of tiles in each frame of the encoded video data stream. 2. The method of claim 1, wherein the method is performed through computer execution of instructions residing on at least one non-transitory computer-readable medium. Methods including:
A virtualized projection generation system receiving a sequence of a plurality of captured surface data frames in real time as an event occurs within a real world scene, wherein said plurality of captured surface data frames. - each of the sequences includes color and depth frames, said color and depth frames depicting said real-world scene according to respective sets of capture parameters, each set of said capture parameters representing a different real-world scene; included in a first plurality of sets of view-related capture parameters, each surface data frame sequence in said plurality of captured surface data frame sequences being different in a plurality of capture devices; captured by a capture device, said capture device positioned at different locations with respect to said real-world scene to capture different views of said real-world scene;
identifying a second plurality of sets of capture parameters, in real-time, when the event occurs within the real-world scene, wherein the second set of capture parameters comprises: A plurality of sets are different from the set of capture parameters included within the first plurality of sets of capture parameters, and each of the second plurality of sets of capture parameters are different from the real-world scene. wherein each customized view of the real-world scene differs from the different views of the real-world scene captured by the plurality of capture devices according to the first The plurality of sets of 2 includes a larger number than is included in the first plurality of sets of capture parameters;
the virtualization projection generation system rendering color and depth frames for a virtualization projection of each customized view of the real-world scene in real-time and upon the occurrence of the event within the real-world scene; wherein said rendering is based on said plurality of captured surface data frame sequences and based on said second plurality of sets of capture parameters; and
The virtual projection generation system generates a plurality of virtualized surface data in real-time, when the event occurs in the real-world scene, and for inclusion in virtual reality media content. providing a sequence of frames to a media player device, wherein the plurality of virtualized surface data frame sequences are rendered for the virtualized projection of each customized view of the real world scene. Includes color and depth frames. 11. The method of claim 10, characterized by:
A particular customized view of the real-world scene associated with a particular set of capture parameters in the second plurality of sets of capture parameters is located at the different locations with respect to the real-world scene. the different views of the real-world scene captured by the capture device; and
The rendering of the color and depth frames for the virtualized projection of the particular customized view of the real-world scene includes rendering color and depth frames, the rendering comprising at least two of the following: based on: said surface data frame sequence within said plurality of captured surface data frame sequences. 11. The method of claim 10, wherein the method is performed through computer execution of instructions residing on at least one non-transitory computer-readable medium. System, including:
At least one physical computing device that:
Receiving a plurality of captured surface data frame sequences, wherein said plurality of captured surface data frame sequences each include a color and depth frame, said color and depth frames representing the real world depicting a scene according to respective sets of capture parameters, said parameters being included in a plurality of sets of capture parameters associated with different views of a real-world scene, and said plurality of captured surface data frames; - each surface data frame sequence in a sequence is captured by a different capture device in a plurality of capture devices, said capture devices capturing different views of said real-world scene; located at different locations with respect to;
identifying an additional set of capture parameters, wherein said additional set of capture parameters is different from said set of capture parameters included within said plurality of sets of capture parameters, and said associated with a customized view of a real-world scene, wherein the real-world scene is distinct from the different views of the real-world scene captured by the plurality of capture devices, and the at least one physical computing device; The device identifies said additional set of capture parameters by:
analyzing the real-world scene with respect to one or more geometric properties of the real-world scene;
generating a plurality of additional sets of capture parameters based on said analysis of said real-world scene, wherein said plurality of additional sets of capture parameters are within said plurality of sets of capture parameters; different from said set of included capture parameters and including said additional set of capture parameters; and
identifying the additional set of capture parameters from the plurality of additional sets of capture parameters ;
customizing the real-world scene based on at least one of the surface data frame sequences in the plurality of captured surface data frame sequences and the additional set of capture parameters; rendering color and depth frames for the virtualized projection of the view; and
providing a surface data frame sequence to be virtualized to a media player device for inclusion within virtual reality media content, wherein said surface data frame sequence is a representation of said real world scene; including rendered color and depth frames for the virtualized projection of the customized view. 14. The system of claim 13, comprising the following features:
the customized view of the real-world scene is misaligned with the different views of the real-world scene captured by the plurality of capture devices positioned at the different locations with respect to the real-world scene; and
The rendering of color and depth frames for the virtualized projection of the customized view of the real-world scene includes rendering color and depth frames, the rendering being based on at least two of: the plurality of The surface data frame sequence in the captured surface data frame sequence of . 14. The system of claim 13, wherein each set of capture parameters included within said plurality of sets of capture parameters associated with said different views of a real-world scene includes at least one of:
a capture parameter representing a location with respect to the real-world scene at which color and depth frames corresponding to a particular view of the real-world scene are captured;
a capture parameter representing an orientation in which the color and depth frames corresponding to the particular view of the real-world scene are captured;
capture parameters representing a field of view in which the color and depth frames corresponding to the particular view of the real-world scene are captured; and
A capture parameter that represents the image quality with which the color and depth frames corresponding to the particular view of the real-world scene are captured. 14. The system of claim 13, wherein the additional set of capture parameters associated with the customized view of the real-world scene includes at least one of:
A capture parameter representing a customized field of view associated with the customized view of the real-world scene, wherein the customized field of view is the basis for the rendering of the color and depth frames for the virtual projection. narrower than the captured field of view associated with at least one of the surface data frame sequences where
a capture parameter representing a customized image quality associated with the customized view of the real-world scene, wherein the customized image quality is of the color and depth frames for the virtualization projection; lower than the captured image quality associated with at least one of said sequence of surface data frames on which said rendering is based. 14. The system of claim 13, wherein the at least one physical computing device converts the rendered color and depth frames for the virtualized projection of the customized view of the real-world scene into providing said virtualized surface data frame sequence comprising: said providing packaging a plurality of surface data frame sequences together into a common transport stream using a tile map ping technique; tiling the frames from each of the plurality of surface data frame sequences together into a single tiled frame sequence using a tile map ping technique. system by packing into multiple frames . 14. The system of claim 13, wherein said virtualized surface data frame sequences provided for inclusion within said virtual reality media content are packaged such that said reality Rendered color and depth frames for the virtualized projection of the customized view of a world scene are contained within a transport stream, the transport stream comprising the virtualized surface data frame sequence. system that does not include color and depth frames representing additional surface data frame sequences other than 14. The system of claim 13, wherein said virtualized surface data frame sequences provided for inclusion within said virtual reality media content are packaged such that said reality Rendered color and depth frames for the virtualized projection of the customized view of a world scene are contained within a transport stream, the transport stream comprising at least one of the virtualized surface data images. The system further comprising color and depth frames representing additional surface data frame sequences other than the frame sequence. 14. The system of claim 13, wherein said virtualized surface data frame sequences provided for inclusion within said virtual reality media content are packaged such that said reality Each color and depth frame rendered for the virtualized projection of the customized view of a world scene is represented as a tile in a tiled video data stream, the tiled video data stream comprising the tiles A system that implements a tile map that represents multiple tiles in each frame of an encoded video data stream.

Images