TW202240431A - Object collision data for virtual camera in virtual interactive scene defined by streamed media data - Google Patents

Abstract

An example device for retrieving media data includes a memory configured to store media data; and one or more processors implemented in circuitry and configured to execute a presentation engine, the presentation engine being configured to: receive streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; receive object collision data representing boundaries of the at least one virtual solid object; receive camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; and using the object collision data, prevent the virtual camera from passing through the at least one virtual solid object in response to the camera movement data.

Description

Object collision data for a virtual camera in a virtual interactive scene defined by streaming data

This application claims the benefit of U.S. Provisional Application No. 63/159,379, filed March 10, 2021, which is hereby incorporated by reference in its entirety.

This disclosure relates to storage and transmission of encoded video data.

Digital video capabilities can be incorporated into a wide variety of devices, including digital television, digital broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, digital cameras, digital recording devices, digital Media players, video game devices, video game consoles, cellular or satellite radio phones, video teleconferencing devices, etc. Digital video devices implementing video compression techniques (such as those developed by MPEG-2, MPEG-4, ITU-T H.263 or ITU-T H.264/MPEG-4 (Part 10, Advanced Video Coding (AVC)), Standards defined by ITU-T H.265 (also known as High Efficiency Video Coding (HEVC)) and those techniques described in extensions to such standards) to transmit and receive digital video information more efficiently.

Video compression techniques perform spatial prediction and/or temporal prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, video frames or slices can be divided into macroblocks. Each macroblock can be further divided. Macroblocks in an intra-coded (I) frame or slice are coded using spatial prediction with respect to neighboring macroblocks. A macroblock in an intercoded (P or B) frame or slice may use spatial prediction with respect to neighboring macroblocks in the same frame or slice, or temporal prediction with respect to other reference frames.

After the video data has been encoded, the video data may be packetized for transmission or storage. Video data may be assembled into a video archive conforming to any of various standards, such as the International Organization for Standardization (ISO) base media archive format and extensions thereof, such as AVC.

In summary, this disclosure describes techniques related to streaming interactive media data. Such interactive media data may be, for example, virtual reality, augmented reality, or other such interactive content (such as other three-dimensional video content). The latest MPEG scene description elements include support for timed media in glTF 2.0. The media access function (MAF) provides an application programming interface (API) to the presentation engine through which the presentation engine can request timed media. The retrieval unit executing the MAF may process the retrieved timed media data, and pass the processed media data to the rendering engine in a desired format through a circular buffer. The current MPEG scene description allows users to consume scene media data in 6 degrees of freedom (6DoF). Thus, the user is generally able to move freely in the 3D scene (eg, through a wall displayed in the 3D scene). However, content authors may wish to impose restrictions on viewer movement in certain areas, for example to prevent movement through displayed walls or other objects. This disclosure describes techniques for imposing such restrictions that can improve a user's experience by making the experience more realistic by preventing the user from traversing obstacles in the virtual world.

In one example, a method of retrieving media data includes: receiving, by a rendering engine, streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving, by the rendering engine, camera control data, the camera control data including data defining restrictions to prevent the virtual camera from traversing the at least one virtual solid object; receiving camera movement data from the user by the rendering engine, the camera movement data requesting the virtual camera to move through the at least one virtual solid object a virtual solid object; and using the camera control data, preventing, by the rendering engine, the virtual camera from traversing the at least one virtual solid object in response to the camera movement data.

In another example, an apparatus for retrieving media data includes: a memory configured to store media data; and one or more processors implemented in a circuit and configured to execute a rendering engine, The rendering engine is configured to: receive streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receive camera control data for the three-dimensional scene, the camera control data including defining constraints to data preventing the virtual camera from traversing the at least one virtual solid object; receiving camera movement data from a user requesting the virtual camera to move through the at least one virtual solid object; and using the camera control data, preventing the virtual camera from responding The camera movement data traverses the at least one virtual solid object.

In another example, a computer-readable storage medium has stored thereon instructions that, when executed, cause a processor of a client device to: receive streaming media data representing A virtual three-dimensional scene of at least one virtual solid object; receiving camera control data for the three-dimensional scene, the camera control data including data defining restrictions to prevent the virtual camera from traversing the at least one virtual solid object; receiving camera movement data from a user, the camera movement data requesting the virtual camera to move through the at least one virtual solid object; and using the camera control data to prevent the virtual camera from passing through the at least one virtual solid object in response to the camera movement data.

In another example, an apparatus for retrieving media data includes: means for receiving streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; means for camera control data of the scene, the camera control data comprising data defining limits to prevent the virtual camera from traversing the at least one virtual solid object; means for receiving camera movement data from the user requesting the virtual camera to move through passing the at least one virtual solid object; and means for using the camera control data to prevent the virtual camera from passing through the at least one virtual solid object in response to the camera movement data.

In another example, a method of retrieving media data includes: receiving, by a rendering engine, streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving, by the rendering engine, data representing the at least one virtual object collision data for boundaries of solid objects; receiving, by the rendering engine, camera movement data from a user, the camera movement data requesting the virtual camera to move through the at least one virtual solid object; and using the object collision data, the rendering engine prevents The virtual camera traverses the at least one virtual solid object in response to the camera movement data.

In another example, an apparatus for retrieving media data includes: a memory configured to store media data; and one or more processors implemented in a circuit and configured to execute a rendering engine, The rendering engine is configured to: receive streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receive object collision data representing boundaries of the at least one virtual solid object; receive a camera from a user movement data requesting the virtual camera to move through the at least one virtual solid object; and using the object collision data to prevent the virtual camera from moving through the at least one virtual solid object in response to the camera movement data.

In another example, a computer-readable storage medium has stored thereon instructions that, when executed, cause a processor of a client device to: receive streaming media data representing a virtual three-dimensional scene of at least one virtual solid object; receiving object collision data representing a boundary of the at least one virtual solid object; receiving camera movement data from a user requesting the virtual camera to move through the at least one virtual solid object; and using the object collision data, preventing the virtual camera from traversing the at least one virtual solid object in response to the camera movement data.

In another example, an apparatus for retrieving media data includes: means for receiving streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; means for object collision data of a boundary of a virtual solid object; means for receiving camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; and for using the object collision data to A component that prevents the virtual camera from traversing the at least one virtual solid object in response to the camera movement data.

The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the scope of claims.

Interactive media data can be streamed over the Internet. For example, a client device may retrieve interactive media data using unicast, broadcast, multicast, etc. Interactive media data may be, for example, three-dimensional (3D) media data for extended reality (XR), augmented reality (AR), virtual reality (VR), and the like. Thus, when presented to the user, the user can navigate the 3D virtual scene rendered from the interactive media data.

MPEG scene descriptions can describe three-dimensional (3D) scenes for virtual worlds or experiences, such as for XR, VR, AR or other interactive media experiences. According to the techniques of this disclosure, an MPEG scene description can describe objects within a 3D scene, such as chairs, walls, tables, counters, doors, windows, or other solid objects. This disclosure describes techniques by which MPEG scene descriptions (or other such description datasets) can be enhanced to impose constraints on the movement of a virtual camera, for example to prevent the camera from passing through solid objects such as walls .

Specifically, the scene description may describe the set of paths that the camera is allowed to move. A path can be described as a collection of anchors connected by path segments. To enhance the expressiveness of camera control, each path segment can be augmented with a bounding volume that allows some freedom of motion along the path.

Additionally or alternatively, the scene description may describe virtual solid objects in the scene. A scene description can provide information that represents, for example, the boundaries of an object, whether the object is likely to be affected by collisions with the user or other objects (such as whether the object moves or will remain stationary in response to such collisions), represents how colliding objects interact with objects The material used for the object, and/or animation data representing animations played or applied to the object in response to collisions.

The techniques of this disclosure may be applied to video archives conforming to video data encapsulated according to any of the following: ISO base media archive format, Scalable Video Coding (SVC) archive format, Advanced Video Coding (AVC) file format, 3rd Generation Partnership Project (3GPP) file format, and/or Multiview Video Coding (MVC) file format, or other similar video file formats.

In HTTP streaming, frequently used operations include HEAD, GET and partial GET. The HEAD operation retrieves the header of the archive associated with a given Uniform Resource Locator (URL) or Uniform Resource Name (URN), without retrieving the payload associated with the URL or URN. The GET operation retrieves the entire archive associated with a given URL or URN. A partial GET operation receives as an input parameter a byte range and retrieves a consecutive number of bytes for the archive, where the number of bytes corresponds to the received byte range. Thus, movie fragments can be provided for HTTP streaming, since a partial GET operation can obtain one or more individual movie fragments. In a movie fragment there may be several track fragments of different tracks. In HTTP streaming, a media presentation may be a client-accessible collection of structured data. The client can request and download the media data information to present the streaming service to the user.

In the example of streaming 3GPP data using HTTP streaming, there may be multiple representations for the video and/or audio data of the multimedia content. As explained below, different representations may correspond to different coding characteristics (e.g., different profiles or levels of video coding standards), different coding standards, or extensions of coding standards (such as multi-view and/or scalable extension), or different bit rates. A list of such representations may be defined in a Media Presentation Description (MPD) data structure. A media presentation may correspond to a collection of structured data accessible to an HTTP streaming client device. The HTTP streaming client device can request and download media data information to present the streaming service to the user of the client device. A media presentation may be described in an MPD data structure, which may include updates to the MPD.

A media presentation may contain a sequence of one or more time periods. Each period may extend until the start of the next period, or until the end of the media presentation (in the case of the last period). Each period may contain one or more representations for the same media content. The representation may be one of several alternative encoded versions of audio, video, timed text, or other such data. The representations may differ in encoding type (eg, bit rate, resolution, and/or codec for video data, and bit rate, language, and/or codec for audio data). The term representation may be used to refer to a portion of encoded audio or video data that corresponds to a particular period of multimedia content and is encoded in a particular manner.

Representations of a particular time period may be assigned to a group indicated by an attribute in the MPD indicating the adapted set to which these representations belong. Representations in the same adaptation set are generally considered to be substitutes for each other, since a client device can dynamically and seamlessly switch between these representations, eg, to perform bandwidth adaptation. For example, each representation of video data for a particular period of time may be assigned to the same adaptation set so that any of these representations may be selected for decoding to render media data for the corresponding period of multimedia content, such as video data or audio data. In some examples, media content within a time period may be represented by any one representation from group 0, if present, or a combination of at most one representation from each non-zero group. Timing data for each representation of a period may be expressed relative to the start time of that period.

A representation can consist of one or more segments. Each representation may include initialization segments, or each segment of the representation may be self-initializing. When present, the initialization section may contain initialization information for accessing representations. Typically, initialization segments contain no media data. A segment may be uniquely referenced by an identifier, such as a Uniform Resource Locator (URL), Uniform Resource Name (URN), or Uniform Resource Identifier (URI). MPD can provide an identifier for each segment. In some examples, the MPD may also provide a byte range in the form of a range attribute, which may correspond to data for a segment within the archive accessible via a URL, URN, or URI.

Different representations can be selected for substantially simultaneous retrieval of different types of media data. For example, a client device may select audio representations, video representations, and timed text representations from which to retrieve segments. In some examples, a client device may select a particular adaptation set to perform bandwidth adaptation. That is, the client device may select an adaptation set that includes video representations, an adaptation set that includes audio representations, and/or an adaptation set that includes timed text. Alternatively, a client device may select adaptation sets for certain types of media (eg, video) and directly select representations for other types of media (eg, audio and/or timed text).

1 is a block diagram illustrating an example system 10 that implements techniques for streaming media data over a network. In this example, system 10 includes content preparation device 20 , server device 60 and client device 40 . Client device 40 and server device 60 are communicatively coupled by network 74, which may include the Internet. In some examples, content preparation device 20 and server device 60 may also be coupled through network 74 or another network, or may be directly communicatively coupled. In some examples, content preparation device 20 and server device 60 may comprise the same device.

In the example of FIG. 1 , content preparation device 20 includes audio source 22 and video source 24 . Audio source 22 may include, for example, a microphone that generates electrical signals representative of captured audio data to be encoded by audio encoder 26 . Alternatively, audio source 22 may comprise a storage medium storing previously recorded audio data, an audio data generator such as a computerized synthesizer, or any other source of audio data. Video source 24 may include a video camera that generates video data to be encoded by video encoder 28, a storage medium encoded with previously recorded video data, a video data generation unit such as a computer graphics source, or any other source of video data. In all examples, content preparation device 20 is not necessarily communicatively coupled to server device 60 , but may store multimedia content to a separate medium that is read by server device 60 .

Raw audio and video data may include analog or digital data. Analog data may be digitized before being encoded by audio encoder 26 and/or video encoder 28 . Audio source 22 may obtain audio data from the speaking participant while the speaking participant is speaking, and video source 24 may simultaneously obtain video data of the speaking participant. In other examples, audio source 22 may include a computer-readable storage medium containing stored audio data, and video source 24 may include a computer-readable storage medium containing stored video data. In this way, the techniques described in this disclosure can be applied to live, streaming, real-time audio and video data or to archived, pre-recorded audio and video data.

An audio frame corresponding to a video frame is typically an audio frame containing audio data captured by audio source 22 (or Generated. For example, audio source 22 captures audio data when a speaking participant would normally produce audio data by speaking, while video source 24 simultaneously (ie, while audio source 22 is capturing audio data) captures video data for the speaking participant. Thus, an audio frame may correspond in time to one or more specific video frames. Accordingly, an audio frame corresponding to a video frame generally corresponds to a situation in which audio data and video data are captured simultaneously, and for which the audio frame and video frame respectively include the simultaneously captured audio data and video data.

In some examples, audio encoder 26 may encode into each encoded audio frame a time stamp representing the time at which the audio data was recorded for the encoded audio frame, and similarly, video encoder 28 may A time stamp representing the time at which video data for each encoded video frame was recorded is encoded in the encoded video frame. In such an example, an audio frame corresponding to a video frame may include an audio frame containing a time stamp and a video frame containing the same time stamp. Content preparation device 20 may include an internal clock from which audio encoder 26 and/or video encoder 28 may generate time stamps, or which audio source 22 and video source 24 may use to convert audio and video data Associated with timestamps respectively.

In some examples, audio source 22 may send data to audio encoder 26 corresponding to a time when audio data was recorded, and video source 24 may send data to video encoder 28 corresponding to a time when video data was recorded. In some examples, audio encoder 26 may encode a sequence identifier into the encoded audio data to indicate the relative temporal order of the encoded audio data, but not necessarily the absolute time at which the audio data was recorded, and similarly , video encoder 28 may also use sequence identifiers to indicate the relative temporal order of the encoded video data. Similarly, in some examples, sequence identifiers may be mapped or otherwise related to timestamps.

Audio encoder 26 typically produces a stream of encoded audio data, while video encoder 28 produces a stream of encoded video data. Each individual data stream (whether audio or video) can be called an elementary stream. An elementary stream is a single, bit-encoded (possibly compressed) component of a representation. For example, the encoded video or audio portion of a representation may be an elementary stream. Before encapsulating an elementary stream within a video archive, it can be converted to a Packetized Elementary Stream (PES). Within the same representation, the stream ID can be used to distinguish PES packets belonging to one elementary stream from PES packets belonging to another elementary stream. The basic data unit of an elementary stream is a packetized elementary stream (PES) packet. Thus, encoded video data generally corresponds to an elementary video stream. Similarly, audio data corresponds to one or more corresponding elementary streams.

Many video coding standards (such as ITU-T H.264/AVC, and the forthcoming High Efficiency Video Coding (HEVC) standard) define the syntax, semantics, and decoding process for error-free bitstreams, any of which One conforms to a certain profile or level. Video coding standards usually do not specify encoders, but encoders are tasked with ensuring that the resulting bitstream is standard-compliant for decoders. In the context of video coding standards, "profiles" correspond to subsets of algorithms, features, or tools and constraints that apply to them. For example, as defined by the H.264 standard, a "profile" is a subset of the entire bitstream syntax specified by the H.264 standard. A "level" corresponds to a limit on decoder resource consumption, such as, for example, decoder memory and computation, related to the resolution, bit rate, and block processing rate of a picture. A profile may be signaled with a profile_idc (profile indicator) value, while a level may be signaled with a level_idc (level indicator) value.

For example, the H.264 standard recognizes that within the bounds imposed by the syntax of a given profile, large variations in the performance of encoders and decoders may still be required, depending on the A value such as the specified decoded picture size. The H.264 standard further recognizes that in many applications it is neither practical nor economical to implement a decoder capable of handling all hypothetical uses of the syntax within a particular profile. Accordingly, the H.264 standard defines a "level" as a specified set of constraints imposed on the values of syntax elements in a bitstream. These constraints may be simple restrictions on values. Alternatively, these constraints may take the form of constraints on arithmetic combinations of values (eg picture width times picture height times number of pictures decoded per second). The H.264 standard further provides that individual implementations may support different levels for each supported profile.

A decoder conforming to a profile typically supports all features defined in the profile. For example, as a coding feature, B-picture coding is not supported in the basic profile of H.264/AVC, but is supported in other profiles of H.264/AVC. A conforming class-compliant decoder shall be able to decode any bitstream that does not require resources beyond the limits defined in that class. The definition of profiles and levels can aid in interpretability. For example, during video transmission, a pair of profile and level definitions may be negotiated and agreed upon for the entire transmission session. More specifically, in H.264/AVC, a level can define limits on: the number of macroblocks that need to be processed, the decoded picture buffer (DPB) size, the coded picture buffer (CPB) Size, vertical motion vector range, maximum number of motion vectors per two consecutive MBs, and whether B-blocks can have sub-macroblock partitions smaller than 8x8 pixels. In this way, a decoder can determine whether the decoder is able to decode the bitstream correctly.

In the example of FIG. 1 , encapsulation unit 30 of content preparation device 20 receives an elementary stream comprising encoded video data from video encoder 28 and an elementary stream comprising encoded audio data from audio encoder 26 . In some examples, video encoder 28 and audio encoder 26 may each include a packetizer for forming PES packets from encoded data. In other examples, video encoder 28 and audio encoder 26 may each interface with a corresponding packetizer for forming PES packets from the encoded data. In yet other examples, encapsulation unit 30 may include a packetizer for forming PES packets from encoded audio and video data.

The video encoder 28 may encode the video data of the multimedia content in various ways to generate images of the multimedia content at various bit rates and with various characteristics (such as pixel resolution, frame rate, compliance with various encoding standards, compliance with various Different representations of individual profiles and/or levels of profiles of coding standards, representations with one or more views (eg, for 2D or 3D playback, or other such characteristics). A representation, as used in this disclosure, may include one of audio data, video data, text data (eg, for closed captioning), or other such data. A representation may include elementary streams, such as audio elementary streams or video elementary streams. Each PES packet may include a stream_id identifying the elementary stream to which the PES packet belongs. Encapsulation unit 30 is responsible for assembling the elementary streams into video archives (eg segments) of the respective representations.

Encapsulation unit 30 receives PES packets for the represented elementary streams from audio encoder 26 and video encoder 28 and forms corresponding network abstraction layer (NAL) units from the PES packets. Coded video segments can be organized into NAL units that provide a "network friendly" video representation addressed to applications such as video telephony, storage, broadcasting or streaming. NAL units may be classified into video coding layer (VCL) NAL units and non-VCL NAL units. A VCL unit may contain a core compression engine, and may include block, macroblock, and/or slice level data. Other NAL units may be non-VCL NAL units. In some examples, a coded picture, generally represented as an elementary coded picture at one instance in time, may be contained in an access unit, which may include one or more NAL units.

Non-VCL NAL units may also include parameter set NAL units and SEI NAL units, among other units. A parameter set can contain sequence-level header information (in a sequence parameter set (SPS)) and infrequently changing picture-level header information (in a picture parameter set (PPS)). With parameter sets (eg, PPS and SPS), infrequently changing information does not need to be repeated for each sequence or picture; thus, coding efficiency can be improved. In addition, the use of parameter sets enables out-of-band transmission of important header information, thereby avoiding the need for redundant transmission for error recovery. In an out-of-band transmission example, parameter set NAL units may be transmitted on a different channel than other NAL units, such as SEI NAL units.

Supplemental Enhancement Information (SEI) may contain information that is not necessary for decoding coded picture samples from VCL NAL units, but may facilitate processes related to decoding, display, error recovery, and other purposes. SEI information can be contained in non-VCL NAL units. SEI messages are a formal part of some standard specifications, and thus are not always mandatory for standard-compliant decoder implementations. SEI messages can be sequence-level SEI messages or picture-level SEI messages. Some sequence level information can be contained in SEI messages, such as scalability information SEI messages in the SVC example, and view scalability information SEI messages in MVC. These example SEI messages may convey information about, for example, the extraction of the operation point and the characteristics of the operation point. Additionally, encapsulation unit 30 may form a manifest file, such as a Media Presentation Descriptor (MPD), which describes characteristics of a representation. The encapsulation unit 30 may format the MPD according to Extensible Markup Language (XML).

Encapsulation unit 30 may provide data for one or more representations of multimedia content to output interface 32 along with a manifest file (eg, MPD). Output interface 32 may include a network interface, or an interface for writing to storage media such as a Universal Serial Bus (USB) interface, a CD or DVD recorder or burner, an interface to magnetic or flash storage media, or other interfaces for storing or transferring media data). The encapsulation unit 30 may provide the data of each of the representations of the multimedia content to the output interface 32, and the output interface 32 may send the data to the server device 60 via network transmission or storage media. In the example of FIG. 1 , server device 60 includes a storage medium 62 for storing various multimedia content 64 each including a corresponding manifest file 66 and one or more representations 68A-68N (representation 68 ). In some examples, the output interface 32 can also directly send data to the network 74 .

In some examples, representation 68 may be divided into adapted sets. That is, each subset of representations 68 may include a corresponding set of common properties, such as codec, profile and level, resolution, number of views, file format for segmentation, may identify the associated representation and/or required Text type information that is decoded and displayed with, for example, audio data presented by a speaker, language or other characteristics of the text, camera angle information that can describe the camera angle or real-world perspective for the scene represented in the adaptation set, describing the content for Rating information on suitability for a particular audience, etc.

Manifest archive 66 may include data indicating a subset of representations 68 corresponding to a particular set of adaptations, as well as common properties for the sets of adaptations. Manifest file 66 may also include data representing individual characteristics used to adapt individual representations in the set, such as bit rates. In this way, adaptation sets can provide simplified network bandwidth adaptation. Sub-elements in the adaptation set element of manifest file 66 may be used to indicate representations in the adaptation set.

The server device 60 includes a request processing unit 70 and a network interface 72 . In some examples, the server device 60 may include a plurality of network interfaces. Additionally, any or all of the features of server device 60 may be implemented on other devices in the content delivery network, such as routers, bridges, proxy devices, switches, or other devices. In some examples, the intermediary device of the content delivery network may cache the data of the multimedia content 64 and include components substantially identical to those of the server device 60 . Generally, network interface 72 is configured to send and receive data over network 74 .

Request handling unit 70 is configured to receive network requests for data of storage medium 62 from a client device, such as client device 40 . For example, the request processing unit 70 may implement the Hypertext Transfer Protocol ( HTTP) version 1.1. That is, request handling unit 70 may be configured to receive an HTTP GET or partial GET request and provide data for multimedia content 64 in response to the request. The request may specify a segment of one of the representations 68 (eg, using the segment's URL). In some examples, the request may also specify one or more byte ranges of segments, thereby including partial GET requests. Request processing unit 70 may further be configured to service HTTP HEAD requests to provide header data for segments of one of representations 68 . In any event, request processing unit 70 may be configured to process requests to provide requested data to a requesting device (such as client device 40 ).

Additionally or alternatively, request processing unit 70 may be configured to deliver media data via a broadcast or multicast protocol such as eMBMS. Content preparation device 20 may create DASH segments and/or sub-segments in substantially the same manner as described, but server device 60 may deliver these segments using eMBMS or another broadcast or multicast network transport protocol or subsegments. For example, request handling unit 70 may be configured to receive a multicast group join request from client device 40 . That is, server device 60 may advertise to client devices, including client device 40, an Internet Protocol (IP) address associated with a multicast group associated with particular media content (e.g., a broadcast of a live event). )Associated. Client device 40 may in turn submit a request to join the multicast group. This request can be propagated throughout the network 74 (e.g., the routers that make up the network 74), causing the routers to direct traffic destined for the IP address associated with the multicast group to a custom client device (such as a client end device 40).

As shown in the example of FIG. 1 , multimedia content 64 includes a manifest file 66 , which may correspond to a media presentation description (MPD). The manifest file 66 may contain descriptions of different alternative representations 68 (e.g., video services with different qualities), and the description may include, for example, codec information, profile values, level values, bit rates, and other descriptions for the representation 68 sexual characteristics. Client device 40 may retrieve the MPD of the media presentation to determine how to access the segments of representation 68 .

Specifically, the retrieval unit 52 can retrieve configuration data (not shown) of the client device 40 to determine the decoding capability of the video decoder 48 and the rendering capability of the video output 44 . Video output 44 may be included in a display device, such as a headset, for extended reality, augmented reality, or virtual reality. Likewise, configuration data may indicate whether video output 44 is capable of presenting three-dimensional video data, eg, for extended reality, augmented reality, virtual reality, and the like. Configuration data may also include any or all of the following: a language preference selected by a user of client device 40, one or more camera angles corresponding to a depth preference set by a user of client device 40 , and/or rating preferences selected by the user of the client device 40.

Retrieval unit 52 may include, for example, a web browser or media client configured to submit HTTP GET and partial GET requests. Retrieval unit 52 may correspond to software instructions executed by one or more processors or processing units (not shown) of client device 40 . In some examples, all or part of the functions described with respect to the retrieval unit 52 may be implemented by hardware, or by a combination of hardware, software and/or firmware, wherein necessary hardware may be provided to execute or firmware instructions.

The retrieval unit 52 may compare the decoding and rendering capabilities of the client device 40 with the characteristics of the representation 68 indicated by the information of the manifest file 66 . Retrieval unit 52 may initially retrieve at least a portion of manifest archive 66 to determine characteristics of representation 68 . For example, retrieval unit 52 may request a portion of manifest archive 66 that describes properties of one or more adaptation sets. Retrieval unit 52 may select a subset of representation 68 (eg, an adapted set) that has properties that may be satisfied by the coding and rendering capabilities of client device 40 . Retrieval unit 52 may then determine the bit rate for the representations in the adapted set, determine the amount of network bandwidth currently available, and retrieve from one of the representations a component with a bit rate that the network bandwidth can satisfy. part.

In general, higher bit rate representations can produce higher quality video playback, while lower bit rate representations can provide adequate quality video playback when available network bandwidth is reduced. Correspondingly, when the available network bandwidth is relatively high, retrieval unit 52 may retrieve data from relatively high bit-rate representations, and when available network bandwidth is low, retrieval unit 52 may retrieve data from relatively low-bit-rate representations. Retrieve data in the representation. In this manner, the client device 40 can stream multimedia data over the network 74 while also adapting to the varying network bandwidth availability of the network 74 .

Additionally or alternatively, retrieval unit 52 may be configured to receive data according to a broadcast or multicast network protocol, such as eMBMS or IP multicast. In such an example, retrieval unit 52 may submit a request to join a multicast network group associated with particular media content. After joining the multicast group, the retrieval unit 52 can receive the data of the multicast group without making an additional request to the server device 60 or the content preparation device 20 . When the data of the multicast group is no longer needed, the retrieval unit 52 may submit a request to leave the multicast group, eg, stop playback or change the channel to a different multicast group.

Network interface 54 may receive data representing segments of the selected representation and provide them to retrieval unit 52 , which in turn may provide the segments to decapsulation unit 50 . Decapsulation unit 50 may decapsulate the elements of the video archive into a constituent PES stream, decapsulate the PES stream to retrieve the encoded data, and send the encoded data to audio decoder 46 or video decoder 48, which Depends on whether the encoded data is part of an audio stream or a video stream (eg, as indicated by the stream's PES packet header). Audio decoder 46 decodes encoded audio data and sends the decoded audio data to audio output 42, while video decoder 48 decodes encoded video data and sends the decoded video data (which may include stream multiple views) to the video output 44.

Video encoder 28, video decoder 48, audio encoder 26, audio decoder 46, encapsulation unit 30, retrieval unit 52, and decapsulation unit 50 may be implemented as any of a variety of suitable processing circuits, as applicable. One or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic circuits, software, hardware, firmware, or any combination thereof. Each of video encoder 28 and video decoder 48 may be included in one or more encoders or decoders, either of which may be integrated into a combined video encoder/decoder (CODEC) part. Likewise, each of audio encoder 26 and audio decoder 46 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined CODEC. Appliances including video encoder 28, video decoder 48, audio encoder 26, audio decoder 46, encapsulation unit 30, retrieval unit 52, and/or decapsulation unit 50 may include integrated circuits, microprocessors, and/or wireless Communication devices (such as cellular phones).

Client device 40, server device 60, and/or content preparation device 20 may be configured to operate in accordance with the techniques of this disclosure. This disclosure describes these techniques with respect to client device 40 and server device 60 for purposes of example. It should be understood, however, that content preparation device 20 may be configured to perform these techniques in place of (or in addition to) server device 60 .

Encapsulation unit 30 may form a NAL unit that includes a header identifying the program to which the NAL unit belongs and payload (eg, audio data, video data, or data describing the transport or program stream to which the NAL unit corresponds). For example, in H.264/AVC, a NAL unit includes a 1-byte header and a variable-sized payload. A NAL unit that includes video data in its payload may include video data at various levels of granularity. For example, a NAL unit may include a block of video data, a plurality of blocks, a slice of video data, or an entire picture of video data. Encapsulation unit 30 may receive encoded video data from video encoder 28 in the form of PES packets of elementary streams. Encapsulation unit 30 may associate each elementary stream with a corresponding program.

Encapsulation unit 30 may also assemble access units from a plurality of NAL units. In general, an access unit may include one or more NAL units for representing a frame of video data, and (when such audio data is available) the corresponding audio data for that frame. An access unit typically includes all NAL units for one output time instance, eg, all audio and video data for one time instance. For example, if each view has a frame rate of 20 frames per second (fps), each time instance may correspond to a time interval of 0.05 seconds. During this time interval, a particular frame for all views of the same access unit (same time instance) may be rendered simultaneously. In one example, an access unit may include a coded picture at one instance of time, which may be presented as an elementary coded picture.

Accordingly, an access unit may include all audio and video frames at a common time instance, eg, all views corresponding to time X. This disclosure also refers to encoded pictures of a particular view as "view components." That is, a view component may include encoded pictures (or frames) for a particular view at a particular time. Accordingly, an access unit may be defined to include all view components for a common time instance. The decoding order of the access units does not necessarily need to be the same as the output or display order.

A media presentation may include a media presentation description (MPD), which may contain descriptions of different alternative representations (eg, video services with different qualities), and which may include, for example, codec information, profile values, and level values. MPD is an example of a manifest file, such as manifest file 66 . Client device 40 may retrieve the MPD of the media presentation to determine how to access the movie segments for each presentation. Movie fragments can be located in a movie fragment box (moof box) in a video archive.

Manifest archive 66 (which may include, for example, MPD) may advertise the availability of segments of representation 68 . That is, the MPD may include information indicating the clock time at which a first segment of one of the representations 68 became available, and information indicating the duration of a segment within representation 68 . In this way, the retrieval unit 52 of the client device 40 can decide when each segment is available based on the start times and durations of the segments preceding the particular segment.

After encapsulation unit 30 has assembled the NAL units and/or access units into a video file based on the received data, encapsulation unit 30 passes the video file to output interface 32 for output. In some examples, the encapsulation unit 30 may store the video file locally or send the video file to a remote server through the output interface 32 instead of directly sending the video file to the client device 40 . Output interface 32 may include, for example, a transmitter, a transceiver, a device for writing data to a computer-readable medium such as, for example, an optical drive, a magnetic media drive (e.g., a floppy disk drive), a Universal Serial Bus (USB) port , network interface or other output interface. Output interface 32 outputs the video file to a computer-readable medium such as, for example, a transmission signal, magnetic media, optical media, memory, flash drive, or other computer-readable media.

Network interface 54 may receive NAL units or access units via network 74 and provide the NAL units or access units to decapsulation unit 50 via retrieval unit 52 . Decapsulation unit 50 may decapsulate the elements of the video archive into constituent PES streams, decapsulate the PES streams to retrieve encoded data, and send audio decoder 46 or video decoder 48 (depending on whether the encoded data is An audio stream is also part of a video stream, eg sending encoded data as indicated by the stream's PES packet header. Audio decoder 46 decodes encoded audio data and sends the decoded audio data to audio output 42, while video decoder 48 decodes encoded video data and sends the decoded video data (which may include stream multiple views) to the video output 44.

According to the techniques of this disclosure, a user of client device 40 may obtain media data related to a 3D virtual scene, such as for extended reality (XR), augmented reality (AR), virtual reality (VR), and the like. A user may navigate through the 3D virtual scene using one or more devices, such as controllers, in communication with client device 40 . Additionally or alternatively, client device 40 may include sensors, cameras, etc. for determining that the user has moved in real-world space, and client device 40 may translate such real-world movement into virtual space movement.

A 3D virtual scene may include one or more virtual solid objects. Such objects may include, for example, walls, windows, tables, chairs, or any other such objects that may appear in a virtual scene. According to the techniques of this disclosure, the media data retrieved by retrieval unit 52 may include scene descriptions describing such virtual solid objects. The scene description may conform to eg the MPEG scene description element of glTF 2.0.

In some examples, the scene description may include a description of allowable camera movement. For example, the scene description may describe one or more bounding volumes within which the virtual camera is allowed to move such that the virtual camera is not allowed to move beyond the bounds of the shape (e.g., volumes according to shapes such as spheres, cubes, cones, frustums, etc. ). That is, the bounding volume may describe the allowable camera movement volume within which the virtual camera is permitted to move. Additionally or alternatively, the scene description may describe one or more vertices or anchors and allowed paths (eg, segments) between the vertices or anchors. Client device 40 may only allow the virtual camera to move along the allowed path and/or within the bounding volume.

In some examples, additionally or alternatively, the scene description may describe one or more virtual solid objects in the scene that the virtual camera cannot pass through.

FIG. 2 is a block diagram illustrating an example set of components of the retrieval unit 52 of FIG. 1 in more detail. In this example, the retrieval unit 52 includes an eMBMS middleware unit 100 , a DASH client 110 , a media application 112 and a presentation engine 114 .

In this example, the eMBMS middleware unit 100 further includes an eMBMS receiving unit 106 , a cache 104 and a proxy server unit 102 . In this example, the eMBMS receiving unit 106 is configured to receive data via eMBMS, for example, according to File Delivery over Unidirectional Transport (FLUTE), as specified in November 2012, RFC 6726, Network Working Group, T . Paila et al., "FLUTE—File Delivery over Unidirectional Transport," available at tools.ietf.org/html/rfc6726. That is, the eMBMS receiving unit 106 may receive files via broadcast from, for example, the server device 60 , and the server device 60 may function as a Broadcast/Multicast Service Center (BM-SC).

As the eMBMS middleware unit 100 receives data for archiving, the eMBMS middleware unit may store the received data in the cache 104 . Cache 104 may include computer-readable storage media, such as flash memory, hard drive, RAM, or any other suitable storage media.

The proxy server unit 102 may act as a server for the DASH client 110 . For example, proxy server unit 102 may provide an MPD file or other manifest file to DASH client 110 . The proxy server unit 102 may advertise in the MPD file the availability times for the segments and the hyperlinks from which the segments can be retrieved. These hyperlinks may include a local host address prefix corresponding to client device 40 (eg, 127.0.0.1 for IPv4). In this manner, DASH client 110 may request segments from proxy server unit 102 using HTTP GET or partial GET requests. For example, for a segment available from link http://127.0.0.1/rep1/seg3, DASH client 110 may construct an HTTP GET request including a request for http://127.0.0.1/rep1/seg3, and send The proxy server unit 102 submits the request. Proxy server unit 102 may retrieve the requested data from cache 104 and provide the data to DASH client 110 in response to such a request.

DASH client 110 provides the retrieved media data to media application 112 . For example, media application 112 may be a web browser, a game engine, or another application that receives and renders media data. Additionally, rendering engine 114 represents an application that interacts with media application 112 to render the retrieved media data in a 3D virtual environment. Rendering engine 114 may, for example, map two-dimensional media data onto a 3D projection. The presentation engine 114 may also receive input from other components of the client device 40 to determine the position of the user in the 3D virtual environment and the orientation the user is facing in that position. For example, presentation engine 114 may determine the X, Y, and Z coordinates of the user's location, and the orientation the user is viewing in order to determine appropriate media data to display to the user. Additionally, the rendering engine 114 may receive camera movement data representing real-world user movement data and convert the real-world user movement data into 3D virtual space movement data.

According to the techniques of this disclosure, the eMBMS middleware unit 100 may receive media data via broadcast or multicast (eg, according to glTF 2.0), and then the DASH client 110 may retrieve the media data from the eMBMS middleware unit 100 . The media data may include a scene description including camera control information indicating how the virtual camera may move through the virtual scene. For example, a scene description may include data describing allowable paths through the virtual scene, eg, along defined paths between anchor points. Additionally or alternatively, the scene description may include data describing a bounding volume, representing a volume within which the virtual camera is allowed to move. Additionally or alternatively, the scene description may include data describing one or more solid virtual objects (such as walls, tables, chairs, etc.) in the 3D virtual environment. For example, scene description data can define collision boundaries for 3D virtual objects. The scene description may further include data representing what happens in the case of a collision with such an object, such as an animation to be played with the object, whether the object is static (e.g. in the case of a wall) or dynamic (for example, in the case of chairs).

The rendering engine 114 may use the scene description to decide what to render in the event of a collision with a 3D virtual object and/or an attempt to move out of an allowable path or volume. For example, if the scene description includes data for an allowable path or bounding volume, and the user attempts to move beyond the allowable path or bounding volume, the rendering engine 114 may simply refrain from updating the display, indicating that such movement is not permitted. As another example, if the scene description includes data for a 3D virtual solid object, and the user attempts to move through the 3D virtual solid object, the rendering engine 114 may refrain from updating the display if the 3D virtual solid object is static. If the 3D virtual solid object is not static, rendering engine 114 may determine animations to be displayed for the object, eg, translational movement and/or rotational movement to be applied to the object. For example, if the 3D virtual solid object is a chair, the animation data may indicate that the chair will be pushed along the floor, or topple over in the event of a collision.

FIG. 3 is a conceptual diagram illustrating elements of example multimedia content 120 . Multimedia content 120 may correspond to multimedia content 64 ( FIG. 1 ) or another multimedia content stored in storage medium 62 . In the example of FIG. 3 , multimedia content 120 includes a media presentation description (MPD) 122 and a plurality of representations 124A- 124N (representations 124 ). Representation 124A includes optional header data 126 and segments 128A-128N (segment 128 ), while representation 124N includes optional header data 130 and segments 132A-132N (segment 132 ). For convenience, the letter N is used to designate the last movie segment in each of representations 124 . In some examples, there may be different numbers of movie segments between representations 124 .

MPD 122 may include a separate data structure from representation 124 . MPD 122 may correspond to manifest file 66 of FIG. 1 . Likewise, representation 124 may correspond to representation 68 of FIG. 1 . In general, MPD 122 may include data that generally describes characteristics of representation 124, such as coding and rendering characteristics, adaptation sets, profiles to which MPD 122 corresponds, text type information, camera angle information, rating information, trick mode information (e.g. , indicating information comprising representations of temporal subsequences), and/or information for retrieving remote time periods (eg, for inserting targeted advertisements into media content during playback).

Header data 126 (when present) may describe characteristics of segments 128, for example, the temporal location of a random access point (RAP, also known as a stream access point (SAP)), which of the segments 128 includes random An access point, a byte offset from a random access point within segment 128 , a uniform resource locator (URL) for segment 128 , or other aspect of segment 128 . Header data 130 (when present) may describe similar characteristics of segments 132 . Additionally or alternatively, such features may be fully included in MPD 122 .

Segments 128, 132 include one or more encoded video samples, each of which may include a frame or slice of video data. Each of the coded video samples of segment 128 may have similar characteristics, such as height, width, and bandwidth requirements. Such characteristics may be described by data from MPD 122, although such data is not shown in the example of FIG. MPD 122 may include features as described by the 3GPP specification with any or all of the signaled information described in this disclosure added.

Each of the segments 128, 132 may be associated with a unique Uniform Resource Locator (URL). Accordingly, each of the segments 128, 132 may be independently retrievable using a streaming network protocol such as DASH. In this manner, a destination device, such as client device 40, may retrieve segment 128 or 132 using an HTTP GET request. In some examples, client device 40 may retrieve a particular byte range of segment 128 or 132 using an HTTP partial GET request.

FIG. 4 is a block diagram illustrating elements of an example video archive 150, which may correspond to a segment of a representation, such as one of the segments 128, 132 of FIG. Each of the segments 128 , 132 may include data that substantially conforms to the arrangement of data shown in the example of FIG. 4 . Video archive 150 may be thought of as packaged segments. As mentioned above, data is stored in a series of objects called "boxes" according to the ISO base media file format and its extension, the video file. In the example of FIG. 4 , a video archive 150 includes a file type (FTYP) box 152, a movie (MOOV) box 154, a segment index (sidx) box 162, a movie fragment (MOOF) box 164, and a movie fragment random access ( MFRA) Box 166. Although FIG. 4 represents an example of a video archive, it should be understood that other media archives may include other types of media data (e.g., audio data) structured similarly to the data of video archive 150 according to the ISO base media archive format and extensions thereof. , timed text data, etc.).

A file type (FTYP) box 152 generally describes the file type for the video file 150 . The dossier type box 152 may include data identifying a specification describing the best use for the video dossier 150 . File type box 152 may alternatively be placed before MOOV box 154 , movie fragment box 164 and/or MFRA box 166 .

In some examples, a segment such as video archive 150 may include an MPD update box (not shown) prior to FTYP box 152 . The MPD update box may include information indicating that the MPD corresponding to the representation including video dossier 150 is to be updated and information for updating the MPD. For example, the MPD update box may provide a URI or URL for a resource that will be used to update the MPD. As another example, an MPD update box may include data for updating the MPD. In some examples, an MPD update box may immediately follow a segment type (STYP) box (not shown) of video archive 150 , where the STYP box may define a segment type for video archive 150 .

In the example of FIG. 4 , MOOV boxes 154 include a movie header (MVHD) box 156 , a track (TRAK) box 158 , and one or more movie extension (MVEX) boxes 160 . In general, MVHD box 156 may describe the general characteristics of video archive 150 . For example, MVHD box 156 may include data describing when video archive 150 was originally created, when video archive 150 was most recently modified, the time scale for video archive 150, the duration of playback for video archive 150, or generally describe Other data of the video file 150 .

TRAK box 158 may contain data for a track of video archive 150 . The TRAK box 158 may include a track header (TKHD) box describing the characteristics of the track corresponding to the TRAK box 158 . In some examples, TRAK box 158 may include coded video pictures, while in other examples, the coded video pictures of a track may be included in movie fragments 164, which may pass through TRAK box 158 and/or or sidx box 162 data to reference.

In some examples, video archive 150 may include more than one track. Thus, the MOOV box 154 may include a number of TRAK boxes equal to the number of tracks in the video archive 150 . The TRAK box 158 may describe the characteristics of the corresponding track of the video archive 150 . For example, TRAK box 158 may describe temporal and/or spatial information for a corresponding track. When encapsulation unit 30 ( FIG. 3 ) includes parameter set tracks in a video archive such as video archive 150 , a TRAK box similar to TRAK box 158 of MOOV box 154 may describe characteristics of the parameter set track. Encapsulation unit 30 may signal the presence of sequence-level SEI messages in the parameter set track within the TRAK box describing the parameter set track.

The MVEX box 160 may describe the characteristics of the corresponding movie fragment 164 , eg, to signal that the video archive 150 includes the movie fragment 164 in addition to the video data (if any) included within the MOOV box 154 . In the context of streaming video data, encoded video pictures may be included in movie segments 164 rather than in MOOV boxes 154 . Therefore, all coded video samples may be included in movie fragment 164 instead of in MOOV box 154 .

The MOOV box 154 may include a number of MVEX boxes 160 equal to the number of movie segments 164 in the video archive 150 . Each of MVEX boxes 160 may describe characteristics of a corresponding one of movie fragments 164 . For example, each MVEX box may include a movie extension header box (MEHD) box that describes a temporal duration for a corresponding movie fragment in movie fragments 164 .

As noted above, encapsulation unit 30 may store sequence data sets in video samples that do not include actual encoded video data. A video sample may generally correspond to an access unit, which is a representation of a coded picture at a particular instance of time. In the context of AVC, a coded picture includes one or more VCL NAL units and other associated non-VCL NAL units (such as SEI information) that contain information for all pixels from which an access unit is to be constructed. Correspondingly, the encapsulation unit 30 may include a sequence data set in one of the movie segments 164, and the sequence data set may include sequence-level SEI information. Encapsulation unit 30 may further signal the presence of a sequence data set and/or sequence level SEI information in an MVEX box within MVEX box 160 corresponding to one of movie fragments 164 as present in that movie fragment 164 middle.

SIDX box 162 is an optional element of video archive 150 . That is, a video file conforming to the 3GPP file format or other such file format does not necessarily include the SIDX box 162 . According to an example of the 3GPP archive format, SIDX boxes may be used to identify sub-segments of a segment (eg, a segment contained within video archive 150). The 3GPP archive format defines a subsegment as "a self-contained collection of one or more consecutive movie fragment boxes with corresponding media data boxes, and the media data box containing the data referenced by the movie fragment box must follow the movie fragment box and Before the next movie fragment box containing information about the same track." The 3GPP file format also indicates that a SIDX box "contains a sequence of references to the subsegments of the (sub)segments recorded by the box. The referenced subsegments Segments are contiguous in presentation time. Similarly, bytes referenced by a segment index box are always contiguous within a segment. The referenced size gives the number of bytes in the referenced material count."

SIDX box 162 generally provides information representing one or more sub-segments of a segment included in video archive 150 . For example, such information may include the playback time at which the subsegment begins and/or ends, the byte offset for the subsegment, whether the subsegment includes (e.g., starts from), a stream access point (SAP), Type of SAP used (e.g., is the SAP an Instantaneous Decoder Refresh (IDR) picture, Clean Random Access (CRA) picture, Broken Link Access (BLA) picture, or something else), the position of the SAP in the subsegment (based on playback time and/or byte offset), etc.

Movie segment 164 may include one or more encoded video pictures. In some examples, movie segment 164 may include one or more groups of pictures (GOPs), each of which may include a number of encoded video pictures, eg, frames or pictures. Additionally, as noted above, in some examples, movie segments 164 may include sequence data sets. Each of movie fragments 164 may include a movie fragment header box (MFHD, not shown in FIG. 4 ). The MFHD box may describe properties of the corresponding movie fragment, such as a serial number for that movie fragment. Movie clips 164 may be included in video archive 150 in order of serial number.

MFRA boxes 166 may describe random access points within movie fragments 164 of video archive 150 . This may assist in performing trick modes, such as performing searches for specific temporal positions (ie, playback times) within segments encapsulated by the video archive 150 . MFRA box 166 is generally optional, and in some examples need not be included in the video archive. Likewise, a client device (eg, client device 40 ) does not necessarily need to reference MFRA box 166 to correctly decode and display the video data of video archive 150 . MFRA box 166 may include a number of Track Fragment Random Access (TFRA) boxes (not shown), the number of TFRA boxes equal to the number of tracks of video archive 150, or in some examples, equal to the number of media tracks of video archive 150 (e.g., number of non-hinted tracks).

In some examples, movie fragment 164 may include one or more stream access points (SAPs), such as IDR pictures. Likewise, MFRA box 166 may provide an indication of the location of the SAP within video archive 150 . Accordingly, a temporal subsequence of the video archive 150 may be formed from the SAP of the video archive 150 . The temporal subsequence may also include other pictures, such as P-frames and/or B-frames depending on SAP. Frames and/or slices of a temporal subsequence may be arranged within segments such that frames/slices of a temporal subsequence that are dependent on other frames/slices of the subsequence can be correctly decoded. For example, in a hierarchical arrangement of data, data used for predictions for other data may also be included in temporal subsequences.

FIG. 5 is a conceptual diagram illustrating an example camera path segment 212 with a bounding volume in accordance with the techniques of this disclosure. Specifically, in 3D scene 200 , camera 202 represents a point of view from which a user can watch a part of 3D scene 200 . In this example, path segment 212 is defined between point 204 and point 206 . Furthermore, a bounding volume is defined by extruding points from bounding box 208 to bounding box 210 along path segment 212 . Thus, in this example, the camera 202 is allowed to move within the bounding volume along the path segment 212, but is restricted from moving beyond the bounding volume.

A scene description may describe a set of paths along which a camera, such as camera 202 , is allowed to move. A path may be described as a set of anchor points (such as points 204 , 206 ), which are connected by path segments (such as path segment 212 ). In some examples, such as the example of FIG. 5 , each path segment may be augmented with a bounding volume that allows some freedom of movement along the path.

Thus, the scene camera, and thus the viewer, will be able to move freely along the path segments within the bounding volume. Path segments can be described using more complex geometries to allow finer control over the path.

Additionally, camera parameters may be constrained at each point along the path. Parameters for each anchor point can be provided and then used with an interpolation function to calculate corresponding parameters for each point segmented along the path. Interpolation functions can be applied to all parameters, including bounding volumes.

The camera control extension mechanism of the present disclosure can be implemented as a glTF 2.0 extension that defines camera control for a scene. The camera control extension may be identified by an 'MPEG_camera_control' tag, which may be included in the extensionsUsed element, and may be included in the extensionsRequired element for 3D scenes.

An example "MPEG_camera_control" extension is shown in Table 1 below and may be defined on the "camera" element of a scene description. Table 1 name Types of default describe anchor number N/A The number of anchor points in the camera path. section number N/A The number of path segments in the camera path. bounding volume number BV_NONE The type of bounding volume used for path segmentation. Possible types are: l BV_NONE: no bounding volume l BV_CONE: capped cone bounding volume, which is defined by circles at each anchor point. l BV_FRUSTUM: Frustum bounding volume, which is defined by two rectangles each containing an anchor point. l BV_SPHERE: segmented along the path, spherical bounding volume around each point. The bounding volume is defined by the radius of the sphere. intrinsic parameters Boolean false When set to true, indicates that intrinsic camera parameters are modified at each anchor point. Parameters should be provided as camera based on the camera type defined as camera.perspective or camera.orthographic in [glTF 2.0]. accessor number N/A A reference to an accessor or timed accessor that provides camera control information.

Camera control information can be structured as follows: l For each anchor point, the (x, y, z) coordinates of the anchor point can be represented by floating point values l For each path segment, the (i, j) index of the first anchor point and the second anchor point of the path segment can be expressed as an integer value l For bounding volumes: o If the bounding volume is BV_CONE, you can provide the (r1,r2) radii of the circles of the first anchor point and the second anchor point. o If the bounding volume is BV_FRUSTUM, ((x,y,z)_topleft,w,h) can be provided for each anchor point of the path segment. o If the bounding volume is BV_SPHERE, r can be provided as the radius of the sphere for each anchor point of the path segment. l If the intrinsic parameter is true, the intrinsic parameter object can be modified.

A presentation engine (eg, presentation engine 114 of FIG. 2 or another element of client device 40, which may be different than the components shown in FIGS. 1 and 2) may support the MPEG_camera_control extension or other such data structures. If the scene provides camera control information, the rendering engine can restrict camera movement to the indicated path such that the camera's (x, y, z) coordinates are always on the path segment or within the path segment's bounding volume. The rendering engine can provide visual, auditory and/or tactile feedback to viewers as they approach the boundaries of the bounding volume.

FIG. 6 is a conceptual diagram illustrating an example virtual object 220, which in this example is a chair. In order to provide an immersive experience to the viewer, it is important that the viewer interacts correctly with the objects in the scene. Viewers should not be able to walk through solid objects in the scene such as walls, chairs and tables, or other such solid objects.

Figure 6 depicts the 3D mesh representation of the chair and the collision boundaries defined as a collection of cuboids. An MPEG_mesh_collision extension data structure may be defined to provide a description of the collision boundaries of such 3D meshes. Extended data structures can be defined on mesh objects as collections of cuboids around the mesh geometry. Table 2 below represents an example set of attributes that may be included in such an extended data structure. Table 2 name Types of default describe boundary array(object) N/A An array of bounding shapes used to define the collision bounds of the mesh. The boundary can be a sphere or a cuboid. static Boolean True Determines whether objects are affected by collisions. Static objects will not be affected by collisions, meaning that when the viewer or another object collides with it, its position will not change. Material number N/A Index of the collision material, which defines how the collision object or viewer will interact with the object. This can include bounce, friction, etc. animation array(object) N/A Defines animations triggered by collisions or actions on this object. Animations can be limited to a subset of other objects, for example, only the viewer can trigger the animation. It also contains a pointer to the animation that will be executed when triggered.

Mesh collision information may include cuboid vertex coordinates (x, y, z) for cuboid boundaries, or sphere center and radius for spherical boundaries. These values can be provided as floating point numbers.

The rendering engine may support the MPEG_mesh_collision extension or other such data structures. The rendering engine can ensure that the camera position (x,y,z) does not become contained within one of the defined mesh cuboids at any point in time. The collision may be signaled to the viewer by visual, audible and/or tactile feedback. The rendering engine can use the information about the bounds for the nodes to initialize and configure the 3D physics engine that will detect collisions.

7 is a flowchart illustrating an example method of retrieving media data in accordance with the techniques of this disclosure. The method of FIG. 7 is explained with respect to the client device 40 of FIG. 1 and the retrieval unit 52 of FIG. 2 . Other such devices may be configured to perform this or similar methods.

Initially, client device 40 may retrieve media data (250). For example, the retrieval unit 52 may retrieve media data conforming to glTF 2.0, for example. In some examples, retrieval unit 52 may directly retrieve the media data, eg, from unicast, such as using DASH. In some examples, the middleware unit of the retrieval unit 52 (such as the eMBMS middleware 100 of FIG. 2 ) can receive the media data via broadcast or multicast, and then the DASH client (for example, the DASH client 110 of FIG. 2 ) can receive the media data from the intermediate file unit to retrieve media data.

Media data may include scene descriptions. Accordingly, retrieval unit 52 or another component of client device 40 may extract the scene description from the media data (252). According to the techniques of this disclosure, the scene description may be an MPEG scene description including camera control data. The retrieval unit 52 may provide the scene description to the rendering engine 114 . Rendering engine 114 may thus receive the scene description and, in turn, determine camera control data for the three-dimensional scene from the scene description (254). The camera control data may conform to Table 1 above. That is, for example, camera control data may include one or more anchor points for the camera path, one or more segments between anchor points for the camera path, such as a cone, frustum, or sphere. Class bounding volume, intrinsic parameters that can be modified at each anchor point, and/or accessors that provide camera control information.

Rendering engine 114 may further determine movement restrictions based on the camera control data ( 256 ). For example, the rendering engine 114 may determine two or more anchor points and allowed paths between the anchor points based on movement constraints of the camera control data. Additionally or alternatively, the rendering engine 114 may determine a bounding volume, such as a cube, sphere, frustum, cone, etc., based on the movement constraints of the camera control data. The rendering engine 114 may use the allowed paths to determine paths along which the virtual camera is allowed to move and/or paths that the virtual camera is allowed to move within but not outside of the bounding volume. Allowable paths and/or bounding volumes can be defined to ensure that the virtual camera does not go beyond 3D solid virtual objects, such as walls. That is, a bounding volume or allowed path may be defined within one or more 3D solid virtual objects such as walls, floors, ceilings, or other objects in the 3D virtual scene.

Rendering engine 114 may then receive camera movement data (258). For example, rendering engine 114 may receive data from one or more controllers (such as handheld controllers and/or headsets that include displays) that represent the orientation of the headset and movement of the headset and/or virtual camera ( such as directional movement and/or rotational movement). The rendering engine 114 may determine camera movement data requesting movement of the camera through the 3D solid virtual object (such as beyond the bounds of the bounding volume or along a path that is not one of the defined allowed paths) ( 260 ). In response, rendering engine 114 may prevent the virtual camera from traversing the 3D solid virtual object (262).

In this manner, the method of FIG. 7 represents an example of a method of retrieving media data, comprising receiving, by a rendering engine, streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; Camera control data for a three-dimensional scene, the camera control data including data defining constraints to prevent the virtual camera from passing through at least one virtual solid object; camera movement data received by the rendering engine from the user, the camera movement data requesting the virtual camera to move through at least one virtual solid object objects; and using the camera control data, preventing, by the rendering engine, the virtual camera from traversing the at least one virtual solid object in response to the camera movement data.

8 is a flowchart illustrating an example method of retrieving media data in accordance with the techniques of this disclosure. The method of FIG. 8 is explained with respect to the client device 40 of FIG. 1 and the retrieval unit 52 of FIG. 2 . Other such devices may be configured to perform this or similar methods.

Initially, client device 40 may retrieve media data (280). For example, the retrieval unit 52 may retrieve media data conforming to glTF 2.0, for example. In some examples, retrieval unit 52 may retrieve the media data directly, eg, from unicast, such as using DASH. In some examples, the middleware unit of the retrieval unit 52 (such as the eMBMS middleware 100 of FIG. 2 ) can receive the media data via broadcast or multicast, and then the DASH client (for example, the DASH client 110 of FIG. 2 ) can receive the media data from the intermediate file unit to retrieve media data.

Media data may include scene descriptions. Accordingly, retrieval unit 52 or another component of client device 40 may extract the scene description from the media data (282). According to the techniques of this disclosure, the scene description may be an MPEG scene description including object collision data. The retrieval unit 52 may provide the scene description to the rendering engine 114 . Rendering engine 114 may thus receive the scene description and, in turn, determine object collision data for the one or more 3D solid virtual objects based on the scene description ( 284 ). Object collision data may conform to Table 2 above. That is, object collision data may include data representing, for example: the bounds of an array of bounding shapes defining the collision bounds of a mesh (3D virtual solid) object; data indicating whether the object is static (i.e., movable); The material representing the collision material to use for the object; and/or the animation to render for the object in case of collision.

The rendering engine 114 may further determine object collision data from the camera control data (286). For example, the rendering engine 114 may determine a boundary representing an array of boundary shapes defining a collision boundary for a mesh (3D virtual solid) object, data indicating whether the object is static (i.e., movable), representing a collision material for the object material, and/or the animation to render for the object in case of collision. Rendering engine 114 may use object collision data to decide how to react in the event of a collision with a 3D solid virtual object.

The rendering engine 114 may then receive the camera movement data (288). For example, rendering engine 114 may receive data from one or more controllers (such as handheld controllers and/or headsets that include displays) that represent the orientation of the headset and movement of the headset and/or virtual camera ( such as directional movement and/or rotational movement). The rendering engine 114 may determine the camera movement data to request movement of the camera through the 3D solid virtual object (such as into the 3D solid virtual object defined by the object collision data) ( 290 ). In response, rendering engine 114 may prevent the virtual camera from traversing the 3D solid virtual object (292). For example, if the object is static (as indicated by object collision data), rendering engine 114 may prevent the virtual camera from moving into and through the object. As another example, if the object is not static (e.g., movable), the rendering engine 114 may use the object collision data to determine a reaction, such as an animation to play on the object, in response to a collision with the object, e.g., if Object will tip over or move.

In this manner, the method of FIG. 8 represents an example of a method of retrieving media data, the method comprising: receiving, by a rendering engine, streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving, by the engine, object collision data representing a boundary of at least one virtual solid object; receiving, by the rendering engine, camera movement data from the user, the camera movement data requesting that the virtual camera move through the at least one virtual solid object; and using the object collision data, the rendering engine responds The camera movement data is used to prevent the virtual camera from passing through at least one virtual solid object.

Some examples of the techniques of this disclosure are outlined in the following clauses:

Clause 1: A method of retrieving media data, the method comprising: receiving, by a rendering engine, streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving, by the rendering engine, a camera control data, the camera control data including data defining allowable positions for the virtual camera; receiving camera movement data from the user by the rendering engine, the camera movement data requesting the virtual camera to move through the at least one virtual solid object and updating, by the rendering engine, the position of the virtual camera using the camera control data to ensure that the virtual camera remains within the allowable position.

Clause 2: The method of Clause 1, wherein updating the position of the virtual camera includes preventing the virtual camera from passing through the at least one virtual solid object.

Clause 3: The method of Clause 1, wherein the streaming media data includes glTF 2.0 media data.

Clause 4: The method of Clause 1, wherein receiving the streaming media data comprises: requesting the streaming media data from a retrieval unit via an Application Programming Interface (API).

Clause 5: The method of Clause 1, wherein the camera control data is included in an MPEG scene description.

Clause 6: The method of Clause 1, wherein the camera control data includes data defining two or more anchor points and one or more segments between the anchor points, the segments representing an allowable camera movement vector for the camera, and wherein updating the position of the virtual camera includes allowing the virtual camera to traverse only the segment between the anchor points.

Clause 7: The method of Clause 1, wherein the camera control data includes data defining a bounding volume representing an allowable camera movement volume for the virtual camera, and wherein updating the position of the virtual camera includes allowing The virtual camera only crosses the allowable camera movement volume.

Clause 8: The method of Clause 7, wherein the data defining the bounding volume comprises data defining at least one of a cone, frustum, or sphere.

Clause 9: The method of clause 1, wherein the camera control data is included in an MPEG_camera_control extension.

Clause 10: The method of Clause 9, wherein the MPEG_camera_control extension includes one or more of: anchor point data representing the number of anchor points for allowable paths for the virtual camera; segment data, It represents the number of path segments for the allowable path between the anchor points; bounding volume data, which represents the bounding volume for the virtual camera; intrinsic parameters, which indicate whether the camera parameters are within the anchor points modified at each anchor point; and accessor data representing the index of the accessor that provided the camera control data.

Clause 11: The method of Clause 1, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 12: The method of Clause 1, further comprising: determining an allowable path for the virtual camera based on the camera control data, wherein updating the position of the virtual camera comprises ensuring that the virtual camera only follows Virtual path movement within the allowable path defined in the data.

Clause 13: The method of Clause 1, wherein the camera control data is included in an MPEG_mesh_collision extension.

Clause 14: An apparatus for retrieving media data, the apparatus comprising: a memory configured to store the media data; and one or more processors implemented in a circuit and configured to execute a rendering engine, The rendering engine is configured to: receive streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receive camera control data for the three-dimensional scene, the camera control data including definitions for data of allowable positions of the virtual camera; receiving camera movement data from a user requesting the virtual camera to move through the at least one virtual solid object; and using the camera control data, updating the position of the virtual camera to ensure The virtual camera remains within the allowable position.

Clause 15: The apparatus of Clause 14, wherein the rendering engine is configured to: prevent the virtual camera from passing through the at least one virtual solid object.

Clause 16: The device of Clause 14, wherein the streaming media data comprises glTF 2.0 media data.

Clause 17: The device of Clause 14, wherein the rendering engine is configured to: request the streaming media data from the retrieval unit via an application programming interface (API).

Clause 18: The apparatus of Clause 14, wherein the camera control data is included in an MPEG scene description.

Clause 19: The apparatus of Clause 14, wherein the camera control data includes data defining two or more anchor points and one or more segments between the anchor points, the segments representing an allowable camera movement vector for the camera, and wherein, to update the position of the virtual camera, the rendering engine is configured to allow the virtual camera to only traverse the segment between the anchor points.

Clause 20: The apparatus of Clause 14, wherein the camera control data includes data defining a bounding volume representing an allowable camera movement volume for the virtual camera, and wherein, to update the position of the virtual camera, The rendering engine is configured to allow the virtual camera only beyond the allowable camera movement volume.

Clause 21: The device of Clause 20, wherein the data defining the bounding volume comprises data defining at least one of a cone, frustum, or sphere.

Clause 22: The apparatus of Clause 14, wherein the camera control data is included in an MPEG_camera_control extension.

Clause 23: The apparatus of Clause 22, wherein the MPEG_camera_control extension includes one or more of: anchor point data representing the number of anchor points for allowable paths for the virtual camera; segment data, It represents the number of path segments for the allowable path between the anchor points; bounding volume data, which represents the bounding volume for the virtual camera; intrinsic parameters, which indicate whether the camera parameters are within the anchor points modified at each anchor point; and accessor data representing the index of the accessor that provided the camera control data.

Clause 24: The apparatus of Clause 14, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 25: The apparatus of Clause 14, wherein the rendering engine is further configured to determine allowable paths for the virtual camera as the camera control data, wherein, to update the position of the virtual camera, the rendering The engine is configured to ensure that the virtual camera only moves along virtual paths within the allowable paths defined in the camera control data.

Clause 26: The apparatus of Clause 14, wherein the camera control data is included in an MPEG_mesh_collision extension.

Clause 27: A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor executing a rendering engine to: receive streaming media data representing at least one A virtual three-dimensional scene of virtual solid objects; receiving camera control data for the three-dimensional scene, the camera control data including data defining allowable positions for the virtual camera; receiving camera movement data from a user, the camera movement data requesting the virtual moving a camera through the at least one virtual solid object; and updating a position of the virtual camera using the camera control data to ensure that the virtual camera remains within the allowable position.

Clause 28: The computer-readable storage medium of Clause 27, wherein the instructions causing the processor to update the position of the virtual camera comprise instructions causing the processor to prevent the virtual camera from passing through the at least one virtual solid object.

Clause 29: The computer-readable medium of Clause 27, wherein the streaming media data comprises glTF 2.0 media data.

Clause 30: The computer-readable medium of clause 27, wherein the instructions causing the processor to receive the streaming media data comprise instructions causing the processor to request the streaming media data from a retrieval unit via an application programming interface (API). .

Clause 31: The computer-readable medium of Clause 27, wherein the camera control data is included in an MPEG scene description.

Clause 32: The computer-readable medium of Clause 27, wherein the camera control data includes data defining two or more anchor points and one or more segments between the anchor points, the segments representing an allowable camera movement vector for the virtual camera, and wherein the instruction causing the processor to update the position of the virtual camera comprises causing the processor to allow the virtual camera to traverse only the segment between the anchor points instruction.

Clause 33: The computer-readable medium of Clause 27, wherein the camera control data includes data defining a bounding volume representing an allowable camera movement volume for the virtual camera, and wherein the processor is caused to update the The instructions for the position of the virtual camera include instructions for causing the processor to allow the virtual camera to only cross the allowable camera movement volume.

Clause 34: The computer-readable medium of Clause 20, wherein the data defining the bounding volume comprises data defining at least one of a cone, frustum, or sphere.

Clause 35: The computer-readable medium of Clause 27, wherein the camera control data is included in an MPEG_camera_control extension.

Clause 36: The computer-readable medium of Clause 22, wherein the MPEG_camera_control extension includes one or more of: anchor point data representing the number of anchor points for allowable paths for the virtual camera; segment data, which represents the number of path segments for the allowable path between the anchor points; bounding volume data, which represents the bounding volume for the virtual camera; intrinsic parameters, which indicate whether the camera parameters are within the anchor points points are modified at each anchor point; and accessor data representing the index of the accessor that provided the camera control data.

Clause 37: The computer-readable medium of Clause 27, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 38: The computer-readable medium of Clause 27, further comprising: instructions causing the processor to determine allowable paths for the virtual camera based on the camera control data, wherein the processor is caused to update the virtual camera's The instructions for position include instructions for the processor to ensure that the virtual camera moves only along virtual paths within the allowable paths defined in the camera control data.

Clause 39: The computer-readable medium of Clause 27, wherein the camera control data is included in an MPEG_mesh_collision extension.

Clause 40: An apparatus for retrieving media data, the apparatus comprising: means for receiving streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; means for camera control data of the scene, the camera control data including data defining allowable positions for the virtual camera; means for receiving camera movement data from the user, the camera movement data requesting the virtual camera to move through the at least one a virtual solid object; and means for updating the position of the virtual camera using the camera control data to ensure that the virtual camera remains within the allowable position.

Clause 41: A method of retrieving media data, the method comprising: receiving, by a rendering engine, streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving, by the rendering engine, representation of the at least one virtual object collision data for boundaries of solid objects; receiving, by the rendering engine, camera movement data from a user, the camera movement data requesting a virtual camera to move through the at least one virtual solid object; and using the object collision data, updating, by the rendering engine, the A position of the virtual camera to ensure that the virtual camera remains outside the at least one virtual solid object in response to the camera movement data.

Clause 42: The method of Clause 41, wherein updating the position of the virtual camera comprises preventing the virtual camera from passing through the at least one virtual solid object.

Clause 43: The method of Clause 41, wherein receiving the object collision data comprises: receiving an MPEG_mesh_collision extension.

Clause 44: The method of Clause 43, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object.

Clause 45: The method of Clause 44, wherein the MPEG_mesh_collision extension includes data defining at least one of: a boundary of a 3D mesh for the at least one virtual solid object, a material for the 3D mesh , or an animation that will be rendered in response to the virtual camera touching the 3D mesh.

Clause 46: The method of Clause 41, wherein receiving the object collision data comprises receiving data comprising one or more of: boundary data representing one or more collision boundaries of the at least one virtual solid object; static data indicating whether the at least one virtual solid object is affected by the collision; material data indicating how the colliding object interacts with the at least one virtual solid object; or animation data indicating being triggered by a collision with the at least one virtual solid object animation.

Clause 47: The method of Clause 41, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 48: The method of Clause 41, wherein the streaming media data comprises glTF 2.0 media data.

Clause 49: The method of Clause 41, wherein receiving the streaming media data comprises: requesting the streaming media data from a retrieval unit via an Application Programming Interface (API).

Clause 50: The method of Clause 41, wherein the object collision data is included in an MPEG scene description.

Clause 51: An apparatus for retrieving media data, the apparatus comprising: a memory for storing the media data; and one or more processors implemented in a circuit and configured to execute a rendering engine, the rendering engine configured to: receive streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receive object collision data representing boundaries of the at least one virtual solid object; receive camera movement data from a user, the camera movement data requests a virtual camera to move through the at least one virtual solid object; and updating the position of the virtual camera using the object collision data to ensure that the virtual camera remains on the at least one virtual solid in response to the camera movement data outside the object.

Clause 52: The device of Clause 51, wherein, to update the position of the virtual camera, the rendering engine is configured to: prevent the virtual camera from passing through the at least one virtual solid object.

Clause 53: The apparatus of Clause 51, wherein, to receive the object collision data, the rendering engine is configured to: receive an MPEG_mesh_collision extension.

Clause 54: The apparatus of Clause 53, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object.

Clause 55: The apparatus of Clause 54, wherein the MPEG_mesh_collision extension includes data defining at least one of: a boundary of a 3D mesh for the at least one virtual solid object, a material for the 3D mesh , or an animation that will be rendered in response to the virtual camera touching the 3D mesh.

Clause 56: The apparatus of Clause 51, wherein, to receive the object collision data, the rendering engine is configured to receive data comprising one or more of the following: boundary data representing the at least one virtual solid object one or more collision boundaries; static data indicating whether the at least one virtual solid object is affected by the collision; material data indicating how the collision object interacts with the at least one virtual solid object; An animation triggered by the collision of at least one virtual solid object.

Clause 57: The apparatus of Clause 51, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 58: The device of Clause 51, wherein the streaming media data comprises glTF 2.0 media data.

Clause 59: The device of Clause 51, wherein, to receive the streaming media data, the rendering engine is configured to: request the streaming media data from the retrieval unit via an Application Programming Interface (API).

Clause 60: The apparatus of Clause 51, wherein the object collision data is included in an MPEG scene description.

Clause 61: A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor to: receive streaming media data representing an a virtual three-dimensional scene; receiving object collision data representing a boundary of the at least one virtual solid object; receiving camera movement data from a user requesting the virtual camera to move through the at least one virtual solid object; and using the object collision data, The position of the virtual camera is updated to ensure that the virtual camera remains outside the at least one virtual solid object in response to the camera movement data.

Clause 62: The computer-readable medium of Clause 61, wherein the instructions causing the processor to update the position of the virtual camera comprise instructions causing the processor to prevent the virtual camera from passing through the at least one virtual solid object.

Clause 63: The computer-readable medium of Clause 61, wherein the instructions causing the processor to receive the object collision data comprise instructions causing the processor to receive an MPEG_mesh_collision extension.

Clause 64: The computer-readable medium of Clause 62, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object.

Clause 65: The computer-readable medium of Clause 63, wherein the MPEG_mesh_collision extension includes data defining at least one of: a boundary of a 3D mesh for the at least one virtual solid object, a boundary for the 3D mesh The material of the mesh, or the animation that will appear in response to the virtual camera touching the 3D mesh.

Clause 66: The computer-readable medium of Clause 61, wherein the instructions causing the processor to receive the object collision data comprise instructions causing the processor to receive data comprising one or more of: boundary data, which represents one or more collision boundaries of the at least one virtual solid object; static data which represents whether the at least one virtual solid object is affected by the collision; material data which represents how the collision object interacts with the at least one virtual solid object; or animation data representing an animation triggered by a collision with the at least one virtual solid object.

Clause 67: The computer-readable medium of Clause 61, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 68: The computer-readable medium of Clause 61, wherein the streaming media data comprises glTF 2.0 media data.

Clause 69: The computer-readable medium of clause 61, wherein the instructions causing the processor to receive the streaming media data comprise: causing the processor to request the streaming media data from a retrieval unit via an application programming interface (API) instruction.

Clause 70: The computer-readable medium of Clause 61, wherein the object collision data is included in an MPEG scene description.

Clause 71: An apparatus for retrieving media data, the apparatus comprising: means for receiving streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; means for object collision data of a boundary of a virtual solid object; means for receiving camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; and for updating a position of the virtual camera, A means to ensure that the virtual camera remains outside the at least one virtual solid object in response to the camera movement data.

Clause 72: A method of retrieving media data, the method comprising: receiving, by a rendering engine, streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving, by the rendering engine, a camera control data, the camera control data including data defining allowable positions for the virtual camera; receiving camera movement data from the user by the rendering engine, the camera movement data requesting the virtual camera to move through the at least one virtual solid object and updating, by the rendering engine, the position of the virtual camera using the camera control data to ensure that the virtual camera remains within the allowable position.

Clause 73: The method of Clause 72, wherein updating the position of the virtual camera comprises preventing the virtual camera from passing through the at least one virtual solid object.

Clause 74: The method of any one of clauses 72 and 73, wherein the streaming media data comprises glTF 2.0 media data.

Clause 75: The method of any one of clauses 72-74, wherein receiving the streaming media data comprises: requesting the streaming media data from a retrieval unit via an application programming interface (API).

Clause 76: The method of any one of clauses 72-75, wherein the camera control data is included in an MPEG scene description.

Clause 77: The method of any of clauses 72-76, wherein the camera control data includes data defining two or more anchor points and one or more segments between the anchor points, the segment Segments represent allowable camera movement vectors for the virtual camera, and wherein updating the position of the virtual camera includes allowing the virtual camera to traverse only the segment between the anchor points.

Clause 78: The method of any of clauses 72-77, wherein the camera control data includes data defining a bounding volume representing an allowable camera movement volume for the virtual camera, and wherein updating the virtual The position of the camera includes allowing the virtual camera to pass only the allowable camera movement volume.

Clause 79: The method of Clause 78, wherein the data defining the bounding volume comprises data defining at least one of a cone, frustum, or sphere.

Clause 80: The method of any one of clauses 72-79, wherein the camera control data is included in an MPEG_camera_control extension.

Clause 81: The method of Clause 80, wherein the MPEG_camera_control extension includes one or more of: anchor point data representing the number of anchor points for allowable paths for the virtual camera; segment data, It represents the number of path segments for the allowable path between the anchor points; bounding volume data, which represents the bounding volume for the virtual camera; intrinsic parameters, which indicate whether the camera parameters are within the anchor points modified at each anchor point; and accessor data representing the index of the accessor that provided the camera control data.

Clause 82: The method of any one of clauses 72-81, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 83: The method of Clause 72, further comprising: determining an allowable path for the virtual camera based on the camera control data, wherein updating the position of the virtual camera comprises ensuring that the virtual camera only follows Virtual path movement within the allowable path defined in the data.

Clause 84: The method of any of clauses 72-83, wherein the camera control data is included in an MPEG_mesh_collision extension.

Clause 85: An apparatus for retrieving media data, the apparatus comprising: a memory configured to store the media data; and one or more processors implemented in a circuit and configured to execute a rendering engine, The rendering engine is configured to: receive streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receive camera control data for the three-dimensional scene, the camera control data including definitions for data of allowable positions of the virtual camera; receiving camera movement data from a user requesting the virtual camera to move through the at least one virtual solid object; and using the camera control data, updating the position of the virtual camera to ensure The virtual camera remains within the allowable position.

Clause 86: The device of Clause 85, wherein the rendering engine is configured to: prevent the virtual camera from passing through the at least one virtual solid object.

Clause 87: The device of any one of clauses 85 and 86, wherein the streaming media data comprises glTF 2.0 media data.

Clause 88: The device of any one of clauses 85-87, wherein the presentation engine is configured to: request the streaming media data from the retrieval unit via an application programming interface (API).

Clause 89: The apparatus of any one of clauses 85-88, wherein the camera control data is included in an MPEG scene description.

Clause 90: The device of any of Clauses 85-89, wherein the camera control data includes data defining two or more anchor points and one or more segments between the anchor points, the segment Segments represent allowable camera movement vectors for the virtual camera, and wherein, to update the position of the virtual camera, the rendering engine is configured to allow the virtual camera to only traverse the segment between the anchor points.

Clause 91: The apparatus of any of clauses 85-90, wherein the camera control data includes data defining a bounding volume representing an allowable camera movement volume for the virtual camera, and wherein, to update the At the location of the virtual camera, the rendering engine is configured to allow the virtual camera to only cross the allowable camera movement volume.

Clause 92: The device of Clause 91, wherein the data defining the bounding volume comprises data defining at least one of a cone, frustum, or sphere.

Clause 93: The apparatus of any one of clauses 85-92, wherein the camera control data is included in an MPEG_camera_control extension.

Clause 94: The apparatus of Clause 93, wherein the MPEG_camera_control extension includes one or more of: anchor point data representing the number of anchor points for allowable paths for the virtual camera; segment data, It represents the number of path segments for the allowable path between the anchor points; bounding volume data, which represents the bounding volume for the virtual camera; intrinsic parameters, which indicate whether the camera parameters are within the anchor points modified at each anchor point; and accessor data representing the index of the accessor that provided the camera control data.

Clause 95: The apparatus of any of clauses 85-94, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 96: The apparatus of any one of clauses 85-95, wherein the rendering engine is further configured to: determine an allowable path for the virtual camera based on the camera control data, wherein, for updating the virtual camera For this position, the rendering engine is configured to ensure that the virtual camera moves only along virtual paths within the allowable paths defined in the camera control data.

Clause 97: The apparatus of any one of clauses 85-96, wherein the camera control data is included in an MPEG_mesh_collision extension.

Clause 98: A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor executing a rendering engine to: receive streaming media data representing at least one A virtual three-dimensional scene of virtual solid objects; receiving camera control data for the three-dimensional scene, the camera control data including data defining allowable positions for the virtual camera; receiving camera movement data from a user, the camera movement data requesting the virtual moving a camera through the at least one virtual solid object; and updating a position of the virtual camera using the camera control data to ensure that the virtual camera remains within the allowable position.

Clause 99: The computer-readable storage medium of Clause 98, wherein the instructions causing the processor to update the position of the virtual camera comprise instructions causing the processor to prevent the virtual camera from passing through the at least one virtual solid object.

Clause 100: The computer-readable medium of any one of clauses 98 and 99, wherein the streaming media data comprises glTF 2.0 media data.

Clause 101: The computer-readable medium of any one of clauses 98-100, wherein the instructions causing the processor to receive the streaming media data comprise causing the processor to request the stream from a retrieval unit via an application programming interface (API). Instructions for streaming media data.

Clause 102: The computer-readable medium of any one of Clauses 98-101, wherein the camera control data is included in an MPEG scene description.

Clause 103: The computer-readable medium of any one of clauses 98-102, wherein the camera control data includes data defining two or more anchor points and one or more segments between the anchor points , the segment represents an allowable camera movement vector for the virtual camera, and wherein the instruction causing the processor to update the position of the virtual camera includes causing the processor to allow the virtual camera to pass only between the anchor points Instructions for this segment between.

Clause 104: The computer-readable medium of Clause 103, wherein the camera control data includes data defining a bounding volume representing an allowable camera movement volume for the virtual camera, and wherein the processor is caused to update the The instructions for the position of the virtual camera include instructions for causing the processor to allow the virtual camera to only cross the allowable camera movement volume.

Clause 105: The computer-readable medium of any one of clauses 98-104, wherein the data defining the bounding volume comprises data defining at least one of a cone, frustum, or sphere.

Clause 106: The computer-readable medium of Clause 105, wherein the camera control data is included in an MPEG_camera_control extension.

Clause 107: The computer-readable medium of any one of clauses 98-106, wherein the MPEG_camera_control extension includes one or more of: anchor point data representing allowable paths for the virtual camera number of anchor points; segment data, which represents the number of path segments for the allowable path between the anchor points; bounding volume data, which represents the bounding volume for the virtual camera; intrinsic parameters, which indicate the camera whether the parameter is modified at each of the anchor points; and accessor data representing the index of the accessor that provided the camera control data.

Clause 108: The computer-readable medium of any one of clauses 98-107, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 109: The computer-readable medium of any one of clauses 98-108, further comprising: instructions that cause the processor to determine an allowable path for the virtual camera based on the camera control data, wherein the processor is caused to The instructions to update the position of the virtual camera include instructions to cause the processor to ensure that the virtual camera moves only along virtual paths within the allowable paths defined in the camera control data.

Clause 110: The computer-readable medium of any one of clauses 98-109, wherein the camera control data is included in an MPEG_mesh_collision extension.

Clause 111: An apparatus for retrieving media data, the apparatus comprising: means for receiving streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; means for camera control data of the scene, the camera control data including data defining allowable positions for the virtual camera; means for receiving camera movement data from the user, the camera movement data requesting the virtual camera to move through the at least one a virtual solid object; and means for updating the position of the virtual camera using the camera control data to ensure that the virtual camera remains within the allowable position.

Clause 112: A method of retrieving media data, the method comprising: receiving, by a rendering engine, streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving, by the rendering engine, representation of the at least one virtual object collision data for boundaries of solid objects; receiving, by the rendering engine, camera movement data from a user, the camera movement data requesting a virtual camera to move through the at least one virtual solid object; and using the object collision data, updating, by the rendering engine, the A position of the virtual camera to ensure that the virtual camera remains outside the at least one virtual solid object in response to the camera movement data.

Clause 113: A method comprising a combination of the method of any one of clauses 72-84 and the method of clause 112.

Clause 114: The method of any one of clauses 112 and 113, wherein updating the position of the virtual camera comprises preventing the virtual camera from passing through the at least one virtual solid object.

Clause 115: The method of any one of clauses 112-114, wherein receiving the object collision data comprises: receiving an MPEG_mesh_collision extension.

Clause 116: The method of Clause 115, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object.

Clause 117: The method of Clause 116, wherein the MPEG_mesh_collision extension includes data defining at least one of: a boundary of a 3D mesh for the at least one virtual solid object, a material for the 3D mesh , or an animation that will be rendered in response to the virtual camera touching the 3D mesh.

Clause 118: The method of any of clauses 112-117, wherein receiving the object collision data comprises receiving data comprising one or more of: boundary data representing one of the at least one virtual solid object or a plurality of collision boundaries; static data representing whether the at least one virtual solid object is affected by the collision; material data representing how the collision object interacts with the at least one virtual solid object; or animation data representing the interaction between the at least one virtual solid object Animations triggered by collisions of virtual solid objects.

Clause 119: The method of any one of clauses 112-118, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 120: The method of any one of clauses 112-119, wherein the streaming media data comprises glTF 2.0 media data.

Clause 121: The method of any one of clauses 112-120, wherein receiving the streaming media data comprises: requesting the streaming media data from a retrieval unit via an application programming interface (API).

Clause 122: The method of any of clauses 112-121, wherein the object collision data is included in an MPEG scene description.

Clause 123: An apparatus for retrieving media data, the apparatus comprising: memory for storing media data; and one or more processors implemented in circuitry and configured to execute a rendering engine, the rendering engine configured to: receive streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receive object collision data representing boundaries of the at least one virtual solid object; receive camera movement data from a user, the camera movement data requests a virtual camera to move through the at least one virtual solid object; and updating the position of the virtual camera using the object collision data to ensure that the virtual camera remains on the at least one virtual solid in response to the camera movement data outside the object.

Clause 124: A device comprising a combination of the device of any one of clauses 85-97 and the device of clause 123.

Clause 125: The device of any one of clauses 123 and 124, wherein, to update the position of the virtual camera, the rendering engine is configured to: prevent the virtual camera from passing through the at least one virtual solid object.

Clause 126: The apparatus of any one of clauses 123-125, wherein, to receive the object collision data, the rendering engine is configured to: receive an MPEG_mesh_collision extension.

Clause 127: The device of Clause 126, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object.

Clause 128: The apparatus of Clause 127, wherein the MPEG_mesh_collision extension includes data defining at least one of: a boundary of a 3D mesh for the at least one virtual solid object, a material for the 3D mesh , or an animation that will be rendered in response to the virtual camera touching the 3D mesh.

Clause 129: The apparatus of any of Clauses 123-128, wherein, to receive the object collision data, the rendering engine is configured to receive data comprising one or more of the following: boundary data representing One or more collision boundaries of the at least one virtual solid object; static data indicating whether the at least one virtual solid object is affected by the collision; material data indicating how the collision object interacts with the at least one virtual solid object; or animation data , which represents an animation triggered by a collision with the at least one virtual solid object.

Clause 130: The apparatus of any of clauses 123-129, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 131: The device of any one of clauses 123-130, wherein the streaming media data comprises glTF 2.0 media data.

Clause 132: The device of any one of clauses 123-131, wherein, to receive the streaming media data, the rendering engine is configured to: request the streaming media data from the retrieval unit via an application programming interface (API).

Clause 133: The apparatus of any of Clauses 123-132, wherein the object collision data is included in an MPEG scene description.

Clause 134: A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor to: receive streaming media data representing a a virtual three-dimensional scene; receiving object collision data representing a boundary of the at least one virtual solid object; receiving camera movement data from a user requesting the virtual camera to move through the at least one virtual solid object; and using the object collision data, The position of the virtual camera is updated to ensure that the virtual camera remains outside the at least one virtual solid object in response to the camera movement data.

Clause 135: A computer-readable storage medium comprising a combination of the computer-readable storage medium of any one of clauses 98-110 and the computer-readable storage medium of clause 134.

Clause 136: The computer-readable medium of any one of clauses 134 and 135, wherein the instruction causing the processor to update the position of the virtual camera comprises: causing the processor to prevent the virtual camera from traversing the at least one virtual solid object instructions.

Clause 137: The computer-readable medium of any one of clauses 134-136, wherein the instructions causing the processor to receive the object collision data comprise instructions causing the processor to receive an MPEG_mesh_collision extension.

Clause 138: The computer-readable medium of any one of clauses 134-137, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object.

Clause 139: The computer-readable medium of any of Clauses 134-138, wherein the MPEG_mesh_collision extension includes data defining at least one of: a boundary of a 3D mesh for the at least one virtual solid object , a material for the 3D mesh, or an animation to be rendered in response to the virtual camera touching the 3D mesh.

Clause 140: The computer-readable medium of any one of clauses 134-139, wherein the instructions causing the processor to receive the object collision data comprise causing the processor to receive data comprising one or more of the following Instructions for: boundary data, which represents one or more collision boundaries of the at least one virtual solid object; static data, which represents whether the at least one virtual solid object is affected by collision; material data, which represents how the collision object interacts with the at least one virtual solid object interaction; or animation data representing an animation triggered by a collision with the at least one virtual solid object.

Clause 141: The computer-readable medium of any one of clauses 134-140, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 142: The computer-readable medium of any one of clauses 134-141, wherein the streaming media data comprises glTF 2.0 media data.

Clause 143: The computer-readable medium of any one of clauses 134-142, wherein the instructions causing the processor to receive the streaming media data comprise: causing the processor to request from the retrieval unit via an application programming interface (API) The instruction to stream media data.

Clause 144: The computer-readable medium of any one of clauses 134-143, wherein the object collision data is included in an MPEG scene description.

Clause 145: A method of retrieving media data, the method comprising: receiving, by a rendering engine, streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving, by the rendering engine, a camera control data, the camera control data including data defining restrictions to prevent the virtual camera from traversing the at least one virtual solid object; receiving camera movement data from the user by the rendering engine, the camera movement data requesting the virtual camera to move through the at least one virtual solid object a virtual solid object; and using the camera control data, preventing the virtual camera from traversing the at least one virtual solid object in response to the camera movement data.

Clause 146: The method of Clause 145, wherein the streaming media data comprises glTF 2.0 media data.

Clause 147: The method of any one of clauses 145 and 146, wherein receiving the streaming media data comprises: requesting the streaming media data from a retrieval unit via an application programming interface (API).

Clause 148: The method of any one of clauses 145-147, wherein the camera control data is included in an MPEG scene description.

Clause 149: The method of any one of clauses 145-148, wherein the camera control data is included in an MPEG_camera_control extension.

Clause 150: The method of Clause 149, wherein the MPEG_camera_control extension includes data defining two or more anchor points and one or more segments between the anchor points, the segments representing allowable camera movement vectors .

Clause 151: The method of any one of clauses 145-148, wherein the MPEG_camera_control extension includes data defining a bounding volume representing an allowable camera movement volume.

Clause 152: The method of Clause 151, wherein the data defining the bounding volume comprises data defining at least one of a cone, frustum, or sphere.

Clause 153: The method of any one of clauses 149-152, wherein the MPEG_camera_control extension conforms to the data of Table 1 above.

Clause 154: The method of any one of clauses 149-153, wherein the at least one virtual solid object comprises a virtual wall.

Clause 155: The method of any one of clauses 149-154, wherein preventing the virtual camera from traversing the at least one virtual solid object comprises: preventing the virtual camera from following a virtual path that exceeds an allowable path defined in the MPEG_camera_control extension move.

Clause 156: The method of any one of clauses 145-155, wherein the camera control data is included in an MPEG_mesh_collision extension.

Clause 157: The method of any of clauses 145-155, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object.

Clause 158: The method of Clause 157, wherein the MPEG_mesh_collision extension includes data defining at least one of: boundaries of the 3D mesh, materials for the 3D mesh, or will respond to the virtual camera An animation rendered by touching the 3D mesh.

Clause 159: The method of any one of clauses 156-158, wherein the MPEG_mesh_collision extension conforms to Table 2 above.

Clause 160: The method of any of clauses 156-159, wherein preventing the virtual camera from passing through the at least one virtual solid object comprises: using the MPEG_mesh_collision extension to prevent the virtual camera from entering the at least one virtual solid object.

Clause 161: An apparatus for retrieving media data, the apparatus comprising one or more means for performing the method of any one of clauses 145-160.

Clause 162: The apparatus of Clause 161, wherein the one or more means comprise one or more processors implemented in circuitry.

Clause 163: The device of Clause 161, wherein the apparatus comprises at least one of: an integrated circuit; a microprocessor; and a wireless communication device.

Clause 164: An apparatus for retrieving media data, the apparatus comprising: means for receiving streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; means for camera control data of the scene, the camera control data comprising data defining limits to prevent the virtual camera from traversing the at least one virtual solid object; means for receiving camera movement data from the user requesting the virtual camera to move through passing the at least one virtual solid object; and means for using the camera control data to prevent the virtual camera from passing through the at least one virtual solid object in response to the camera movement data.

In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, eg, according to a communication protocol. In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include computer readable media.

By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or may be used in Store desired program code in the form of instructions or data structures and any other medium that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cables, fiber optic cables, twisted pair cables, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then coaxial cables, Fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs use Injection to optically replicate data. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor," as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into a combined codec. Additionally, the technology may be implemented entirely in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or appliances, including a wireless handset, an integrated circuit (IC), or a group of ICs (eg, a chipset). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit, or by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and /or firmware to provide.

Various examples have been described. These and other examples are within the scope of the claims that follow.

10: System 20: Content preparation device 22: Audio source 24: Video source 26:Audio encoder 28:Video Encoder 30: Encapsulation unit 32: output interface 40: Client device 42: Audio output 44: Video output 46:Audio decoder 48:Video decoder 50: Decapsulation unit 52: retrieval unit 54: Network interface 60: Server device 62: Storage media 64: Multimedia content 66:Manifest file 68A-68N: Indicates 70: request processing unit 72: Network interface 74: Network 100: eMBMS middleware unit 102: proxy server unit 104: Cache 106: eMBMS receiving unit 110: DASH client 112:Media application 114: Rendering engine 120: Multimedia content 122:Media Presentation Description 124A-124N: Indicates 126, 130: header data 128A-128N, 132A-132N: Segmentation 150:Video Archives 152: File Type (FTYP) Box 154: Movie (MOOV) box 156:Movie header (MVHD) box 158: Track (TRAK) box 160: Movie Extended (MVEX) Box 162:Segmented Index (SIDX) Box 164:Movie fragment (MOOF) box 166:Movie Fragment Random Access (MFRA) Box 200:3D scene 202: camera 204, 206: points 208, 210: bounding box 212: Path segmentation 220: Virtual objects 250: Retrieve media data 252: Extract scene description 254:Determine camera control data according to scene description 256: Determine movement restrictions based on camera control data 258: Receive camera movement data 260: Determining Camera Movement Data Requests Movement Through 3D Solid Virtual Objects 262: Prevent the virtual camera from passing through 3D solid virtual objects 280: Retrieve media data 282: Extract scene description 284:Determine camera control data according to scene description 286: Determine object collision data based on camera control data 288: Receive camera movement data 290:Determine camera movement data request movement through 3D solid virtual objects 292:Prevent virtual camera from passing through 3D solid virtual objects

1 is a block diagram illustrating an example system that implements techniques for streaming media data over a network.

FIG. 2 is a block diagram illustrating an example component set of the retrieval unit of FIG. 1 in more detail.

FIG. 3 is a conceptual diagram illustrating elements of example multimedia content.

4 is a block diagram illustrating elements of an example video archive that may correspond to segments of a representation.

5 is a conceptual diagram illustrating an example camera path segmentation with a bounding volume in accordance with the techniques of this disclosure.

FIG. 6 is a conceptual diagram illustrating an example virtual object, which in this example is a chair.

7 is a flowchart illustrating an example method of retrieving media data in accordance with the techniques of this disclosure.

8 is a flowchart illustrating an example method of retrieving media data in accordance with the techniques of this disclosure.

280: Retrieve media data

282: Extract scene description

284:Determine camera control data according to scene description

286: Determine object collision data based on camera control data

288: Receive camera movement data

290:Determine camera movement data request movement through 3D solid virtual objects

292:Prevent virtual camera from passing through 3D solid virtual objects

Claims (31)

A method of retrieving media data, the method comprising: receiving, by the rendering engine, streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving, by the rendering engine, object collision data representing a boundary of the at least one virtual solid object; receiving, by the rendering engine, camera movement data from a user requesting movement of a virtual camera through the at least one virtual solid object; and Using the object collision data, the position of the virtual camera is updated by the rendering engine to ensure that the virtual camera remains outside the at least one virtual solid object in response to the camera movement data. The method of claim 1, wherein updating the position of the virtual camera includes: preventing the virtual camera from passing through the at least one virtual solid object. The method of claim 1, wherein receiving the object collision data includes: receiving an MPEG_mesh_collision extension. The method of claim 3, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object. The method of claim 4, wherein the MPEG_mesh_collision extension includes data defining at least one of: a boundary of a 3D mesh for the at least one virtual solid object, a material for the 3D mesh, or An animation that will be rendered in response to the virtual camera touching the 3D mesh. The method of claim 1, wherein receiving the object collision data includes receiving data including one or more of the following items: boundary data representing one or more collision boundaries of the at least one virtual solid object; static data indicating whether the at least one virtual solid object is affected by a collision; material data representing how the collision object interacts with the at least one virtual solid object; or animation data representing an animation triggered by a collision with the at least one virtual solid object. The method according to claim 1, wherein the at least one virtual solid object includes one of a virtual wall, a virtual chair, or a virtual table. The method of claim 1, wherein the streaming media data includes glTF 2.0 media data. The method of claim 1, wherein receiving the streaming media data includes: requesting the streaming media data from a retrieval unit through an application program interface (API). The method of claim 1, wherein the object collision data is included in an MPEG scene description. A device for retrieving media data, the device comprising: memory for storing media data; and One or more processors implemented in circuitry and configured to execute a rendering engine configured to: receiving streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving object collision data representing a boundary of the at least one virtual solid object; receiving camera movement data from a user requesting virtual camera movement through the at least one virtual solid object; and Using the object collision data, the position of the virtual camera is updated to ensure that the virtual camera remains outside the at least one virtual solid object in response to the camera movement data. The device of claim 11, wherein, in order to update the position of the virtual camera, the rendering engine is configured to prevent the virtual camera from passing through the at least one virtual solid object. The apparatus of claim 11, wherein, to receive the object collision data, the rendering engine is configured to receive an MPEG_mesh_collision extension. The apparatus of claim 13, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object. The apparatus of claim 14, wherein the MPEG_mesh_collision extension includes data defining at least one of: a boundary of a 3D mesh for the at least one virtual solid object, a material for the 3D mesh, or An animation that will be rendered in response to the virtual camera touching the 3D mesh. The device of claim 11, wherein, in order to receive the object collision data, the rendering engine is configured to receive data comprising one or more of the following: boundary data representing one or more collision boundaries of the at least one virtual solid object; static data indicating whether the at least one virtual solid object is affected by a collision; material data representing how the collision object interacts with the at least one virtual solid object; or animation data representing an animation triggered by a collision with the at least one virtual solid object. The device according to claim 11, wherein the at least one virtual solid object includes one of a virtual wall, a virtual chair, or a virtual table. The device according to claim 11, wherein the streaming media data includes glTF 2.0 media data. The device of claim 11, wherein, in order to receive the streaming media data, the presentation engine is configured to request the streaming media data from a retrieval unit via an Application Programming Interface (API). The apparatus of claim 11, wherein the object collision data is included in an MPEG scene description. A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor to: receiving streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving object collision data representing a boundary of the at least one virtual solid object; receiving camera movement data from a user requesting virtual camera movement through the at least one virtual solid object; and Using the object collision data, the position of the virtual camera is updated to ensure that the virtual camera remains outside the at least one virtual solid object in response to the camera movement data. The computer-readable medium of claim 21, wherein the instructions for causing the processor to update the position of the virtual camera comprise instructions for causing the processor to prevent the virtual camera from passing through the at least one virtual solid object. The computer-readable medium of claim 21, wherein the instruction for enabling the processor to receive the object collision data comprises: an instruction for enabling the processor to receive an MPEG_mesh_collision extension. The computer readable medium of claim 22, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object. The computer-readable medium of claim 23, wherein the MPEG_mesh_collision extension includes data defining at least one of: a boundary of a 3D mesh for the at least one virtual solid object, a boundary for the 3D mesh material, or animation that will be rendered in response to the virtual camera touching the 3D mesh. The computer-readable medium of claim 21, wherein the instructions for causing the processor to receive the object collision data include instructions for causing the processor to receive data comprising one or more of the following: boundary data representing one or more collision boundaries of the at least one virtual solid object; static data indicating whether the at least one virtual solid object is affected by a collision; material data representing how the collision object interacts with the at least one virtual solid object; or animation data representing an animation triggered by a collision with the at least one virtual solid object. The computer-readable medium of claim 21, wherein the at least one virtual solid object includes one of a virtual wall, a virtual chair, or a virtual table. The computer-readable medium of claim 21, wherein the streaming media data includes glTF 2.0 media data. The computer-readable medium of claim 21, wherein the instruction for causing the processor to receive the streaming media data comprises: an instruction for causing the processor to request the streaming media data from a retrieval unit via an application program interface (API). The computer readable medium of claim 21, wherein the object collision data is included in an MPEG scene description. A device for retrieving media data, the device comprising: means for receiving streaming media data representing a virtual three-dimensional scene including at least one virtual solid object; means for receiving object collision data representing a boundary of the at least one virtual solid object; means for receiving camera movement data from a user requesting virtual camera movement through the at least one virtual solid object; and means for updating the position of the virtual camera using the object collision data to ensure that the virtual camera remains outside of the at least one virtual solid object in response to the camera movement data.

Images