KR20230155444A - Camera control data for virtual cameras in a virtual interactive scene defined by streamed media data - Google Patents

Abstract

Exemplary devices for retrieving media data include a memory configured to store media data; and one or more processors implemented in circuitry and configured to execute a presentation engine, the presentation engine configured to: receive streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; Receive camera control data for the three-dimensional scene, the camera control data including data defining constraints to prevent the virtual camera from passing through at least one virtual solid object; receive camera movement data from a user requesting that the virtual camera move through at least one virtual solid object; and, using the camera control data, prevent the virtual camera from passing the at least one virtual solid object in response to the camera movement data.

Description

Camera control data for virtual cameras in a virtual interactive scene defined by streamed media data

This application claims priority to U.S. Patent Application No. 17/654,020, filed March 8, 2022, and U.S. Provisional Application No. 63/159,379, filed March 10, 2021, each of which The entire contents are incorporated herein by reference. U.S. Patent Application No. 17/654,020, filed March 8, 2022, claims the benefit of U.S. Provisional Application No. 63/159,379, filed March 10, 2021.

technology field

This disclosure relates to storage and transmission of encoded video data.

background

Digital video capabilities include digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, digital cameras, digital recording devices, digital media players, It can be included in a wide range of devices, including video gaming devices, video game consoles, cellular or satellite wireless phones, teleconferencing devices, etc. Digital video devices must support MPEG-2, MPEG-4, ITU-T H.263 or ITU-T H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 (High Efficiency Video Implement video compression techniques, such as techniques described in the standards defined by (also referred to as HEVC), and extensions of those standards, to transmit and receive digital video information more efficiently.

Video compression techniques perform spatial prediction and/or temporal prediction to reduce or eliminate redundancy inherent in video sequences. For block-based video coding, a video frame or slice may be partitioned into macroblocks. Each macroblock can be further partitioned. Macroblocks in an intra-coded (I) frame or slice are encoded using spatial prediction with respect to neighboring macroblocks. Macroblocks in an inter-coded (P or B) frame or slice may use spatial prediction relative to neighboring macroblocks in the same frame or slice or temporal prediction relative to other reference frames.

After the video data is encoded, the video data may be packetized for transmission or storage. Video data may be assembled into a video file that conforms to any of a variety of standards, such as the International Organization for Standardization (ISO) base media file format and its extensions, such as AVC.

outline

In general, this disclosure describes techniques related to streaming interactive media data. Such interactive media data may be, for example, virtual reality, augmented reality, or other such interactive content, for example, other three-dimensional video content. The latest MPEG Scene Description element includes support for timed media in glTF 2.0. The media access function (MAF) provides an application programming interface (API) to the presentation engine, through which the presentation engine may request timed media. A retrieval unit executing the MAF may process the retrieved timed media data and deliver the processed media data in the desired format via circular buffers to the presentation engine. Current MPEG scene descriptions allow users to consume scene media data with six degrees of freedom (6DoF). Accordingly, users can typically move freely within the 3D scene (eg, through walls displayed in the 3D scene). However, content creators may wish to impose restrictions on the viewer's movement to certain areas, for example, to prevent movement through displayed walls or other objects. This disclosure describes techniques for imposing such restrictions that may improve the user's experience because the experience may be made more realistic by preventing the user from traversing obstacles in the virtual world.

In one example, a method of retrieving media data includes receiving, by a presentation engine, streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; Receiving, by the presentation engine, camera control data for the three-dimensional scene, the camera control data comprising data defining constraints to prevent the virtual camera from passing through at least one virtual solid object; Receiving, by the presentation engine, camera movement data from a user requesting that the virtual camera move through at least one virtual solid object; and preventing, by the presentation engine, using the camera control data, the virtual camera from passing the at least one virtual solid object in response to the camera movement data.

In another example, a device for retrieving media data includes a memory configured to store media data; and one or more processors implemented in circuitry and configured to execute a presentation engine, the presentation engine configured to: receive streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; Receive camera control data for the three-dimensional scene, the camera control data including data defining constraints to prevent the virtual camera from passing through at least one virtual solid object; receive camera movement data from a user requesting that the virtual camera move through at least one virtual solid object; and, using the camera control data, prevent the virtual camera from passing the at least one virtual solid object in response to the camera movement data.

In another example, a computer-readable storage medium stores instructions that, when executed, cause a processor of a client device to display streamed media data representing a virtual three-dimensional scene including at least one virtual solid object. to receive; Receive camera control data for the three-dimensional scene, the camera control data including data defining constraints to prevent the virtual camera from passing through at least one virtual solid object; receive camera movement data from a user requesting that the virtual camera move through at least one virtual solid object; Use the camera control data to prevent the virtual camera from passing at least one virtual solid object in response to the camera movement data.

In another example, a device for retrieving media data includes means for receiving streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; means for receiving camera control data for a three-dimensional scene, the camera control data comprising data defining constraints to prevent the virtual camera from passing through at least one virtual solid object; means for receiving camera movement data from a user requesting that the virtual camera move through at least one virtual solid object; and means for using the camera control data to prevent the virtual camera from passing the at least one virtual solid object in response to the camera movement data.

In another example, a method of retrieving media data includes receiving, by a presentation engine, streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; Receiving, by the presentation engine, object collision data representing boundaries of at least one virtual solid object; Receiving, by the presentation engine, camera movement data from a user requesting that the virtual camera move through at least one virtual solid object; and preventing, by the presentation engine, using the object collision data, the virtual camera from passing the at least one virtual solid object in response to the camera movement data.

In another example, a device for retrieving media data includes a memory configured to store media data; and one or more processors implemented in circuitry and configured to execute a presentation engine, the presentation engine configured to: receive streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; receive object collision data representing boundaries of at least one virtual solid object; receive camera movement data from a user requesting that the virtual camera move through at least one virtual solid object; and prevent the virtual camera from passing the at least one virtual solid object in response to the camera movement data, using the object collision data.

In another example, a computer-readable storage medium having instructions stored thereon that, when executed, cause a processor of a client device to: receive streamed media data representing a virtual three-dimensional scene including at least one virtual solid object. do; receive object collision data representing boundaries of at least one virtual solid object; receive camera movement data from a user requesting that the virtual camera move through at least one virtual solid object; The object collision data is used to prevent the virtual camera from passing at least one virtual solid object in response to the camera movement data.

In another example, a device for retrieving media data includes means for receiving streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; means for receiving object collision data representing boundaries of at least one virtual solid object; means for receiving camera movement data from a user requesting that the virtual camera move through at least one virtual solid object; and means for using object collision data to prevent the virtual camera from passing the at least one virtual solid object in response to the camera movement data.

Details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects and advantages will become apparent from the detailed description and drawings and the claims.

Brief description of the drawings
1 is a block diagram illustrating an example system implementing techniques for streaming media data over a network.
FIG. 2 is a block diagram illustrating an example set of components of the extraction unit of FIG. 1 in more detail.
3 is a conceptual diagram showing elements of example multimedia content.
4 is a block diagram illustrating elements of an example video file, which may correspond to segments of a representation.
5 is a conceptual diagram illustrating an example camera path segment with a bounding volume, according to the techniques of this disclosure.
6 is a conceptual diagram illustrating an example virtual object, which in this example is a chair.
7 is a flow chart illustrating an example method of retrieving media data according to the techniques of this disclosure.
8 is a flow chart illustrating an example method of retrieving media data according to the techniques of this disclosure.

details

Interactive media data may also be streamed over a network. For example, a client device may retrieve interactive media data using unicast, broadcast, multicast, etc. Interactive media data may be, for example, three-dimensional (3D) media data for extended reality (XR), augmented reality (AR), virtual reality (VR), etc. Accordingly, when presented to a user, the user may navigate a 3D virtual scene rendered according to the interactive media data.

An MPEG scene description may describe a three-dimensional (3D) scene for a virtual world or experience, such as XR, VR, AR, or other interactive media experiences. According to the techniques of this disclosure, an MPEG scene description may describe objects within a 3D scene, such as chairs, walls, tables, counters, doors, windows, or other solid objects. This disclosure describes techniques in which an MPEG scene description (or other such descriptive data sets) may be enhanced to impose restrictions on virtual camera movement, e.g., to prevent the camera from passing through solid objects such as walls. explain them.

In particular, a scene description may describe a set of paths along which the camera is allowed to move. The paths may be described as a set of anchor points connected through path segments. For improved expressiveness of camera control, each path segment may be enhanced with a bounding volume that allows some degree of freedom in motion along the path.

Additionally or alternatively, a scene description may describe virtual solid objects in the scene. The scene description can include, for example, the boundaries of objects, whether an object can be affected by collisions with the user or other objects (e.g., whether the object moves in response to such collisions or remains stationary). information representing information representing objects, material about objects indicating how colliding objects interact with them, and/or animation data indicating animations to be played or applied to the objects in response to the collision.

The techniques of the present disclosure include an ISO base media file format, SVC (Scalable Video Coding) file format, AVC (Advanced Video Coding) file format, 3GPP (Third Generation Partnership Project) file format, and/or MVC (Multiview Video Coding) file format. , or may be applied to video files that follow video data encapsulated according to any of the file formats.

In HTTP streaming, frequently used operations include HEAD, GET, and partial GET. The HEAD operation retrieves the header of the file associated with a given uniform resource locator (URL) or uniform resource name (URN), without retrieving the payload associated with that URL or URN. The GET operation retrieves the entire file associated with a given URL or URN. A partial GET operation receives a byte range as an input parameter and retrieves a consecutive number of bytes from the file, the number of bytes corresponding to the received byte range. Accordingly, movie fragments may be provided for HTTP streaming, since a partial GET operation can obtain one or more individual movie fragments. In a movie fragment, there may be several track fragments of different tracks. In HTTP streaming, media presentation may be a structured collection of data accessible to the client. The client may request and download media data information to present the streaming service to the user.

In the example of streaming 3GPP data using HTTP streaming, there may be multiple representations of the video and/or audio data of the multimedia content. As explained below, different representations may have different coding characteristics (e.g., different profiles or levels of a video coding standard), different coding standards or extensions of coding standards (e.g., multiview and/or scalable extensions), or may correspond to different bitrates. The manifest of such representations may be defined in a Media Presentation Description (MPD) data structure. A media presentation may correspond to a structured collection of data accessible to an HTTP streaming client device. An HTTP streaming client device may request and download media data information to present streaming services to a user of the client device. A media presentation may be described in an MPD data structure, which may include updates of the MPD.

A media presentation may include a sequence of one or more periods. Each period may extend until the beginning of the next period, or in the case of the last period, until the end of the media presentation. Each period may include one or more representations of the same media content. The representation may be audio, video, timed text, or other one of a number of alternative encoded versions of such data. Representations may differ by encoding types, such as bitrate, resolution, and/or codec for video data and bitrate, language, and/or codec for audio data. The term representation may be used to refer to a section of encoded audio or video data that corresponds to a specific period of multimedia content and is encoded in a specific way.

Representations of a particular period may be assigned to a group indicated by an attribute in the MPD that indicates the adaptation set to which the representations belong. Representations in the same adaptation set are generally alternatives to each other, in that the client device can dynamically and seamlessly switch between these representations, for example, to perform bandwidth adaptation. is considered. For example, each representation of video data for a particular period of time may be assigned to the same adaptation set, such that any of the representations may contain video data or audio of the multimedia content for the corresponding period. It may also be selected for decoding to present media data such as data. Media content within a period may, in some examples, be represented by either one representation from group 0, if present, or a combination of at most one representation from each non-zero group. . Timing data for each representation of a period may be expressed relative to the start time of that period.

A representation may include one or more segments. Each representation may include an initialization segment, or each segment of the representation may be self-initialized. If present, the initialization segment may contain initialization information for accessing the representation. Typically, the initialization segment does not contain media data. A segment may be uniquely referenced by an identifier, such as a uniform resource locator (URL), uniform resource name (URN), or uniform resource identifier (URI). MPD may provide identifiers for each segment. In some examples, MPD may provide byte ranges in the form of a range attribute that may correspond to data for a URL, URN, or segment within a file accessible by a URI.

Different representations may be selected for substantially simultaneous retrieval of different types of media data. For example, a client device may select an audio representation, a video representation, and a timed text representation from which to retrieve segments. In some examples, a client device may select specific adaptation sets to perform bandwidth adaptation. That is, the client device may select an adaptation set containing video representations, an adaptation set containing audio representations, and/or an adaptation set containing timed text. Alternatively, the client device may select adaptation sets for some types of media (e.g., video) and directly select representations for other types of media (e.g., audio and/or timed text). .

1 is a block diagram illustrating an example system implementing techniques for streaming media data over a network. In this example, system 10 includes content preparation device 20, server device 60, and client device 40. Client device 40 and server device 60 are communicatively coupled by network 74, which may include the Internet. In some examples, content preparation device 20 and server device 60 may also be coupled by network 74 or another network, or directly communicatively coupled. In some examples, content preparation device 20 and server device 60 may include the same device.

Content preparation device 20, in the example of FIG. 1, includes audio source 22 and video source 24. Audio source 22 may include, for example, a microphone that generates electrical signals representative of captured audio data to be encoded by audio encoder 26. Alternatively, audio source 22 may include a storage medium that stores previously recorded audio data, an audio data generator such as a computerized synthesizer, or any other source of audio data. Video source 24 may be a video data generating unit such as a video camera that produces video data to be encoded by video encoder 28, a storage medium encoded with previously recorded video data, a computer graphics source, or any of the video data. It may also include other sources. Content preparation device 20 need not be communicatively coupled to server device 60 in all examples, but may store multimedia content on a separate medium that is read by server device 60 .

Raw audio and video data may include analog or digital data. Analog data may be digitized before being encoded by audio encoder 26 and/or video encoder 28. Audio source 22 may obtain audio data from the speaking participant while the speaking participant is speaking, and video source 24 may simultaneously obtain video data of the speaking participant. In other examples, audio source 22 may include a computer-readable storage medium containing stored audio data, and video source 24 may include a computer-readable storage medium containing stored video data. In this way, the techniques described in this disclosure may be applied to live, streaming, real-time audio and video data or to archived, pre-recorded audio and video data.

Audio frames corresponding to video frames typically include video data captured (or generated) by video source 24 and audio that was captured (or generated) by audio source 22 simultaneously with the video data contained within the video frames. These are audio frames that contain data. For example, while a speaking participant typically generates audio data by speaking, audio source 22 captures the audio data, and video source 24 simultaneously captures the audio data, i.e., audio source 22 captures the audio data. While capturing, capture video data of the speaking participant. So, an audio frame may correspond temporally to one or more specific video frames. Accordingly, an audio frame corresponding to a video frame generally corresponds to a situation where audio data and video data have been captured simultaneously and the audio frame and video frame include simultaneously captured audio data and video data, respectively.

In some examples, audio encoder 26 may encode a timestamp within each encoded audio frame that indicates the time the audio data for the encoded audio frame was recorded, and similarly, video encoder 28 may encode the encoded audio frame. A timestamp may be encoded within each encoded video frame that indicates when video data for the video frame was recorded. In these examples, the audio frame that corresponds to the video frame may include an audio frame that includes a timestamp and a video frame that includes the same timestamp. Content preparation device 20 may include an internal clock from which audio encoder 26 and/or video encoder 28 may generate timestamps, or audio source 22 and video encoder 28 may generate timestamps. Source 24 may utilize audio and video data to associate each with a timestamp.

In some examples, audio source 22 may transmit data corresponding to the time the audio data was recorded to audio encoder 26, and video source 24 may transmit data corresponding to the time the video data was recorded to the video encoder. It can also be sent to (28). In some examples, audio encoder 26 may encode a sequence identifier within the encoded audio data that indicates the relative temporal ordering of the encoded audio data, but does not necessarily indicate the absolute time at which the audio data was recorded, and similarly , video encoder 28 may also use a sequence identifier to indicate the relative temporal ordering of the encoded video data. Similarly, in some examples, a sequence identifier may be mapped to or otherwise correlated with a timestamp.

Audio encoder 26 generally produces a stream of encoded audio data, while video encoder 28 produces a stream of encoded video data. Each individual stream of data (whether audio or video) may be referred to as an elementary stream. An elementary stream is a single, digitally coded (possibly compressed) component of the representation. For example, the coded video or audio portion of the representation may be an elementary stream. *The elementary stream may be converted to a packetized elementary stream (PES) before being encapsulated within the video file. Within the same representation, the stream ID may be used to distinguish PES packets belonging to one elementary stream from others. The basic unit of data in an elementary stream is a packetized elementary stream (PES) packet. Accordingly, coded video data generally corresponds to elementary video streams. Similarly, audio data corresponds to one or more respective elementary streams.

Many video coding standards, such as ITU-T H.264/AVC and the emerging High Efficiency Video Coding (HEVC) standard, define syntax, semantics, and decoding processes for error-free bitstreams, some of which require predetermined Conforms to the profile or level of Video coding standards typically do not specify the encoder, but the encoder is responsible for ensuring that the generated bitstreams are standards compatible for the decoder. In the context of a video coding standard, “profile” corresponds to a subset of algorithms, features, or tools and the constraints that apply to them. As defined by the H.264 standard, for example, a “profile” is a subset of the overall bitstream syntax specified by the H.264 standard. “Level” corresponds to the limits of decoder resource consumption, for example decoder memory and computation, which are related to the resolution of the pictures, bit rate, and block processing rate. A profile may be signaled with a profile_idc (profile indicator) value, while a level may be signaled with a level_idc (level indicator) value.

The H.264 standard specifies the performance of encoders and decoders depending on the values taken by syntax elements in the bitstream, for example, the specified size of the decoded pictures, within the boundaries imposed by the syntax of a given profile. We recognize that it is still possible to require large changes in . Additionally, the H.264 standard recognizes that in many applications, it is neither practical nor economical to implement a decoder capable of handling all hypothetical uses of syntax within a particular profile. Accordingly, the H.264 standard defines a “level” as a specified set of constraints imposed on the values of syntax elements in the bitstream. These constraints may be simple restrictions on values. Alternatively, these constraints may take the form of constraints on arithmetic combinations of values (e.g., picture width times picture height times number of pictures decoded per second). Additionally, the H.264 standard specifies that individual implementations may support different levels for each supported profile.

A decoder conforming to a profile usually supports all features defined in the profile. For example, as a coding feature, B-picture coding is not supported in the baseline profile of H.264/AVC, but is supported in other profiles of H.264/AVC. A level-compliant decoder should be able to decode arbitrary bitstreams without requiring resources beyond the limits defined at that level. Definitions of profiles and levels may aid interpretability. For example, during video transmission, a pair of profile and level definitions may be negotiated and agreed upon during the entire transmission session. More specifically, in H.264/AVC, the level is the number of macroblocks that need to be processed, the decoded picture buffer (DPB) size, and the coded picture buffer (CPB) size. , vertical motion vector range, maximum number of motion vectors per two consecutive MBs, and whether a B-block can have sub-macroblock partitions of less than 8x8 pixels. In this way, the decoder may determine whether it is possible for the decoder to properly decode the bitstream.

In the example of FIG. 1 , encapsulation unit 30 of content preparation device 20 encodes elementary streams containing coded video data from video encoder 28 and encodes elementary streams containing coded audio data from audio encoder 26. ) is received from. In some examples, video encoder 28 and audio encoder 26 may each include packetizers to form PES packets from encoded data. In other examples, video encoder 28 and audio encoder 26 may each interface with a respective packetizer to form PES packets from encoded data. In still other examples, encapsulation unit 30 may include packetizers for forming PES packets from encoded audio and video data.

Video encoder 28 encodes video data of multimedia content in various ways, at various bitrates and with various characteristics, such as pixel resolutions, frame rates, conformance to various coding standards, and various Compliance with various profiles and/or levels of profiles for coding standards, representations with one or multiple views (e.g. for two-dimensional or three-dimensional playback), or multimedia content with other such characteristics. You can also create different representations of . A representation, as used in this disclosure, may include one of audio data, video data, text data (e.g., for closed captions), or other such data. A representation may also include elementary streams, such as an audio elementary stream or a video elementary stream. Each PES packet may include a stream_id that identifies the elementary stream to which the PES packet belongs. Encapsulation unit 30 is responsible for assembling the elementary streams into video files (eg, segments) of various representations.

Encapsulation unit 30 receives PES packets for the elementary streams of the representation from audio encoder 26 and video encoder 28 and generates corresponding network abstraction layer (NAL) units from the PES packets. form Coded video segments may be organized into NAL units, which provide “network-friendly” video representation to address applications such as video telephony, storage, broadcast, or streaming. NAL units can be classified into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL units may contain a core compression engine and may contain block, macroblock, and/or slice level data. Other NAL units may be non-VCL NAL units. In some examples, a coded picture in one time instance, usually presented as a primary coded picture, may be included within an access unit, which may include one or more NAL units.

Non-VCL NAL units may include parameter set NAL units and SEI NAL units, among others. Parameter sets may contain sequence-level header information (in sequence parameter sets (SPS)) and rarely changing picture-level header information (in picture parameter sets (PPS)). For parameter sets (eg, PPS and SPS), infrequently changing information does not need to be repeated for each sequence or picture, and thus coding efficiency may be improved. Moreover, the use of parameter sets may enable out-of-band transmission of important header information, avoiding the need for redundant transmissions for error tolerance. In out-of-band transmission examples, parameter set NAL units may be transmitted on a different channel than other NAL units, such as SEI NAL units.

Supplemental Enhancement Information (SEI) may include information that is not required to decode samples of coded pictures from VCL NAL units, but may support processes related to decoding, display, error resilience, and other purposes. SEI messages may also be included in non-VCL NAL units. SEI messages are a normative part of some standard specifications and, therefore, are not always mandatory for standard-compliant decoder implementation. SEI messages may be sequence level SEI messages or picture level SEI messages. Some sequence level information may be included in SEI messages, such as scalability information SEI messages in the example of SVC and view scalability information SEI messages in MVC. These example SEI messages may carry information about, for example, extraction of operation points and characteristics of the operation points. Additionally, encapsulation unit 30 may form a manifest file, such as a Media Presentation Descriptor (MPD), that describes the characteristics of the representations. Encapsulation unit 30 may format the MPD according to extensible markup language (XML).

Encapsulation unit 30 may provide data for one or more representations of multimedia content along with a manifest file (e.g., MPD) to output interface 32. Output interface 32 is a network interface or an interface for writing to a storage medium, such as a universal serial bus (USB) interface, a CD or DVD writer or burner, an interface to magnetic or flash storage media, or media data. It may also include other interfaces for storing or transmitting. Encapsulation unit 30 may provide data of respective representations of multimedia content to output interface 32, which may transmit the data to server device 60 via network transmission or storage media. You can also send it. In the example of FIG. 1 , server device 60 hosts various multimedia content 64, each including a respective manifest file 66 and one or more representations 68A-68N (representations 68). It includes a storage medium 62 that stores . In some examples, output interface 32 may also transmit data directly to network 74.

In some examples, representations 68 may be separated into adaptation sets. That is, the various subsets of representations 68 each have a set of common characteristics, such as codec, profile and level, resolution, number of views, file format for segments, audio to be decoded and presented, e.g., by speakers. Text type information, which may identify the language or other characteristics of the text to be displayed in the data and/or representation, and camera angle information, which may describe the camera angle or real-world camera perspective of the scene for representations in the adaptation set. , rating information that describes content suitability for a specific audience, etc.

Manifest file 66 may include data representing subsets of representations 68 that correspond to specific adaptation sets and common characteristics for the adaptation sets. Manifest file 66 may also include data indicating individual characteristics, such as bitrate, for individual representations of adaptation sets. In this way, the adaptation set may provide simplified network bandwidth adaptation. Representations in an adaptation set may be indicated using child elements of the adaptation set elements of manifest file 66.

Server device 60 includes a request processing unit 70 and a network interface 72. In some examples, server device 60 may include multiple network interfaces. Moreover, any or all of the features of server device 60 may be implemented on other devices in a content delivery network, such as routers, bridges, proxy devices, switches, or other devices. In some examples, intermediate devices in the content delivery network cache data of multimedia content 64 and may include components that substantially correspond to those of server device 60. Generally, network interface 72 is configured to transmit and receive data over network 74.

Request processing unit 70 is configured to receive network requests from client devices, such as client device 40, for data on storage medium 62. For example, request processing unit 70 may support the Hypertext Transfer Protocol, as described in RFC 2616, "Hypertext Transfer Protocol - HTTP/1.1" by R. Fielding et al., Network Working Group, IETF, June 1999. (HTTP) version 1.1 can also be implemented. In other words, request processing unit 70 may be configured to receive HTTP GET or partial GET requests and provide data of multimedia content 64 in response to the requests. The requests may specify a segment of one of the representations 68, e.g., using that segment's URL. In some examples, requests may also specify one or more byte ranges of a segment and thus may include partial GET requests. Request processing unit 70 may also be configured to service HTTP HEAD requests to provide header data of a segment of one of the representations 68. In either case, request processing unit 70 may be configured to process requests to provide the requested data to a requesting device, such as client device 40.

Additionally or alternatively, request processing unit 70 may be configured to deliver media data via a broadcast or multicast protocol, such as eMBMS. Content preparation device 20 may form DASH segments and/or sub-segments in substantially the same manner as described, but server device 60 may use eMBMS or another broadcast or multicast network transport protocol. Thus, these segments or sub-segments may be delivered. For example, request processing unit 70 may be configured to receive a multicast group join request from client device 40. That is, server device 60 advertises an Internet Protocol (IP) address associated with a multicast group to client devices, including client device 40, associated with specific media content (e.g., a broadcast of a live event). You may. Client device 40 may, again, submit a request to join the multicast group. This request may be propagated through network 74, e.g., through the routers that make up network 74, which may then target the IP address associated with the multicast group to subscribing client devices, such as client device 40. This causes traffic to be sent.

As illustrated in the example of FIG. 1, multimedia content 64 includes a manifest file 66, which may correspond to a Media Presentation Description (MPD). Manifest file 66 may include descriptions of different alternative representations 68 (e.g., video services with different qualities), such as codec information, profile value, level value, It may also include bitrate, and other descriptive characteristics of representations 68. Client device 40 may retrieve the MPD of the media presentation to determine how to access segments of representations 68.

In particular, retrieval unit 52 retrieves configuration data (not shown) of client device 40 to determine the decoding capabilities of video decoder 48 and rendering capabilities of video output 44. You can also decide. Video output 44 may be included in a display device for extended reality, augmented reality, or virtual reality, such as a headset. Likewise, the configuration data may indicate whether video output 44 is capable of rendering 3D video data, for example, for extended reality, augmented reality, virtual reality, etc. The configuration data may include language preferences selected by the user of client device 40, one or more camera perspectives corresponding to depth preferences set by the user of client device 40, and/or Any or all of the selected rating preferences may also be included.

Retrieval unit 52 may include, for example, a web browser or a media client configured to submit HTTP GET and partial GET requests. Retrieval unit 52 may correspond to software instructions executed by one or more processors or processing units (not shown) of client device 40. In some examples, all or portions of the functionality described with respect to retrieval unit 52 may be implemented in hardware, or in a combination of hardware, software, and/or firmware, where the necessary hardware may execute instructions for the software or firmware. It may be provided for execution.

Retrieval unit 52 may compare the decoding and rendering capabilities of client device 40 to the characteristics of representations 68 indicated by information in manifest file 66. Retrieval unit 52 may initially retrieve at least a portion of manifest file 66 to determine characteristics of representations 68. For example, retrieval unit 52 may request a portion of manifest file 66 that describes the characteristics of one or more adaptation sets. Retrieval unit 52 may select a subset of representations 68 (e.g., an adaptation set) whose characteristics can be satisfied by the coding and rendering capabilities of client device 40. Retrieval unit 52 then determines the bitrates for the representations in the adaptation set, determines the currently available amount of network bandwidth, and generates a reprint with a bitrate that can be satisfied by the network bandwidth. Segments may be retrieved from one of the statements.

In general, higher bitrate representations may yield higher quality video playback, while lower bitrate representations may provide sufficient quality video playback when available network bandwidth decreases. Accordingly, when the available network bandwidth is relatively high, retrieval unit 52 may retrieve data from relatively high bitrate representations, whereas when the available network bandwidth is low, retrieval unit 52 ) may retrieve data from relatively low bitrate representations. In this manner, client device 40 may stream multimedia data over network 74 while adapting to the fluctuating network bandwidth availability of network 74.

Additionally or alternatively, retrieval unit 52 may be configured to receive data according to a broadcast or multicast network protocol, such as eMBMS or IP multicast. In these examples, retrieval unit 52 may submit a request to join a multicast network group associated with specific media content. After joining a multicast group, retrieval unit 52 may receive the multicast group's data without additional requests issued to server device 60 or content preparation device 20. Retrieval unit 52 may submit a request to leave a multicast group when data in the multicast group is no longer needed, such as to change channels to another multicast group or to stop playback.

Network interface 54 may receive and provide data of segments of the selected representation to retrieval unit 52, which in turn may provide the segments to decapsulation unit 50. You can also provide it. Decapsulation unit 50 decapsulates elements of the video file into component PES streams and depacketizes the PES streams to retrieve encoded data, e.g., PES packets in the stream. Depending on whether the encoded data is part of an audio or video stream, as indicated by headers, encoded data may be sent to either audio decoder 46 or video decoder 48. Audio decoder 46 decodes the encoded audio data and transmits the decoded audio data to audio output 42, while video decoder 48 decodes the encoded video data and transmits a plurality of views of the stream. The decoded video data it may contain is transmitted to the video output unit 44.

Video encoder 28, video decoder 48, audio encoder 26, audio decoder 46, encapsulation unit 30, retrieval unit 52, and decapsulation unit 50 each have various functions as applicable. Suitable processing circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic circuits, software, hardware, firmware, or any It may be implemented as any of these combinations. Video encoder 28 and video decoder 48 may each be included within one or more encoders or decoders, any of which may be integrated as part of a combined video encoder/decoder (CODEC). Likewise, each of audio encoder 26 and audio decoder 46 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined CODEC. A device comprising a video encoder (28), a video decoder (48), an audio encoder (26), an audio decoder (46), an encapsulation unit (30), an extraction unit (52), and/or a decapsulation unit (50). It may also include integrated circuits, microprocessors, and/or wireless communication devices, such as cellular telephones.

Client device 40, server device 60, and/or content preparation device 20 may be configured to operate in accordance with the techniques of this disclosure. For purposes of illustration, this disclosure describes these techniques for client device 40 and server device 60. However, content preparation device 20 may be configured to perform these techniques instead of (or in addition to) server device 60.

Encapsulation unit 30 stores NAL units that include a header identifying the program to which the NAL unit belongs and a payload, such as audio data, video data, or data describing the transport or program stream to which the NAL unit corresponds. It can also be formed. For example, in H.264/AVC, a NAL unit includes a 1-byte header and a payload of variable size. A NAL unit that includes video data in its payload may include video data at various granularity levels. For example, a NAL unit may include a block of video data, a plurality of blocks, a slice of video data, or an entire picture of video data. Encapsulation unit 30 may receive encoded video data from video encoder 28 in the form of PES packets of elementary streams. Encapsulation unit 30 may associate each elementary stream with a corresponding program.

Encapsulation unit 30 may also assemble access units from a plurality of NAL units. In general, an access unit may include one or more NAL units for representing a frame of video data, as well as audio data corresponding to that frame when audio data is available. An access unit generally contains all NAL units for one output time instance, eg, all audio and video data for one output time instance. For example, if each view has a frame rate of 20 frames per second (fps), each time instance may correspond to a time interval of 0.05 seconds. During this time interval, certain frames for all views of the same access unit (same time instance) may be rendered simultaneously. In one example, an access unit may include a coded picture in one time instance, which may be presented as a primary coded picture.

Accordingly, an access unit may include all audio and video frames of a common time instance, eg, all views corresponding to time X. This disclosure also refers to an encoded picture of a particular view as a “view component.” That is, a view component may contain an encoded picture (or frame) for a specific view at a specific time. Accordingly, an access unit may be defined as containing all view components of a common time instance. The decoding order of access units is not necessarily the same as the output or display order.

The media presentation may include a media presentation description (MPD), which may include a description of different alternative representations (e.g. video services with different qualities), The description may include codec information, profile values, and level values, for example. MPD is an example of a manifest file, such as manifest file 66. Client device 40 may retrieve the MPD of the media presentation to determine how to access movie fragments of the various presentations. Movie fragments may be located in movie fragment boxes (moof boxes) of video files.

Manifest file 66 (which may include, for example, an MPD) may advertise the availability of segments of representations 68. In other words, the MPD contains information indicating the wall-clock time at which the first segment of one of the representations 68 becomes available, as well as information indicating the wall-clock time within the representations 68. It may also contain information indicating the durations of the segments. In this way, retrieval unit 52 of client device 40 may determine when each segment is available based on the start time as well as durations of the segments preceding a particular segment.

After encapsulation unit 30 assembles the NAL units and/or access units into a video file based on the received data, encapsulation unit 30 sends the video file to output interface 32 for output. In some examples, encapsulation unit 30 may store the video file locally, or may send the video file to a remote server via output interface 32 rather than transmitting the video file directly to client device 40. You can also send it. Output interface 32 may include, for example, a transmitter, a transceiver, a device for recording data to a computer-readable medium such as an optical drive, a magnetic media drive (e.g., a floppy drive), a universal serial bus (USB) port, It may also include a network interface or other output interface. Output interface 32 outputs the video file to a computer-readable medium, such as, for example, a transmission signal, magnetic medium, optical medium, memory, flash drive, or other computer-readable medium.

Network interface 54 receives a NAL unit or access unit via network 74 and provides the NAL unit or access unit to decapsulation unit 50, via retrieval unit 52. Decapsulation unit 50 decapsulates elements of the video file into constituent PES streams and depacketizes the PES streams to retrieve the encoded data, e.g., as indicated by the stream's PES packet headers. Likewise, depending on whether the encoded data is part of an audio or video stream, the encoded data may be transmitted to either audio decoder 46 or video decoder 48. Audio decoder 46 decodes the encoded audio data and transmits the decoded audio data to audio output 42, while video decoder 48 decodes the encoded video data and transmits a plurality of views of the stream. The decoded video data it may contain is transmitted to the video output unit 44.

According to the techniques of this disclosure, a user of client device 40 may obtain media data related to a 3D virtual scene, e.g., for extended reality (XR), augmented reality (AR), virtual reality (VR), etc. It may be possible. A user may navigate through a 3D virtual scene using one or more devices that communicate with client device 40, such as controllers. Additionally or alternatively, client device 40 may include sensors, cameras, etc. to determine that the user has moved in real-world space, and client device 40 converts these real-world movements into virtual space movements. You may.

A 3D virtual scene may contain one or more virtual solid objects. These objects may include, for example, walls, windows, tables, chairs, or any other such objects that may appear in the virtual scene. According to the techniques of this disclosure, media data retrieved by retrieval unit 52 may include a scene description that describes these virtual solid objects. The scene description may follow the MPEG scene description element of glTF 2.0, for example.

In some examples, the scene description may include a description of allowable camera movements. For example, a scene description may specify one or more sets of shapes (e.g., along the volume of a shape such as a sphere, cube, cone, frustum, etc.) in which the virtual camera is allowed to move such that the virtual camera is not allowed to move beyond the boundaries of the shapes. Bounding volumes may also be described. In other words, the bounding volume may describe a camera movement volume in which movement of the virtual camera is allowed. Additionally or alternatively, a scene description may describe one or more vertices or anchor points as well as allowed paths (e.g., segments) between vertices or anchor points. Client device 40 may only allow the virtual camera to move along permitted paths and/or within the bounding volume.

In some examples, additionally or alternatively, the scene description may describe one or more virtual solid objects in the scene that the virtual camera cannot pass through.

FIG. 2 is a block diagram showing an example set of components of extraction unit 52 of FIG. 1 in more detail. In this example, retrieval unit 52 includes eMBMS middleware unit 100, DASH client 110, media application 112, and presentation engine 114.

In this example, eMBMS middleware unit 100 further includes an eMBMS receiving unit 106, cache 104, and proxy server unit 102. In this example, the eMBMS receiving unit 106 is configured to, for example, use the Network Transmission Protocol, “FLUTE-File Delivery over Unidirectional Transport”, T. Paila et al., November 2012, available at tools.ietf.org/html/rfc6726. It is configured to receive data via eMBMS according to FLUTE (File Delivery over Unidirectional Transport), described in Working Group, RFC 6726. That is, eMBMS receiving unit 106 may receive files via broadcast, for example, from server device 60, which may act as a broadcast/multicast service center (BM-SC).

When eMBMS middleware unit 100 receives data for files, eMBMS middleware unit may store the received data in cache 104. Cache 104 may include a computer-readable storage medium, such as flash memory, hard disk, RAM, or any other suitable storage medium.

Proxy server unit 102 may act as a server for DASH client 110. For example, proxy server unit 102 may provide an MPD file or other manifest file to DASH client 110. Proxy server unit 102 may advertise availability times for segments within the MPD file, as well as hyperlinks from which the segments can be retrieved. These hyperlinks may include a localhost address prefix corresponding to client device 40 (e.g., 127.0.0.1 for IPv4). In this manner, DASH client 110 may request segments from proxy server unit 102 using HTTP GET or partial GET requests. For example, for a segment available from the link http://127.0.0.1/rep1/seg3, DASH client 110 would make an HTTP GET request containing a request for http://127.0.0.1/rep1/seg3. Configure and submit the request to proxy server unit 102. Proxy server unit 102 may retrieve requested data from cache 104 and provide data to DASH client 110 in response to such requests.

The DASH client 110 provides the retrieved media data to the media application 112. Media application 112 may be, for example, a web browser, game engine, or other application that receives and presents media data. Additionally, the presentation engine 114 represents an application that interacts with the media application 112 to provide media data retrieved from the 3D virtual environment. Presentation engine 114 may, for example, map two-dimensional media data onto a 3D projection. Presentation engine 114 may also receive input from other elements of client device 40 to determine the user's location in the 3D virtual environment and the orientation the user is facing at that location. For example, presentation engine 114 may determine X-, Y-, and Z-coordinates for the user's location, as well as the orientation from which the user is viewing, to determine appropriate media data to display to the user. Additionally, the presentation engine 114 may receive camera movement data representing real-world user movement data and convert the real-world user movement data into 3D virtual space movement data.

According to the techniques of this disclosure, eMBMS middleware unit 100 may receive media data via broadcast or multicast (e.g., according to glTF 2.0), and then DASH client 110 may Media data may be retrieved from unit 100. Media data may also include a scene description that includes camera control information indicating how the virtual camera may move through the virtual scene. For example, a scene description may include data describing allowable paths through the virtual scene, such as along defined paths between anchor points. Additionally or alternatively, the scene description may include data describing a bounding volume that represents the volume through which the virtual camera is allowed to move. Additionally or alternatively, the scene description may include data describing one or more solid virtual objects in the 3D virtual environment, such as walls, tables, chairs, etc. For example, data in a scene description may define collision boundaries of a 3D virtual object. The scene description describes the animations to be played using the objects, collisions with those objects, such as whether the objects are static (as in the case of a wall, for example) or dynamic (as in the case of a chair, for example). It may further include data indicating what happens in the event.

Presentation engine 114 may use the scene description to determine what to present in case of a collision with a 3D virtual object and/or an attempt to move out of an acceptable path or volume. For example, if the scene description includes data about allowable paths or bounding volumes and the user attempts to move beyond the allowable paths or bounding volumes, presentation engine 114 may simply avoid updating the display. may be possible, thereby indicating that such movement is not permitted. As another example, if the scene description contains data for a 3D virtual solid object and the user attempts to move through the 3D virtual solid object, if the 3D virtual solid object is static, presentation engine 114 may not update the display. You can also avoid it. If the 3D virtual solid object is not static, presentation engine 114 may determine an animation to display for the object, e.g., translation and/or rotation movement to be applied to the object. For example, if the 3D virtual solid object is a chair, the animation data may indicate that the chair will be pushed along the floor or fall over in the event of a collision.

3 is a conceptual diagram illustrating elements of example multimedia content 120. Multimedia content 120 may correspond to multimedia content 64 (Figure 1), or another multimedia content stored on storage medium 62. In the example of FIG. 3 , multimedia content 120 includes a media presentation description (MPD) 122 and a plurality of representations 124A-124N (representations 124). Representation 124A includes optional header data 126 and segments 128A-128N (segments 128), while representation 124N includes optional header data 130 and segments 128A-128N. 132A-132N (segments 132). The letter N is used to designate the final movie fragment in each of the representations 124 for convenience. In some examples, there may be a different number of movie fragments between representations 124.

MPD 122 may include a data structure separate from representations 124. MPD 122 may correspond to manifest file 66 of FIG. 1 . Likewise, representations 124 may correspond to representations 68 of FIG. 1 . In general, MPD 122 stores data that generally describes the characteristics of representations 124, such as coding and rendering characteristics, adaptation sets, profiles to which MPD 122 corresponds, text type information, camera angles, etc. targeting into media content to retrieve information, rating information, trick mode information (e.g., information indicating representations containing temporal sub-sequences) and/or remote periods (e.g., during playback) It may also include information (for inserting advertisements).

When present, header data 126 may include characteristics of segments 128, e.g., temporal locations of random access points (RAPs) (also referred to as stream access points (SAPs)), segments ( 128) which includes random access points, byte offsets for random access points within segments 128, uniform resource locators (URLs) of segments 128, or other Aspects may also be described. If present, header data 130 may describe similar characteristics for segments 132. Additionally or alternatively, these features may be included entirely within MPD 122.

Segments 128, 132 include one or more coded video samples, each of which may include frames or slices of video data. Each of the coded video samples of segments 128 may have similar characteristics, such as height, width, and bandwidth requirements. These characteristics may be described by data in MPD 122, but such data is not illustrated in the example of FIG. 3. MPD 122 may include features as described by the 3GPP specification, with the addition of any or all of the signaled information described in this disclosure.

Each of segments 128, 132 may be associated with a unique uniform resource locator (URL). Accordingly, each of segments 128, 132 may be independently retrievable using a streaming network protocol, such as DASH. In this manner, a destination device, such as client device 40, may retrieve segments 128 or 132 using an HTTP GET request. In some examples, client device 40 may use HTTP partial GET requests to retrieve specific byte ranges of segments 128 or 132.

FIG. 4 is a block diagram illustrating elements of an example video file 150 that may correspond to a segment of the representation, such as one of segments 128 and 132 of FIG. 3 . Each of segments 128, 132 may include data that substantially conforms to the arrangement of data shown in the example of FIG. 4. Video file 150 may also be said to encapsulate a segment. As described above, video files according to the ISO Base Media File Format and its extensions store data in a series of objects referred to as “boxes.” In the example of Figure 4, video file 150 has a file type (FTYP) box 152, a movie (MOOV) box 154, segment index (sidx) boxes 162, and a movie fragment (MOOF) box ( 164) and a Movie Fragment Random Access (MFRA) box 166. 4 shows an example of a video file, but other media files may include other types of media data structured similarly to the data of video file 150 (e.g., audio data, It should be understood that it may also include timed text data, etc.).

File Type (FTYP) box 152 generally describes the file type for video file 150. File type box 152 may include data identifying specifications that describe the best use for video file 150. File type box 152 may alternatively be placed before MOOV box 154, movie fragment boxes 164, and/or MFRA box 166.

In some examples, a segment, such as video file 150, may include an MPD Update box (not shown) before FTYP box 152. The MPD update box may include information indicating that the MPD corresponding to the representation containing video file 150 will be updated along with information to update the MPD. For example, the MPD update box may provide a URI or URL to a resource that will be used to update the MPD. As another example, the MPD update box may include data for updating the MPD. In some examples, the MPD Update box may immediately follow a Segment Type (STYP) box (not shown) of video file 150, where the STYP box may define the segment type for video file 150. .

MOOV box 154, in the example of FIG. 4, includes a movie header (MVHD) box 156, a track (TRAK) box 158, and one or more movie extensions (MVEX) boxes 160. In general, MVHD box 156 may describe general characteristics of video file 150. For example, MVHD box 156 may display the timescale for video file 150, when video file 150 was originally created, when video file 150 was last modified, and the playback for video file 150. may include data describing the duration of the video file 150, or other data generally describing the video file 150.

TRAK box 158 may contain data for a track of video file 150. TRAK box 158 may include a Track Header (TKHD) box that describes characteristics of the track that corresponds to TRAK box 158. In some examples, TRAK box 158 may include coded video pictures, while in other examples, the coded video pictures of a track are separated by data in TRAK box 158 and/or sidx boxes 162. It may be included in movie fragments 164 that may be referenced.

In some examples, video file 150 may include more than one track. Accordingly, MOOV box 154 may include the same number of TRAK boxes in video file 150 as the number of tracks. TRAK box 158 may describe the characteristics of the corresponding track of video file 150. For example, TRAK box 158 may describe temporal and/or spatial information for the corresponding track. A TRAK box, similar to the TRAK box 158 of the MOOV box 154, provides the characteristics of the parameter set track when the encapsulation unit 30 (FIG. 3) includes a parameter set track in a video file, such as video file 150. You can also describe them. Encapsulation unit 30 may signal the presence of sequence level SEI messages in the parameter set track in a TRAK box describing the parameter set track.

MVEX boxes 160 may have corresponding Characteristics of movie fragments 164 may also be described. In the context of streaming video data, coded video pictures may be included within movie fragments 164 rather than within a MOOV box 154. Accordingly, all coded video samples may be included within movie fragments 164, rather than within MOOV box 154.

MOOV box 154 may include a number of MVEX boxes 160 equal to the number of movie fragments 164 in video file 150. Each of MVEX boxes 160 may describe properties of a corresponding one of movie fragments 164. For example, each MVEX box may include a movie extends header box (MEHD) box that describes the temporal duration for the corresponding one of the movie fragments 164.

As mentioned above, encapsulation unit 30 may store sequence data sets in video samples that do not contain actual coded video data. A video sample may generally correspond to an access unit, which is a representation of a coded picture at a specific time instance. In the context of AVC, a coded picture includes one or more VCL NAL units containing information for configuring all pixels of an access unit, and other associated non-VCL NAL units, such as SEI messages. Accordingly, encapsulation unit 30 may include a sequence data set, which may include sequence level SEI messages, in one of the movie fragments 164. The encapsulation unit 30 is configured to store a movie fragment as present in one of the movie fragments 164 within an MVEX box of one of the MVEX boxes 160 corresponding to one of the movie fragments 164. The presence of sequence data sets and/or sequence level SEI messages may be additionally signaled.

SIDX boxes 162 are optional elements of video file 150. In other words, video files conforming to the 3GPP file format, or other such file formats, do not necessarily include SIDX boxes 162. According to the example of the 3GPP file format, a SIDX box may be used to identify a sub-segment of a segment (e.g., a segment included within video file 150). The 3GPP file format defines a sub-segment as "a self-contained set of one or more consecutive movie fragment boxes with corresponding media data box(s), and the media data box containing the data referenced by the movie fragment box is must follow and must precede the next movie fragment box containing information about the same track. The 3GPP file format also indicates that a SIDX box "contains a sequence of references to subsegments of the (sub)segment documented by the box." The referenced subsegments are consecutive in presentation time. Likewise, bytes referenced by a segment index box are always consecutive within a segment. The referenced size provides a count of the number of bytes in the referenced material.

SIDX boxes 162 generally provide information representing one or more sub-segments of a segment included in video file 150. For example, this information may include playback times at which the sub-segments start and/or end, byte offsets for the sub-segments, whether the sub-segments contain a stream access point (SAP) (e.g. the type of SAP (e.g., whether the SAP is an instantaneous decoder refresh (IDR) picture, a clean random access (CRA) picture, a broken link access (BLA) picture, etc.), the type of SAP in the sub-segment position (in terms of playback time and/or byte offset), etc.

Movie fragments 164 may include one or more coded video pictures. In some examples, movie fragments 164 may include one or more groups of pictures (GOPs), each of which may include multiple coded video pictures, e.g., frames or pictures. there is. Additionally, as described above, movie fragments 164 may include sequence data sets in some examples. Each of movie fragments 164 may include a movie fragment header box (MFHD, not shown in FIG. 4). The MFHD box may describe characteristics of the corresponding movie fragment, such as a sequence number for the movie fragment. Movie fragments 164 may be included in video file 150 in order of sequence number.

MFRA box 166 may describe random access points within movie fragments 164 of video file 150. This may support performing trick modes, such as performing searches for specific temporal locations (i.e., playback times) within the segment encapsulated by video file 150. MFRA box 166 is generally optional and in some examples does not need to be included in video files. Similarly, a client device, such as client device 40, does not necessarily need to reference MFRA box 166 to correctly decode and display the video data of video file 150. MFRA box 166 represents a random number of track fragments equal to the number of tracks in video file 150, or in some examples, equal to the number of media tracks (e.g., non-hint tracks) in video file 150. Access (TFRA) boxes (not shown) may also be included.

In some examples, movie fragments 164 may include one or more stream access points (SAPs), such as IDR pictures. Similarly, MFRA box 166 may provide indications of locations within SAPs' video file 150. Accordingly, a temporal sub-sequence of video file 150 may be formed from the SAPs of video file 150. A temporal sub-sequence may also include other pictures such as P-frames and/or B-frames that depend on SAPs. Frames and/or slices of a temporal sub-sequence may be arranged within segments so that frames/slices of a temporal sub-sequence that depend on other frames/slices of the sub-sequence can be properly decoded. For example, in a hierarchical arrangement of data, data used for prediction of other data may also be included in the temporal sub-sequence.

5 is a conceptual diagram illustrating an example camera path segment with a bounding volume, according to the techniques of this disclosure. In particular, in a 3D scene 200, a camera 202 represents a viewpoint from which a user can view a portion of the 3D scene 200. In this example, path segment 212 is defined between points 204 and 206. Additionally, the bounding volume is defined by the extrusion of points from bounding box 208 to bounding box 210 along path segment 212. Accordingly, in this example, camera 202 is allowed to move within the bounding volume along path segment 212, but is restricted from moving beyond the bounding volume.

A scene description may describe a set of paths along which a camera, such as camera 202, is allowed to move. Paths may be described as a set of anchor points, such as points 204 and 206, that are connected by a path segment, such as path segment 212. In some examples, such as the example of Figure 5, each path segment may be reinforced with a bounding volume that allows some degree of freedom of motion along the path.

The scene camera, and thereby the viewer, will be able to move freely within the bounding volume along the path segment. Path segments may be described using more complex geometries to allow finer control of the path.

Additionally, camera parameters may be limited at each point along the path. Parameters may be provided for every anchor point and then used with an interpolation function to calculate the corresponding parameters for every point along the path segment. An interpolation function may be applied to all parameters including the bounding volume.

The camera control extension mechanism of this disclosure may be implemented as a glTF 2.0 extension that defines camera control for a scene. The camera control extension may be identified by the "MPEG_camera_control" tag, which may be included in the extensionsUsed element or in the extensionsRequired element for 3D scenes.

An example of the “MPEG_camera_control” extension is shown in Table 1, and may be defined in “camera” elements of the scene description.

table 1 designation type default description anchors number Not applicable Number of anchor points in camera paths. segments number Not applicable Number of path segments in camera paths. boundingVolume number BV_NONE Type of bounding volume for path segments. The possible types are:
· BV_NONE: No bounding volume
· BV_CONE: A closed conical bounding volume defined by a circle at each anchor point.
· BV_FRUSTUM: A frustum bounding volume defined by two rectangles, each containing an anchor point.
· BV_SPHERE: A spherical bounding volume around each point along a path segment. The bounding volume is defined by the radius of the sphere. intrinsicParameters boolean false When set to true, indicates that the underlying camera parameters are modified at each anchor point. Parameters must be provided based on the type of camera as defined in camera.perspective or camera.orthographic in [glTF 2.0]. accessor number Not applicable Index of the accessor or timed accessor that provides camera control information.

Camera control information may be structured as follows:

· For each anchor point, the (x, y, z) coordinates of the anchor points may be expressed using floating point values.

· For each path segment, the (i,j) indices of the first and second anchor points of the path segment may be expressed as integer values.

· About bounding volumes:

o If boundingVolume is BV_CONE, (r1,r2) radii of the circles of the first anchor point and the second anchor point may be provided.

o If boundingVolume is BV_FRUSTUM, ((x,y,z)_topleft,w,h) may be provided for each anchor point of the path segment.

o If boundingVolume is BV_SPHERE, r may be provided as the radius of the sphere for each anchor point of the path segment.

· If intrinsicParameters is true, the intrinsic parameter object may be modified.

The presentation engine (e.g., presentation engine 114 of Figure 2 or other elements of client device 40 that may be different from the components shown in Figures 1 and 2) may include the MPEG_camera_control extension or other such data structure. You can also support . If the scene provides camera control information, the presentation engine may constrain camera movement to marked paths such that the camera's (x, y, z) coordinates always lie on the path segment or within the bounding volume of the path segment. . The presentation engine may provide visual, auditory and/or haptic feedback to the viewer as the viewer approaches the boundary of the bounding volume.

6 is a conceptual diagram illustrating an example virtual object 220, which in this example is a chair. To provide the viewer with an immersive experience, it is important for the viewer to interact appropriately with objects within the scene. The viewer should not be able to walk through solid objects in the scene, such as walls, chairs, tables, or other such solid objects.

Figure 6 shows a 3D mesh representation of a chair with collision boundaries defined as a set of cubes. The MPEG_mesh_collision extended data structure may be defined to provide a description of the collision boundaries of this 3D mesh. An extension data structure may be defined on mesh objects as a set of cuboids around the mesh geometry. Table 2 below presents an example set of attributes that may be included in this extended data structure.

table 2 designation type default description boundaries array (object) Not applicable An array of boundary shapes used to define the collision boundaries of a mesh object. Boundaries may be spheres or cuboids. static boolean really Determines whether an object is affected by collisions. A static object is not affected by collisions, so its position does not change when the viewer or another object collides with it. material number Not applicable An index of conflicting objects or collision data that defines how viewers will interact with this object. This may include bounceness, friction, etc. animations array (object) Not applicable Defines animations triggered by collisions or actions of this object. The animation may be limited to a subset of other objects, for example, only the viewer may trigger the animation. It also contains a pointer to the animation that will run when triggered.

Mesh collision information may include cuboid vertex coordinates (x,y,z) for cuboid boundaries or sphere center and radius for sphere boundaries. Values may be provided as floating point numbers.

The presentation engine may support the MPEG_mesh_collision extension or other such data structures. The presentation engine may ensure that the camera position (x, y, z) is not contained within one of the defined mesh cuboids at any point in time. Collisions may be signaled to the viewer through visual, auditory, and/or haptic feedback. The presentation engine may use information about boundaries for nodes to initialize and configure a 3D physics engine to detect collisions.

7 is a flow chart illustrating an example method of retrieving media data according to the techniques of this disclosure. The method of FIG. 7 is described with respect to the client device 40 of FIG. 1 and the extraction unit 52 of FIG. 2 . Other such devices may be configured to perform this or similar methods.

Initially, client device 40 may retrieve media data (250). For example, retrieval unit 52 may retrieve media data that conforms to glTF 2.0, for example. In some examples, retrieval unit 52 may retrieve media data directly according to unicast, such as using DASH, for example. In some examples, a middleware unit of retrieval unit 52, such as eMBMS middleware 100 of FIG. 2, may receive media data via broadcast or multicast and then transmit the media data to a DASH client, e.g., FIG. 2 The DASH client 110 of may retrieve media data from the middleware unit.

Media data may include a scene description. Accordingly, retrieval unit 52 or another component of client device 40 may extract the scene description from the media data (252). The scene description may be an MPEG scene description containing camera control data according to the techniques of this disclosure. Retrieval unit 52 may provide a scene description to presentation engine 114. Accordingly, presentation engine 114 may receive the scene description and, in turn, determine camera control data for the three-dimensional scene from the scene description (254). Camera control data may follow <Table 1> above. That is, for example, camera control data may include one or more anchor points for camera paths, one or more segments between anchor points for camera paths, a bounding volume such as a cone, frustum, or sphere, and each anchor point. may include intrinsic parameters that may be modified, and/or an accessor that provides camera control information.

Presentation engine 114 may further determine movement limitations from camera control data (256). For example, presentation engine 114 may determine two or more anchor points and allowable paths between anchor points from movement constraints in camera control data. Additionally or alternatively, presentation engine 114 may determine a bounding volume, such as a cube, sphere, frustum, cone, etc., from movement constraints in the camera control data. Presentation engine 114 may use the allowable paths to determine paths on which the virtual camera is allowed to move and/or paths on which the virtual camera is allowed to move within the bounding volume but not outside the bounding volume. Allowable paths and/or bounding volumes may be defined to ensure that the virtual camera does not pass through a 3D solid virtual object, such as a wall. That is, the bounding volume or allowable paths may be defined to be within one or more 3D solid virtual objects, such as walls, floors, ceilings, or other objects within a 3D virtual scene.

Presentation engine 114 may then receive camera movement data (258). For example, presentation engine 114 may be one such as a headset that includes handheld controllers and/or a display that represents movements of the headset and/or a virtual camera, such as orientation and directional movement and/or rotational movement of the headset. Data may be received from the above controllers. Presentation engine 114 may determine that the camera movement data requests camera movement through the 3D solid virtual object 260, such as across the boundaries of a bounding volume or along a path that is not one of the defined allowable paths. . In response, presentation engine 114 may prevent the virtual camera from passing through the 3D solid virtual object (262).

In this way, the method of Figure 7 includes receiving, by a presentation engine, streamed media data representing a virtual three-dimensional scene comprising at least one virtual solid object; Receiving, by the presentation engine, camera control data for the three-dimensional scene, the camera control data comprising data defining constraints to prevent the virtual camera from passing through at least one virtual solid object; Receiving, by the presentation engine, camera movement data from a user requesting that the virtual camera move through at least one virtual solid object; and preventing the virtual camera from passing the at least one virtual solid object in response to the camera movement data, by the presentation engine, using the camera control data. indicates.

8 is a flow chart illustrating an example method of retrieving media data according to the techniques of this disclosure. The method of FIG. 8 is described with respect to the client device 40 of FIG. 1 and the extraction unit 52 of FIG. 2 . Other such devices may be configured to perform this or similar methods.

Initially, client device 40 may retrieve media data (280). For example, retrieval unit 52 may retrieve media data that conforms to glTF 2.0, for example. In some examples, retrieval unit 52 may retrieve media data directly according to unicast, such as using DASH, for example. In some examples, a middleware unit of retrieval unit 52, such as eMBMS middleware 100 of FIG. 2, may receive media data via broadcast or multicast and then transmit the media data to a DASH client, e.g., FIG. 2 The DASH client 110 of may retrieve media data from the middleware unit.

Media data may include a scene description. Accordingly, retrieval unit 52 or another component of client device 40 may extract the scene description from the media data (282). The scene description may be an MPEG scene description containing object collision data according to the techniques of this disclosure. Retrieval unit 52 may provide a scene description to presentation engine 114. Accordingly, presentation engine 114 may receive a scene description and then determine object collision data for one or more 3D solid virtual objects from the scene description (284). Object collision data may follow Table 2 above. That is, object collision data includes, for example, data representing boundaries representing an array of boundary shapes defining the collision boundaries of a mesh (3D virtual solid) object, data indicating whether the object is static (i.e., movable), It may also include material representing a collision material for the object, and/or animations to be presented for the object upon collision.

Presentation engine 114 may additionally determine object collision data from camera control data (286). For example, the presentation engine 114 may include bounds representing an array of boundary shapes that define the collision boundaries of a mesh (3D virtual solid) object, data indicating whether the object is static (i.e., moveable), and Material representing the collision material for the object, and/or animations to be presented for the object in case of a collision may be determined. Presentation engine 114 may use object collision data to determine how to react in the event of a collision with a 3D solid virtual object.

Presentation engine 114 may then receive camera movement data (288). For example, presentation engine 114 may be one such as a headset that includes handheld controllers and/or a display that represents movements of the headset and/or a virtual camera, such as orientation and directional movement and/or rotational movement of the headset. Data may be received from the above controllers. Presentation engine 114 may determine 290 that the camera movement data requests camera movement through a 3D solid virtual object, such as a 3D solid virtual object defined by object collision data. In response, presentation engine 114 may prevent the virtual camera from passing through the 3D solid virtual object (292). For example, if an object is static as indicated by object collision data, presentation engine 114 may prevent the virtual camera from moving into and through the object. As another example, if the object is not static (e.g., movable), presentation engine 114 may provide a device to play on the object, e.g., if the object tips over or moves. A response from object collision data may also be determined in response to a collision with an object, such as an animation.

In this way, the method of FIG. 8 represents an example of a method of retrieving media data, which method comprises, by a presentation engine, streamed media data representing a virtual three-dimensional scene comprising at least one virtual solid object. receiving; Receiving, by the presentation engine, object collision data representing boundaries of at least one virtual solid object; Receiving, by the presentation engine, camera movement data from a user requesting that the virtual camera move through at least one virtual solid object; and preventing, by the presentation engine, using the object collision data, the virtual camera from passing the at least one virtual solid object in response to the camera movement data.

Specific examples of the techniques of this disclosure are summarized in the following provisions:

Clause 1: A method of retrieving media data, the method comprising: receiving, by a presentation engine, streamed media data representing a virtual three-dimensional scene comprising at least one virtual solid object; Receiving, by the presentation engine, camera control data for the three-dimensional scene, the camera control data including data defining acceptable positions for a virtual camera; receiving, by the presentation engine, camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; and using the camera control data to update, by the presentation engine, the position of the virtual camera to ensure that the virtual camera remains within the acceptable positions. method.

Clause 2: The method of clause 1, wherein updating the position of the virtual camera comprises preventing the virtual camera from passing the at least one virtual solid object.

Clause 3: The method of clause 1, wherein the streamed media data comprises glTF 2.0 media data.

Clause 4: The method of clause 1, wherein receiving streamed media data comprises requesting the streamed media data from a retrieval unit via an application programming interface (API).

Clause 5: The method of clause 1, wherein the camera control data is included in an MPEG scene description.

Clause 6: The method of clause 1, wherein the camera control data includes data defining two or more anchor points and one or more segments between the anchor points, wherein the segments define acceptable camera characteristics for the virtual camera. Indicating movement vectors and updating the position of the virtual camera includes allowing the virtual camera to only traverse the segments between the anchor points.

Clause 7: The method of clause 1, wherein the camera control data includes data defining a bounding volume indicative of an allowable camera movement volume for the virtual camera, and a position of the virtual camera. updating comprises allowing the virtual camera to only traverse the allowable camera movement volume.

Clause 8: The method of clause 7, wherein the data defining the bounding volume includes data defining at least one of a cone, a frustrum, or a sphere.

Clause 9: The method of clause 1, wherein the camera control data is included in an MPEG_camera_control extension.

Clause 10: The method of clause 9, wherein the MPEG_camera_control extension comprises: anchor data indicating a number of anchor points for allowable paths for the virtual camera; segment data indicating the number of path segments for the allowable paths between the anchor points; Bounding volume data representing a bounding volume for the virtual camera; intrinsic parameters indicating whether camera parameters are modified at each of the anchor points; and accessor data indicating an index of an accessor providing the camera control data.

Clause 11: The method of clause 1, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 12: The method of clause 1, further comprising determining allowable paths for a virtual camera from camera control data, wherein updating the position of the virtual camera comprises the step of updating the position of the virtual camera when the virtual camera is defined in the camera control data. A method comprising ensuring movement only along virtual paths that are within allowable paths.

Clause 13: The method of clause 1, wherein the camera control data is included in an MPEG_mesh_collision extension.

Clause 14: A device for retrieving media data, the device comprising: a memory configured to store media data; and one or more processors implemented in circuitry and configured to execute a presentation engine, the presentation engine configured to: receive streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving camera control data for the three-dimensional scene, wherein the camera control data includes data defining acceptable positions for a virtual camera; receive camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; The device for retrieving media data, configured to use the camera control data to update the position of the virtual camera to ensure that the virtual camera remains within the acceptable positions.

Clause 15: The device of clause 14, wherein the presentation engine is configured to prevent the virtual camera from passing the at least one virtual solid object.

Clause 16: The device of clause 14, wherein the streamed media data comprises glTF 2.0 media data.

Clause 17: The device of clause 14, wherein the presentation engine is configured to request the streamed media data from a retrieval unit via an application programming interface (API).

Clause 18: The device of clause 14, wherein the camera control data is included in an MPEG scene description.

Clause 19: The device of clause 14, wherein the camera control data includes data defining two or more anchor points and one or more segments between the anchor points, wherein the segments represent allowable cameras for the virtual camera. In order to represent movement vectors and update the position of the virtual camera, the presentation engine is configured to allow the virtual camera to only traverse segments between the anchor points.

Clause 20: The device of clause 14, wherein the camera control data includes data defining a bounding volume indicative of an allowable camera movement volume for the virtual camera and configured to update the position of the virtual camera. wherein the engine is configured to allow the virtual camera to traverse only the allowable camera movement volume.

Clause 21: The device of clause 20, wherein the data defining the bounding volume includes data defining at least one of a cone, a frustum, or a sphere.

Clause 22: The device of clause 14, wherein the camera control data is included in the MPEG_camera_control extension.

Clause 23: The device of clause 22, wherein the MPEG_camera_control extension includes: anchor data indicating a number of anchor points for allowable paths for the virtual camera; segment data indicating the number of path segments for the allowable paths between the anchor points; Bounding volume data representing a bounding volume for the virtual camera; intrinsic parameters indicating whether camera parameters are modified at each of the anchor points; and accessor data indicating an index of an accessor providing the camera control data.

Clause 24: The device of clause 14, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 25: The device of clause 14, wherein the presentation engine is further configured to determine allowable paths for the virtual camera from the camera control data, and to update the position of the virtual camera: is configured to ensure that the virtual camera moves only along virtual paths that are within the allowable paths defined in the camera control data.

Clause 26: The device of clause 14, wherein the camera control data is included in the MPEG_mesh_collision extension.

Clause 27: A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor executing a presentation engine to: display streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; to receive; receive camera control data for the three-dimensional scene, wherein the camera control data includes data defining acceptable positions for a virtual camera; receive camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; and use the camera control data to update the position of the virtual camera to ensure that the virtual camera remains within the acceptable positions.

Clause 28: The computer-readable storage medium of clause 27, wherein the instructions causing the processor to update the position of the virtual camera cause the processor to cause the virtual camera to pass the at least one virtual solid object. A computer-readable storage medium containing instructions to prevent a computer-readable storage medium.

Clause 29: The computer-readable medium of clause 27, wherein the streamed media data comprises glTF 2.0 media data.

Clause 30: The computer-readable medium of clause 27, wherein the instructions causing the processor to receive the streamed media data cause the processor to receive the streamed media data from a retrieval unit via an application programming interface (API). A computer-readable medium comprising instructions that cause a request.

Clause 31: The computer-readable medium of clause 27, wherein the camera control data is included in an MPEG scene description.

Clause 32: The computer-readable medium of clause 27, wherein the camera control data includes data defining two or more anchor points and one or more segments between the anchor points, the segments being The instructions indicating allowable camera movement vectors and causing the processor to update the position of the virtual camera include instructions causing the processor to allow the virtual camera to traverse only segments between the anchor points. computer readable media.

Clause 33: The computer-readable medium of clause 27, wherein the camera control data includes data defining a bounding volume indicative of an allowable camera movement volume for the virtual camera, and causing the processor to determine the position of the virtual camera. The instructions for causing an update include instructions for causing the processor to allow the virtual camera to traverse only the allowable camera movement volume.

Clause 34: The computer-readable medium of clause 20, wherein the data defining the bounding volume comprises data defining at least one of a cone, a frustum, or a sphere.

Clause 35: The computer-readable medium of clause 27, wherein the camera control data is included in an MPEG_camera_control extension.

Clause 36: The computer-readable medium of clause 22, wherein the MPEG_camera_control extension comprises: anchor data indicating a number of anchor points for allowable paths for the virtual camera; segment data indicating the number of path segments for the allowable paths between the anchor points; Bounding volume data representing a bounding volume for the virtual camera; intrinsic parameters indicating whether camera parameters are modified at each of the anchor points; and accessor data indicating an index of an accessor providing the camera control data.

Clause 37: The computer-readable medium of clause 27, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 38: The computer-readable medium of clause 27, further comprising instructions that cause the processor to determine allowable paths for the virtual camera from the camera control data, and cause the processor to determine the location of the virtual camera. The instructions for updating include instructions for causing the processor to ensure that the virtual camera moves only along virtual paths that are within the allowable paths defined in the camera control data.

Clause 39: The computer-readable medium of clause 27, wherein the camera control data is included in an MPEG_mesh_collision extension.

Clause 40: A device for retrieving media data, the device comprising: means for receiving streamed media data representing a virtual three-dimensional scene comprising at least one virtual solid object; means for receiving camera control data for the three-dimensional scene, the camera control data comprising data defining acceptable positions for a virtual camera; means for receiving camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; and means for using the camera control data to update the position of the virtual camera to ensure that the virtual camera remains within the acceptable positions.

Clause 41: A method of retrieving media data, comprising: receiving, by a presentation engine, streamed media data representing a virtual three-dimensional scene comprising at least one virtual solid object; receiving, by the presentation engine, object collision data representing boundaries of the at least one virtual solid object; receiving, by the presentation engine, camera movement data from a user requesting that a virtual camera move through the at least one virtual solid object; and, using the object collision data, update the position of the virtual camera in response to the camera movement data, by the presentation engine, to ensure that the virtual camera remains outside of the at least one virtual solid object. A method for retrieving media data, comprising the steps of:

Clause 42: The method of clause 41, wherein updating the position of the virtual camera comprises preventing the virtual camera from passing the at least one virtual solid object.

Clause 43: The method of clause 41, wherein receiving object collision data comprises receiving an MPEG_mesh_collision extension.

Clause 44: The method of clause 43, wherein the MPEG_mesh_collision extension comprises data defining at least one 3D mesh for the at least one virtual solid object.

Clause 45: The method of clause 44, wherein the MPEG_mesh_collision extension comprises boundaries of a 3D mesh for the at least one virtual solid object, a material for the 3D mesh, or a contact point of the virtual camera with the 3D mesh. A method comprising data defining at least one of the animations to be presented in response.

Clause 46: The method of clause 41, wherein receiving object collision data comprises: boundary data representing one or more collision boundaries of the at least one virtual solid object; static data indicating whether the at least one virtual solid object is affected by collisions; material data indicating how collision objects interact with the at least one virtual solid object; or receiving data comprising one or more of animation data representing animations triggered by a collision with the at least one virtual solid object.

Clause 47: The method of clause 41, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 48: The method of clause 41, wherein the streamed media data comprises glTF 2.0 media data.

Clause 49: The method of clause 41, wherein receiving streamed media data comprises requesting the streamed media data from a retrieval unit via an application programming interface (API).

Clause 50: The method of clause 41, wherein the object collision data is included in an MPEG scene description.

Clause 51: A device for retrieving media data, the device comprising: a memory configured to store media data; and one or more processors implemented in circuitry and configured to execute a presentation engine, the presentation engine configured to: receive streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; receive object collision data representing boundaries of the at least one virtual solid object; receive camera movement data from a user requesting that a virtual camera move through the at least one virtual solid object; configured to use the object collision data to update the position of the virtual camera in response to the camera movement data to ensure that the virtual camera remains outside of the at least one virtual solid object. A device for doing so.

Clause 52: The device of clause 51, wherein to update the position of the virtual camera, the presentation engine is configured to prevent the virtual camera from passing the at least one virtual solid object.

Clause 53: The device of clause 51, wherein, to receive the object collision data, the presentation engine is configured to receive an MPEG_mesh_collision extension.

Clause 54: The device of clause 53, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object.

Clause 55: The device of clause 54, wherein the MPEG_mesh_collision extension is in response to boundaries of a 3D mesh for the at least one virtual solid object, a material for the 3D mesh, or the virtual camera contacting the 3D mesh. A device containing data defining at least one of the animations to be presented.

Clause 56: The device of clause 51, wherein, to receive the object collision data, the presentation engine comprises: boundary data indicative of one or more collision boundaries of the at least one virtual solid object; static data indicating whether the at least one virtual solid object is affected by collisions; material data indicating how collision objects interact with the at least one virtual solid object; or animation data representing animations triggered by a collision with the at least one virtual solid object.

Clause 57: The device of clause 51, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 58: The device of clause 51, wherein the streamed media data comprises glTF 2.0 media data.

Clause 59: The device of clause 51, wherein, to receive the streamed media data, the presentation engine is configured to request the streamed media data from an retrieval unit via an application programming interface (API).

Clause 60: The device of clause 51, wherein the object collision data is included in an MPEG scene description.

Clause 61: A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor to: receive streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; receive object collision data representing boundaries of the at least one virtual solid object; receive camera movement data from a user requesting that a virtual camera move through the at least one virtual solid object; A computer-readable storage medium that uses the object collision data to update the position of the virtual camera in response to the camera movement data to ensure that the virtual camera remains outside of the at least one virtual solid object. .

Clause 62: The computer-readable medium of clause 61, wherein the instructions causing the processor to update the position of the virtual camera cause the processor to prevent the virtual camera from passing the at least one virtual solid object. A computer-readable medium containing instructions to do so.

Clause 63: The computer-readable medium of clause 61, wherein the instructions causing the processor to receive the object collision data include instructions to cause the processor to receive an MPEG_mesh_collision extension.

Clause 64: The computer-readable medium of clause 62, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object.

Clause 65: The computer-readable medium of clause 63, wherein the MPEG_mesh_collision extension includes boundaries of a 3D mesh for the at least one virtual solid object, a material for the 3D mesh, or a point where the virtual camera contacts the 3D mesh. A computer-readable medium comprising data defining at least one of animations to be presented in response.

Clause 66: The computer-readable medium of clause 61, wherein the instructions causing the processor to receive the object collision data cause the processor to: Boundary data representative of one or more collision boundaries of the at least one virtual solid object. ; static data indicating whether the at least one virtual solid object is affected by collisions; material data indicating how collision objects interact with the at least one virtual solid object; or animation data representing animations triggered by a collision with the at least one virtual solid object.

Clause 67: The computer-readable medium of clause 61, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 68: The computer-readable medium of clause 61, wherein the streamed media data comprises glTF 2.0 media data.

Clause 69: The computer-readable medium of clause 61, wherein the instructions causing the processor to receive the streamed media data cause the processor to receive the streamed media data from a retrieval unit via an application programming interface (API). A computer-readable medium comprising instructions that cause a request.

Clause 70: The computer-readable medium of clause 61, wherein the object collision data is included in an MPEG scene description.

Clause 71: A device for retrieving media data, the device comprising: means for receiving streamed media data representing a virtual three-dimensional scene comprising at least one virtual solid object; means for receiving object collision data representing boundaries of the at least one virtual solid object; means for receiving camera movement data from a user requesting that a virtual camera move through the at least one virtual solid object; and means for updating a position of the virtual camera in response to the camera movement data to ensure that the virtual camera remains outside the at least one virtual solid object.

Clause 72: A method of retrieving media data, the method comprising: receiving, by a presentation engine, streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; Receiving, by the presentation engine, camera control data for the three-dimensional scene, the camera control data including data defining acceptable positions for a virtual camera; receiving, by the presentation engine, camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; and using the camera control data to update, by the presentation engine, the position of the virtual camera to ensure that the virtual camera remains within the acceptable positions. method.

Clause 73: The method of clause 72, wherein updating the position of the virtual camera comprises preventing the virtual camera from passing the at least one virtual solid object.

Clause 74: The method of any of clauses 72 and 73, wherein the streamed media data comprises glTF 2.0 media data.

Clause 75: The method of any of clauses 72-74, wherein receiving streamed media data comprises requesting the streamed media data from a retrieval unit via an application programming interface (API). .

Clause 76: The method of any of clauses 72-75, wherein the camera control data is included in an MPEG scene description.

Clause 77: The method of any of clauses 72 to 76, wherein the camera control data includes data defining two or more anchor points and one or more segments between the anchor points, the segments being for the virtual camera. Indicating acceptable camera movement vectors, updating the position of the virtual camera includes allowing the virtual camera to traverse only segments between anchor points.

Clause 78: The method of any of clauses 72-77, wherein the camera control data includes data defining a bounding volume indicative of an allowable camera movement volume for a virtual camera, and updating the position of the virtual camera. The method includes allowing the virtual camera to traverse only an allowable camera movement volume.

Clause 79: The method of clause 78, wherein the data defining the bounding volume includes data defining at least one of a cone, a frustrum, or a sphere.

Clause 80: The method of any of clauses 72 to 79, wherein the camera control data is included in an MPEG_camera_control extension.

Clause 81: The method of clause 80, wherein the MPEG_camera_control extension comprises: anchor data indicating a number of anchor points for allowable paths for the virtual camera; segment data indicating the number of path segments for the allowable paths between the anchor points; Bounding volume data representing a bounding volume for the virtual camera; intrinsic parameters indicating whether camera parameters are modified at each of the anchor points; and accessor data indicating an index of an accessor providing the camera control data.

Clause 82: The method of any of clauses 72-81, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 83: The method of clause 72, further comprising determining allowable paths for a virtual camera from camera control data, wherein updating the position of the virtual camera includes the virtual camera being defined in the camera control data. A method comprising ensuring movement only along virtual paths that are within allowable paths.

Clause 84: The method of any of clauses 72 to 83, wherein the camera control data is included in an MPEG_mesh_collision extension.

Clause 85: A device for retrieving media data, the device comprising: a memory configured to store media data; and one or more processors implemented in circuitry and configured to execute a presentation engine, the presentation engine configured to: receive streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving camera control data for the three-dimensional scene, wherein the camera control data includes data defining acceptable positions for a virtual camera; receive camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; The device for retrieving media data, configured to use the camera control data to update the position of the virtual camera to ensure that the virtual camera remains within the acceptable positions.

Clause 86: The device of clause 85, wherein the presentation engine is configured to prevent the virtual camera from passing the at least one virtual solid object.

Clause 87: The device of any of clauses 85 and 86, wherein the streamed media data comprises glTF 2.0 media data.

Clause 88: The device of any of clauses 85-87, wherein the presentation engine is configured to request the streamed media data from a retrieval unit via an application programming interface (API).

Clause 89: The device of any of clauses 85-88, wherein the camera control data is included in an MPEG scene description.

Clause 90: The device of any of clauses 85-89, wherein the camera control data includes data defining two or more anchor points and one or more segments between the anchor points, wherein the segments allow for a virtual camera. In order to indicate possible camera movement vectors and update the position of the virtual camera, the presentation engine is configured to allow the virtual camera to traverse only segments between anchor points.

Clause 91: The device of any of clauses 85-90, wherein the camera control data includes data defining a bounding volume indicative of an allowable camera movement volume for a virtual camera, to update the position of the virtual camera, wherein the presentation engine is configured to allow the virtual camera to traverse only an allowable volume of camera movement.

Clause 92: The device of clause 91, wherein the data defining the bounding volume includes data defining at least one of a cone, a frustum, or a sphere.

Clause 93: The device of any of clauses 85 to 92, wherein the camera control data is included in an MPEG_camera_control extension.

Clause 94: The device of clause 93, wherein the MPEG_camera_control extension comprises: anchor data indicating a number of anchor points for allowable paths for the virtual camera; segment data indicating the number of path segments for the allowable paths between the anchor points; Bounding volume data representing a bounding volume for the virtual camera; intrinsic parameters indicating whether camera parameters are modified at each of the anchor points; and accessor data indicating an index of an accessor providing the camera control data.

Clause 95: The device of any of clauses 85-94, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 96: The device of any of clauses 85-95, wherein the presentation engine is further configured to determine allowable paths for the virtual camera from the camera control data, and to update the position of the virtual camera. , wherein the presentation engine is configured to ensure that the virtual camera moves only along virtual paths that are within the allowable paths defined in the camera control data.

Clause 97: The device of any of clauses 85 to 96, wherein the camera control data is included in an MPEG_mesh_collision extension.

Clause 98: A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor executing a presentation engine to: display streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; to receive; receive camera control data for the three-dimensional scene, wherein the camera control data includes data defining acceptable positions for a virtual camera; receive camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; and use the camera control data to update the position of the virtual camera to ensure that the virtual camera remains within the acceptable positions.

Clause 99: The computer-readable storage medium of clause 98, wherein the instructions causing the processor to update the position of the virtual camera cause the processor to cause the virtual camera to pass the at least one virtual solid object. A computer-readable storage medium containing instructions to prevent a computer-readable storage medium.

Clause 100: The computer-readable medium of any of clauses 98 and 99, wherein the streamed media data comprises glTF 2.0 media data.

Clause 101: The computer readable medium of any of clauses 98-100, wherein the instructions causing the processor to receive the streamed media data cause the processor to execute a retrieval unit via an application programming interface (API). A computer-readable medium comprising instructions to request the streamed media data from.

Clause 102: The computer-readable medium of any of clauses 98-101, wherein the camera control data is included in an MPEG scene description.

Clause 103: The computer readable medium of any of clauses 98-102, wherein the camera control data includes data defining two or more anchor points and one or more segments between the anchor points, the segments comprising: The instructions that indicate allowable camera movement vectors for the virtual camera and cause the processor to update the position of the virtual camera allow the processor to traverse only segments between the anchor points. A computer-readable medium containing instructions to do so.

Clause 104: The computer-readable medium of clause 103, wherein the camera control data includes data defining a bounding volume indicative of an allowable camera movement volume for the virtual camera, and causing the processor to determine the position of the virtual camera. The instructions for causing an update include instructions for causing the processor to allow the virtual camera to traverse only the allowable camera movement volume.

Clause 105: The computer-readable medium of any of clauses 98-104, wherein the data defining the bounding volume includes data defining at least one of a cone, a frustum, or a sphere.

Clause 106: The computer-readable medium of clause 105, wherein the camera control data is included in an MPEG_camera_control extension.

Clause 107: The computer-readable medium of any of clauses 98-106, wherein the MPEG_camera_control extension comprises: anchor data indicating a number of anchor points for allowable paths for the virtual camera; segment data indicating the number of path segments for the allowable paths between the anchor points; Bounding volume data representing a bounding volume for the virtual camera; intrinsic parameters indicating whether camera parameters are modified at each of the anchor points; and accessor data indicating an index of an accessor providing the camera control data.

Clause 108: The computer-readable medium of any of clauses 98-107, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 109: The computer-readable medium of any of clauses 98-108, further comprising instructions that cause the processor to determine allowable paths for the virtual camera from the camera control data, and cause the processor to: The instructions for updating the position of a virtual camera include instructions for causing the processor to ensure that the virtual camera moves only along virtual paths that are within the allowable paths defined in the camera control data. Computer-readable media.

Clause 110: The computer-readable medium of any of clauses 98-109, wherein the camera control data is included in an MPEG_mesh_collision extension.

Clause 111: A device for retrieving media data, the device comprising: means for receiving streamed media data representing a virtual three-dimensional scene comprising at least one virtual solid object; means for receiving camera control data for the three-dimensional scene, the camera control data comprising data defining acceptable positions for a virtual camera; means for receiving camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; and means for using the camera control data to update the position of the virtual camera to ensure that the virtual camera remains within the acceptable positions.

Clause 112: A method of retrieving media data, comprising: receiving, by a presentation engine, streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; receiving, by the presentation engine, object collision data representing boundaries of the at least one virtual solid object; receiving, by the presentation engine, camera movement data from a user requesting that a virtual camera move through the at least one virtual solid object; and, using the object collision data, update the position of the virtual camera in response to the camera movement data, by the presentation engine, to ensure that the virtual camera remains outside of the at least one virtual solid object. A method for retrieving media data, comprising the steps of:

Clause 113: A method comprising a combination of the method of any of Clauses 72 to 84 and the method of Clause 112.

Clause 114: The method of any of clauses 112 and 113, wherein updating the position of the virtual camera comprises preventing the virtual camera from passing the at least the virtual solid object.

Clause 115: The method of any of clauses 112-114, wherein receiving object collision data comprises receiving an MPEG_mesh_collision extension.

Clause 116: The method of clause 115, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object.

Clause 117: The method of clause 116, wherein the MPEG_mesh_collision extension is in response to boundaries of a 3D mesh for the at least one virtual solid object, a material for the 3D mesh, or the virtual camera contacting the 3D mesh. A method, comprising data defining at least one of the animations to be presented.

Clause 118: The method of any of clauses 112-117, wherein receiving object collision data comprises: boundary data representing one or more collision boundaries of the at least one virtual solid object; static data indicating whether the at least one virtual solid object is affected by collisions; material data indicating how collision objects interact with the at least one virtual solid object; or receiving data comprising one or more of animation data representing animations triggered by a collision with the at least one virtual solid object.

Clause 119: The method of any of clauses 112-118, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 120: The method of any of clauses 112-119, wherein the streamed media data comprises glTF 2.0 media data.

Clause 121: The method of any of clauses 112-120, wherein receiving streamed media data comprises requesting the streamed media data from a retrieval unit via an application programming interface (API). .

Clause 122: The method of any of clauses 112-121, wherein the object collision data is included in an MPEG scene description.

Clause 123: A device for retrieving media data, the device comprising: a memory configured to store media data; and one or more processors implemented in circuitry and configured to execute a presentation engine, the presentation engine configured to: receive streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; receive object collision data indicating boundaries of the at least one virtual solid object; receive camera movement data from a user requesting that a virtual camera move through the at least one virtual solid object; configured to use the object collision data to update the position of the virtual camera in response to the camera movement data to ensure that the virtual camera remains outside of the at least one virtual solid object. A device for doing so.

Clause 124: A device comprising a combination of the device of any of clauses 85-97 and the device of clause 123.

Clause 125: The device of any of clauses 123 and 124, wherein to update the position of the virtual camera, the presentation engine is configured to prevent the virtual camera from passing the at least the virtual solid object.

Clause 126: The device of any of clauses 123-125, wherein, to receive object collision data, the presentation engine is configured to receive an MPEG_mesh_collision extension.

Clause 127: The device of clause 126, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object.

Clause 128: The device of clause 127, wherein the MPEG_mesh_collision extension is in response to boundaries of a 3D mesh for the at least one virtual solid object, a material for the 3D mesh, or the virtual camera contacting the 3D mesh. A device containing data defining at least one of the animations to be presented.

Clause 129: The device of any of clauses 123-128, wherein to receive the object collision data, the presentation engine comprises: boundary data indicative of one or more collision boundaries of the at least one virtual solid object; static data indicating whether the at least one virtual solid object is affected by collisions; material data representing how colliding objects interact with the at least one virtual solid object; or animation data representing animations triggered by a collision with the at least one virtual solid object.

Clause 130: The device of any of clauses 123-129, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 131: The device of any of clauses 123-130, wherein the streamed media data comprises glTF 2.0 media data.

Clause 132: The device of any of clauses 123-131, wherein, to receive the streamed media data, the presentation engine is configured to request the streamed media data from the retrieval unit via an application programming interface (API). .

Clause 133: The device of any of clauses 123-132, wherein the object collision data is included in an MPEG scene description.

Clause 134: A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor to: receive streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; receive object collision data representing boundaries of the at least one virtual solid object; receive camera movement data from a user requesting that a virtual camera move through the at least one virtual solid object; A computer-readable storage medium that uses the object collision data to update the position of the virtual camera in response to the camera movement data to ensure that the virtual camera remains outside of the at least one virtual solid object. .

Clause 135: A computer-readable storage medium comprising a combination of the computer-readable storage medium of any of clauses 98-110 and the computer-readable storage medium of clause 134.

Clause 136: The computer readable medium of any of clauses 134 and 135, wherein the instructions causing the processor to update the position of the virtual camera cause the processor to cause the virtual camera to pass the at least a virtual solid object. A computer-readable medium containing instructions that prevent the computer from doing so.

Clause 137: The computer-readable medium of any of clauses 134-136, wherein the instructions causing the processor to receive the object collision data include instructions to cause the processor to receive an MPEG_mesh_collision extension. Readable media.

Clause 138: The computer-readable medium of any of clauses 134-137, wherein the MPEG_mesh_collision extension includes data defining at least one 3D mesh for the at least one virtual solid object.

Clause 139: The computer-readable medium of any of clauses 134-138, wherein the MPEG_mesh_collision extension includes boundaries of a 3D mesh for the at least one virtual solid object, a material for the 3D mesh, or the virtual camera. A computer-readable medium comprising data defining at least one of animations to be presented in response to contacting a 3D mesh.

Clause 140: The computer-readable medium of any of clauses 134-139, wherein the instructions cause a processor to receive object collision data, wherein the instructions cause the processor to: detect a boundary representing one or more collision boundaries of at least one virtual solid object; data; static data indicating whether at least one virtual solid object is affected by collisions; material data representing how colliding objects interact with at least one virtual solid object; or animation data representing animations triggered by a collision with at least one virtual solid object.

Clause 141: The computer-readable medium of any of clauses 134-140, wherein the at least one virtual solid object comprises one of a virtual wall, a virtual chair, or a virtual table.

Clause 142: The computer-readable medium of any of clauses 134-141, wherein the streamed media data comprises glTF 2.0 media data.

Clause 143: The computer-readable medium of any of clauses 134-142, wherein the instructions causing the processor to receive the streamed media data cause the processor to receive the streamed media data from a retrieval unit via an application programming interface (API). A computer-readable medium comprising instructions to request the streamed media data.

Clause 144: The computer-readable medium of any of clauses 134-143, wherein the object collision data is included in an MPEG scene description.

Clause 145: A method of retrieving media data, the method comprising: receiving, by a presentation engine, streamed media data representing a virtual three-dimensional scene including at least one virtual solid object; Receiving, by the presentation engine, camera control data for the three-dimensional scene, the camera control data including data defining constraints to prevent the virtual camera from passing through at least one virtual solid object; Receiving, by the presentation engine, camera movement data from a user requesting that the virtual camera move through at least one virtual solid object; and using the camera control data to prevent the virtual camera from passing the at least one virtual solid object in response to the camera movement data.

Clause 146: The method of clause 145, wherein the streamed media data comprises glTF 2.0 media data.

Clause 147: The method of any of clauses 145 and 146, wherein receiving streamed media data comprises requesting the streamed media data from a retrieval unit via an application programming interface (API). .

Clause 148: The method of any of clauses 145-147, wherein the camera control data is included in the MPEG scene description.

Clause 149: The method of any of clauses 145 to 148, wherein the camera control data is included in an MPEG_camera_control extension.

Clause 150: The method of clause 149, wherein the MPEG_camera_control extension includes data defining two or more anchor points and one or more segments between the anchor points, the segments representing allowable camera movement vectors. .

Clause 151: The method of any of clauses 149 and 150, wherein the MPEG_camera_control extension includes data defining a bounding volume indicative of an allowable camera movement volume.

Clause 152: The method of clause 151, wherein the data defining the bounding volume comprises data defining at least one of a cone, a frustum, or a sphere.

Clause 153: The method of any of clauses 149 to 152, wherein the MPEG_camera_control extension is in accordance with the data in Table 1.

Clause 154: The method of any of clauses 149-153, wherein the at least one virtual solid object comprises a virtual wall.

Clause 155: The method of any of clauses 149-154, wherein preventing the virtual camera from passing through the at least one virtual solid object comprises causing the virtual camera to follow virtual paths that exceed allowable paths defined in the MPEG_camera_control extension. A method comprising preventing movement.

Clause 156: The method of any of clauses 145 to 155, wherein the camera control data is included in the MPEG_mesh_collision extension.

Clause 157: The method of clause 156, wherein the MPEG_mesh_collision extension comprises data defining at least one 3D mesh for the at least one virtual solid object.

Clause 158: The method of clause 157, wherein the MPEG_mesh_collision extension configures at least one of the boundaries of the 3D mesh, a material for the 3D mesh, or animations to be presented in response to the virtual camera contacting the 3D mesh. A method containing defining data.

Clause 159: The method of any of clauses 156 to 158, wherein the MPEG_mesh_collision extension is according to Table 2 above.

Clause 160: The method of any of clauses 156-159, wherein preventing the virtual camera from passing through the at least one virtual solid object comprises: the virtual camera entering the at least one virtual solid object using the MPEG_mesh_collision extension. A method comprising steps of preventing.

Clause 161: A device for retrieving media data, wherein the device comprises one or more means for performing the method of any of clauses 145 to 160.

Clause 162: The device of clause 161, wherein the one or more means comprises one or more processors implemented in circuitry.

Clause 163: The device of clause 161, wherein the device comprises: an integrated circuit; microprocessor; and at least one of a wireless communication device.

Clause 164: A device for retrieving media data, the device comprising: means for receiving streamed media data representing a virtual three-dimensional scene comprising at least one virtual solid object; means for receiving camera control data for a three-dimensional scene, the camera control data comprising data defining constraints to prevent the virtual camera from passing through at least one virtual solid object; means for receiving camera movement data from a user requesting that the virtual camera move through at least one virtual solid object; and means for using the camera control data to prevent the virtual camera from passing the at least one virtual solid object in response to the camera movement data.

In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media refers to a computer-readable storage medium, such as a tangible medium, such as data storage media, or any device that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. It may also include communication media including media. In this manner, computer-readable media may generally correspond to (1) non-transitory computer-readable storage media or (2) communication media, such as a signal or carrier wave. A data storage medium may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include computer-readable media.

By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or in the form of instructions or data structures. It can be used to store desired program code and can include any other medium that can be accessed by a computer. Any connection is also properly termed a computer-readable medium. For example, if commands are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the commands are coaxial. Cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, wireless, and microwave are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other transient media, but instead relate to non-transitory, tangible storage media. Disk and disc as used herein include compact disk (CD), laser disk, optical disk, digital versatile disk (DVD), floppy disk and Blu-ray disk, where disk ( A disk usually reproduces data magnetically, but a disc reproduces data optically using a laser. Combinations of the above should also be included within the scope of computer-readable media.

The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term “processor,” as used herein, may refer to any of the above-described structure or any other structure suitable for implementation of the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding or integrated in a combined codec. Additionally, the techniques may be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to highlight functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors as described above, together with suitable software and/or firmware. It may be possible.

Various examples have been described. These and other examples are within the scope of the following claims.

Claims (40)

As a method of retrieving media data,
The above method is:
Receiving, by a presentation engine, streamed media data representing a virtual three-dimensional scene including at least one virtual solid object;
Receiving, by the presentation engine, camera control data for the three-dimensional scene, wherein the camera control data includes data defining acceptable positions for a virtual camera. ;
receiving, by the presentation engine, camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; and
Using the camera control data, update, by the presentation engine, the position of the virtual camera to ensure that the virtual camera remains within the acceptable positions. . According to claim 1,
Wherein updating the position of the virtual camera includes preventing the virtual camera from passing the at least one virtual solid object. According to claim 1,
A method for retrieving media data, wherein the streamed media data includes glTF 2.0 media data. According to claim 1,
The method of retrieving media data, wherein receiving the streamed media data includes requesting the streamed media data from a retrieval unit via an application programming interface (API). According to claim 1,
A method for retrieving media data, wherein the camera control data is included in an MPEG scene description. According to claim 1,
The camera control data includes data defining two or more anchor points and one or more segments between the anchor points, the segments representing allowable camera movement vectors for the virtual camera, and the position of the virtual camera. wherein updating includes allowing the virtual camera to only traverse the segments between the anchor points. According to claim 1,
The camera control data includes data defining a bounding volume representing an allowable camera movement volume for the virtual camera, and updating the position of the virtual camera causes the virtual camera to only traverse the allowable camera movement volume. A method of retrieving media data, comprising allowing: According to claim 7,
The method of retrieving media data, wherein the data defining the bounding volume includes data defining at least one of a cone, a frustum, or a sphere. According to claim 1,
A method of retrieving media data, wherein the camera control data is included in the MPEG_camera_control extension. According to clause 9,
The MPEG_camera_control extension above:
anchors data indicating the number of anchor points for allowable paths for the virtual camera;
segments data indicating the number of path segments for allowable paths between the anchor points;
Bounding volume data representing a bounding volume for the virtual camera;
intrinsic parameters indicating whether camera parameters are modified at each of the anchor points; and
Accessor data indicating the index of the accessor providing the camera control data
A method of retrieving media data, including one or more of: According to claim 1,
The method of retrieving media data, wherein the at least one virtual solid object includes one of a virtual wall, a virtual chair, or a virtual table. According to claim 1,
Further comprising determining allowable paths for the virtual camera from the camera control data, wherein updating the position of the virtual camera ensures that the virtual camera is within the allowable paths defined in the camera control data. A method of retrieving media data, comprising ensuring that it moves only along virtual paths that exist. According to claim 1,
A method of retrieving media data, wherein the camera control data is included in the MPEG_mesh_collision extension. A device for retrieving media data,
The device:
a memory configured to store media data; and
comprising one or more processors implemented in circuitry and configured to execute a presentation engine;
The presentation engine:
receive streamed media data representing a virtual three-dimensional scene including at least one virtual solid object;
receive camera control data for the three-dimensional scene, wherein the camera control data includes data defining acceptable positions for a virtual camera;
receive camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; and
Use the camera control data to update the position of the virtual camera to ensure that the virtual camera remains within the acceptable positions.
A device configured to retrieve media data. According to claim 14,
wherein the presentation engine is configured to prevent the virtual camera from passing the at least one virtual solid object. According to claim 14,
A device for retrieving media data, wherein the streamed media data includes glTF 2.0 media data. According to claim 14,
The device for retrieving media data, wherein the presentation engine is configured to request the streamed media data from a retrieval unit via an application programming interface (API). According to claim 14,
A device for retrieving media data, wherein the camera control data is included in an MPEG scene description. According to claim 14,
The camera control data includes data defining two or more anchor points and one or more segments between the anchor points, the segments representing allowable camera movement vectors for the virtual camera, and the position of the virtual camera. wherein the presentation engine is configured to allow the virtual camera to only traverse the segments between the anchor points. According to claim 14,
The camera control data includes data defining a bounding volume representing the allowable camera movement volume for the virtual camera, and to update the position of the virtual camera, the presentation engine determines that the virtual camera is the allowable camera movement. A device for retrieving media data, configured to allow only traversing a moving volume. According to claim 20,
The device for retrieving media data, wherein the data defining the bounding volume includes data defining at least one of a cone, a frustum, or a sphere. According to claim 14,
A device for retrieving media data, wherein the camera control data is included in the MPEG_camera_control extension. According to claim 22,
The MPEG_camera_control extension above:
anchors data indicating the number of anchor points for allowable paths for the virtual camera;
segments data indicating the number of path segments for allowable paths between the anchor points;
Bounding volume data representing a bounding volume for the virtual camera;
intrinsic parameters indicating whether camera parameters are modified at each of the anchor points; and
Accessor data indicating the index of the accessor providing the camera control data
A device for retrieving media data, including one or more of the following. According to claim 14,
The device for retrieving media data, wherein the at least one virtual solid object includes one of a virtual wall, a virtual chair, or a virtual table. According to claim 14,
The presentation engine is further configured to determine allowable paths for the virtual camera from the camera control data, and to update the position of the virtual camera, the presentation engine is configured to determine the virtual camera based on the camera control data. A device for retrieving media data, configured to ensure that it moves only along virtual paths that are within the defined allowable paths. According to claim 14,
A device for retrieving media data, wherein the camera control data is included in the MPEG_mesh_collision extension. A computer-readable storage medium storing instructions, comprising:
The instructions, when executed, cause the processor running the presentation engine to:
receive streamed media data representing a virtual three-dimensional scene including at least one virtual solid object;
receive camera control data for the three-dimensional scene, wherein the camera control data includes data defining acceptable positions for a virtual camera;
receive camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; and
and use the camera control data to update the position of the virtual camera to ensure that the virtual camera remains within the acceptable positions. According to clause 27,
The instructions causing the processor to update the position of the virtual camera include instructions to cause the processor to prevent the virtual camera from passing the at least one virtual solid object. According to clause 27,
Wherein the streamed media data includes glTF 2.0 media data. According to clause 27,
wherein the instructions causing the processor to receive the streamed media data include instructions causing the processor to request the streamed media data from a retrieval unit via an application programming interface (API). media. According to clause 27,
Wherein the camera control data is included in an MPEG scene description. According to clause 27,
The camera control data includes data defining two or more anchor points and one or more segments between the anchor points, the segments representing allowable camera movement vectors for the virtual camera, and causing the processor to: The instructions for updating the position of a virtual camera include instructions for causing the processor to allow the virtual camera to only traverse the segments between the anchor points. According to clause 27,
The camera control data includes data defining a bounding volume indicating an allowable camera movement volume for the virtual camera, and the instructions causing the processor to update the position of the virtual camera cause the processor to update the virtual camera. A computer-readable storage medium comprising instructions for allowing a camera to only traverse the allowable camera movement volume. According to claim 20,
The data defining the bounding volume includes data defining at least one of a cone, a frustum, or a sphere. According to clause 27,
The computer-readable storage medium wherein the camera control data is included in the MPEG_camera_control extension. According to claim 22,
The MPEG_camera_control extension above:
anchors data indicating the number of anchor points for allowable paths for the virtual camera;
segments data indicating the number of path segments for allowable paths between the anchor points;
Bounding volume data representing a bounding volume for the virtual camera;
intrinsic parameters indicating whether camera parameters are modified at each of the anchor points; and
Accessor data indicating the index of the accessor providing the camera control data
A computer-readable storage medium comprising one or more of the following. According to clause 27,
wherein the at least one virtual solid object includes one of a virtual wall, a virtual chair, or a virtual table. According to clause 27,
further comprising instructions that cause the processor to determine allowable paths for the virtual camera from the camera control data, wherein the instructions cause the processor to update the position of the virtual camera; A computer-readable storage medium comprising instructions to ensure that a camera moves only along virtual paths that are within the allowable paths defined in the camera control data. According to clause 27,
The computer-readable storage medium wherein the camera control data is included in the MPEG_mesh_collision extension. A device for retrieving media data,
The device:
means for receiving streamed media data representing a virtual three-dimensional scene including at least one virtual solid object;
means for receiving camera control data for the three-dimensional scene, wherein the camera control data includes data defining acceptable positions for a virtual camera;
means for receiving camera movement data from a user requesting that the virtual camera move through the at least one virtual solid object; and
and means for using the camera control data to update the position of the virtual camera to ensure that the virtual camera remains within the acceptable positions.

Images