RU2795052C2 - Methods and device for adaptive point cloud streaming - Google Patents

Abstract

FIELD: data transmission.

SUBSTANCE: invention relates to means for adaptive streaming video-based point cloud compression (V-PCC) using an adaptive HTTP streaming protocol such as DASH MPEG. PCD PC related information is transmitted in a Media Presentation Description (MPD) for Dynamic Adaptive Streaming over HTTP (DASH), wherein the DASH MPD includes information indicating at least: (1) a basic adaptation set (AS) for the point cloud, the basic AS including information indicating at least: (i) a unique codec attribute value indicating that the basic AS corresponds to the V-PCC data, (ii) an initialization segment comprising at least one set of V-PCC sequence parameters for representing the PC, and (2) a plurality of AS set components, wherein each of the plurality of AS components corresponds to one of the plurality of V-PCC components and includes information indicative of at least: (a) a V-PCC Component descriptor specifying the type of the corresponding V-PCC component, and (b) at least one property of the corresponding V-PCC component. MPD DASH is transmitted over the network.

EFFECT: increased transmission efficiency of 3D point clouds.

20 cl, 8 dwg, 8 tbl

Description

Prerequisites for the creation of the invention

Recently, high-quality 3D point clouds have emerged as an advanced representation of an immersive environment. A point cloud consists of a collection of points represented in 3D space using coordinates indicating the location of each point along with one or more attributes such as the color associated with each point, transparency, acquisition time, laser reflection or material property, etc. n. Data for creating point clouds can be obtained in different ways. For example, one technique for capturing point clouds is to use multiple cameras and depth sensors. Laser light detection and ranging (LiDAR) scanners are also often used to capture point clouds. The number of points needed to realistically restore objects and scenes using point clouds is on the order of millions (or even billions). Therefore, efficient representation and compression is essential for storing and transmitting point cloud data.

Recent advances in 3D point capture and visualization technologies have led to new applications in the areas of telepresence, virtual reality, and large-scale dynamic 3D maps. The 3D Graphics subgroup of the Moving Picture Experts Group (MPEG), which oversees the ISO/IEC JTC1/SC29/WG11 standard, is currently working on the development of two 3D Point Cloud Compression (PCC) standards: a geometry-based compression standard for static point clouds (point clouds for stationary objects); and a video-based compression standard for dynamic point clouds (point clouds for moving objects). The purpose of these standards is to provide efficient and interactive storage and transmission of 3D point clouds. Among the requirements of these standards is support for lossy and/or lossless encoding of geometric coordinates and point cloud attributes.

Brief description of graphic materials

For a more detailed explanation, see the detailed description below, using examples in conjunction with the accompanying graphics. The figures in such drawings, as well as the detailed description, are examples. Thus, these figures and the detailed description should not be construed as limiting, and other equally effective examples are possible and likely. In addition, like positions in the figures indicate like elements, with

in fig. 1A is a block diagram illustrating an example of a video encoding and decoding system in which one or more embodiments may be provided and/or implemented;

in fig. 1B is a block diagram illustrating an example of a video encoder block for use with the video encoding and/or decoding system shown in FIG. 1A;

in fig. 2 is a block diagram of a universal hybrid block video coding system;

in fig. 3 is a general block diagram of a block video decoder;

in fig. 4 shows a bitstream structure for video-based point cloud compression (V-PCC);

in fig. 5 shows a hierarchical MPD data model;

in fig. 6 shows an example DASH configuration for grouping V-PCC components belonging to a single point cloud in an MPD MPEG-DASH file; And

in fig. 7 is a flowchart illustrating an example of a decoding method for streaming point cloud content according to an embodiment.

Detailed description

Examples of systems in which implementation of the embodiments is possible

In FIG. 1A is a block diagram illustrating an example of a video encoding and decoding system 100 in which one or more embodiments may be provided and/or implemented. System 100 may include a source device 112 that can transmit encoded video information to destination device 114 over communication channel 116 .

Source device 112 and/or destination device 114 can be any of a wide variety of devices. In some exemplary embodiments, source device 112 and/or destination device 114 may include wireless transmit and/or receive units (WTRUs), such as wireless telephones or any wireless device capable of transmitting video information over communication channel 116, and in this In this case, the link 116 includes a wireless link. However, the methods, devices, and systems described, disclosed, or otherwise provided, either expressly, implicitly, and/or inherently (collectively referred to as "provided") herein, are not necessarily limited to wireless applications or settings. For example, these techniques can be applied to terrestrial television broadcasts, cable television transmissions, satellite television transmissions, Internet video transmissions, encoded digital video that is encoded on a storage medium, and/or other scenarios. Communication channel 116 may include and/or may be any combination of wireless or wired media suitable for transmitting encoded video data.

Source device 112 may include a video encoder unit 118, a transmit and/or receive (Tx/Rx) unit 120, and/or a Tx/Rx element 122. As shown, source device 112 may include video source 124. Destination device 114 may include a Tx/Rx element 126, a Tx/Rx block 128, and/or a video decoder block 130. As shown, the destination device 114 may include a display device 132. Each of the Tx/Rx blocks 120, 128 may be or may include a transmitter, a receiver, or a combination of transmitter and receiver (eg, a transceiver or a transmitter/receiver). Each of the elements 122, 126 Tx/Rx can be, for example, an antenna. As used herein, video encoder unit 118 of source device 112 and/or video decoder unit 130 of destination device 114 may be configured and/or adapted (generally "adapted") to apply the encoding techniques proposed herein.

Source and destination devices 112 and 114 may include other elements/components or structures. For example, source device 112 may be configured to receive video data from an external video source. Destination device 114 may communicate with an external display device (not shown) and/or may include and/or use (eg, integrated) display device 132. In some embodiments, the data stream generated by video encoder unit 118 may be transmitted to other devices without data carrier modulation, such as direct digital transmission, and other devices may or may not modulate the data for transmission.

The techniques proposed herein can be implemented using any digital video encoding and/or decoding device. Although the techniques proposed herein are essentially performed with separate video encoders and/or video decoders, these techniques can also be performed with a combined video encoder/decoder, commonly referred to as a "CODEC". The techniques provided herein may also be performed by a video pre-processor or the like. Source device 112 and destination device 114 are only examples of such encoders in which source device 112 can generate (and/or can receive video data and can generate) encoded video information for transmission to destination device 114. In some exemplary embodiments, source and destination devices 112, 114 may operate in a substantially symmetrical manner such that each of devices 112, 114 may include components and/or elements of both video encoding and decoding (collectively referred to as "elements") . Thus, system 100 can support any one-way and two-way video transmission between source and destination devices 112, 114 (eg, but not limited to any of video streaming, video playback, video broadcast, video telephony, and/or video conferencing). In some exemplary embodiments, source device 112 may be, for example, a video streaming server configured to generate (and/or receive video data and generate) encoded video information for one or more destination devices, wherein the destination devices can communicate with the device. -source 112 via wired and/or wireless communication systems.

The external video source and/or video source 124 may be and/or include a video capture device such as a video camera, a video archive containing previously captured video, and/or a video stream from a video content provider. In some exemplary embodiments, the external video source and/or video source 124 may generate data from computer graphics as source video, or a combination of live video, archived video, and/or computer generated video. In some exemplary embodiments, if video source 124 is a video camera, source device 112 and destination device 114 may be or include camera phones or video phones.

The captured, pre-captured, computer-generated video, video stream, and/or other type of video data (collectively referred to as "non-encoded video") may be encoded by the video encoder unit 118 to generate encoded video information. The Tx/Rx unit 120 may modulate the encoded video information (eg, in accordance with a communication standard to generate one or more modulated signals containing encoded video information). The Tx/Rx unit 120 may send the modulated signals to its transmitter for transmission. The transmitter may transmit the modulated signals to destination device 114 via Tx/Rx element 122.

At destination device 114, Tx/Rx unit 128 can receive modulated signals over channel 116 via Tx/Rx element 126. Block 128 Tx/Rx may demodulate the modulated signals to obtain encoded video information. Block 128 Tx/Rx can transmit encoded video information in block 130 of the video decoder.

The video decoder unit 130 may decode the encoded video information to obtain decoded video data. The encoded video information may include syntax information defined by video encoder block 118. This syntax information may include one or more elements ("syntax elements"), some or all of which may be useful for decoding the encoded video information. The syntax elements may include, for example, characteristics of the encoded video information. The syntax elements may also include characteristics and/or describe the processing of unencoded video used to generate encoded video information.

The video decoder unit 130 may output the decoded video data for later storage and/or display on an external display (not shown). In some exemplary embodiments, video decoder unit 130 may output decoded video data to display device 132. Display device 132 may be and/or may include any single display device, a plurality, combination of different display devices configured to display decoded video data to a user. Examples of such display devices include, but are not limited to, a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, and/or a cathode ray tube (CRT).

Communication channel 116 may be any wireless or wired media, such as the radio frequency (RF) spectrum, or one or more physical transmission lines, or any combination of wireless and wired media. Communication channel 116 may be part of a packet network such as a local area network, a wide area network, or a wide area network such as the Internet. Communication channel 116 is essentially any suitable communication medium or set of different communication media for transmitting video data from source device 112 to destination device 114, including any suitable combination of wired and/or wireless media. Communication channel 116 may include routers, switches, base stations, and/or any other equipment that may be useful in facilitating data transfer from source device 112 to destination device 114. Details of an exemplary communications system that may facilitate such communications between devices 112, 114 are provided below with reference to FIG. 15A–15E. Below are also detailed information about the devices 112, 114 source and destination.

Video encoder unit 118 and video decoder unit 130 may operate in accordance with one or more standards and/or specifications such as, for example, but not limited to MPEG-2, H.261, H.263, H.264, H.264/AVC and/or H.264 extended according to the SVC extensions (H.264/SVC). The person skilled in the art will appreciate that the methods, devices, and/or systems described herein are applicable to other video encoders, decoders, and/or CODECs implemented in accordance with (and/or compatible with) different standards, or to proprietary video encoders. , decoders and/or CODECs, including future video encoders, decoders and/or CODECs. The techniques outlined in this document are not limited to any particular coding standard.

The relevant parts of H.264/AVC mentioned above are available from the International Telecommunication Union as ITU-T Recommendation H.264 or more specifically ITU-T Rec. H.264 and ISO/IEC 14496-10 (MPEG4-AVC), Advanced Video Coding for Generic Audiovisual Services,'v5, March 2010, which is incorporated herein by reference and which may be referred to herein as the H. 264, H.264 specification, standard and/or H.264/AVC specification. The techniques presented herein can be applied to devices that conform (eg, generally conform) to the H.264 standard.

Although not shown in FIG. 1A, each of the video encoder and video decoder blocks 118, 130 may include and/or be combined with an audio encoder and/or audio decoder (if necessary). The video encoder and video decoder units 118, 130 may include respective MUX-DEMUX units or other hardware and/or software for processing both audio and video encoding in a common data stream and/or in separate data streams. If applicable, the MUX-DEMUX units may conform to, for example, the multiplexer protocol and/or other protocols such as User Datagram Protocol (UDP) as per ITU-T Recommendation H.223.

One or more video encoder and/or video decoder blocks 118, 130 may be included in one or more encoders and/or decoders, any of which may be integrated into a CODEC and may be integrated and/or combined with, among other things, a corresponding camera, computer, mobile device, subscriber device, broadcast device, set-top box and/or server. Video encoder unit 118 and/or video decoder unit 130 may be implemented as any of a variety of suitable encoder and/or decoder circuits, respectively, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gates. arrays (FPGAs), discrete logic circuits, software, hardware, firmware, or any combination thereof. One or both of the video encoder and video decoder units 118, 130 may be substantially implemented in software, and the operations of the elements of video encoder unit 118 and/or video decoder unit 130 may be performed by appropriate software instructions executed by one or more processors (not shown). ). Such an embodiment may include off-chip components in addition to the processor, such as, but not limited to, external storage (eg, in the form of non-volatile storage) and/or I/O interfaces.

In any embodiment in which the operations of the elements of video encoder and/or video decoder blocks 118, 130 may be performed by software instructions executable by one or more processors, the software instructions may be stored on a computer-readable medium, including, for example, but not limited to, magnetic disks, optical disks, any other volatile (eg, random access memory (RAM)), non-volatile (eg, read only memory (ROM)) and/or mass storage device readable by the CPU. The computer-readable medium may include an interoperable or interconnected computer-readable medium that may be used exclusively within a processing system and/or distributed among a plurality of interconnected processing systems, which may be local or remote to said processing system.

In FIG. 1B is a block diagram illustrating an example video encoder block 118 for use with a video encoding and/or decoding system, such as system 100. Video encoder block 118 may include a video encoder 133, an output buffer 134, and a system controller 136. Video encoder 133 (or one or more of its elements) may be implemented in accordance with one or more standards and/or specifications, such as, for example, but not limited to, H.261, H.263, H.264, H.264/AVC, SVC H. 264/AVC (H.264/AVC Annex G), HEVC and/or Scalable HEVC Extensions (SHVC). One skilled in the art will appreciate that the methods, devices, and/or systems presented herein may be applicable to other video encoders implemented in accordance with various standards and/or proprietary CODECs, including future CODECs.

Video encoder 133 may receive video from a video source, such as video source 124 and/or an external video source. This video signal may include unencoded video. Video encoder 133 may encode unencoded video and provide encoded (ie, compressed) video bitstream (BS) at its output.

The encoded video bitstream of the BS may be sent to an output buffer 134. The output buffer 134 may buffer the encoded video bitstream of the BS and may provide such encoded video bitstream of the BS as a buffered bitstream (BBS) for transmission over the communication channel 116.

The buffered BBS bitstream output from output buffer 134 may be sent to a storage device (not shown) for later viewing or transmission. In some exemplary embodiments, video encoder unit 118 may be configured for visual communication, in which a buffered BBS bitstream may be transmitted over communication channel 116 at a predetermined constant and/or variable bit rate (e.g., latency (e.g., very low or minimum delay.

The encoded video bitstream BS and in turn the buffered bitstream BBS may carry bits of encoded video information. The bits of the buffered BBS bitstream may be ordered as a stream of encoded video frames. The encoded video frames may be intra-coded frames (eg, I-frames) or inter-coded frames (eg, B-frames and/or P-frames). The encoded video frame stream may be ordered, for example, as a series of groups of pictures (GOPs), with the encoded video frames of each GOP arranged in that order. Typically, each GOP may begin with an intra-coded frame (eg, I-frame) followed by one or more inter-coded frames (eg, P-frames and/or B-frames). Each GOP may include only one intra-coded frame, although any of the GOPs may include multiple such frames. It is contemplated that B-frames may not be used for low latency real-time applications because, for example, bidirectional prediction may result in additional coding delay compared to unidirectional prediction (P-frames). Additional and/or other types of frames may be used, and the specific order of the encoded video frames may be changed, as will be appreciated by one of ordinary skill in the art.

Each GOP may include syntax data ("GOP syntax data"). The GOP syntax data may be located in the GOP header, in the header of one or more GOP frames, and/or elsewhere. The GOP syntax data may indicate the order, number, or type and/or describe the encoded video frames of the corresponding GOP. Each encoded video frame may include syntax data ("encoded frame syntax data"). The encoded frame syntax data may indicate and/or describe an encoding mode for a corresponding encoded video frame.

The system controller 136 may monitor various parameters and/or constraints associated with the channel 116, the processing capability of the video encoder unit 118, user requirements, and the like, and may set targets to provide a concomitant Quality of Experience (QoE) that is appropriate for the specified constraints and /or channel conditions 116. One or more target parameters may be adjusted over time or periodically depending on specified constraints and/or channel conditions. As an example, QoE can be quantified using one or more metrics for evaluating video quality, including, for example, a metric commonly referred to as relative perceptual quality of coded video sequences. The relative perceptual quality of encoded video sequences, measured for example by peak signal-to-noise ratio (PSNR), may be monitored by the bit rate (BR) of the encoded BS bitstream. One or more target parameters (including, for example, a quantization parameter (QP)) may be adjusted to maximize the relative perceptual quality of the video within limits associated with the bit rate in the encoded BS bitstream.

In FIG. 2 shows a block diagram of a hybrid block video encoder 200 for use with a video encoding and/or decoding system, such as system 100.

As shown in FIG. 2, the hybrid block coding system 200 may include, but is not limited to, a transform block 204, a quantizer 206, an entropy coding block 208, an inverse quantizer 210, an inverse transform block 212, a first adder 216, a second adder 226, a spatial prediction block 260. , a motion predictor 262, a reference image store 264, one or more filters 266 (eg, loop filters), and/or a mode selector and encoder control 280.

The details of video encoder 200 are provided for illustrative purposes only and actual implementations may vary. For example, a practical implementation may include more, fewer, and/or different elements, and/or it may be organized differently from the arrangement shown in FIG. 2. For example, some or all of the functionality of both transform block 204 and quantizer 206, although shown separately, can be largely integrated into some practical implementations, such as, for example, implementations that use a basic transform. according to the H.264 standard. Similarly, inverse quantizer 210 and inverse transform 212 can be largely integrated into some practical implementations (eg, H.264 or HEVC compliant implementations), but they are also illustrated separately for conceptual purposes.

As described above, video encoder 200 may receive video at its input 202. Video encoder 200 may generate encoded video information from the received unencoded video and output encoded video information (eg, any of the intraframes or interframes) from its output 220 in the form of an encoded video bitstream BS. Video encoder 200 may function as a hybrid video encoder, for example, and use a block coding technique to encode unencoded video. When performing such an encoding method, video encoder 200 may operate on individual frames, pictures, and/or images (collectively referred to as "non-encoded pictures") of non-encoded video.

To simplify the block coding method, video encoder 200 may slice, split, split, and/or segment (generally, "segment") each non-encoded image received at its input 202 into a plurality of non-encoded video blocks. For example, video encoder 200 may segment a non-encoded image into multiple non-encoded video segments (eg, slices) and may (eg, subsequently may) segment each of the non-encoded video segments into non-encoded video blocks. Video encoder 200 may transmit, submit, send, or provide uncoded video blocks to spatial predictor 260, motion predictor 262, mode selector and encoder control 280, and/or first adder 216. As described in more detail below, uncoded video blocks may be provided block by block.

Spatial predictor 260 may receive uncoded video blocks and encode such video blocks in intra mode. The term "intra mode" refers to any of several spatial compression modes, and intra mode coding provides spatial compression of an uncoded image. Spatial compression, if present, may result from the reduction or removal of spatial redundancy of video information in the uncoded image. When generating prediction blocks, spatial predictor 260 may perform spatial prediction (or "intra-prediction") of each non-encoded video block with respect to one or more video blocks of the non-encoded image that have already been encoded ("encoded video blocks") and/or have been reconstructed ("reconstructed video blocks"). "). The encoded and/or reconstructed video blocks may be adjacent, contiguous, or adjacent to (eg, in close proximity to) an unencoded video block.

Motion predictor 262 may receive unencoded video blocks from input 202 and encode them in intermode. The term "intermode" refers to any of several temporal compression modes, including, for example, P-mode (unidirectional prediction) and/or B-mode (bidirectional prediction). Intermode encoding provides temporal compression of the non-encoded image. Temporal compression, if present, may be the result of reducing or removing temporal video redundancy between an uncoded picture and one or more reference (eg, adjacent) pictures. The temporal prediction/motion prediction unit 262 may perform temporal prediction (or “inter prediction”) of each non-encoded video block with respect to one or more reference picture video blocks (“reference video blocks”). The temporal prediction performed may be unidirectional prediction (eg, for P-mode) and/or bidirectional prediction (eg, for B-mode).

For unidirectional prediction, reference video blocks may be obtained from one or more previously encoded and/or reconstructed pictures. The encoded and/or reconstructed image or images may be neighboring(s), adjacent(s), or adjacent(s) to the unencoded image.

For bidirectional prediction, reference video blocks may be obtained from one or more previously encoded and/or reconstructed pictures. The encoded and/or reconstructed pictures may be adjacent, contiguous, or adjacent to the non-encoded picture.

When using multiple reference pictures (which may be the latest video coding standards such as H.264/AVC and/or HEVC), a reference picture index for each video block may be sent to entropy encoding unit 208 for subsequent output and/or transmission. The reference index may be used to identify from which reference image or images in the reference image store 264 the temporal prediction signal is derived.

While typically highly integrated, the functions of temporal prediction/motion prediction unit 262 for motion estimation and motion compensation may be performed by separate entities or units (not shown). Motion estimation may be performed to estimate the motion for each non-encoded video block relative to the video blocks of the reference picture, and this may include generating a motion vector for the non-encoded video block. The motion vector may indicate the offset of the prediction block relative to the encoded uncoded video block. This prediction block is a reference picture video block found to have a close match, for example, in terms of pixel difference of the encoded non-coded video block. Compliance can be determined by the sum of the absolute difference (SAD), the sum of the quadratic difference (SSD) and/or other indicators of the difference. Motion compensation may include fetching and/or generating a prediction block based on a motion vector determined by motion estimation.

Motion predictor 262 may calculate a motion vector for an uncoded video block by comparing the uncoded video block with reference video blocks from reference pictures in reference picture store 264 . The motion predictor 262 may calculate values for the fractional pixel positions of the reference image included in the reference image store 264 . In some cases, adder 226 or another unit of video encoder 200 may calculate fractional pixel position values for reconstructed video blocks and may store the reconstructed video blocks with calculated values for fractional pixel positions in reference image store 264 . Motion predictor 262 may interpolate sub-integer pixels of a reference image (eg, I-frame and/or P-frame and/or B-frame).

The motion predictor 262 may be configured to encode a motion vector with respect to a selected motion prediction parameter. The motion prediction parameter selected by the temporal prediction/motion prediction block 262 may be, for example, a vector equivalent to the average of the motion vectors of adjacent blocks that have already been coded. To encode the motion vector of a non-encoded video block, temporal prediction/motion prediction unit 262 may calculate a difference between the motion vector and the motion prediction parameter to generate a motion vector difference value.

In the H.264 and HEVC standards, a set of potential reference frames is considered as a "list". Such a list of reference frames may correspond to a set of reference images in the reference image store 264 . The temporal/motion predictor 262 may compare the reference video blocks of the reference pictures from the reference picture store 264 with an uncoded video block (eg, a P-frame or a B-frame). When the reference images in the reference image store 264 include values for sub-integer pixels, the motion vector computed by the temporal/motion prediction unit 262 may refer to the sub-integer pixel location of the reference image. The temporal/motion predictor 262 may send the computed motion vector to the entropy encoding unit 208 and the motion compensation functions of the temporal/motion prediction unit 262. The motion predictor 262 (or its motion compensation functions) may compute error values for the prediction block with respect to the encoded uncoded video block. The motion prediction block 262 may calculate prediction data based on the prediction block.

Block 280 mode selection and encoder control can select one of the coding modes - intramode or intermode. The mode selector and encoder control unit 280 may do this based on, for example, a method for optimizing the distortion-to-data rate ratio and/or the error results obtained in each mode.

Video encoder 200 may generate a residual block ("residual video block") by subtracting the prediction data provided by motion predictor 262 from the encoded uncoded video block. The adder 216 is an element or a plurality of elements that can perform this subtraction operation.

A transform block 204 may apply a transform to the residual video block to transform such a residual video block from a pixel domain to a transform domain, such as a frequency domain. The transform may be, for example, any of the transforms presented herein, a discrete cosine transform (DCT), or a conceptually similar transform. Other transform examples include those defined in H.264 and/or HEVC, wavelet transforms, integer transforms, and/or subband transforms, among others. When transform block 204 applies a transform to a residual video block, a corresponding block of residual video block transform coefficients ("residual transform coefficients") is created. These residual transform coefficients may be the magnitudes of the frequency components of the residual video block. Transform block 204 may send the residual transform coefficients to quantizer 206 .

Quantizer 206 may quantize the residual transform coefficients to further reduce the encoded bit rate. For example, the quantization process may reduce the bit depth associated with some or all of the residual transform coefficients. In certain cases, the quantizer 206 may divide the value of the residual transform coefficients by the quantization level corresponding to QP to form a block of quantized transform coefficients. The degree of quantization can be changed by adjusting the QP value. Quantizer 206 may apply quantization to represent residual transform coefficients using a desired number of quantization steps; the number of steps used (or quantization level value, respectively) may determine the number of encoded video bits used to represent the residual video block. Quantizer 206 may receive the QP value from a rate controller (not shown). After quantization, quantizer 206 may provide the quantized transform coefficients to entropy coding unit 208 and to inverse quantizer 210 .

Entropy encoding unit 208 may apply entropy encoding to the quantized transform coefficients to generate entropy-coded coefficients (ie, a bitstream). Entropy encoding unit 208 may use adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), and/or other entropy coding technique to generate the entropy coding coefficients. CABAC encoding, as those skilled in the art will appreciate, may require the input of contextual information ("context"). For example, this context may be based on adjacent video blocks.

Entropy encoding unit 208 may provide entropy-coded coefficients along with motion vectors and one or more reference picture indices as a raw encoded video bitstream for an internal bitstream format (not shown). This bitstream format can generate an encoded BS video bitstream input to output buffer 134 (FIG. 1B) by adding additional information about the raw encoded video bitstream, including headers and/or other information allowing, for example, video decoder unit 300 (FIG. 3) decode the encoded video block from the raw encoded video bitstream. After entropy encoding, the encoded BS video bitstream obtained from entropy encoding unit 208 may be output to, for example, output buffer 134 and may be transmitted, for example, to destination device 114 over channel 116, or archived for later transmission or retrieval.

In certain exemplary embodiments, entropy encoding unit 208 or another video encoder unit 133, 200 may be configured to perform other encoding functions in addition to entropy encoding. For example, entropy encoding block 208 may be configured to determine code block template (CBP) values for video blocks. In some exemplary embodiments, entropy encoding block 208 may perform run-length encoding of quantized transform coefficients in a video block. As an example, entropy encoding unit 208 may apply a zigzag scan or other scanning pattern to arrange the quantized transform coefficients in a video block and encode runs of zeros for additional compression. Entropy encoding unit 208 may generate header information with appropriate syntax elements for transmission in the encoded video bitstream BS.

An inverse quantizer 210 and an inverse transform 212 may apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel region, for example, for later use as one of the reference video blocks (for example, in one of the reference pictures in the reference picture list).

The mode selection and encoder control block 280 may calculate a reference video block by adding the reconstructed residual video block to a prediction block of one of the reference pictures in the reference picture store 264 . The mode selection and encoder control block 280 may apply one or more interpolation filters to the reconstructed residual video block to calculate sub-integer pixel values (eg, for half-pixel positions) when used in motion estimation.

Adder 226 may add the reconstructed residual video block to the motion-compensated prediction video block to generate a reconstructed video block for storage in reference image store 264. The reconstructed video block (in pixel range) may be used by motion prediction block 262 (or its motion estimation functions and/or its motion compensation functions) as one of the reference blocks for intercoding an uncoded video block in subsequent uncoded video.

Filters 266 (eg, loop filters) may include a deblocking filter. The deblocking filter may be operable to remove visual artifacts that may be present in the reconstructed macroblocks. These artifacts may be introduced into the encoding process, for example, due to the use of different encoding modes such as I-type, P-type, or B-type. Artifacts may be present, for example, at the boundaries and/or edges of the received video blocks, and the deblocking filter may function to smooth the boundaries and/or edges of the video blocks to improve visual quality. The deblocking filter may filter the output of adder 226. Filters 266 may include other in-loop filters such as the adaptive sample offset (SAO) filter supported by the HEVC standard.

In FIG. 3 is a block diagram illustrating an example of a video decoder 300 for use with a video decoder unit, such as the video decoder unit 130 shown in FIG. 1A. Video decoder 300 may include an input 302, an entropy decoder 308, a motion compensated predictor 362, a spatial predictor 360, an inverse quantizer 310, an inverse transform 312, a reference image store 364, filters 366, an adder 326, and an output 320. Video decoder 300 may perform a decoding method that is essentially the reverse of the encoding method applied to video encoder 133, 200. This decoding process may be performed as described below.

The motion compensated predictor 362 may generate prediction data based on the motion vectors received from the entropy decoder 308 . The motion vectors may be encoded with respect to a motion prediction parameter for the video block corresponding to the encoded motion vector. The motion compensated predictor 362 may determine a motion prediction parameter, for example, as a median of motion vectors of blocks adjacent to the video block to be decoded. After determining the motion prediction parameter, the motion compensation predictor 362 may decode the encoded motion vector by extracting the motion vector difference value from the encoded video bitstream BS and adding the motion vector difference value to the motion prediction parameter. The motion compensated predictor 362 may quantize the motion prediction parameter to the same resolution as the encoded motion vector. In certain exemplary embodiments, motion compensated predictor 362 may use the same accuracy for some or all of the encoded motion prediction parameters. As another example, motion compensation predictor 362 may be configured to use any of the above methods and determine which method to use by analyzing data included in a sequence parameter set, a slice parameter set, or a picture parameter set derived from the encoded video bitstream BS.

After the motion vector is decoded, the motion compensated predictor 362 may extract the prediction video block identified by the motion vector from the reference image of the reference image store 364 . If the motion vector points to a position of a fractional pixel, such as a half pixel, the motion compensated predictor 362 may interpolate values for the positions of the fractional pixels. Motion compensated predictor 362 may use adaptive interpolation filters or fixed interpolation filters to interpolate these values. The motion compensated predictor 362 may receive indications as to which of the filters 366 to use and, in various exemplary embodiments, the coefficients for the filters 366 from the received encoded video bitstream BS.

The spatial predictor 360 may use the intra prediction modes received in the encoded video bitstream of the BS to form a prediction video block from spatially contiguous blocks. An inverse quantizer 310 may inverse quantize (eg, dequantize the quantized block coefficients provided in the encoded BS video bitstream and decoded by entropy decoder 308). The inverse quantization method may include a standard method, such as according to the H.264 standard. The inverse quantization method may include using the quantization parameter QP calculated by video encoder 133, 200 for each video block to determine the quantization rate and/or inverse quantization rate to be used.

At inverse transform block 312, an inverse transform (eg, inverse of any of the transforms presented herein, inverse DCT, inverse integer transform, or a conceptually similar inverse transform) may be applied to the transform coefficients to obtain residual video blocks in the pixel domain. The motion compensated predictor 362 may create motion compensated blocks and may interpolate based on the interpolation filters. Interpolation filter identifiers for use in sub-pixel motion estimation may be included in the video block syntax elements. Motion compensated predictor 362 may use the interpolation filters used by video encoder 133, 200 during video block encoding to calculate interpolated values for the sub-integer pixels of the reference block. The motion-compensated predictor 362 may determine the interpolation filters used by the video encoder 133, 200 according to the received syntax information and use the interpolation filters to create prediction blocks.

The motion-compensated predictor 262 may use: (1) syntax information to determine the sizes of video blocks used to encode one or more pictures of the encoded video sequence; (2) section information that describes the division of each video block of a frame of the encoded video sequence; (3) modes (or mode information) indicating how each section is encoded; (4) one or more reference pictures for each intercoded video block; and/or (5) other information for decoding the encoded video sequence.

An adder 326 may add the residual blocks with the corresponding prediction blocks generated by the motion compensation predictor 362 or the spatial predictor 360 to generate decoded video blocks. Loop filters 366 (eg, deblocking filters or SAO filters) can be applied to filter decoded video blocks to remove blocking artifacts and/or to improve visual quality. The decoded video blocks may be stored in a reference image store 364, which may provide reference video blocks for subsequent motion compensation, and decoded video may be generated for presentation on a display device (not shown).

Point cloud compression

In FIG. 4 shows a bitstream structure for video-based point cloud compression (V-PCC). The generated video bitstreams and metadata are multiplexed together to generate the final V-PCC bitstream.

The V-PCC bitstream consists of a set of V-PCC blocks as shown in FIG. 4. The V-PCC block syntax as defined in the latest version of the V-PCC Standard Community Draft (V-PCC CD) is shown in Table 1, where each V-PCC block has a V-PCC block header and a V-PCC block payload. The V-PCC block header describes the V-PCC block type (Table 2). V-PCC blocks with block types 2, 3, and 4 are blocks of occupancy, geometry, and attribute data as defined by the community project. These data blocks are the three main components needed to restore a point cloud. In addition to the V-PCC box type, the V-PCC box header for an attribute also defines the attribute type and its index, which allows multiple instances of the same attribute type to be supported.

The payload of the Occupancy, Geometry, and Attribute V-PCC blocks (Table 3) corresponds to video data blocks (e.g., HEVC NAL blocks) that can be decoded by the video decoder specified in the corresponding V-PCC block of the Occupancy, Geometry Parameter Set and attributes.

Table 1. V-PCC Block Syntax

vpcc_unit( ) { Descriptor vpcc_unit_header( ) vpcc_unit_payload( ) }

Table 2. V-PCC Block Header Syntax

vpcc_unit_header( ) { Descriptor vpcc_unit_type u(5) if( vpcc_unit_type = = VPCC_AVD | | vpcc_unit_type = = VPCC_GVD | |
vpcc_unit_type == VPCC_OVD | | vpcc_unit_type == VPCC_PSD) vpcc_sequence_parameter_set_id u(4) if(vpcc_unit_type==VPCC_AVD) { vpcc_attribute_index u(7) if( sps_multiple_layer_streams_present_flag ) { vpcc_layer_index u(4) pcm_separate_video_data( 11 ) } else pcm_separate_video_data( 15 ) } else if( vpcc_unit_type = = VPCC_GVD ) { if( sps_multiple_layer_streams_present_flag ) { vpcc_layer_index u(4) pcm_separate_video_data( 18 ) } else pcm_separate_video_data( 22 ) } else if( vpcc_unit_type = = VPCC_OVD | vpcc_reserved_zero_23bits u(23) } else vpcc_reserved_zero_27bits u(27) }

Table 3. V-PCC Block Payload Syntax

vpcc_unit_payload( ) { Descriptor if(vpcc_unit_type==VPCC_SPS) sequence_parameter_set( ) else if( vpcc_unit_type = = VPCC_PSD ) patch_sequence_data_unit( ) else if( vpcc_unit_type == VPCC_OVD | |
vpcc_unit_type == VPCC_GVD | |
vpcc_unit_type == VPCC_AVD) video_data_unit( ) }

Dynamic Streaming over HTTP (DASH)

MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH) is a generic delivery format that provides end users with the best possible video experience by dynamically adapting to changing network conditions.

Adaptive streaming over HTTP, such as MPEG-DASH, requires that different bit rates of the media content be available on the server. In addition, the multimedia content may contain several multimedia components (eg, audio, video, text), each of which may have different characteristics. In MPEG-DASH, these characteristics are represented in the multimedia presentation description (MPD).

In FIG. 5 shows the hierarchical data model of the MPD. The MPD describes a sequence of periods in which the agreed set of coded versions of the media content components does not change during the period. Each period has a start time and duration and consists of one or more adaptation sets (AdaptationSet).

An AdaptationSet is a set of coded versions of one or more media content components with identical properties such as language, media type, aspect ratio, role, accessibility, and rating score. For example, an AdaptationSet may contain different bit rates of a video component with the same media content. Another AdaptationSet may contain different audio component bit rates (eg lower quality for stereo and higher quality for surround sound) with the same media content. Each AdaptationSet usually includes a set of representations (Representation).

A representation describes a delivered encoded version of one or more media components that differ from other representations in bit rate, resolution, number of channels, or other characteristics. Each view consists of one or more segments. Representation element attributes such as @id, @bandwidth, @qualityRanking, and @dependencyId are used to specify the properties of the associated Representation. Views may also include subviews that are part of the view to describe and extract partial information from the view. Subviews may provide access to a lower quality version of the view in which they are contained.

A Segment is the largest block of data that can be retrieved with a single HTTP request. Each segment has a URL, i.e. addressable location on the server, it can be downloaded using HTTP GET or HTTP GET with byte ranges.

To use this data model, the DASH client parses the MPD XML document, selects a selection of AdaptationSets appropriate for its environment, based on the information provided in each of the AdaptationSet elements. In each AdaptationSet, the client selects one Representation, typically based on the value of the @bandwidth attribute (bandwidth), and also considers the client's decoding and rendering capabilities. The client loads the initialization segment of the selected views and then accesses the content by requesting all segments or byte ranges of the segments. After the presentation is started, the client continues to consume the media content by continuously requesting media segments or portions of media segments and playing the content according to the media presentation timeline. The client can switch views based on updated information from its environment. The client must play the content continuously for all periods. As soon as the client receives the media contained in the Segment near the end of the declared media in the Representation, either the presentation of the media is terminated, a new Period is started, or an MPD refetch is required.

Descriptors in DASH

MPEG-DASH introduces the concept of descriptors to provide application-specific information about multimedia content. All descriptor elements are structured in the same way: they contain a @schemeIdUri attribute that provides a URI to identify the scheme, an optional @value attribute, and an optional @id attribute. The semantics of the element depends on the schema used. The URI identifying the scheme can be a Uniform Resource Name (URN) or a Uniform Resource Locator (URL). MPD does not provide any specific information on how to use these elements. It depends on the application that uses DASH formats to instantiate descriptor elements with appropriate schema information. DASH applications that use one of these elements must first define a schema identifier as a URI, and then determine the value space for the element when using that schema identifier. If structured data is required, then any extension element or attribute can be defined in a separate namespace. Descriptors can appear at several levels in the MPD.

- The presence of an element at the MPD level means that this element is a child of the MPD element.

- The presence of an element at the adaptation set level means that this element is a child of the AdaptationSet element.

- The presence of an element in the presentation layer means that this element is a child of the Representation element.

Presets

In MPEG-DASH, a package is a collection of media components that can be shared by a single decoder instance. Each package includes a main media component that contains decoder-specific information and loads the decoder. A PreSelection defines a subset of media components in a package that are expected to be shared.

The AdaptationSet that contains the main media component is called the main AdaptationSet. The main media component is always included in any PreSelection associated with a package. In addition, each package may include one or more partial AdaptationSets. Partial AdaptationSets can only be processed in combination with the main AdaptationSet.

A PreSelection may be defined using a PreSelection element as defined in Table 4. The PreSelection selection is based on the attributes and elements contained in the PreSelection element.

Table 4. Semantics of the PreSelection element

Element or attribute name Application Description Preselection @id OD
default = 1 Specifies the id of the Preselection element. It will be unique within one time period. @preselectionComponents M Specifies the identifiers of the contained customization sets or content components belonging to this preset as a separate, space-separated list in processing order, where the first identifier is the identifier for the primary media component. @lang O Declares a language code for a PreSelection according to the syntax and semantics of IETF RFC 5646. Availability 0…N Specifies information about the accessibility scheme. Role 0…N Specifies information about the commenting scheme of a role. Rating 0…N Specifies information about the rating scheme. Viewpoint 0…N Specifies information about the viewpoint commenting scheme. CommonAttributesElements - Specifies common attributes and elements (attributes and elements from the underlying RepresentationBaseType). Legend:
For attributes: M = required, O = optional, OD = optional with default value, CM = conditionally required.
For elements: <minOccurs>..<maxOccurs> (N=unbounded)
Elements are in bold; attributes are not in bold and are preceded by an @ sign.

Adaptive Point Cloud Streaming

While traditional media applications such as video are still popular, there has been significant interest in new media such as virtual reality (VR) and immersive 3D graphics. Recently, high-quality 3D point clouds have emerged - an advanced representation of immersive multimedia, which allows you to create new forms of interaction and data exchange with virtual worlds. The large amount of information required to represent such dynamic point clouds requires efficient coding algorithms. The MPEG 3DG Working Group is currently working on developing a standard for video-based point cloud compression with a community draft (CD) version released at MPEG meeting #124. The latest version of the CD defines a bitstream for compressed dynamic point clouds. In parallel, MPEG is also developing a system standard for transporting point cloud data.

The point cloud standards mentioned above only deal with the encoding and storage aspects of point clouds. However, practical point cloud applications are expected to require point cloud data to be streamed over a network. Such applications can stream point cloud content in real time or on demand, depending on how the content was generated. In addition, due to the large amount of information required to represent point clouds, such applications must support adaptive streaming techniques to avoid network congestion and provide an optimal viewing experience at any given time in terms of network bandwidth at that moment.

One good potential way to adaptively deliver point clouds is Dynamic Adaptive Streaming over HTTP (DASH). However, the current MPEG-DASH standard does not provide any signaling mechanism for a point cloud environment, including point cloud streams based on the V-PCC MPEG standard. Thus, it is important to define new signaling elements that will allow streaming clients to identify point cloud streams and their component substreams in a media presentation descriptor (MPD) file. In addition, it is also necessary to signal the various kinds of metadata associated with the point cloud components so that the streaming client can select the best version(s) of the point cloud or its components that it is capable of supporting.

Unlike traditional multimedia content, V-PCC multimedia content consists of a number of components, with some components having multiple layers. Each component (and/or layer) is separately encoded as a substream of the V-PCC bitstream. Some component substreams such as geometry and occupancy map (in addition to some attributes such as texture) are encoded using traditional video encoders (eg H.264/AVC or HEVC). However, these substreams need to be jointly decoded along with additional metadata in order to render the point cloud.

A number of XML elements and attributes are defined. These XML elements are defined in a separate namespace urn:mpeg:mpegI:vpcc:2019. This document uses the vpccc: namespace descriptor to refer to this namespace.

V-PCC component signaling in MPD DASH

Each V-PCC component and/or component level may be represented in a DASH manifest (MPD) file as a separate AdaptationSet (hereinafter "AdaptationSet component") with an additional AdaptationSet as the primary access point for the contents of the V-PCC (hereinafter "primary AdaptationSet") . In another embodiment, one adaptation set is sent on each component for each grant.

In one embodiment, adaptation sets for a V-PCC flow, including all V-PCC AdaptationSet components, must have the @codecs attribute value (e.g., as defined for V-PCC) set to vpc1, which would indicate that the MPD refers to the point clouds. In another embodiment, only the underlying AdaptationSet has the @codecs attribute set to vpc1, while the @codecs attribute for the point cloud AdaptationSet component (or the corresponding Representations if no @codecs is passed for the AdaptationSet element) is set based on the corresponding codec used to encode the component . For video encoded components, the value of @codecs should be set to resv.pccv.XXXX, where XXXX corresponds to the four-character code (4CC) of the video codec (eg avc1 or hvc1).

An EssentialProperty descriptor with a @schemeIdUri attribute of urn:mpeg:mpegI:vpcc:2019:component can be used to define the type of V-PCC component(s) (for example, occupancy map, geometry, or attribute) in an AdaptationSet component. This descriptor is called the VPCCComponent Descriptor.

At the adaptation set level, one VPCCComponent descriptor may be signaled for each point cloud component that is present in the Representation of the adaptation set.

In an embodiment, the @value attribute of the VPCCComponent descriptor shall not be present. The VPCCComponent Descriptor may include the elements and attributes specified in Table 5.

Table 5. Elements and attributes for the VPCCComponent descriptor

Elements and Attributes for the VPCCComponent Descriptor Note-
knowledge Data type Description component 0..N vpcc:vpccComponentType An element whose attributes specify information for one of the point cloud components present in the adaptation set representation(s). component@component_type M xs:string Specifies the type of point cloud component. The geom value denotes a geometry component, occp denotes an occupancy component, and attr denotes an attribute component. component@min_layer_index O xs:integer Specifies the index of the first level of the component represented by the AdaptationSet that contains the VPCCComponent descriptor.
If there is only one layer in the AdaptationSet views, min_layer_index and max_layer_index must be the same value. component@max_layer_index CM xs:integer Specifies the index of the last level of the component represented by the AdaptationSet that contains the VPCCComponent descriptor.
Should only be present if min_layer is present.
If there is only one layer in the country set views, min_layer_index and max_layer_index must be the same value. component@attribute_type CM xs:unsignedByte Indicates the attribute type (see Table 7.2 in the V-PCC CD). Only values between 0 and 15 are allowed.
Should only be present if the component is a point cloud attribute (i.e. component_type is attr). component@attribute_index CM xs:unsignedByte Specifies the index of the attribute. Must be between 0 and 127 inclusive.
Should only be present if the component is a point cloud attribute (i.e. component_type is attr). Legend:
For attributes: M = required, O = optional, OD = optional with default value, CM = conditionally required.
For elements: <minOccurs>..<maxOccurs> (N=unbounded)
Elements are in bold; attributes are not in bold and are preceded by an @ sign.

The data types for the various elements and attributes of the VPCCComponent descriptor may be as defined in the following XML schema.

In one embodiment, the main AdaptationSet must contain either one initialization segment at the adaptation set level or multiple initialization segments at the presentation layer (one for each Representation). In one embodiment, the initialization segments must contain sets of V-PCC sequence parameters, as defined in the community draft, that are used to initialize the V-PCC decoder. In the case of a single initialization segment, the V-PCC sequence parameter sets for all Representations may be contained in the initialization segment. When more than one Representation is sent in the main AdaptationSet, the initialization segment of each Representation may contain sets of V-PCC sequence parameters for that particular Representation. When using the ISO Basic Media File Format (ISOBMFF) as the media container for V-PCC content as defined in ISO/IEC 23090-10 WD, initialization segments may also include a metadata container as defined in ISO/IEC 14496-12 . This metabox contains one or more instances of the VPCCGroupBox, as defined in the VPCC CD, which provides metadata information describing the tracks at the file format level and the relationship between them.

In an embodiment, the media segments for the Representation of the main AdaptationSet comprise one or more track tiles of a V-PCC track defined in a community project. Media segments for Representation AdaptationSet components contain one or more track fragments of the corresponding component track at the file format level.

In another embodiment, an additional attribute is defined for the VPCCComponent descriptor, in this case called the @videoCodec attribute, the value of which indicates the codec used to encode the corresponding point cloud component. This allows you to support scenarios where there is more than one point cloud component in an AdaptationSet or Representation.

In another embodiment, the Role descriptor element may be used with newly defined values for V-PCC components to indicate the role of the associated AdaptationSet or Representation (eg, geometry, occupancy map, or attribute). For example, the geometry, occupancy map, and attribute components may have the following corresponding values, respectively: vpcc-geometry, vpcc-occupancy, and vpcc-attribute. An optional element of the EssentialProperty descriptor, similar to the element described in Table 5, except for the component_type attribute, may be reported at the customization set level to identify component levels and attribute type (if the component is a point cloud attribute).

Grouping V-PCC Adaptation Sets

The streaming client is able to identify the type of point cloud component in an AdaptationSet or Representation by examining the VPCCComponent descriptor within the corresponding element. However, the streaming client must also distinguish between the various point cloud streams present in the MPD file and identify their respective component streams.

An EssentialProperty element with an @schemeIdUri attribute of urn:mpeg:mpegI:vpcc:2019:vpc, which is referred to as a VPCC descriptor in this document, may be introduced. At the adaptation set level, a maximum of one VPCC descriptor may be present for the main AdaptationSet of the point cloud. If more than one Representation is present in the main AdaptationSet, then no more than one VPCC descriptor may be present at the presentation layer (ie, within each Representation element). Table 6 illustrates the attributes of a VPCC descriptor according to an embodiment.

Table 6. Attributes for the VPCC Descriptor

Attributes for the VPCC Descriptor Application Data type Description vpcc:@pcId CM xs:string Identifier (id) of the point cloud. This attribute shall be present if multiple versions of the same point cloud are reported in separate AdaptationSets. vpcc:@occupancyId M StringVectorType The identifier for the AdaptationSet or Representation component of the point cloud occupancy map. vpcc:@geometryId M StringVectorType A space-separated list of identifiers corresponding to the values of the @id attribute for the AdaptationSet and/or Representation of the point cloud geometry component. vpcc:@attributeId M StringVectorType A space-separated list of identifiers corresponding to the @id attribute values for the AdaptationSet and/or Representation attribute components of the point cloud.

If there is more than one version of the point cloud (for example, with different resolutions), each version may be present in a separate AdaptationSet containing one Representation, as well as a VPCC descriptor with the same value for the @pcId attribute. In another embodiment, different versions of the point cloud may be reported as a Representation of the same (main) AdaptationSet. In such a case, the VPCC Descriptor shall be present in each Representation, and the @pcId attribute may either be signaled with the same value for all Representations in the main AdaptationSet or omitted.

In another embodiment, the PreSelection is signaled in MPD with the value of the @preselectionComponents attribute, including the identifier of the main AdaptationSet for the point cloud, followed by the identifiers of the AdaptationSet components corresponding to the components of the point cloud. The @codecs attribute for PreSelection should be set to vpc1, indicating that the PreSelection media is a video-based point cloud. PreSelection can be advertised through the PreSelection element in the Period element, or a preset descriptor at the adaptation set level (or presentation level when multiple versions/views are available for the same point cloud). When a PreSelection element is used and there is more than one version of the same point cloud, each version is advertised in a separate PreSelection element, with the first identifier in the @preselectionComponents attribute's list of identifiers being the Representation identifier of the corresponding point cloud version in the main AdaptationSet. In FIG. 6 shows an example of a DASH configuration for grouping V-PCC components belonging to a single point cloud in an MPEG-DASH MPD file.

Using the preset descriptor, this grouping/this association can be signaled as follows.

In another embodiment, the main AdaptationSet of the point cloud or its Representation may enumerate the identifiers of the AdaptationSet and/or Representation component sets using the @associationId attribute defined in ISO/IEC 23009-1, with @associationType values set to 4CC at V-PCC( i.e. vpc1).

In another embodiment, the point cloud's main AdaptationSet or its Representation may list the identifiers of the AdaptationSet and/or Representation component sets using the @dependencyId attribute defined in ISO/IEC 23009-1. This is due to the existence of an internal dependency, since in order to restore the point cloud, it is necessary to decode the segments in the main AdaptationSet along with the segments from the component sets of the AdaptationSet of the point cloud.

Component metadata signaling

Metadata about geometry and attributes is commonly used for visualization. They are advertised in the V-PCC bitstream parameter sets. However, it may be necessary to advertise these metadata elements in the MPD so that the streaming client can receive the information as early as possible. In addition, the streaming client may make decisions to choose between multiple point cloud versions with different geometry and attribute metadata values (eg, based on whether the client supports the reported values or not).

Geometry metadata signaling

A SupplementalProperty element with a @schemeIdUri attribute of urn:mpeg:mpegI:vpcc:2019:geom_meta may be introduced, which is referred to in this document as a geometry metadata descriptor or geoMeta descriptor. At most one geomMeta descriptor may be present at the MPD level, in which case it is applicable to the geometry components for all point clouds advertised in the MPD, unless it is overridden by a geomMeta descriptor at a lower level, as described below. At most one geomMeta descriptor may be present at the adaptation set level in the main AdaptationSet. At most one geomMeta descriptor may be present at the presentation level in the main AdaptationSet. If a geomMeta descriptor is present at a particular level, it overrides any geomMeta descriptor passed at a higher level.

In an embodiment, the @value attribute of the geomMeta descriptor shall not be present. In an embodiment, the geomMeta descriptor includes the elements and attributes specified in Table 7.

Table 7. Elements and attributes for the geomMeta descriptor

Elements and Attributes for the geomMeta Descriptor Application Data type Description geom 0..1 vpcc:geometryMetadataType A container element whose attributes and elements define metadata information about the geometry. geom@point_shape O xs:unsignedByte Specifies the point geometry to use for rendering. Supported values range from 0 to 15 inclusive. The corresponding form is derived from table 7-2 in the community draft. If absent, the default value should be 0. geom@point_size O xs:unsignedShort Specifies the point geometric dimension to use for rendering. Valid values must be between 1 and 65535 inclusive. If absent, the default value should be inferred to be 1. geom.geomSmoothing 0..1 vpcc:geometrySmoothingType An element whose attributes provide geometry smoothing information. geom.geomSmoothing@grid_size M xs:unsignedByte Specifies the mesh size for geometry smoothing. Valid values must be between 2 and 128 inclusive.
If there is no geom.geomSmoothing element, the grid size should default to 8. geom. geomSmoothing@threshold M xs:unsignedByte Smoothing threshold.
If there is no geom.geomSmoothing element, the default threshold should be inferred to be 64. geom.geomScale 0..1 vpcc:geometryScaleType An element whose attributes provide geometry scaling information. geom. geomScale@x M xs:unsignedInt X scale value. If there is no geom.geomSmoothing element, the default value should be 1. geom. geomScale@y M xs:unsignedInt Y-axis scale value. If there is no geom.geomSmoothing element, the default value should be 1. geom. geomScale@z M xs:unsignedInt Scale value along the Z axis. If the geom.geomSmoothing element is not present, the default value should be 1. geom.geomOffset 0..1 vpcc:GeometryOffsetType An element whose attributes provide information about the offset of the geometry. geom.geomOffset@x M xs:int X offset value. If there is no geom.geomSmoothing element, the default value should be 0. geom.geomOffset@y M xs:int Y-axis offset value. If there is no geom.geomSmoothing element, the default value should be 0. geom.geomOffset@z M xs:int Z offset value. If there is no geom.geomSmoothing element, the default value should be 0. geom.geomRotation 0..1 vpcc:geometryRotationType An element whose attributes provide information about the rotation of the geometry. geom. geomRotation@x M xs:int X-axis rotation value in units of 2 ^-16 degrees. If there is no geom.geomSmoothing element, the default value should be 0. geom. geomRotation@y M xs:int The Y-axis rotation value is 2 ^-16 degrees. If there is no geom.geomSmoothing element, the default value should be 0. geom. geomRotation@z M xs:int The Z rotation value is 2 ^-16 degrees. If there is no geom.geomSmoothing element, the default value should be 0. Legend:
For attributes: M = required, O = optional, OD = optional with default value, CM = conditionally required.
For elements: <minOccurs>..<maxOccurs> (N=unbounded)
Elements are in bold; attributes are not in bold and are preceded by an @ sign.

In an embodiment, the data types for the various elements and attributes of the geomMeta descriptor may be as defined in the following XML schema.

Attribute metadata signaling

A SupplementalProperty element with a @schemeIdUri attribute of urn:mpeg:mpegI:vpcc:2019:attr_meta may be introduced, which is referred to in this document as an attribute metadata descriptor or attrMeta descriptor. At most one attrMeta descriptor may be present at the adaptation set level in the main AdaptationSet. At most one attrMeta descriptor may be present at the presentation layer in the main AdaptationSet. If an attrMeta descriptor is present at the presentation layer, it overrides any attrMeta descriptor reported at the adaptation set layer for the AdaptationSet to which the Representation belongs.

In an embodiment, the @value attribute of the attrMeta descriptor is omitted. In an embodiment, the attrMeta descriptor may include the elements and attributes specified in Table 8.

Table 8. Elements and attributes for the attrMeta descriptor

Elements and Attributes for the attrMeta Descriptor Application Data type Description attm 0..N vpcc:attributeMetadataType A container element whose attributes and elements specify metadata information for a point cloud attribute. attm@index M xs:unsignedByte Specifies the index of the attribute. Must be between 0 and 127 inclusive. attm@num_dimensions M xs:unsignedByte The number of measurements for the point cloud attribute. attm.attrSmoothing 0..1 vpcc:attributeSmoothingType An element whose attributes provide anti-aliasing information for the point cloud attribute. atm. attrSmoothing@radius M xs:unsignedByte The radius to detect neighbors for the smoothing attribute. If there is no attm.attmSmoothing element, then the default value should be inferred to be 0. atm. attrSmoothing@neighbor_count M xs:unsignedByte The maximum number of neighboring points used for the smoothing attribute. If there is no attm.attmSmoothing element, then the default value should be inferred to be 0. atm. attrSmoothing@radius2_boundary M xs:unsignedByte Radius to detect edge points. If there is no attm.attmSmoothing element, then the default value should be inferred to be 0. atm. attrSmoothing@threshold M xs:unsignedByte Smoothing attribute threshold. If there is no attm.attmSmoothing element, then the default value should be inferred to be 0. atm. attrSmoothing@threshold_local_entropy M xs:unsignedByte Local entropy threshold in the neighborhood of a boundary point. The value for this attribute must be between 0 and 7, inclusive. If there is no attm.attmSmoothing element, then the default value should be inferred to be 0. attm.attrScale 0..1 vpcc:attributeScaleType An element whose attribute provides scaling information along each dimension of the point cloud attribute. attm.attrScale@values M xs:string A string of comma-separated scale values for each point cloud attribute dimension. attm.attrOffset 0..1 vpcc:attributeOffsetType An element whose attribute provides information about the offset along each dimension of the point cloud attribute. attm.attrOffset@values M xs:string A string of point-separated offset values for each point cloud attribute dimension. Legend:
For attributes: M = required, O = optional, OD = optional with default value, CM = conditionally required.
For elements: <minOccurs>..<maxOccurs> (N=unbounded)
Elements are in bold; attributes are not in bold and are preceded by an @ sign.

In an embodiment, the data types for the various elements and attributes of the attrMeta descriptor may be as defined in the following XML schema.

Streaming Client Behavior

The DASH client (decoder node) is guided by the information provided in the MPD. The following is an example of client behavior when processing streaming point cloud content in accordance with the signaling provided herein, given an embodiment in which the association of AdaptationSets with the main AdaptationSet of the point cloud is advertised using a VPCC descriptor. In FIG. 7 is a flowchart illustrating an example of a streaming client process according to an embodiment.

At block 711, the client first issues an HTTP request and downloads the MPD file from the content server. The client then parses the MPD file to create the appropriate in-memory representation of the XML elements in the MPD file.

Then, at block 713, to determine the available point cloud media content in the Period, the streaming client scans the AdaptationSet elements for AdaptationSets with the @codecs attribute set to vpc1 and the VPCC descriptor element. The resulting subgroup is a set of basic AdaptationSets for point cloud content.

Next, at step 715, the streaming client identifies the number of unique point clouds by checking the VPCC descriptors of these AdaptationSet and AdaptationSet groups with the same @pcId value in their VPCC descriptor as in versions of the same content.

At block 717, an AdaptationSet is identified with an @pcId value corresponding to the content of the point cloud that the user wishes to stream. If the group contains more than one AdaptationSet, the streaming client selects an AdaptationSet with a supported version (eg, video resolution). Otherwise, a single AdaptationSet of the group is chosen.

Then, at step 719, the streaming client checks the VPCC descriptor of the selected AdaptationSet to identify the AdaptationSets for the point cloud components. They are identified by the values of the @occupancyId, @geometryId and @attributeId attributes. If geomMeta and/or attrMeta descriptors are present in the selected main AdaptationSet, the streaming client may determine whether it supports the reported rendering configuration for the point cloud stream before downloading any segments. Otherwise, the client needs to extract this information from the initialization segment.

Then, in step 721, the client starts streaming the point cloud by downloading an initialization segment for the main AdaptationSet containing the parameter sets needed to initialize the V-PCC decoder.

At block 723, the initialization segments for the video encoded component streams are downloaded and cached in a storage device.

At block 725, the streaming client then starts downloading the time-synchronized media segments from the main AdaptatationSet and components of the AdatptationSet in parallel over HTTP, and the downloaded segments are stored in a segment buffer in storage.

At 727, the time-synchronized media segments are retrieved from their respective buffers and concatenated with their respective initialization segments.

Finally, at step 729, the media container (eg, ISOBMFF) is parsed to extract elementary stream information and structure the V-PCC bitstream according to the V-PCC standard, after which the bitstream is passed to the V-PCC decoder.

While the features and elements are described above in specific combinations, it will be apparent to one skilled in the art that each feature or element can be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware embedded in a computer-readable medium and designed to be executed by a computer or processor. Examples of volatile computer-readable storage media include, without limitation, read-only memory (ROM), random access memory (RAM), register, cache memory, semiconductor storage devices, magnetic media such as internal hard drives and removable drives, magneto-optical media and optical media such as CD-ROMs and digital versatile discs (DVDs). The processor, in conjunction with software, may be used to implement an RF transceiver for use in WTRU module 102, user equipment (UE), terminal, base station, RNC, and/or any host computer.

In addition, in the embodiments described above, processing platforms, computing systems, controllers, and other devices containing processors are noted. These devices may include at least one central processing unit (CPU) and a storage device. As the practice of specialists in the field of computer programming shows, indications of actions and symbolic representations of steps or commands can be implemented using various CPUs and storage devices. Such actions and steps or commands may be referred to as "executable", "computer-executable", or "CPU-executable".

It will be apparent to a person skilled in the art that said actions and symbolically represented steps or commands involve the control of electrical signals by the CPU. The electrical system provides data bits that can cause the resulting conversion or attenuation of the electrical signals and the storage of the data bits in storage cells in the storage system to thereby reconfigure or otherwise alter CPU operation as well as other signal processing. Memory cells that store data bits are physical locations that have certain electrical, magnetic, optical, or organic properties corresponding to or characteristic of data bits. It should be understood that the exemplary embodiments are not limited to the aforementioned platforms or CPUs, and that other platforms and CPUs may also support the proposed methods.

Data bits may also be stored on a computer-readable medium, including magnetic disks, optical disks, and any other volatile (eg, random access memory (RAM)) or non-volatile (eg, read only memory (ROM)) mass storage device for reading by the CPU. . Computer-readable media may include co-operating or interconnected computer-readable media used exclusively within a processing system or distributed among a plurality of interconnected processing systems, which may be local or remote to said processing system. This implies that exemplary embodiments are not limited to the aforementioned storage devices and that other platforms and storage devices may also support the described methods.

In an exemplary embodiment, any of the steps, methods, and the like described herein may be implemented as computer-readable instructions stored on a computer-readable medium. The mobile device processor, network element, and/or any other computing device may be configured to execute machine-readable instructions.

Minor differences remain between hardware and software implementations of aspects of systems. The use of hardware or software typically (but not always, as the distinction between hardware and software can become significant in certain contexts) involves choosing a design that represents a trade-off between cost and efficiency. There may be various means by which the methods and/or systems and/or other technologies described herein can be implemented (e.g., hardware, software, and/or firmware), and the preferred means may vary depending on the context in which said methods and/or systems and/or other technologies are deployed. For example, if the developer determines that speed and accuracy are of paramount importance, he may use mostly hardware and/or firmware. If flexibility is the most important aspect, the developer may choose to implement it primarily in software. In an alternative embodiment, the developer may use a combination of hardware, software, and/or firmware.

In the above detailed description, various embodiments of devices and/or methods are set forth using block diagrams, block diagrams, and/or examples. Since such block diagrams, block diagrams and/or examples contain one or more functions and/or operations, it will be obvious to those skilled in the art that each function and/or operation in such block diagrams, block diagrams or examples can be implemented , separately and/or together, using a wide range of hardware, software, firmware, or essentially any combination thereof. By way of example, suitable processors include a general purpose processor, a special purpose processor, a standard processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, application specific integrated circuits (ASICs), standard parts application specific integrated circuit (ASSP); a field programmable gate array (FPGA), any other type of integrated circuit (IC), and/or a state machine.

While the features and elements are presented above in specific combinations, it will be apparent to one skilled in the art that each feature or element may be used alone or in any combination with other features and elements. The present invention is not limited to the specific embodiments described in this application, which are intended to illustrate various aspects. It will be apparent to those skilled in the art that many modifications and changes are possible without departing from the spirit and scope of the present invention. None of the elements, actions or commands used in the description of this application should not be considered critical or essential to the invention, unless expressly stated otherwise. Functionally equivalent methods and devices within the scope of the description, in addition to those listed herein, will become apparent to those skilled in the art upon review of the above descriptions. Such modifications and variations are intended to be included within the scope of the appended claims. The present description is limited solely by the appended claims, as well as the full range of equivalents to which such claims apply. It should be understood that the present description is not limited to specific methods or systems.

In addition, it should be understood that the terms used herein are used only for the purpose of describing specific embodiments of the present invention and they are not restrictive. As used herein, the terms "station" and its abbreviation STA, "user equipment" and its abbreviation UE may mean (i) a wireless transmit/receive unit (WTRU), for example, as described below; (ii) any of a number of WTRU embodiments, eg, as described below; (iii) a wireless and/or wired (eg, pluggable) device specifically configured using some or all of the WTRU designs and functionality, eg, as described below; (iii) a wireless and/or wired device not configured with all of the WTRU designs and functionality, eg as described below; or (iv) etc.

In certain exemplary embodiments, certain portions of the subject matter described herein may be implemented with application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), and/or other integrated circuit formats. However, those skilled in the art will appreciate that certain aspects of the embodiments described herein, in whole or in part, may equivalently be implemented in integrated circuits as one or more computer programs executing on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as software. essentially any combination thereof, and that designing circuits and/or writing code for software and/or firmware would be well within the ability of a person skilled in the art upon reading this disclosure. In addition, it will be apparent to those skilled in the art that the mechanisms of the subject matter described herein may be distributed as a software product in a variety of forms and that the exemplary embodiment of the subject matter described herein is applicable regardless of the particular type of environment. the signaling used to actually make that allocation. Examples of the signal medium include, without limitation, the following: a recordable medium such as floppy disk, hard disk drive, CD, DVD, digital tape, computer storage device, etc., and a medium capable of transmission, such as a digital and/or analog data transmission medium (eg, fiber optic cable, waveguide, wired link, wireless link, etc.).

The subject matter described herein sometimes illustrates various components contained within or connected to various other components. It should be understood that such illustrated architectures are merely examples and that various other architectures with the same functionality could in fact be implemented. In a conceptual sense, any design of components to achieve the same functionality is practically "related" to the ability to provide the desired functionality. Therefore, any two components combined herein to achieve a particular functionality can be considered to be "related" to each other to provide the desired functionality, regardless of architectures or intermediate components. Similarly, any two components that are connected in this way can also be considered to be "operably connected" or "operably linked" to each other to provide the desired functionality, and any two components that can be connected in this way can also be considered to be "capable of being functionally connections" with each other to provide the desired functionality. Specific examples of operably connected components include, without limitation, components capable of physical interfacing and/or physical and/or logical and/or wireless interaction and/or components that interact logically and/or wirelessly.

With respect to the application of essentially any plural and/or singular variations for terms herein, those skilled in the art may change plural to singular and/or singular to plural as required by the context and/or scope. In this document, various combinations of singular/plural may be explicitly indicated for clarity.

It will be apparent to those skilled in the art that, in general, terms used herein and in particular in the appended claims (e.g., in the main body of the appended claims) are generally considered to be "non-limiting" terms (e.g., the term " including" should be interpreted as "including, without limitation", the term "having" should be interpreted as "having at least", the term "includes" should be interpreted as "includes, without limitation", etc.). In addition, those skilled in the art will appreciate that if a specific number of claims are intended to be included, that intent will be explicitly stated in the claims, and in the absence of such mention, there is no such intention. For example, if only one element is involved, the term "one" or a similar term may be used. For ease of understanding, the following appended claims and/or descriptions herein may contain the introductory phrases "at least one" and "one or more" to introduce a listing of claims. However, the use of such phrases should not be construed as implying that the introduction of a claim enumeration with singular grammatical forms limits any particular claim containing such an introduced enumeration of claims to embodiments containing only one such enumeration, even if the same claim includes the introductory phrases "one or more" or "at least one" and singular grammatical forms (eg, singular grammatical forms should be interpreted to mean "at least" or "one or more"). The same is true for the use of certain grammatical forms used to introduce a listing of claims. In addition, even if a specific number of included listed claims is explicitly indicated, it will be obvious to those skilled in the art that such a listing should be interpreted to mean at least the specified number (for example, simply indicating "two claims" without other definitions means by at least two points and two or more points). In addition, in the case of using a rule similar to the rule "at least one of A, B and C, etc.", in general, a person skilled in the art understands the rule implied by such a construction (for example, "a system containing at least one of A, B and C" will include, without limitation, systems that contain only A, only B, only C, both A and B, both A and C, both B and C, and/or both A, B and C, etc.). In the case of using a rule similar to the rule "at least one of A, B or C, etc.", in general, a person skilled in the art understands the rule implied by such a design (for example, "a system containing at least one of A, B or C" will include, without limitation, systems that contain only A, only B, only C, both A and B, both A and C, both B and C, and/or both A, B and C, etc.). In addition, it will be obvious to those skilled in the art that essentially any separating word and/or separating phrase representing two or more alternative terms, whether in the description, claims, or drawings, is intended to include one of the terms, any of terms or both terms. For example, the phrase "A or B" would imply the inclusion of the possibilities "A" or "B" or "A and B". In addition, as used herein, the term "any of" followed by a list of a plurality of elements and/or a plurality of categories of elements shall include "any of", "any combination of", "any plurality of", and/or "any combination of from a plurality of elements and/or categories of elements, either alone or in combination with other elements and/or other categories of elements. In addition, as used herein, the term "set" or "group" includes any number of elements, including zero. In addition, as used herein, the term "amount" includes any number, including zero.

In addition, if features or aspects of the description are presented in terms of Markush groups, it will be apparent to those skilled in the art that the description is thus also presented in terms of any individual member or subgroup of members of the Markush group.

As will be apparent to one of skill in the art, for all purposes, such as providing a written description, all ranges described herein also cover all of their possible subranges and combinations of subranges. Any of the listed ranges can be easily recognized as providing a sufficient description and as a range that can be broken down into at least equal halves, thirds, quarters, fifths, tenths, etc. In a non-limiting example, each range described herein can be easily broken down into lower third, middle third, and upper third, and so on. As will be apparent to one of ordinary skill in the art, all terms such as "up to", "at least", "more than", "less than", etc., include the cited number and refer to ranges which can be further divided into subranges as described above. Finally, as will be apparent to those skilled in the art, the range includes every single element. Thus, for example, a group containing 1-3 cells refers to groups containing 1, 2, or 3 cells. Similarly, a group containing 1-5 cells refers to groups containing 1, 2, 3, 4 or 5 cells, etc.

In addition, the claims should not be construed as being limited to the proposed order or elements, unless otherwise indicated. In addition, use of the term "intended for" in any claim implies reference to 35 U.S.C. "intended for" does not indicate this.

Although the invention has been illustrated and described herein in connection with specific embodiments, the invention is not limited to the detailed description provided. On the contrary, various changes may be made to the detailed description within the scope and range of equivalents of the claims without departing from the present invention.

It will be apparent to those skilled in the art that, in the present disclosure, certain exemplary embodiments may be used in an alternative embodiment or in combination with other exemplary embodiments.

While the features and elements are described above in specific combinations, it will be apparent to one skilled in the art that each feature or element can be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware embedded in a computer-readable medium and designed to be executed by a computer or processor. Examples of volatile computer-readable storage media include, without limitation, read-only memory (ROM), random access memory (RAM), register, cache memory, semiconductor storage devices, magnetic media such as internal hard drives and removable drives, magneto-optical media and optical media such as CD-ROMs and digital versatile discs (DVDs). The processor, in conjunction with software, may be used to implement an RF transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

In addition, in the embodiments described above, processing platforms, computing systems, controllers, and other devices containing processors are noted. These devices may include at least one central processing unit (CPU) and a storage device. As the practice of specialists in the field of computer programming shows, indications of actions and symbolic representations of steps or commands can be implemented using various CPUs and storage devices. Such actions and steps or commands may be referred to as "executable", "computer-executable", or "CPU-executable".

It will be apparent to a person skilled in the art that said actions and symbolically represented steps or commands involve the control of electrical signals by the CPU. The electrical system provides data bits that can cause the resulting conversion or attenuation of the electrical signals and the storage of the data bits in storage cells in the storage system to thereby reconfigure or otherwise alter CPU operation as well as other signal processing. Memory cells that store data bits are physical locations that have certain electrical, magnetic, optical, or organic properties corresponding to or characteristic of data bits.

Data bits may also be stored on a computer-readable medium, including magnetic disks, optical disks, and any other volatile (eg, random access memory (RAM)) or non-volatile (eg, read only memory (ROM)) mass storage device readable by the CPU. . Computer-readable media may include co-operating or interconnected computer-readable media used exclusively within a processing system or distributed among a plurality of interconnected processing systems, which may be local or remote to said processing system. This implies that exemplary embodiments are not limited to the aforementioned storage devices and that other platforms and storage devices may also support the described methods.

None of the elements, acts or instructions used in the description of this application should not be considered critical or essential to the present invention, unless expressly described otherwise. In addition, as used herein, the singular grammatical form is intended to include one or more elements. If only one element is intended, use the term "one" or a similar term. In addition, as used herein, the term "any of" followed by a list of a plurality of elements and/or a plurality of categories of elements shall include "any of", "any combination of", "any plurality of", and/or "any combination of "multiple" elements and/or categories of elements, either alone or in combination with other elements and/or other categories of elements. In addition, as used herein, the term "set" includes any number of elements, including zero. In addition, as used herein, the term "amount" includes any number, including zero.

In addition, the claims should not be construed as being limited to the order or elements described, unless otherwise indicated. In addition, the use of the term “remedy” in any claim implies reference to 35 U.S.C.

By way of example, suitable processors include a general purpose processor, a special purpose processor, a standard processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, application specific integrated circuits (ASICs), standard parts application specific integrated circuit (ASSP); a field programmable gate array (FPGA), any other type of integrated circuit (IC), and/or a state machine.

The processor, in combination with software, can be used to implement an RF transceiver for use in a wireless transmit/receive unit (WRTU), user equipment (UE), terminal, base station, mobility management entity (MME), or evolved packet core (EPC), or any main computer. The WRTU may be used in conjunction with modules implemented in hardware and/or software, including Software Defined Radio (SDR) and other components such as a camera, video camera module, video phone, loudspeaker phone, vibration device, speaker, microphone, TV transceiver, headphones with microphone, keyboard, Bluetooth® module, frequency modulation (FM) radio module, near field communication (NFC) module, liquid crystal display (LCD) unit, organic light emitting diode (OLED) display unit, digital music player, media player , a video game playback device module, an Internet browser, and/or any wireless local area network (WLAN) or ultra-wideband (UWB) module.

While the invention has been described in the context of communications systems, it is contemplated that these systems may be implemented as software in general purpose microprocessors/computers (not shown). In certain embodiments, one or more functions of the various components may be implemented in software that controls a general purpose computer.

In addition, although the invention has been illustrated and described herein in connection with specific embodiments, the invention is not limited to the detailed description provided. On the contrary, various changes may be made to the detailed description within the scope and range of equivalents of the claims without departing from the present invention.

Claims (39)

1. A network node comprising a circuit including any of a transmitter, a receiver, a processor, and a storage device for streaming data (PCD) of a point cloud (PC) corresponding to a PC over a network using a Hypertext Transfer Protocol (HTTP), including a plurality of video-based point cloud compression (V-PCC) components constituting a PC, wherein the network node is configured to: transmitting PC cloud PCD related information in a Media Presentation Description (MPD) for Dynamic Adaptive Streaming over HTTP (DASH), wherein the DASH MPD includes information indicating at least: (1) a basic adaptation set (AS) for a PC, where the basic AS includes information indicating at least: (i) a codec attribute value indicating that the basic AS corresponds to the V-PCC data, and (ii) a segment initialization containing at least one set of V-PCC sequence parameters for representing the PC, and (2) a plurality of AS set components, wherein each of the plurality of AS components corresponds to one of the V-PCC components and includes information indicating at least: (a) a V-PCC Component descriptor defining the type of the corresponding V-PCC component, which contains a geometry, occupancy, or attribute, and (b) at least one property of a corresponding V-PCC component; And transferring MPD DASH over the network. 2. The network node according to claim 1, configured to transmit the PCD over the network, wherein, as the media container for the V-PCC content, an International Standards Organization (ISO) Basic Media File Format (ISOBMFF) is used, and wherein the base AS initialization segment includes a metadata container containing one or more V-PCCGroupBox instances providing metadata information associated with V-PCC tracks. 3. The network node of claim 1, wherein the primary AS initialization segment includes information indicating: (1) a single initialization segment in the adaptation layer, and (2) sets of V-PCC sequence parameters for all representations of the primary AS. 4. The network node of claim. 1, wherein the main AS includes information indicating an initialization segment for each of the plurality of PC representations, wherein each initialization segment that corresponds to a PC representation includes a set of V-PCC sequence parameters for that representation. 5. The network node of claim 1, wherein the V-PCC component descriptor includes information indicating a video codec attribute for the codec used to encode the corresponding V-PCC component. 6. The network node of claim 1, wherein the AS includes information indicating a DASH element of the role descriptor with a value indicating one of the geometry, occupancy map, or attribute of the corresponding component. 7. The network node of claim 1, wherein the V-PCC component descriptor information identifies any of the levels of the corresponding V-PCC component or the attribute type of the corresponding V-PPC component. 8. The network node of claim 1, wherein the primary AS specifies a V-PCC descriptor for the specific point cloud to which the primary AS corresponds, and identifiers for AS set components. 9. The network node of claim 8, if the PC V-PCC data includes more than one PC version, further configured to signal each PC version to separate primary ASs, respectively containing a single representation corresponding to the PC version and a corresponding V descriptor. -PCC, with all distinct base ASs corresponding to different PC versions having the same value for the PC ID attribute. 10. The network node according to claim 8, further configured to signal different versions of the PC as different representations of the main AS, wherein for each representation of the base AS, there is a corresponding V-PCC descriptor, and where the PC identifier attribute meets either of the conditions: (1) present in every representation and passed with the same value for all representations in the main AS, or (2) omitted. 11. The network node of claim 8, wherein the MPD DASH includes preset information indicating an attribute value including a base AS identifier (ID) for the PC and identifiers associated with a plurality of AS set components corresponding to the PC components, where the attribute is the preselectionComponents attribute, more than one version of the same PC is available, wherein each of more than one version is signaled in the corresponding PreSelection element, and wherein the first identifier in the list of identifiers of the preselectionComponents attribute is the presentation identifier of the corresponding one of more than one version of the same PC in the main AS. 12. The network node of claim 8, wherein the main PC AS includes information indicating a list of AS component identifiers in an association identifier attribute, with the value of the association type attribute set to 4CC for the V-PCC. 13. The network node of claim 8, wherein the main PC AS includes information indicating a list of identifiers of any of the AS set components using a dependency identifier attribute. 14. The network node of claim 1, wherein the MPD DASH includes information indicative of at least one of geometry metadata and attribute metadata for the PC. 15. The network node of claim 14, wherein information indicative of geometry metadata is included in a URI attribute of the schema identifier. 16. The network node of claim 14, wherein information indicative of attribute metadata is included in a URI attribute of the schema identifier. 17. A method for streaming data (PCD) of a point cloud (PC) of a network node corresponding to a PC over a network using a hypertext transfer protocol (HTTP), including a plurality of video-based PC compression components (V-PCC) constituting a PC, wherein way includes: transmitting PCD related information in a Media Presentation Description (MPD) for Dynamic Adaptive Streaming over HTTP (DASH), wherein the DASH MPD includes information indicating at least: (1) a basic adaptation set (AS) for the point cloud, the basic AS including information indicating at least: (i) a unique codec attribute value indicating that the basic AS corresponds to the V-PCC data, (ii) an initialization segment containing at least one set of V-PCC sequence parameters for representing the PC, and (2) a plurality of AS set components, wherein each of the plurality of AS components corresponds to one of the plurality of V-PCC components and includes information indicating at least: (a) a V-PCC Component descriptor specifying the type of the corresponding V-PCC component , and (b) at least one property of a corresponding V-PCC component; And transmission of MPD DASH over the network. 18. The method of claim 17, further comprising transmitting the PCD over the network, wherein the media container for the V-PCC content is using an International Organization for Standardization (ISO) Basic Media File Format (ISOBMFF), wherein, in the initialization segment of the main AS, the metadata container contains one or more VPCCGroupBox instances providing metadata information associated with V-PCC tracks. 19. The method of claim 17, wherein the base AS initialization segment includes information indicating a single initialization segment in the adaptation layer, the single initialization segment including sets of VPCC sequence parameters for all core AS representations. 20. The method of claim 17, wherein the base AS includes information indicative of an initialization segment for each of the plurality of PC representations, wherein each initialization segment that corresponds to a PC representation includes a set of V-PCC sequence parameters for that representation.

Images