KR20230127982A - Cross Random Access Point Sample Group - Google Patents

Abstract

A mechanism for processing video data is disclosed. A description of the Crossed Random Access Point Reference (CRR) samples is signaled within a visual media data file conforming to the International Organization for Standardization (ISO) Base Media File Format (ISOBMFF). Based on the CRR sample group, conversion between visual media data and the visual media data file is performed.

Description

Cross Random Access Point Sample Group

CROSS REFERENCES TO RELATED APPLICATIONS

This patent application is filed on December 28, 2020 by Ye-Kui Wang et al., entitled "Signaling of Cross Random Access Point References in Video Bitstreams and Media Files", International Application No. It is made to claim the priority of PCT/CN2020/139893. The entire disclosure of the above international application is hereby incorporated by reference as part of the disclosure of this application.

This patent document relates to the creation, storage and consumption of digital audio video media information in a file format.

Digital video uses the most bandwidth on the Internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the demand for bandwidth for digital video usage is expected to continue to increase.

This patent document relates to the creation, storage and consumption of digital audio video media information in a file format.

A first aspect relates to a method for processing video data, the method comprising a crossed random access point reference (CRR: Determining a description of Cross Random Access Point Referencing samples, and performing conversion between the visual media data file and the visual media data based on the CRR sample group.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the description of the CRR samples is included within a CRR sample group.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the description of the CRR samples is included within a dependent random access point (DRAP) sample group.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the description of the CRR samples is included within a type 2 DRAP sample group.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the description of the CRR samples is included within an enhanced dependent random access point (EDRAP) sample group.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the description of the CRR samples is contained within a SampleToGroupBox.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the description of the CRR samples is contained within a CompactSampleToGroupBox.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the description of the CRR samples is included within a group_type_parameter.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the CRR samples are represented as type 2 DRAP samples.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the CRR samples are represented as Enhanced Dependent Random Access Point (EDRAP) samples.

Alternatively, in any of the preceding aspects, another implementation of the above aspect provides that each sample comprises a picture.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the description of CRR samples includes one or more sample identifiers for identifying samples belonging to a certain sample group.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the description of the CRR samples includes identifiers of reference pictures for the CRR samples.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the description of CRR samples includes a number of samples needed as a reference to decode the current sample.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the description of the CRR samples is included within a sample entry within a sample group.

Alternatively, in any of the preceding aspects, another implementation of the aspect refers only to a nearest preceding initial sample, one or more CRR samples preceding the current sample in decoding order, or a combination thereof. , the current sample is one of the CRR samples.

Or, in any of the preceding aspects, another implementation of the above aspect is that when decoding starts at a current sample, if the current sample and all samples after the current sample in decoding order and output order can be decoded correctly, It is provided that the current sample is one of the CRR samples.

Alternatively, in any of the preceding aspects, another implementation of the aspect may, after decoding the nearest preceding initial sample, one or more CRR samples preceding the current sample in decoding order, or a combination thereof, the current sample and that all samples after the current sample are correctly decoded.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the converting comprises generating the visual media data file according to the visual media data.

Alternatively, in any of the preceding aspects, another implementation of the aspect provides that the transforming comprises parsing the visual media data file to obtain the visual media data.

A second aspect relates to an apparatus for processing video data, the apparatus comprising a processor and a non-transitory memory comprising instructions, wherein the instructions, when executed by the processor, cause the processor to perform any of the preceding aspects to be carried out in any way.

A third aspect relates to a non-transitory computer readable medium comprising a computer program product to be used by a video coding device, the computer program product comprising computer executable instructions stored on the computer readable medium to be executed by a processor. cause the video coding device to implement any of the preceding aspects.

For clarity, any one of the preceding embodiments may be combined with any one or more of the other preceding embodiments to create a new embodiment within the scope of the present disclosure.

These and other features will become more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.

For a more complete understanding of the present disclosure, reference is now made to the accompanying drawings in which like reference numbers indicate like parts and the following brief description taken in connection with the detailed description.
1 is a schematic diagram of an example mechanism for random access when decoding a bitstream using IRAP pictures.
2 is a schematic diagram of an example mechanism for random access when decoding a bitstream using DRAP pictures.
3 is a schematic diagram of an example mechanism for random access when decoding a bitstream using CRR pictures.
4 is a schematic diagram of an example mechanism for signaling an outer bitstream to support CRR-based random access.
5 is a schematic diagram showing potential decoding errors when a picture follows a DRAP and/or CRR picture in decoding order and precedes the DRAP and/or CRR picture in output order.
6 is a schematic diagram of a media file stored in the International Organization for Standardization (ISO) based media file format (ISOBMFF).
7 is a schematic diagram of a bitstream containing encoded visual media data.
8 is a schematic diagram showing an exemplary video processing system.
Fig. 9 is a schematic diagram showing an exemplary video processing device.
10 is a flow chart for an exemplary method of video processing.
11 is a schematic diagram illustrating an exemplary video coding system.
12 is a schematic diagram illustrating an exemplary encoder.
13 is a schematic diagram illustrating an exemplary decoder.
14 is a schematic diagram of an exemplary encoder.

An example implementation of one or more embodiments is provided below, but it should be understood from the outset that the disclosed systems and/or methods may be implemented using any number of techniques, whether now known or to be developed. This disclosure should in no way be limited to the example implementations, drawings and techniques illustrated below, including the example designs and implementations illustrated and described herein, but to the full scope of equivalents, along with the appended claims It can be modified within the range of scopes.

The term Versatile Video Coding (VVC), also known as H.266, is used in some technologies only to facilitate understanding of the disclosed technologies and is not intended to limit the scope of the disclosed technologies. As such, the techniques disclosed herein are applicable to other video codec protocols and designs. In the current document, editorial changes are made in relation to the VVC specification or the International Organization for Standardization (ISO) Based Media File Format (ISOBMFF) file format specification, with deleted text appearing in bold italics and added text underlined.

This patent document relates to video coding, video file format, video signaling, and video applications. Specifically, this document provides enhanced signaling of cross random access point (RAP) reference in video coding based on Supplemental Enhancement Information (SEI) messages and cross RAP reference in media files. It is related to the signaling of (Cross RAP Referencing (CRR)). The disclosed examples may be applied individually or in various combinations to media files conforming to any video coding standard or non-standard video codec, such as VVC, and any media file formats, such as ISOBMFF.

This disclosure includes the following abbreviations. Adaptive color transform (ACT), adaptive loop filter (ALF), adaptive motion vector resolution (AMVR), adaptation parameter set (APS), access unit (AU; access unit), access unit delimiter (AUD; access unit delimiter), advanced video coding (Rec. ITU-T H.264 | ISO/IEC 14496-10) (AVC; advanced video coding), bidirectional prediction ( B), bi-prediction with coding unit (CU)-level weights (BCW), bi-directional optical flow (BDOF), block-based delta pulse code Block-based delta pulse code modulation (BDPCM), buffering period (BP), context-based adaptive binary arithmetic coding (CABAC), coding block (CB) block), constant bit rate (CBR), cross-component adaptive loop filter (CCALF), coded layer video sequence (CLVS), coded layer video sequence Start (CLVSS: coded layer video sequence start), coded picture buffer (CPB; coded picture buffer), clean random access (CRA), cyclic redundancy check (CRC; cyclic redundancy check), cross RAP reference (CRR : cross RAP referencing), coding tree block (CTB; coding tree block), coding tree unit (CTU), coding unit (CU), coded video sequence (CVS), coded video sequence start (CVSS) , decoding capability information (DCI), decoding initialization information (DII), decoded picture buffer (DPB), dependent random access point (DRAP), decoding unit (DU; decoding unit), decoding unit information (DUI; decoding unit information), exponential-Golomb (EG), k-th order exponential-Golomb (EGk), end of bitstream (EOB) ; end of bitstream), end of sequence (EOS), filler data (FD), first-in, first-out (FIFO), fixed-length (FL), green, blue, and red (GBR), general constraints information (GCI), gradual decoding refresh (GDR), geometric partitioning mode (GPM), Rec . ITU-T H.265 | Also known as ISO/IEC 23008-2, high efficiency video coding (HEVC), hypothetical reference decoder (HRD), hypothetical stream scheduler (HSS), intra (I), Intra block copy (IBC), instantaneous decoding refresh (IDR), inter layer reference picture (ILRP), intra random access point (IRAP), low frequency separation Low frequency non-separable transform (LFNST), least probable symbol (LPS), least significant bit (LSB), long-term reference picture (LTRP), chroma scaling Luma mapping with chroma scaling (LMCS), matrix-based intra prediction (MIP), most probable symbol (MPS), most significant bit (MSB), multiple Multiple transform selection (MTS), motion vector prediction (MVP), network abstraction layer (NAL), output layer set (OLS), operation point (OP) ), operating point information (OPI), predictive (P), picture header (PH), picture order count (POC; picture order count), picture parameter set (PPS), prediction refinement with optical flow (PROF), picture timing (PT), picture unit (PU), Quantization parameter (QP), random access decodable leading picture (RADL), random access point (RAP), random access skipped leading picture (RASL), low bytes Raw byte sequence payload (RBSP), red, green, and blue (RGB), reference picture list (RPL), sample adaptive offset (SAO) , sample aspect ratio (SAR), supplemental enhancement information (SEI), slice header (SH), subpicture level information (SLI), a string of data bits (SODB; string of data bits), sequence parameter set (SPS), short-term reference picture (STRP), step-wise temporal sublayer access (STSA), cut rice ( truncated rice (TR), transform unit (TU), variable bit rate (VBR), video coding layer (VCL), video parameter set (VPS; video parameter set), Rec. ITU-T H.274 | Versatile supplemental enhancement information, also known as ISO/IEC 23002-7 (VSEI), video usability information (VUI), and Rec. ITU-T H.266 | ISO/IEC 23090-3, versatile video coding, also known as (VVC).

Video coding standards have evolved mainly through the development of International Telecommunication Union (ITU) Telecommunications Standardization Sector (ITU-T) and International Electrotechnical Commission (ISO/IEC) standards. ITU-T created H.261 and H.263, ISO/IEC created Motion Picture Experts Group (MPEG)-1 and MPEG-4 Visual, and the two organizations jointly created H.262/MPEG-2 Created Video and H.264/MPEG-4 Advanced Video Coding (AVC) and H.265/HEVC standards. Since H.262, video coding standards are based on a hybrid video coding scheme in which temporal prediction and transform coding are used. To explore additional video coding technologies beyond HEVC, a Joint Video Exploration Team (JVET) was jointly established by the Video Coding Experts Group (VCEG) and MPEG. A number of methods have been adopted by JVET and incorporated into standard software called the Joint Exploration Model (JEM). JVET was later renamed JVET (Joint Video Experts Team) when the Versatile Video Coding (VVC) project officially started. VVC is a coding standard that aims for a 50% bitrate reduction compared to HEVC. VVC was finalized by JVET.

VVC and VSEI standards are used for television broadcasting, videoconferencing, playback from storage media, adaptive bit rate streaming, video region extraction, synthesis and merging of content from multiple coded video bitstreams, multiview video, scalable layered It is designed for use in a wide range of applications including coding and viewport adaptive 360 degree (360°) immersive media.

The Essential Video Coding (EVC) standard (ISO/IEC 23094-1) is another video coding standard developed by MPEG.

File format standards are discussed below. Media streaming applications are typically based on Internet Protocol (IP), Transmission Control Protocol (TCP) and Hypertext Transfer Protocol (HTTP) transport methods, and typically rely on a file format such as ISOBMFF. One such streaming system is dynamic adaptive streaming over HTTP (DASH). Video may be encoded in a video format, such as AVC and/or HEVC. Encoded video can be encapsulated in ISOBMFF tracks and included in DASH representations and segments. Important information about video bitstreams, such as profile, tier and level, can be exposed as file format level metadata and/or in a DASH media presentation description (MPD) for content selection purposes. For example, this information can be used for selection of appropriate media segments both for initialization at the beginning of a streaming session and for stream adaptation during a streaming session.

Similarly, when using an image format as ISOBMFF, file format specifications specific to image formats, such as AVC image file format and HEVC image file format, may be employed. The VVC video file format, which is a file format for storing VVC video contents based on ISOBMFF, is being developed by MPEG. A VVC image file format, which is a file format for storing image contents coded using VVC based on ISOBMFF, is also being developed by MPEG.

Support for random access in HEVC and VVC is discussed below. Random access refers to starting access and decoding of a bitstream from a picture that is not the first picture of the bitstream in decoding order. In order to support tuning and channel switching in broadcast/multicast and multiparty videoconferencing, local playback and seeking in streaming, as well as stream adaptation in streaming, the bitstream must contain random access points. These random access points are typically intra coded pictures, but may also be inter coded pictures (eg for gradual decoding refresh). Intra-coded pictures are pictures coded with reference to blocks within a picture, and inter-coded pictures are pictures coded with reference to blocks in other pictures.

HEVC involves signaling intra random access point (IRAP) pictures in the NAL unit header, via NAL unit types. HEVC supports three types of IRAP pictures: instantaneous decoder refresh (IDR) pictures, clean random access (CRA) pictures, and broken link access (BLA) pictures. IDR pictures constrain the inter-picture prediction structure to not refer to any picture before the current group-of-pictures (GOP), also known as closed-GOP (closed-GOP) random access points. CRA pictures are less restrictive by allowing certain pictures to refer to pictures before the current GOP, which are all discarded in case of random access. CRA pictures are referred to as open-GOP random access points. BLA pictures generally result from splicing of two bitstreams or parts thereof in a CRA picture, for example during stream switching. To enable better system usage of IRAP pictures, all 6 different NAL units are defined to signal attributes of IRAP pictures that can be used to better match stream access point types as defined in ISOBMFF. . These stream access point types are also utilized for random access support in dynamic adaptive streaming over HTTP (DASH).

VVC supports three types of IRAP pictures, two types of IDR pictures (one type with associated RADL pictures or the other type without associated RADL pictures) and one type of CRA picture. They are used in a similar way as in HEVC. BLA picture types in HEVC are not included in VVC. This is because the basic function of BLA pictures can be realized by CRA pictures and the end of a sequence NAL unit whose presence indicates that a subsequent picture starts a new CVS in a single layer bitstream. Further, during the development of VVC there was a desire to specify fewer NAL unit types than HEVC, as indicated by the use of 5 bits instead of 6 bits for the NAL unit type field in the NAL unit header.

Another difference in random access support between VVC and HEVC is that VVC supports GDR in a more prescriptive manner. In GDR, decoding of a bitstream can start from an inter-coded picture. For the first picture at the random access point, only part of the picture can be decoded correctly. However, after a number of pictures, the entire picture area can be accurately decoded and displayed. AVC and HEVC also support GDR, using the recovery point SEI message for signaling of GDR random access points and recovery points. In VVC, the NAL unit type is specified for the indication of GDR pictures, and the recovery point is signaled in the picture header syntax structure. CVS and bitstreams are allowed to start with GDR pictures. Therefore, it is allowed for the entire bitstream to contain only inter-coded pictures without a single intra-coded picture. The main advantage of specifying GDR support in this way is that it provides suitable behavior for GDR. Unlike intra-coding entire pictures, GDR distributes intra-coded slices or blocks (which are less compressed than inter-coded slices/blocks) over a large number of pictures, so that encoders can even out the bit rate of the bitstream. do. This enables significant end-to-end latency reduction, which is considered more important than ever with the increasing use of ultra-low latency applications such as wireless display, online gaming and drone-based applications.

Another GDR-related feature in VVC is virtual boundary signaling. The boundary between a refreshed region (ie, a correctly decoded region in GDR) and a non-refreshed region in a picture between a GDR picture and its recovery point may be signaled as a virtual boundary. If signaled, cross-border in-loop filtering will not be applied. This avoids decoding mismatches for some samples at or near the boundary. This can be useful when an application decides to mark correctly decoded regions during the GDR process. IRAP pictures and GDR pictures may be collectively referred to as RAP (Random Access Point) pictures.

VUI and SEI messages are discussed below. A VUI is a syntax structure that is transmitted as part of the SPS (and possibly also in the VPS in HEVC). The VUI does not affect the canonical decoding process, but carries information that can be used for proper rendering of the coded video. SEI supports processes related to decoding, display or other purposes. Like VUI, SEI does not affect the canonical decoding process. SEI is carried in SEI messages. Decoder support of SEI messages is optional. However, SEI messages affect bitstream conformance. For example, if the syntax of the SEI message in the bitstream does not conform to the specification, the bitstream is not conforming. Some SEI messages are used in the HRD specification.

The VUI syntax structure and most SEI messages used with VVC are specified in the VSEI specification, not the VVC specification. SEI messages required for HRD conformance testing are specified in the VVC specification. VVC defines 5 SEI messages related to HRD conformance testing and VSEI specifies 20 additional SEI messages. The SEI messages conveyed in the VSEI specification do not directly affect conforming decoder behavior and are defined to be used in a coding format agnostic way, allowing VSEI to be used with other video coding standards besides VVC in the future. Instead of specifically mentioning VVC syntax element names, the VSEI specification refers to variables whose values are set within the VVC specification.

Compared to HEVC, the VUI syntax structure of VVC focuses only on information related to proper rendering of pictures and does not contain any timing information or bitstream restriction indications. In VVC, the VUI is signaled within the SPS which includes a length field before the VUI syntax structure to signal the length of the VUI payload in bytes. This allows the decoder to skip information easily and enables VUI syntax extensions by adding new syntax elements directly to the end of the VUI syntax structure in a manner similar to SEI message syntax extensions.

The VUI syntax structure contains the following information: whether the content is interlaced or progressive; an indication of whether the content includes frame packed stereoscopic video or projection omnidirectional video; sample aspect ratio; an indication of whether the content is appropriate for an overscan display; color description including primaries, matrices and transfer characteristics that support signaling high dynamic range (HDR) as well as ultra high definition (UHD) to high definition (HD) color space; and indication of chroma position compared to luma (signaling clearer for progressive content compared to HEVC).

When the SPS does not contain any VUI, the information is considered unspecified and either passed through external means or specified by the application if the content of the bitstream is intended to be rendered on the display.

Table 1 lists the SEI messages specified for VVC, as well as the specification including their syntax and semantics. Of the 20 SEI messages specified in the VSEI specification, many have been inherited from HEVC (eg, filler payload and both user data SEI messages). Some SEI messages are used for precise processing or rendering of coded video content. This is the case for mastering display color volume, content brightness level information and/or alternative transfer characteristic SEI messages specifically related to HDR content. Other examples include equirectangular projection, spherical rotation, per-area packing and/or omni-viewport SEI messages related to signaling and processing of 360° video content.

Table 1. List of SEI messages in VVC v1

Name of SEI message Purpose of SEI messages SEI messages specified in the VVC specification buffering period Initial CPR removal delays for HRD picture timing CPB removal delays and DPB output delays for HRD decoding unit information CPB removal delays and DPB output delays for DU-based HRD scalable nesting A mechanism to associate SEI messages with specific output layer sets, layers or sets of subpictures. Subpicture level information Information about levels for subpicture sequences SEI messages specified in the VESI specification filler payload Filler data to adjust bit rate Rec. User data registered by ITU-T T.35 User data not registered Carries user data and can be used as a container for data by other entities film grain characteristics Model for Film Grain Synthesis frame packing arrangement Information about how the stereoscopic video is coded into the bitstream, e.g. two views packing two pictures for each time instance into one picture Display including parameter sets Indication of whether the sequence contains all NAL units needed for decoding Decoded picture hash Hash of decoded pictures for error detection Mastering Display Color Volume A description of the color volume of the display used to author content Content Brightness Level Information Upper bounds on the nominal target luminance brightness level of the content Show dependent RAP Displays pictures that use only preceding IRAP pictures for inter prediction reference Alternative transfer characteristics Preferred Alternative Values for Delivery Characteristics of Content ambient viewing environment Characteristics of a nominal ambient viewing environment for display of content, which may be used to assist a receiver in processing content according to the local viewing environment. content color volume Color volume characteristics of the associated picture Equirectangular Projection Generalized Cubemap Projection An indication of the applied projection format, including information needed to remap content to a sphere for rendering in omnidirectional video applications. sphere rotation Information about rotation angles for transformation between global and local axes for use in omnidirectional video applications. Packing by area Information needed to remap cropped decoded pictures onto projected pictures for use in omnidirectional video applications, including region-specific operations such as repositioning, scaling, and rotation. omni viewport Coordinates of one or more regions corresponding to recommended viewports for displaying, for use in omnidirectional video applications. frame field information Indicate how the associated picture should be displayed, its source scan, and whether it is a duplicate of the previous picture. Sample aspect ratio information Information about the sample aspect ratio of the associated picture

SEI messages specified for VVC v1 include a frame field information SEI message, a sample aspect ratio information SEI message, and a subpicture level information SEI message. The frame field information SEI message includes information indicating how the associated picture should be displayed (eg, field parity or frame repetition period), the source scan type of the associated picture, and whether the associated picture is a duplicate of the previous picture. This information, together with the associated picture's timing information, can be signaled in the picture timing SEI message in previous video coding standards. However, frame field information and timing information are two different kinds of information that do not necessarily need to be signaled together. As a typical example, timing information is signaled at the system level, while frame field information is signaled within the bitstream. Therefore, the frame field information is removed from the picture timing SEI message and instead signaled within a dedicated SEI message. This change also supports modifying the syntax of frame field information to convey additional and more specific commands to the display, such as pairing fields together or more values for frame repetition.

A sample aspect ratio SEI message makes it possible to signal different sample aspect ratios for different pictures within the same sequence, whereas the corresponding information contained in the VUI applies to the entire sequence. This may be relevant when using the reference picture resampling feature with scaling factors that cause different pictures of the same sequence to have different sample aspect ratios.

The subpicture level information SEI message provides levels of information for subpicture sequences.

DRAP indication SEI messages are discussed below. The VSEI specification includes DRAP indication SEI messages, specified as follows:

A picture associated with a dependent random access point (DRAP) indication SEI message is referred to as a DRAP picture. The presence of the DRAP indication SEI message indicates that the constraints on picture order and picture reference specified in this clause apply. These constraints allow the decoder to properly decode pictures that follow a DRAP picture in both decoding order and output order, without having to decode any other pictures except for the DRAP picture's associated IRAP picture.

The constraints indicated by the presence of the DRAP indication SEI message, which must all apply, are as follows. A DRAP picture is a trailing picture. A DRAP picture has a temporal sub-layer identifier equal to 0. A DRAP picture does not contain any pictures in the active entries of its reference picture lists, except for the DRAP picture's associated IRAP picture. Any picture that follows a DRAP picture in both decoding order and output order, in active entries of its reference picture lists, excluding the DRAP picture's associated IRAP picture, precedes the DRAP picture in decoding order or output order. It does not contain any pictures.

DRAP signaling in media files is discussed below. ISOBMFF includes a signaling mechanism for DRAP based on sample groups as follows. A DRAP sample group is defined as follows. A DRAP sample is a sample from which all samples following it in decoding order can be correctly decoded, if the nearest initial sample preceding the DRAP sample is available for reference. The initial sample is a stream access point (SAP) sample of SAP type 1, 2 or 3, indicated by itself by being a sync sample or by a group of SAP samples. For example, if the 32nd sample in the file is an initial sample containing an I picture, the 48th sample may contain a P picture and be marked as a member of the dependent random access point sample group. What this indicates is that random access can be done at the 48th sample by first decoding the 32nd sample (ignoring samples 33 to 47) and then continuing decoding from the 48th sample.

A sample may become a member of a dependent random access point sample group (and may be referred to as a DRAP sample for this reason) only if the following conditions are true: DRAP samples refer only to the nearest preceding initial sample. DRAP sample and all samples following the DRAP sample in output order can be correctly decoded when decoding starts from the DRAP sample. This may occur after decoding the nearest preceding SAP sample of type 1, 2 or 3, which may be indicated by itself by being a sync sample or by a group of SAP samples. DRAP samples can only be used in conjunction with types 1, 2 and 3 SAP samples. This is to activate the function of creating a decodable sequence of samples by concatenating a preceding SAP sample with a DRAP sample and samples following that DRAP sample in output order. Exemplary syntax of the DRAP sample group is as follows.

Exemplary semantics for DRAP sample groups are as follows. DRAP_type is a non-negative integer. If DRAP_type is in the range 1 to 3, indicates the SAP_type (as specified in Annex I) to which the DRAP sample would have corresponded if the DRAP sample had not depended on the nearest preceding SAP. Other type values are preserved. Conserved must be equal to zero. The semantics of this subclause apply only to sample group description entries with a preserved value equal to 0. Parsers MUST accept and ignore sample group description entries with a preserved value greater than 0 when parsing this sample group.

A video coding approach based on cross-RAP reference, also referred to as external decoding refresh (EDR) and/or type 2 DRAP, is discussed below. The basic idea of this video coding approach is as follows. Instead of coding random access points as intra-coded IRAP pictures (except for the very first picture in the bitstream), to avoid unusability of previous pictures when random access points are coded as IRAP pictures, random access point are coded using inter prediction. The mechanism provides a limited number of previous pictures representing different scenes of video content, typically via a separate video bitstream, which may be referred to as an external stream and/or external means. These previous pictures are referred to as external pictures. As a result, each outer picture can be used for inter prediction reference by pictures across random access points. The coding efficiency gain comes from having random access points coded as inter-predicted pictures and having more available reference pictures for pictures that follow EDR pictures in decoding order. A bitstream coded with such a video coding approach can be used in applications based on ISOBMFF and DASH as described below.

DASH content preparation operations are discussed below. Video content is encoded into one or more representations, each having a specific spatial resolution, temporal resolution and quality. Each presentation of video content is represented by a main stream and possibly also an external stream. The main stream contains coded pictures that may or may not contain EDR pictures. When at least one EDR picture is included in the main stream, the outer stream also exists and includes the outer pictures. When EDR pictures are not included in the main stream, there is no external stream. Each main stream is delivered in a Main Stream Representation (MSR). Each EDR picture in MSR is the first picture of a segment.

Each external stream, when present, is carried in an External Stream Representation (ESR). For each segment in the MSR starting with an EDR picture, in the corresponding ESR with the same segment start time derived from the MPD, the external pictures required for decoding of that EDR picture and the decoding order in the bitstream delivered in the MSR There is a segment that carries subsequent pictures in . MSRs of the same video content are included in one Adaptation Set (AS). ESRs of the same video content are included in one AS.

DASH streaming operations are discussed below. The client obtains the MPD of the DASH media presentation, parses the MPD, selects the MSR, and determines the starting presentation time for the content to be consumed. The client requests segments of the MSR, starting with the segment containing the picture with a presentation time equal to (or sufficiently close to) the starting presentation time. If the first picture in the start segment is an EDR picture, preferably before requesting the MSR segments, the corresponding segment (with the same segment start time derived from the MPD) in the associated ESR is also requested. Otherwise, the segment of the associated ESR is not requested.

When switching to a different MSR, the client starts with the first segment with a segment start time greater than the last requested segment start time of the switch-from MSR, Request segments. If the first picture in the starting segment in the MSR to be switched is an EDR picture, preferably before requesting the MSR segments, the corresponding segment in the associated ESR is also requested. Otherwise, the segment of the associated ESR is not requested.

When operating consecutively on the same MSR (after decoding of the starting segment after a seek or stream switching operation), segments of the associated ESR are not requested, including when requesting any segment starting with an EDR picture.

The signaling of cross RAP references in video is discussed below. CRR can be signaled as follows in an SEI message called a type 2 DRAP indication SEI message. The type 2 DRAP indication SEI message syntax is as follows.

Type 2 DRAP indication SEI message semantics are as follows. A picture associated with a Type 2 DRAP indication SEI message is referred to as a Type 2 DRAP picture. Type 1 DRAP pictures (associated with the DRAP indication SEI message) and Type 2 DRAP pictures are collectively referred to as DRAP pictures. The existence of the type 2 DRAP indication SEI message indicates that the constraints on picture order and picture reference specified in this subclause are applied. These constraints enable a decoder to properly decode a type 2 DRAP picture and pictures in the same layer that follow the type 2 DRAP picture in both decoding order and output order. This is in the same CLVS and contains a list of IRAP or DRAP pictures in decoding order, identified by t2drap_ref_rap_id [　i　] syntax elements, excluding the list of pictures referenceablePictures Without the need to decode any other pictures in the same layer can be completed

The above constraints, which are indicated by the presence of the type 2 DRAP indication SEI message and must all be applied, are as follows. A type 2 DRAP picture is a trailing picture. A type 2 DRAP picture has a temporal sublayer identifier equal to 0. A type 2 DRAP picture does not contain any pictures of the same layer in active entries of the reference picture lists of the type 2 DRAP picture except referenceablePictures. Any picture in the same layer that follows a type 2 DRAP picture in both decoding order and output order, excluding referenceablePictures, in active entries of the type 2 DRAP picture's reference picture lists, in decoding order or output order 2 It does not include any picture preceding the DRAP picture. Any picture in the list referenceablePictures does not contain any picture in the same layer that is not a picture at a previous position in the list referenceablePictures, in the active entries of the reference picture lists of that picture. As a result, the first picture in referenceablePictures does not contain any picture from the same layer in the active entries of the picture's reference picture lists, even if the picture is a DRAP picture instead of an IRAP picture.

t2drap_rap_id_in_clvs specifies the RAP picture identifier, denoted as RapPicId, of a type 2 DRAP picture. Each IRAP or DRAP picture is associated with a RapPicId. The value of RapPicId for an IRAP picture is inferred to be equal to 0. The values of RapPicId must be different for any two IRAP or DRAP pictures in CLVS. t2drap_reserved_zero_13bits Must be equal to 0 in bitstreams conforming to this version of this specification. Other values for t2drap_reserved_zero_13bits are retained. Decoders should ignore the value of t2drap_reserved_zero_13bits. t2drap_num_ref_rap_pics_minus1 plus 1 indicates the number of IRAP or DRAP pictures that are in the same CLVS as a type 2 DRAP picture and can be included in active entries of reference picture lists of a type 2 DRAP picture. t2drap_ref_rap_id[i] indicates RapPicId of the i-th IRAP or DRAP picture that is in the same CLVS as the type 2 DRAP picture and can be included in active entries of reference picture lists of the type 2 DRAP picture.

Below are examples of technical problems solved by the disclosed technical solutions. For example, the following problems exist with respect to signaling of CRR and/or DRAP in video bitstreams and media files. The DRAP indication SEI message lacks signaling indicating whether pictures that follow a DRAP picture in decoding order but precede the DRAP picture in output order can be correctly decoded when randomly accessed from the DRAP picture. These pictures may be decoded incorrectly in this case, since they refer for inter prediction to pictures prior to the DRAP picture in decoding order.

5, which shows an example of a picture that follows an associated DRAP picture in decoding order and precedes the associated DRAP picture in output order. Each box is a picture shown in decoding order from left to right. The number in the box is the output order, also known as the picture order count of the picture. An arrow indicates an inter-prediction relationship between two pictures, and the right picture (arrowhead) uses the left picture (arrow starting point) as a reference picture.

In the example presented in FIG. 5 , inter prediction from picture 6 to picture 8 can be turned off (the arrow bringing the two pictures together is removed). In this case, in the case of random access from the DRAP picture (picture 10), picture 8 can be accurately decoded. However, if inter prediction from picture 6 to picture 8 is employed, picture 8 cannot be accurately decoded when a DRAP picture (picture 10) is used as a random access point. An indication of whether this inter prediction is turned off is useful for systems to know when to start presenting video when randomly accessing from a DRAP picture. For example, with this indication, on random access from the DRAP picture (picture 10), the application system can know whether the presentation will start from picture 8 or picture 10.

The type 2 DRAP indication SEI message indicates whether pictures following a type 2 DRAP picture in decoding order but preceding the type 2 DRAP picture in output order can be correctly decoded when randomly accessed from a type 2 DRAP picture A signaling mechanism is also lacking. Such a picture may be decoded incorrectly if the picture refers to pictures prior to a type 2 DRAP picture in decoding order. Such an indication is useful for systems to decide when to present video when randomly accessing from a type 2 DRAP picture. A mechanism for signaling CPP in media files is also lacking.

Furthermore, the semantics of DRAP sample groups in ISOBMFF are incomplete. ISOBMFF states that a DRAP sample is a sample from which all samples following it in decoding order can be decoded correctly, if the nearest initial sample preceding the DRAP sample is available for reference. However, even if the nearest initial sample preceding the DRAP sample is available for reference, samples following the DRAP sample in decoding order but preceding the DRAP sample in output order are within the nearest initial sample for reference. There are cases in which pictures refer to previous pictures. In this case, such samples (pictures) cannot be accurately decoded.

Disclosed herein are mechanisms to address one or more of the problems listed above. For example, a DRAP picture is a random access point picture coded through inter prediction with reference to an IRAP picture. Furthermore, a CRR picture, also known as a type 2 DRAP and/or enhanced dependent random access point (EDRAP) picture, is coded through inter prediction with reference to an IRAP picture and is further allowed to refer to one or more other dependent random access point pictures. is a random access point picture. For this reason, CRR/DRAP/Type 2 DRAP can be considered as one type of DRAP. DRAP and CRR are designed based on the assumption that pictures are managed in a specific order. However, encoders are allowed to rearrange the order of pictures to improve coding efficiency. Accordingly, video pictures may have an output order and a decoding order. The output order is the order in which pictures are presented/displayed, and the decoding order is the order in which the pictures are coded into a bitstream. Some DRAP and CRR designs do not take this distinction into account, so errors can occur when video is coded using DRAP and/or CRR and the encoder decides to rearrange the pictures. Specifically, when an inter-predicted picture follows a DRAP/CRR picture in decoding order and precedes the DRAP/CRR picture in output order, an error may occur. An error may occur because such a picture may be coded with reference to another picture preceding the DRAP/CRR picture in decoding order. When the DRAP/CRR picture is used as a random access point by the decoder, the picture may or may not be fully decodable depending on whether inter prediction referencing other pictures is used. Furthermore, various signaling mechanisms may not fully support DRAP and/or CRR.

Therefore, the present disclosure asks whether an inter-predicted picture that follows a DRAP/CRR picture in decoding order and precedes the DRAP/CRR picture in output order is allowed to refer to other pictures that precede the DRAP/CRR picture in decoding order. It includes a signaling mechanism to indicate whether or not In one example, the signaling mechanism is an SEI message within an encoded bitstream. If such inter-prediction reference is allowed, the inter-predicted picture is not displayed when the DRAP/CRR picture is used as a random access point. If such inter-prediction reference is not allowed, the inter-predicted picture is displayed when the DRAP/CRR picture is used as a random access point. Also, this disclosure describes identical groups and/or identical entries that can be included in ISOBMFF media files to describe DRAP and/or CRR pictures. This allows the encoder to determine the presence and location of DRAP and/or CRR pictures at the file format level.

In order to solve the above problems and other problems, the methods summarized below are disclosed. Items are to be regarded as examples to explain general concepts and not to be construed in a narrow way. Furthermore, these items may be applied individually or in any combination.

Example 1

In one example, an indication is added to the DRAP indication SEI message syntax so that pictures in the same layer as the DRAP picture, which follow the DRAP picture in decoding order and precede the DRAP picture in output order, are in the same layer and are in output order. Indicates whether it is allowed to refer to an earlier picture for inter prediction. If such reference is not allowed, the decoder can accurately decode and display such pictures when DRAP is used as a random access point. If the reference is allowed, decoding may not be possible, and such pictures should not be displayed at the decoder side when DRAP is used as a random access point. In one example, the indication is a 1-bit flag. In one example, the flag is set equal to X (X is 1 or 0) so that pictures in the same layer that follow a DRAP picture in decoding order and precede the DRAP picture in output order are in the same layer and in decoding order DRAP picture. Indicates that it is allowed to refer to a picture that is earlier than the picture for inter prediction. In one example, the flag is set equal to 1-X (X is 1 or 0) so that pictures in the same layer that follow a DRAP picture in decoding order but precede the DRAP picture in output order are in the same layer and are decoded The order indicates that a picture prior to a DRAP picture is not referred to for inter prediction. In one example, the indication is a multi-bit indicator. In one example, the constraint requires that any picture in the same layer that follows a DRAP picture in decoding order must not follow in output order any picture in the same layer that precedes the DRAP picture in decoding order.

example 2

In one example, an additional SEI message is specified, and the presence of this SEI message indicates that pictures that are in the same layer and follow a DRAP picture in the bitstream in decoding order and precede the DRAP picture in output order are in the same layer and in decoding order Indicates that a picture prior to the DRAP picture is not referred to for inter prediction. In one example, the presence of this SEI message indicates that pictures that are in the same layer and follow the DRAP picture in the bitstream in decoding order and precede the DRAP picture in output order are in the same layer and precede the DRAP picture in decoding order. Indicates that it is allowed to refer to a picture for inter prediction. In one example, the constraint requires that any picture that is in the same layer and follows the DRAP picture in decoding order must follow in output order any picture that is in the same layer and precedes the DRAP picture in decoding order .

example 3

In one example, additional SEI messages are specified. The existence of this additional SEI message is such that pictures that are in the same layer, follow in decoding order the DRAP picture associated with both the SEI message and the DRAP indication SEI message, and precede the DRAP picture in output order, are in the same layer and in decoding order. Indicates that any picture located prior to the DRAP picture is not referred for inter prediction. In one example, the absence of this additional SEI message is such that pictures that are in the same layer, that follow in decoding order the DRAP picture associated with both the additional SEI message and the DRAP indication SEI message, and precede the DRAP picture in output order, are in the same layer. Indicates that it is allowed to refer to a picture preceding the DRAP picture in decoding order for inter prediction. In one example, the constraint guarantees that any picture that is in the same layer and follows the DRAP picture in decoding order must follow in output order any picture that is in the same layer and precedes the DRAP picture in decoding order .

example 4

In one example, an additional SEIM message is specified, and an indication is added to the syntax of the additional SEI message to follow the DRAP picture associated with both the additional SEI message and the DRAP indication SEI message in decoding order and to the DRAP picture in output order. It indicates whether or not pictures in the same layer that precede the DRAP picture in decoding order are allowed to refer to pictures in the same layer for inter prediction. In one example, the indication is a 1-bit flag. In one example, the flag is set equal to X (X is 1 or 0) so that pictures in the same layer that follow the DRAP picture in decoding order and precede the DRAP picture in output order are the DRAP picture in decoding order. Indicates that it is allowed to refer to a picture in the same layer that is earlier for inter prediction. In one example, furthermore, the flag is set equal to 1-X (X is 1 or 0) so that pictures in the same layer that follow a DRAP picture in decoding order and precede the DRAP picture in output order are decoded. The order indicates that a picture in the same layer prior to the DRAP picture is not referred to for inter prediction. In one example, the indication is a multi-bit indicator. In one example, the constraint requires that any picture that is in the same layer and follows the DRAP picture in decoding order must follow in output order any picture that is in the same layer and precedes the DRAP picture in decoding order.

example 5

In one example, an indication is added to the type 2 DRAP indication SEI message syntax. The indication is that pictures that are in the same layer, follow the type 2 DRAP picture in decoding order, and precede the type 2 DRAP picture in output order are inter-predicted for pictures that are in the same layer and precede the type 2 DRAP picture in decoding order. Indicates whether referencing is allowed. In one example, the indication is a 1-bit flag. In one example, the flag is set equal to X (X is 1 or 0) so that pictures in the same layer, following the DRAP picture in decoding order, and preceding the DRAP picture in output order, are in the same layer and in decoding order. Indicates that it is allowed to refer to a picture preceding the DRAP picture for inter prediction. In one example, furthermore, the flag is set equal to 1-X (X is 1 or 0) so that pictures in the same layer, following a DRAP picture in decoding order, and preceding the DRAP picture in output order are in the same layer. , and indicates that a picture preceding the DRAP picture in decoding order is not referred to for inter prediction. In one example, the flag is added by reusing 1 bit from the t2drap_reserved_zero_13bits field in the type 2 DRAP indication SEI message syntax. In one example, the indication is a multi-bit indicator. In one example, the constraint requires that any picture that is in the same layer and follows the DRAP picture in decoding order must follow in output order any picture that is in the same layer and precedes the DRAP picture in decoding order.

example 6

In another example, the indication is associated with a DRAP or type 2 DRAP picture. In one example, the indication may be signaled for each DRAP or type 2 DRAP.

Example 7

In one example, an additional sample group is designated to signal CRR (eg, samples containing type 2 DRAP pictures) in the ISOBMFF file.

example 8

In one example, a DRAP sample group is ISOBMFF, eg, using a version field of a sample-to-group box (e.g., SampleToGroupBox or CompactSampleToGroupBox) or using a grouping_type_parameter field (or part thereof) in the sample-to-group box. It is extended to signal CRR (eg, samples containing type 2 DRAP pictures) in a file,

Example 9

In one example, a DRAP sample entry includes a field indicating the number of random access point (RAP) samples required for random access from a member of a DRAP sample group. The required RAP samples are initial samples or DRAP samples. In one example, the DRAP sample entry further includes a field indicating RAP identifiers of members of the DRAP sample group. In one example, the field indicating the RAP identifier is coded using 16 bits. In one example, the field indicating the RAP identifier is coded using 32 bits. In one example, the DRAP sample entry excludes a field indicating the RAP identifiers of members of the DRAP sample group. The RAP identifier may be signaled in a subsample information box, a sample auxiliary information size box, and/or other boxes. In one example, the DRAP sample entry excludes a field indicating the RAP identifiers of members of the DRAP sample group. In one example, the RAP identifier is a sample number. In one example, the DRAP sample entry further includes a number of fields indicating RAP identifiers of required RAP samples needed for random access from a member of a DRAP sample group. In one example, each of the fields indicating RAP identifiers of the requested RAP samples are coded using 16 bits. In one example, each of the fields indicating RAP identifiers of the requested RAP samples are coded using 32 bits. In one example, each of the fields indicating RAP identifiers of the requested RAP samples directly represents the RAP identifier of the requested RAP sample. In one example, each of the fields indicating RAP identifiers of the requested RAP samples represents a difference between RAP identifiers of two RAP samples. In one example, the i-th field of the fields indicating RAP identifiers of the requested RAP samples (i is equal to 0) includes the RAP identifier of the current sample (eg, a sample of the current DRAP sample group) and the first requested RAP sample. It represents the difference between the i-th RAP identifiers of RAP samples. In one example, the i-th field (i is greater than 0) of the fields indicating the RAP identifiers of the requested RAP samples is the RAP identifier of the (i-1)-th required RAP sample and the i-th required RAP sample It expresses the difference between RAP identifiers. In one example, the i-th field (i is greater than 0) of the fields indicating the RAP identifiers of the requested RAP samples is the RAP identifier of the i-th requested RAP sample and the (i-1)-th requested RAP sample It expresses the difference between RAP identifiers.

Example 10

In one example, a dependent random access point (DRAP) sample can be decoded correctly if all samples following it in both decoding and output order are available for reference if the nearest initial sample preceding the DRAP sample is available for reference. is such a sample.

Below are some example embodiments of some of the aspects outlined above. Relevant portions that have been added or modified are underlined and in bold type, and portions that have been deleted are indicated in bold italic type.

In an example implementation, the syntax for the Type 2 DRAP indication SEI message has been modified as follows

Furthermore, the type 2 DRAP indication SEI message semantics have been modified as follows. A picture associated with a Type 2 DRAP indication SEI message is referred to as a Type 2 DRAP picture. Type 1 DRAP pictures (associated with the DRAP indication SEI message) and Type 2 DRAP pictures are collectively referred to as DRAP pictures. The existence of the type 2 DRAP indication SEI message indicates that the constraints on picture order and picture reference specified in this subclause are applied. These constraints cause the decoder to generate a type 2 DRAP picture and the pictures that are in the same layer and that follow it in both decoding order and output order, depending on the decoding order, which are in the same CLVS and identified by the t2drap_ref_rap_id[　i　] syntax elements, either IRAP or It is possible to properly decode any other pictures in the same layer except for the list of pictures referenceablePictures, which is composed of a list of DRAP pictures, without the need to decode them.

The above constraints, which are indicated by the presence of the type 2 DRAP indication SEI message and must all be applied, are as follows. A type 2 DRAP picture is a trailing picture. A type 2 DRAP picture has a temporal sublayer identifier equal to 0. A type 2 DRAP picture does not contain any pictures of the same layer in the active entries of its reference picture lists except for referenceablePictures. Any picture in the same layer that follows a type 2 DRAP picture in both decoding order and output order, excluding referenceablePictures, in active entries of its reference picture lists, is a type 2 DRAP picture in either decoding order or output order. It does not include any picture that precedes . No picture in the list referenceablePictures contains any picture in the same layer that is not a picture at a previous position in the list referenceablePictures, within the active entries of the picture's reference picture lists.

If t2drap_leading_pictures_decodable_flag is equal to 1, the following applies. Any picture that is in the same layer and follows a type 2 DRAP picture in decoding order must follow any picture that is in the same layer and precedes a type 2 DRAP picture in decoding order in output order. Any picture in the same layer that follows a type 2 DRAP picture in decoding order and precedes a type 2 DRAP picture in output order, excluding referenceablePictures, is any picture in the same layer that precedes a type 2 DRAP picture in decoding order. Also does not include active entries in its reference picture lists.

Any picture in the list referenceablePictures does not contain, within the active entries of its reference picture lists, any picture in the same layer that is not a picture at a previous position in the list referenceablePictures. Remarks- As a result, the first picture in referenceablePictures does not contain any picture from the same layer in the active entries of its reference picture lists, even if it is a DRAP picture instead of an IRAP picture.

t2drap_rap_id_in_clvs specifies the RAP picture identifier, denoted as RapPicId, of a type 2 DRAP picture. Each IRAP or DRAP picture is associated with a RapPicId. The value of RapPicId for an IRAP picture is inferred to be equal to 0. The values of RapPicId must be different for any two IRAP or DRAP pictures in CLVS. t2drap_reserved_zero_13bits Must be equal to 0 in bitstreams conforming to this version of this specification. Other values for t2drap_reserved_zero_13bits are ITU-T | It is maintained for future use by ISO/IEC. Decoders should ignore the value of t2drap_reserved_zero_13bits. t2drap_num_ref_rap_pics_minus1 plus 1 indicates the number of IRAP or DRAP pictures that are in the same CLVS as a type 2 DRAP picture and can be included in active entries of reference picture lists of a type 2 DRAP picture. t2drap_ref_rap_id[i] indicates RapPicId of the i-th IRAP or DRAP picture that is in the same CLVS as the type 2 DRAP picture and can be included in active entries of reference picture lists of the type 2 DRAP picture.

In an example implementation, a dependent random access point (DRAP) sample group is defined as: If the classification type of SampleToGroupBox or CompactSampleToGroupBox is equal to 'drap', the following applies. If the version field of SampleToGroupBox or CompactSampleToGroupBox is equal to 0, or if the field grouping_type_parameter is present and has a value equal to 0, the dependent random access point (DRAP) samples are all samples after it in decoding order and output order , the DRAP sample If the nearest initial sample that precedes is available for reference, it is that sample that can be correctly decoded. If the field grouping_type_parameter is present and has a value equal to 1, then a DRAP sample, all samples after it in decoding order and output order, the nearest initial sample that precedes the DRAP sample, and zero or more other identifications that come first in decoding order If available, the DRAP samples for reference are those samples that can be correctly decoded.

The initial sample is a stream access point (SAP) sample of SAP type 1, 2 or 3, indicated by itself by being a sync sample or by a group of SAP samples. For example, if the 32nd sample in the file is an initial sample consisting of I pictures, the 48th sample may consist of P pictures and be marked as a member of the dependent random access point sample group, thereby making the 32nd sample Indicates that random access can be performed at the 48th sample by first decoding (ignoring samples 33 to 47) and then continuing decoding from the 48th sample. Note: DRAP samples can only be used in combination with types 1, 2 and 3 SAP samples. This function creates a decodable sequence of samples by concatenating zero or more identified DRAP samples that precede the preceding SAP sample and the DRAP sample in decoding order with the DRAP sample and samples that follow the DRAP sample in output order. to activate it.

If the version field of SampleToGroupBox or CompactSampleToGroupBox is equal to 0, or if the field grouping_type_parameter is present and its value is equal to 0, then a sample may become a member of the subordinate random access point sample group only if and only if the following are true (and these may be referred to as a DRAP sample for a reason). The DRAP sample only references the nearest preceding initial sample. The DRAP sample and all samples that follow the DRAP sample in decoding order and output order are either themselves indicated by being a sync sample or nearest preceding SAP sample of type 1, 2 or 3 indicated by a SAP sample group. When decoding starts on the DRAP sample side after decoding , it can be accurately decoded.

If the field grouping_type_parameter is present and its value is equal to 1, the sample may become a member of a DRAP sample group (and may be called a DRAP sample for this reason) only if the following conditions are true. A DRAP sample only references the nearest preceding initial sample and zero or more other identified DRAP samples that are earlier in decoding order than the DRAP sample. A DRAP sample and all samples that follow that DRAP sample in decoding order and output order are either indicated by themselves by being sync samples, or by the SAP sample group, which is the nearest preceding SAP sample of type 1, 2, or 3. When decoding starts at the DRAP sample side after decoding and decoding the zero or more identified DRAP samples that are earlier in decoding order than the DRAP sample, it can be correctly decoded.

Exemplary syntax for a DRAP sample group entry is as follows.

Exemplary semantics for a DRAP sample group entry are as follows. DRAP_type is a non-negative integer. If DRAP_type is in the range 1 to 3, it indicates the SAP_type (as specified in Annex I) to which the DRAP sample would have corresponded had it not depended on the nearest preceding SAP or other DRAP samples . Other type values are preserved. num_ref_rap_pics_minus1 plus 1 is earlier than the DRAP sample in decoding order and when starting decoding from the DRAP sample and the DRAP sample, all samples following the DRAP sample in both decoding order and output order Need to be referenced to be able to decode correctly Indicates the number of other DRAP samples present and the initial sample. Conserved must be equal to zero. The semantics of this subclause apply only to sample group description entries whose preserved value is equal to zero. Parsers MUST accept and ignore sample group description entries with a preserved value greater than 0 when parsing this sample group. RAP_id designates RAP sample identifiers of samples belonging to this sample group. A RAP sample is either an initial sample or a DRAP sample. The value of RAP_id for the initial sample is inferred to be equal to zero. ref_RAP_id [i] is earlier than the DRAP sample in decoding order, and all samples following the DRAP sample in both decoding order and output order when decoding is started from the DRAP sample and the DRAP sample Reference to be able to accurately decode Indicates the RAP_id of the i-th RAP sample, which needs to be

In another exemplary implementation, the RAP_id field is not signaled in the VisualDRAPEntry() syntax, in which case the syntax VisualDRAPEntry() is as follows.

Furthermore, RAP_id for each DRAP sample is signaled in a subsample information box, a sample auxiliary information size box, or an additional box.

In another example implementation, the ref_RAP_id[i] field is changed to ref_RAP_id_delta[i], and the semantics of the ref_RAP_id[i] field is changed as follows. ref_RAP_id_delta [i] is earlier than the DRAP sample in decoding order and when decoding starts from the DRAP sample and the DRAP sample, all samples following the DRAP sample in both decoding order and output order can be referenced to accurately decode Displays the delta of RAP_id of the i-th RAP sample that is necessary. A variable RefRapId[i] representing the RAP_id of the i-th RAP sample is derived as follows, where RAP_id is the RAP_id of the current sample (ie, the sample of the current DRAP sample group).

In another example implementation, the semantics of the ref_RAP_id_delta[i] field are changed as follows. The semantics of the ref_RAP_id[i] field is changed as follows. ref_RAP_id_delta [i] is earlier than the DRAP sample in decoding order and when decoding starts from the DRAP sample and the DRAP sample, all samples following the DRAP sample in both decoding order and output order Need to be referenced to accurately decode indicates the delta of RAP_id of the ith RAP sample. The variable RefRapId[i] representing the RAP_id of the i-th RAP sample is derived as follows, where RAP_id is the RAP_id of the current sample (ie, the sample of the current DRAP sample group).

In another exemplary implementation, the RAP sample identifier of the RAP sample is specified equal to the sample number of the RAP sample, the RAP_id of the current sample is the sample number of the current sample, and the variable RefRapId[i] is the sample number of the ith RAP sample. indicate

In another exemplary implementation, the RAP_id field and the ref_RAP_id[i] field when present in the sample group description are coded using 32 bits.

1 is a schematic diagram of an example mechanism for random access when decoding a bitstream using IRAP pictures. Specifically, FIG. 1 shows a bitstream 100 comprising IRAP pictures 101 and non-IRAP pictures 103. The IRAP picture 101 is a picture that is coded according to intra prediction and can be used as an access point to the bitstream 100. Intra-picture coding is a process of coding blocks of a picture with reference to other blocks in the same picture. A picture coded according to intra prediction can be coded without reference to other pictures. On the other hand, the non-IRAP picture 103 is a picture that cannot be used as an access point and can be decoded after the associated IRAP picture 101 has been decoded. For example, non-IRAP pictures 103 are generally coded according to inter prediction. Inter prediction is a process of coding blocks of a picture with reference to blocks of other pictures designated as reference pictures. A picture coded based on inter prediction can be accurately decoded only when all reference pictures of the corresponding picture are also decoded. Both IRAP pictures 101 and non-IRAP pictures 103 may be designated as reference pictures for other non-IRAP pictures 103.

Depending on the coding technology, various types of IRAP pictures 101 may be used. In this example, IRAP pictures 101 include IDR pictures and CRA pictures. An IDR picture is an intra-coded picture that can be used as the first picture in a coded video sequence. A CRA picture is an intra-coded picture that allows the use of associated leading pictures. A leading picture is a picture that precedes the associated IRAP picture 101 in output order but follows the IRAP picture 101 in decoding order. The decoder may start decoding at the beginning of the bitstream 100 . However, users often want to jump to a specific point in the bitstream and start viewing from the selected point. Any point that can be selected by the user as a starting point for decoding is known as a random access point.

In general, any IRAP picture 101 can be used as a random access point. Once an IRAP picture 101 is selected as a random access point, all associated non-IRAP pictures 103 (eg, following the selected IRAP picture 101) can also be decoded. In the example presented, the user has selected CRA4 for random access. The decoder can start decoding at CRA4 without decoding any pictures before CRA4. This is because pictures that follow IRAP pictures generally cannot reference previous IRAP pictures. Accordingly, once CRA4 is selected as a random access point, the decoder can decode CRA4 for display and then based on CRA4 decode non-IRAP pictures 103 following CRA4. This allows the decoder to start presenting the bitstream from the random access point (eg CRA4) without decoding pictures before the random access point.

2 is a schematic diagram of an example mechanism for random access when decoding a bitstream using DRAP pictures. Specifically, FIG. 2 shows a bitstream 200 including IRAP pictures 201 , non-IRAP pictures 203 , and DRAP pictures 205 . IRAP pictures 201 and non-IRAP pictures 203 may be substantially similar to IRAP pictures 101 and non-IRAP pictures 103, respectively. In this example, an IDR picture is used as an IRAP picture (201).

DRAP pictures 205 are also included. The DRAP picture 205 is a picture that can be coded according to inter prediction and used as an access point to the bitstream 200 . For example, each DRAP picture 205 may be coded with reference to the IRAP picture 201. 2 includes arrows pointing to pictures that are coded according to inter prediction and from associated reference pictures. As shown, each DRAP picture 205 is coded with reference to IDR0. By itself, any DRAP picture 205 can be used as a random access point as long as the decoder can decode the associated IRAP picture 201. In the illustrated example, DRAP4 was selected as the random access point. The decoder must know, for example, through signaling, that the DRAP pictures 205 are used in the bitstream 200, and the IRAP picture(s) 201 used as reference pictures for the DRAP pictures 205 should know The decoder can then decode IDR0 for use in random access and decode DRAP4 based on IDR0. The decoder can then decode non-IRAP pictures 203 following DRAP4 based on DRAP4. The decoder may start presenting the decoded video in DRAP4.

Pictures coded according to inter prediction are more compressed than pictures coded according to intra prediction. Accordingly, DRAP pictures 205 are more compressed than IRAP pictures 101 in the bitstream 100. Thus, the use of DRAP pictures 205 is the amount of data (e.g., bits of rate) is reduced.

3 is a schematic diagram of an example mechanism for random access when decoding a bitstream using CRR pictures. Specifically, FIG. 3 shows a bitstream 300 comprising IRAP pictures 301 , non-IRAP pictures 303 , and CRR pictures 305 . IRAP pictures 301 and non-IRAP pictures 303 may be substantially similar to IRAP pictures 101 and non-IRAP pictures 103, respectively. The CRR picture 305 is a picture that can be coded according to inter prediction and used as an access point to the bitstream 300 . A CRR picture 305 may be regarded as a type of DRAP picture. While a DRAP picture is coded with reference to an IRAP picture, a CRR picture 305 may be coded with reference to both the IRAP picture 301 and any other CRR picture 305. Since the CRR picture 305 is a type of DRAP picture, CRR pictures 305 may also be known as EDRAP pictures and/or type 2 DRAP pictures, and these terms may be used interchangeably. 3 includes arrows pointing to pictures that are coded according to inter prediction and from associated reference pictures.

In the illustrated example, all CRR pictures 305 are coded with reference to the IRAP picture 301 denoted IDR0. Furthermore, CRR3, CRR4 and CRR5 are also coded with reference to CRR2. Thus, any CRR picture 305 can be used as a random access point as long as the decoder decodes the associated IRAP picture 301, and any associated CRR picture 305 can be used as a reference picture. In the illustrated example, CRR4 was selected as the random access point. The decoder must know, for example, through signaling, that CRR pictures 305 are used in the bitstream 300, and IRAP picture(s) 310 used as reference pictures for other CRR pictures 305 and CRR picture 305. The decoder can then decode IDR0 and CRR2 for use in random access and decode CRR4 based on IDR0 and CRR2. The decoder can then decode non-IRAP pictures 303 that follow CRR4 based on CRR4. The decoder can start presenting the decoded video at CRR4.

Inter prediction works by matching blocks in a picture with similar reference blocks in reference picture(s). Thus, the encoder can encode a motion vector pointing to a reference block instead of encoding the current block. Any difference between the current block and the reference block is encoded as a residual. The more closely the current block matches the reference block, the smaller the residual to be encoded. By itself, better matching between the current block and the reference block results in less coded data and better compression. The advantage of CRR over DRAP is that more pictures are available, which results in better matching and better compression. The cost of CRR over DRAP is increased complexity in signaling and decoding.

4 is a schematic diagram of an example mechanism for signaling an outer bitstream 401 to support CRR-based random access. As seen above, managing reference pictures for CRR is more complicated than managing reference pictures for DRAP. 4 shows a main bitstream 400 containing encoded video to be decoded by a decoder. Main bitstream 400 is substantially similar to bitstream 300 with references omitted for simplicity. An external bitstream 401 is included to support random access. Specifically, the external bitstream 401 includes a set of reference pictures corresponding to each CRR picture. When random access occurs, the encoder and/or the video server may transmit a main bitstream 400 starting from an access point and a part of the main bitstream 400 corresponding to the access point. For example, the user can select CRR3 for random access. Next, the decoder requests the main bitstream 400 starting at CRR3. Then, the encoder/video server may start transmitting the main bitstream 400 at CRR3. The encoder/video server may also transmit a portion of the external bitstream 401 corresponding to the random access point. In this example, the encoder/video server will send IDR0 and CRR2.

In this way, the decoder receives both the CRR picture at the random access point and all reference pictures necessary for decoding the CRR picture. Next, the decoder can decode CRR3 and start displaying the video from that point. To reduce data transmission, the encoder/video server can send only the part of the outer bitstream 401 needed to decode the random access point, so that random access does not occur again and/or subsequent CRR pictures are now random. Additional data may not be transmitted unless reference pictures not provided by the access point are employed.

5 is a schematic diagram showing potential decoding errors when a picture follows a DRAP and/or CRR picture in decoding order and precedes the DRAP and/or CRR picture in output order. As in the previous pictures, arrows indicate inter prediction, including an arrow pointing to an inter predicted picture and an arrow departing from an associated reference picture.

Encoders are allowed to rearrange pictures to increase compression. As such, the order in which pictures should be presented to the user is known as the output order. The order in which pictures are coded into the bitstream is known as the decoding order. Pictures can be identified by picture order count. The picture order count can be any value in ascending order that uniquely identifies a picture. In diagram 500, pictures are shown in decoding order. On the one hand, pictures are numbered based on their picture order count increasing in output order. As shown by the picture order counts, picture 8 follows picture 10, which is a random access point, out of output order. Thus, picture 8 is the inter-predicted picture 503 that precedes the random access point in output order and follows the random access point in decoding order. In this example, picture 10 is a DRAP/CRR picture 505, which may be a DRAP picture or a CRR/EDRAP/Type 2 DRAP picture, according to the example above. In this example, the inter-predicted picture 503 is coded through inter-prediction with reference to picture 6 (507). Thus, picture 6 is the reference picture 503 for the inter-predicted picture 503.

Diagram 500 shows possible coding errors because inter-predicted picture 503 references 507 reference picture 502 via inter prediction. Specifically, the inter-predicted picture 503 follows the DRAP/CRR picture 505 in decoding order, precedes the DRAP/CRR picture 505 in output order, and follows the DRAP/CRR picture 505 in decoding order. A previously located reference picture 502 is referred to (507). When decoding from picture 4, whose bitstream is an IRAP picture of type IDR, the reference picture 502 is decoded and stored in the reference picture buffer, so the inter-predicted picture 503 can be properly decoded. However, when the DRAP/CRR picture 505 is used for random access, the reference picture 502 is omitted and not decoded. Therefore, the inter-predicted picture 503 cannot be correctly decoded when the inter-predicted picture 503 refers to the reference picture 502 . The encoder has the option of disallowing the reference 507. For example, the encoder can restrict all inter-predicted pictures 503 to refer only to the picture in the associated random access point and pictures following the associated access point in decoding order. If the reference 507 is not allowed, the inter-predicted picture 503 can always be decoded, since the inter-predicted picture 503 is not allowed to refer to any picture before the DRAP/CRR picture 505. because it doesn't However, if the reference 507 is allowed, the inter-predicted picture 503 can be directly decoded if the encoder decides to encode the inter-predicted picture 503 with reference to the reference picture 502. does not exist. It should be noted that allowing the reference 507 is not always error-prone, since the encoder is not required to use the reference 507. However, if the reference 507 is allowed, an error occurs whenever the reference 507 is selected and the DRAP/CRR picture 505 is used for random access. The result of this may be what appears to be random errors from the user's point of view, which reduces the user experience.

This disclosure includes several mechanisms to address this issue. For example, the encoder can signal to the decoder whether the reference 507 is allowed. If the reference 507 is allowed, when the DRAP/CRR picture 505 is used for random access, the inter predicted picture 503 is decodable (depending on whether the encoder chooses to use the reference 507 or not) Since this may or may not be possible, the decoder should not display the inter predicted pictures 503 that precede the DRAP/CRR pictures 505 in output order and follow the DRAP/CRR picture 505 in decoding order. . If the reference 507 is not allowed, when the DRAP/CRR picture 505 is used for random access, the decoder should not display the inter predicted pictures 503 associated with the DRAP/CRR picture 505. do. Moreover, DRAP and CRR signaling mechanisms are not well defined. Accordingly, this disclosure includes mechanisms for signaling descriptions of DRAP and CRR usage in media files to more efficiently decode DRAP/CRR pictures 505 and/or associated pictures to a decoder after random access.

In another example, the coding process can be constrained so that reference 507 does not occur. For example, pictures can be separated into layers, and each layer can be associated with a different frame rate. This allows the decoder to select a layer with a frame rate that the decoder can support. The decoder then displays the mode pictures in the selected layer and all pictures in the layers below the selected layer to achieve the required frame rate. Any picture that is in the same layer as the DRAP/CRR picture 505 and follows the DRAP/CRR picture 505 in decoding order (e.g., the inter-predicted picture 503) is in the same layer and in decoding order. When the encoder requests that any picture preceding the DRAP/CRR picture 505 be followed in output order, the error shown in diagram 500 can be prevented.

6 is a schematic diagram of a media file 600 stored as ISOBMFF. For example, media file 600 can be stored in ISOBMFF and used as a DASH representation. The ISOBMFF media file 600 is stored in a plurality of boxes carrying objects and/or data associated with media content or media presentation. For example, a media file 600 includes a file type box 630 (eg, ftyp), a movie box 610 (eg, moov), and a media data box 620 (eg, mdat). ) may be included.

The file type box 630 can carry data describing the entire file, and thus can carry file level data. Thus, a file level box is any box that contains data related to the entire media file 600. For example, the file type box 630 may include a version number of the ISO specification and/or a file type indicating compatibility information of the media file 600 . Movie box 600 can carry data describing a movie included in a media file, and thus can carry movie level data. The movie level box is an arbitrary box containing data describing the entire movie included in the media file 600 . Movie box 610 can include a wide range of sub-boxes used to contain data for a variety of uses. For example, movie box 610 includes track boxes (traks) carrying meta data describing a track of a media presentation. It should be noted that a track may refer to a temporal sequence of related samples. For example, a media track may contain a sequence of pictures or sampled audio, while a metadata track may contain a sequence of meta data corresponding to the pictures and/or audio. Data describing a track is track level data, so any box describing a track is a track level box.

Media data box 620 contains interleaved and temporally arranged media data (eg, coded video pictures and/or audio) of a media presentation. For example, the media data box 620 may include a bitstream of video data coded according to VVC, AVC, HEVC, or the like. Media data box 620 may contain video pictures, audio, text, or other media data for display to a user. In ISOBMFF, pictures, audio, and text are collectively referred to as samples. This is in contrast to the terminology used in video coding standards which refers to pixels to be encoded/decoded as samples. By itself, the term sample can refer to an entire picture (at the file format level) or a group of pixels (at the bitstream level), depending on the context.

As mentioned above, this disclosure provides additional mechanisms for signaling DRAP and/or CRR usage at the file format level. This allows the decoder to know the DRAP and/or CRR usage by loading the parameters into the moov box 610 prior to actually decoding the bitstream(s) of samples contained in the mdat box 620. For example, the moov box 610 may include a DRAP sample group box 625 and/or an EDRAP sample group box 621 . A Sample Group box may describe which samples are of a matching type with the Sample Group box. In one example, both DRAP and CRR are described in DRAP Sample Group box 625, eg by treating CRR as a DRAP subtype. In another example, CRR samples are described by EDRAP Sample Group box 621 and DRAP samples are described by DRAP Sample Group box 625, respectively. In one example, DRAP sample group 625 may include DRAP sample entries 627 . Then, each DRAP sample entry 627 may describe an associated sample coded according to DRAP. In one example, EDRAP sample group 621 may include EDRAP sample entries 623 . Then, each of the EDRAP sample entries 623 may describe an associated sample coded according to CRR/EDRAP/Type 2 DRAP. The descriptions of each DRAP / CRR sample include the sample identifier of the picture, the identifier of the samples containing the associated reference picture (s), an indication of the number of samples, and / or RAP samples necessary to perform random access from the picture, And/or may include additional information to help the decoder when selecting and performing random access in DRAP/CRR pictures.

The moov box 610 may contain a wide range of other boxes 629 as well. In some examples, descriptions of DRAP/CRR samples may be included in one or more of the other boxes 629 above. For example, the other boxes 629 may include a SampleToGroupBox, and DRAP and/or CRR samples may be described in the SampleToGroupBox. In another example, the other boxes 629 may include a CompactSampleToGroupBox, and DRAP and/or CRR samples may be described in the CompactSampleToGroupBox. As a specific example, the DRAP and/or CRR samples may be described in a group_type_parameter field in SampleToGroupBox and/or CompactSampleToGroupBox. In another example, the other boxes 629 may include a subsample information box, and DRAP and/or CRR samples may be described in the subsample information box. In another example, the other boxes 629 may include a Sample Auxiliary Information Size box, and DRAP and/or CRR samples may be described in the Sample Auxiliary Information Size box. Further, any other box described herein may also be included in the other boxes 629 and may include a description of DRAP and/or CRR samples.

7 is a schematic diagram of a bitstream 700 containing encoded visual media data. The bitstream 700 contains media data that has been coded/compressed by an encoder and to be decoded/decompressed by a decoder. For example, the bitstream 700 may be included in the media data box 620 of the ISOBMFF media file 600. Further, the bitstream 700 may be included in a representation in DASH. The bitstream 700 may be coded according to various coding formats such as VVC, AVC, EVC, and HEVC. In some coding formats, bitstream 700 is represented as a series of NAL units. A NAL unit is a unit of data of a size to be placed in a data packet. For example, VVC contains several types of NAL units. The bitstream 700 is video coding layer (VCL) NAL units containing video data and data describing the VCL NAL units, describing coding tools employed, coding constraints, and the like. It may include non-VCL NAL units including. In one example, bitstream 700 may include pictures 710 coded in VCL NAL units. Pictures 710 may be IRAP pictures, inter predicted pictures, DRAP pictures, CRR pictures, etc. Non-VCL NAL units may contain parameter sets that describe mechanisms used to code various messages and pictures 710 . While many VCL NAL units are included in VVC, this disclosure focuses on SEI NAL units. For example, an SEI NAL unit may contain an SEI message. The SEI NAL message contains data that assists processes related to decoding, display, or other purposes, but is not needed by the decoding process to determine sample values in decoded pictures. In one example, SEI messages may include a DRAP indication SEI message 716 and/or a type 2 DARP indication SEI message 717 . The DRAP indication SEI message 716 is an SEI message containing data describing the use of DRAP pictures. The type 2 DRAP indication SEI message 717 is an SEI message containing data describing the use of CRR/EDRAP/type 2 DRAP pictures. DRAP indication SEI messages 716 and/or type 2 DRAP indication SEI messages 717 may be associated with DRAP and/or CRR/EDRAP/type 2 DRAP pictures and may indicate how such pictures should be treated during decoding. there is.

In one example, the DRAP indication SEI message 716 indicates that a picture that follows a DRAP picture in decoding order and precedes the DRAP picture in output order refers to a reference picture that precedes the DRAP picture in decoding order for inter prediction. It may include an indication as to whether or not this is permitted. In one example, the DRAP indication SEI message 716 indicates that a picture that follows a CRR/EDRAP/type 2 DRAP picture in decoding order and precedes the DRAP picture in output order is a reference picture that precedes the DRAP picture in decoding order. It may include an indication as to whether it is allowed to refer to for inter prediction. In one example, the type 2 DRAP indication SEI message 717 indicates that a picture that follows a CRR/EDRAP/type 2 DRAP picture in decoding order and precedes the DRAP picture in output order precedes the DRAP picture in decoding order. It may include an indication of whether it is allowed to refer to a reference picture for inter prediction. Accordingly, the decoder can read the DRAP indication SEI message 716 and/or the type 2 DRAP indication SEI message 717, following the DRAP/CRR picture in decoding order and in output order, according to the example above. It may be determined whether pictures preceding the DRAP/CRR picture should be presented when the DRAP/CRR picture is used as a random access point.

In a specific example, the DRAP indication SEI message 716 can be associated with a DRAP picture, and the type 2 DRAP indication SEI message 717 can be associated with a CRR/EDRAP/type 2 DRAP picture. In a further example, the type 2 DRAP indication SEI message 717 may include a T2drap_reserved_zero_13bits field 701 , wherein the bitstream from the T2drap_reserved_zero_13bits field 701 follows the CRR/EDRAP/Type 2 DRAP picture in decoding order. It may be used to indicate whether a picture that follows the DRAP picture and precedes the DRAP picture in output order is allowed to refer to a reference picture that precedes the DRAP picture in decoding order for inter prediction. In another example, a field in the DRAP indication SEI message 716 may contain a similar indication for a DRAP picture. In other examples, the multi-bit indicator in the DRAP indication SEI message 716 and/or the type 2 DRAP indication SEI message 717 may be used for this purpose.

8 is a schematic diagram illustrating an example video processing system 800 in which various techniques disclosed herein may be implemented. Various implementations may include some or all components of system 800 . System 800 can include an input 802 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8 or 10 bit multi-component pixel values, or may be received in a compressed or encoded format. Input 802 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces may include wired interfaces such as Ethernet, passive optical network (PON), and the like, and wireless interfaces such as Wi-Fi or cellular interfaces.

System 800 may include a coding component 804 that may implement various coding or encoding methods described herein. Coding component 804 may reduce the average bitrate of video from input 802 to output of coding component 804 to produce a coded representation of the video. Thus, coding techniques are often referred to as video compression or video transcoding techniques. The output of coding component 804 may be stored or transmitted via a coupled communication, as represented by component 806 . The stored or communicated (or coded) bitstream representation of the video received at input 802 is used by component 808 to generate pixel values or displayable video that is transmitted to display interface 810. can do. The process of creating user-viewable video from the bitstream representation is often referred to as video decompression. Further, it will be understood that while certain video processing operations are referred to as “coding” operations or tools, the coding tools or operations are used at the encoder side and the corresponding decoding tools or operations that reverse the results of the coding will be performed by the decoder. will be.

Examples of a peripheral bus interface or display interface may include a universal serial bus (USB) or a high definition multimedia interface (HDMI) or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Connection), PCI, IDE interfaces, and the like. The techniques described in this document may be implemented in a variety of electronic devices such as mobile phones, notebooks, smart phones, or other devices capable of performing digital data processing and/or video display.

9 is a schematic diagram of an exemplary video processing device 900 . Apparatus 900 can be used to practice one or more methods disclosed herein. The device 900 may be implemented in a smartphone, tablet, computer, Internet of Things (IoT) receiver, or the like. Apparatus 900 may include one or more processors 902 , one or more memories 904 , and video processing hardware 906 . Processor(s) 902 may be configured to implement one or more methods disclosed herein. Memory (memories) 904 can be used to store data and code used in practicing the methods and techniques disclosed herein. Video processing hardware 906 may implement some of the techniques disclosed herein in hardware circuitry. In some embodiments, video processing hardware 906 may be included at least in part in processor 902, such as, for example, a graphics coprocessor.

10 is a flow chart for an example method 1000 of video processing. Method 1000 includes determining (e.g. signaling) a description for CRR samples in a visual media data file of an ISOBMFF, at step 1002, at step 1004, the visual media data based on the CRR sample group. Conversion between files and visual media data is performed. A description of the CRR samples may be included in various locations of the ISOBMFF media file. For example, the description of the CRR samples may be included in a CRR sample group, a type 2 DRAP sample group, an EDRAP sample group, and/or a DRAP sample group. In some examples, the description of the CRR samples may be included in SampleToGroupBox and/or CompactSampleToGroupBox, or in group_type_parameter, for example. In some examples, the CRR samples may be indicated as type 2 DRAP samples and/or EDRAP samples. Or, within this context, each sample includes an encoded picture.

The description of the CRR samples may include one or more sample identifiers for identifying samples belonging to (eg, included in) a certain sample group. In another example, the description of the CRR samples may include identifiers of reference pictures for the CRR samples. In another example, the description of the CRR samples may include the number of samples needed as a reference to decode the current sample. As an example, the description of the CRR samples may be included in a sample entry in a sample group. In some examples, a visual media data file may be constrained to support proper decoding. For example, the visual media data file only refers to a current sample, a preceding initial sample that is closest to the current sample, one or more CRR samples preceding the current sample in decoding order, or a combination thereof. It can be limited to be one of the CRR samples. In another example, the visual media data file is configured such that the current sample is the CRR sample only when the current sample and all samples after the current sample in decoding order and output order can be correctly decoded when decoding starts from the current sample. can be limited to be one of them. In another example, the visual media data file may include, after decoding the nearest preceding initial sample, one or more CRR samples preceding the current sample in decoding order, or a combination thereof, the current sample and all subsequent samples. The current sample may be constrained to be one of the CRR samples only if the samples are decoded correctly.

11 is a schematic diagram illustrating an example video coding system 1100 that may utilize the techniques of this disclosure. As shown in FIG. 11 , a video coding system 1100 may include a source device 1110 and a destination device 1120 . The source device 1110 generates encoded video data and may be referred to as a video encoding device. Destination device 1120 can decode the encoded video data generated by source device 1110 and can be referred to as a video decoding device.

The source device 1110 may include a video source 1112 , a video encoder 1114 , and an input/output (I/O) interface 1116 . Video source 1112 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and/or a computer graphics system for generating video data, or a combination of such sources. The video data may include one or more pictures. Video encoder 1114 encodes the video data from video source 1112 to generate a bitstream. The bitstream may include a sequence of bits forming a coded representation of the video data. The bitstream may include coded pictures and associated data. A coded picture is a coded representation of a picture. Associated data may include sequence parameter sets, picture parameter sets, and other syntax structures. I/O interface 1116 may include a modulator/demodulator (modem) and/or a transmitter. The encoded video data may be directly transmitted over network 1130 to destination device 1120 via I/O interface 1116 . The encoded video data may also be stored on storage medium/server 1140 for access by destination device 1120 .

Destination device 1120 may include an I/O interface 1126 , a video decoder 1124 and a display device 1122 . I/O interface 1126 may include a receiver and/or modem. I/O interface 1126 can obtain encoded video data from source device 1110 or storage medium/server 1140 . The video decoder 1124 can decode the encoded video data. The display device 1122 may display the decoded video data to a user. The display device 1122 may be integrated with the destination device 1120 or may be external to the destination device 1120 to be configured to interface with an external display device.

Video encoder 1114 and video decoder 1124 may operate in accordance with video compression standards such as the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVC) standard, and other existing and/or additional standards.

12 is a block diagram illustrating an example of a video encoder 1200, which may be video encoder 1114 in system 1100 shown in FIG. 11 . Video encoder 1200 may be configured to implement any or all of the techniques of this disclosure. In the example of FIG. 12 , video encoder 1200 includes a plurality of functional components. The techniques described in this disclosure may be shared among the various components of video encoder 1200 . In some examples, a processor may be configured to implement any or all of the techniques described in this disclosure.

The functional components of the video encoder 1200 are a division unit 1201, a mode selection unit 1203, a motion estimation unit 1204, a motion compensation unit 1205 and a prediction unit which may include an intra prediction unit 1206. 1202, residual generator 1207, transform processing unit 1208, quantization unit 1209, inverse quantization unit 1210, inverse transform unit 1211, restoration unit 1212, buffer 1213, and entropy encoding may include section 1214 .

In other examples, video encoder 1200 may include more, fewer, or different functional components. In one example, the prediction unit 1202 includes an intra block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode in which at least one reference picture is a picture in which a current video block is located.

Additionally, some components, such as motion estimation unit 1204 and motion compensation unit 1205, may be highly integrated, but are shown separately in the example of FIG. 12 for illustrative purposes.

The division unit 1201 may divide a picture into one or more video blocks. Video encoder 1200 and video decoder 1300 can support various video block sizes.

The mode selector 1203 may select one of the inter or intra coding modes based on, for example, error results, and as a result, provide the intra or inter coded block to the residual generator 1207 to generate the residual Block data may be generated and provided to the reconstruction unit 212 to reconstruct the encoded block for use as a reference picture. In some examples, the mode selector 1203 may select an intra-inter combined prediction (CIIP) mode in which prediction is based on an intra-prediction signal and an inter-prediction signal. The mode selection unit 1203 may also select a resolution (eg, sub-pixel or integer pixel precision) for a motion vector for a block in the case of inter prediction.

To perform inter prediction on the current video block, the motion estimation unit 1204 may compare one or more reference frames from the buffer 1213 to the current video block to generate motion information on the current video block. . The motion compensator 1205 may determine a prediction video block for the current video based on the motion information and decoded samples of pictures other than a picture related to the current block from the buffer 1213 .

Motion estimation unit 1204 and motion compensation unit 1205 may perform different operations on the current video block depending on, for example, whether the current video block is in an I slice, P slice or B slice. .

In some examples, motion estimation unit 1204 may perform uni-directional prediction on the current video block, and motion estimation unit 1204 may perform list 0 or list 0 prediction on a reference video block for the current video block. 1 reference pictures can be searched. Then, the motion estimation unit 1204 includes list 0 including the reference video block or a reference index indicating a reference picture in list 0 and spatial displacement between the current video block and the reference video block. You can create motion vectors. The motion estimation unit 1204 may output the reference index, the prediction direction indicator, and the motion vector as motion information of the video block. The motion compensator 1205 may generate a prediction video block of the current block based on the reference video block indicated by the motion information of the current video block.

In other examples, motion estimation unit 1204 may perform bi-directional prediction on the current video block, and motion estimation unit 1204 may perform a reference video block in List 0 for the current video block. Pictures can be searched, and reference pictures in List 1 can be searched for other reference video blocks for the current video block. Then, the motion estimation unit 1204 determines list 0 containing reference video blocks or a reference index indicating reference pictures in list 0 and spatial displacements between the current video block and the reference video blocks. Motion vectors to be displayed can be generated. The motion estimation unit 1204 may output the reference indices and the motion vectors of the current video block as motion information of the video block. The motion compensator 1205 may generate a prediction video block of the current video block based on the reference video blocks indicated by the motion information of the current video block.

In some examples, motion estimation unit 1204 can output a full set of motion information for the decoder's decoding process. In some examples, motion estimation unit 1204 may not output a full set of motion information for the current video. Rather, the motion estimation unit 1204 may signal motion information of the current video block by referring to motion information of other video blocks. For example, the motion estimation unit 1204 may determine that motion information of the current video block is sufficiently similar to motion information of neighboring video blocks.

In one example, the motion estimation unit 1204 indicates, in a syntax structure related to the current video block, a value indicating to the video decoder 1300 of FIG. 13 that the current video block has the same motion information as other video blocks. can do.

In another example, the motion estimation unit 1204 may identify another video block and a motion vector difference (MVD) in a syntax structure related to the current video block. The motion vector difference indicates a difference between motion vectors of the current video block and the indicated video block. The video decoder 1300 may determine the motion vector of the current video block by using the motion vector of the indicated video block and the motion vector difference.

As discussed above, video encoder 1200 can predictively signal motion vectors. Two examples of predictive signaling techniques that may be implemented by video encoder 1200 are advanced motion vector prediction (AMVP) and merge mode signaling.

The intra prediction unit 1206 may perform intra prediction on the current video block. When the intra prediction unit 1206 performs intra prediction on the current video block, the intra prediction unit 1206 generates prediction data for the current video block based on decoded samples of other video blocks of the same picture. can Prediction data for the current video block may include the predicted video block and various syntax elements.

The residual generator 1207 may generate residual data for the current video block by subtracting the predicted video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample components of samples of the current video block.

In other examples, for example in skip mode, there may be no residual data for the current video block and the residual generator 1207 may not perform a subtraction operation.

Transform processing unit 1208 may apply one or more transforms to the residual video block associated with the current video block to generate one or more transform coefficient video blocks for the current video block.

After the transform processing unit 1208 generates a transform coefficient video block associated with the current video block, the quantization unit 1209 converts the current video block based on one or more quantization parameter (QP) values associated with the current video block. The transform coefficient video block associated with can be quantized.

The inverse quantizer 1210 and the inverse transform unit 1211 may reconstruct a residual video block from the transform coefficient video block by applying inverse quantization and inverse transform to the transform coefficient video block, respectively. The reconstruction unit 1212 adds the reconstructed residual video block to samples corresponding to one or more predicted video blocks generated by the prediction unit 1202 and stores it in the buffer 1213 to restore data associated with the current block. You can create video blocks.

After reconstruction unit 1212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

The entropy encoding unit 1214 may receive data from other functional components of the video encoder 1200 . When the entropy encoding unit 1214 receives the data, the entropy encoding unit 1214 generates entropy-encoded data by performing one or more entropy encoding operations and outputs a bitstream including the entropy-encoded data. can do.

FIG. 13 is a block diagram illustrating an example of a video decoder 1300 , which may be video decoder 1124 in system 1100 shown in FIG. 11 .

Video decoder 1300 may be configured to implement any or all of the techniques of this disclosure. In the example of FIG. 13 , video decoder 1300 includes a plurality of functional components. The techniques described in this disclosure may be shared among the various components of video decoder 1300 . In some examples, a processor may be configured to implement any or all of the techniques described in this disclosure.

In the example of FIG. 13 , the video decoder 1300 includes an entropy decoding unit 1301, a motion compensation unit 1302, an intra prediction unit 1303, an inverse quantization unit 1304, an inverse transform unit 1305, and a reconstruction unit 1306. ), and a buffer 1307. In some examples, video decoder 1300 may perform a decoding pass generally reciprocal to the encoding pass described with respect to video encoder 1200 (FIG. 12).

The entropy decoding unit 1301 may search an encoded bitstream. The encoded bitstream may include entropy coded video data (eg, encoded blocks of video data). The entropy decoding unit 1301 may decode the entropy-coded video data, and from the entropy-decoded video data, the motion compensation unit 1302 may obtain motion information including motion vectors, motion vector precision, and reference picture list indices. and other motion information. For example, the motion compensation unit 1302 can determine this information by executing AMVP and merge mode.

The motion compensation unit 1302 may calculate motion compensated blocks by performing interpolation based on interpolation filters, if possible. Identifiers for interpolation filters to be used with sub-pixel precision may be included in the syntax elements.

Motion compensator 1302 can calculate interpolated values for sub-integer pixels of a reference block using interpolation filters as video encoder 1200 used during encoding of the video block. The motion compensator 1302 may determine the interpolation filters used by the video encoder 1200 according to received syntax information, and may calculate predictive blocks using the interpolation filters.

The motion compensator 1302 includes some syntax information for determining sizes of blocks used to encode the frame(s) and/or slice(s) of the encoded video sequence, and each macro of a picture of the encoded video sequence. Partitioning information describing how the block is divided, modes indicating how each partition was encoded, one or more reference frames (and reference frame lists) for each inter-coded block, and the encoded video Other information may be used to decode the sequence.

The intra prediction unit 1303 may use, for example, intra prediction modes received from a bitstream to form a prediction block from spatially adjacent blocks. The inverse quantization unit 1304 inversely quantizes (ie de-quantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 1301 . The inverse transform unit 1305 applies an inverse transform.

The reconstruction unit 1306 may form decoded blocks by adding the corresponding prediction blocks generated by the motion compensation unit 1302 or the intra prediction unit 1303 and the residual blocks. If required, a deblocking filter may also be applied to the filtering of the decoded blocks to remove blockiness artifacts. The decoded video blocks are then stored in buffer 1307, which provides reference blocks for subsequent motion compensation/intra prediction, and also yields decoded video for presentation on a display device.

14 is a schematic diagram of an example encoder 1400. Encoder 1400 is suitable for implementing the techniques of VVC. The encoder 1400 includes three in-loop filters: a deblocking filter (DF) 1402, a sample adaptive offset (SAO) 1404, and an adaptive loop filter (ALF) 1406. Unlike DF 1402, which uses predefined filters, SAO 1404 and ALF 1406 generate raw-sample output by adding an offset and applying a finite impulse response (FIR) filter. Utilizes the original samples of the current picture to reduce mean square errors between s and reconstructed samples, and the coded side information signals offsets and filter coefficients. The ALF 1406 is located at the last processing stage of each picture and can be regarded as a tool that tries to catch and correct artifacts generated in previous stages.

The encoder 1400 can further include an intra prediction component 1408 and a motion estimation/compensation (ME/MC) component 1410 configured to receive the input video. Intra prediction component 1408 is configured to perform intra prediction, while ME/MC component 1410 is configured to utilize reference pictures obtained from reference picture buffer 1412 to perform inter prediction. Residual blocks from inter prediction or intra prediction are fed to a transform (T) component 1414 and a quantization (Q) component 1416 to generate quantized residual transform coefficients, which are fed to an entropy coding component 1418 . An entropy coding component 1418 entropy codes the prediction results and quantized transform coefficients and sends them to a video decoder (not shown). Quantization components output from quantization component 1416 may be fed to inverse quantization (IQ) components 1420 , inverse transform component 1422 , and reconstruction (REC) component 1424 . The REC component 1424 outputs images to the DF 1402 , SAO 1404 , and ALF 1406 so that the corresponding images may be filtered before being stored in the reference picture buffer 1412 .

A list of preferred solutions by some examples is provided next.

The solutions below show examples of the techniques disclosed herein.

1. A visual media processing method (e.g., method 1000 described in FIG. 10), comprising: converting between a video containing a picture and a bitstream of the video (1004), wherein the picture Coded in the bitstream as a dependent random access point (DRAP) picture, the bitstream conforming to a format rule, the format rule being in the same layer as the DRAP picture, following the DRAP picture in decoding order and output order specifies whether a syntax element is included in an additional enhancement information (SEI) message indicating whether one or more pictures preceding the DRAP picture refer to a picture in the same layer for inter prediction in Is earlier than the DRAP picture in decoding order.

2. The method of solution 1, wherein the SEI message is a DRAP indication SEI message.

3. The method of solution 1, wherein the SEI message is different from a DRAP indication SEI message included in the bitstream.

4. The method of any of solutions 2 to 3, wherein the format rule is one in which the presence of the SEI message is in the same layer as the DRAP picture, follows the DRAP picture in decoding order and precedes the DRAP picture in output order; or Specifies that more pictures indicate that it is allowed to refer to the picture in the same layer for inter prediction, wherein the picture precedes the DRAP picture in decoding order.

5. In any of solutions 2 to 3, the format rule is one in which the presence of the SEI message is in the same layer as the DRAP picture, follows the DRAP picture in decoding order and precedes the DRAP picture in output order, or Specifies that more pictures indicate that it is not allowed to refer to the picture in the same layer for inter prediction, where the picture precedes the DRAP picture in decoding order.

6. The method of any of solutions 1 to 5, wherein the syntax element comprises a 1 bit flag.

The solutions below show example embodiments of the techniques discussed in the previous section.

7. A method of video processing, comprising: performing conversion between a video comprising one or more pictures and a bitstream of the video, the bitstream comprising a type 2 dependent random access point (DRAP) picture; , the bitstream conforms to a format rule, and the format rule determines that pictures in a layer that follow the type 2 DRAP picture in decoding order but precede the type 2 DRAP picture in output order in the bitstream are the layer A specific type of dependent random access point (DRAP) indication syntax message for indicating whether it is allowed to refer for inter prediction to a picture that is in the decoding order and that is earlier than the type 2 DRAP picture in the decoding order. method.

8. The method of solution 7, wherein the specific type of DRAP indication syntax message corresponds to a type 2 DRAP indication syntax message.

9. The method of solution 7, wherein the specific type of DRAP indication syntax message corresponds to a DRAP indication syntax message.

10. The method of any of solutions 7-9, wherein the syntax element comprises a 1-bit flag.

The solutions below show example embodiments of the techniques discussed in the previous section.

11. A method of video processing, comprising the step of performing conversion between a video and a bitstream of the video, the bitstream whether a cross random access point reference (CRR) is signaled in a file format storing the bitstream. conforming to format rules specifying whether and how to be signaled.

12. The method of solution 11, wherein the format rule defines a group of samples indicating the CRR.

13. The method of solution 11, wherein the format rule defines that a dependent random access point (DRAP) sample group includes the CRR.

14. The method of solution 13, wherein the DRAP sample group signaling the CRR includes a version field or grouping_type_parameter field for signaling the CRR.

The solutions below show example embodiments of the techniques discussed in the previous section.

15. A method of video processing comprising: performing conversion between a video and a bitstream of the video, wherein the bitstream comprises a DRAP sample group when the bitstream includes a dependent random access point (DRAP) picture A method that conforms to a format rule specifying the inclusion of a field in a DRAP sample entry indicating the number of random access point (RAP) samples required for random access from a member of.

16. The method of solution 15, wherein the format rule further specifies to include another field indicating RAP identifiers for the members of the DRAP sample group.

The solutions below show example embodiments of the techniques discussed in the previous section.

17. The method of any of solutions 1 to 16, wherein a dependent random access point (DRAP) sample is such that all samples following it in both decoding and output order are such that the nearest initial sample preceding the DRAP sample is available for reference. If the sample can be accurately decoded, the method.

18. The method of any of solutions 1 to 17, further comprising storing the bitstream in a file conforming to a file format.

19. The method of any of solutions 1 to 17, wherein the bitstream is read from a file conforming to a file format.

20. The method of any of solutions 18 to 19, wherein the file format is an International Standards Organization Base Media File Format (ISOBMFF).

21. A video decoding device comprising a processor configured to implement the method recited in one or more of solutions 1 to 20.

22. A video encoding device comprising a processor configured to implement the method recited in one or more of solutions 1 to 20.

23. A computer program product storing computer code, which, when executed by a processor, causes the processor to implement the method recited in any of solutions 1-20.

24. A computer readable medium storing a bitstream conforming to the bitstream format generated according to any one of solutions 1 to 20.

25. A method comprising generating a bitstream according to the method recited in any of solutions 1 to 20 and writing the bitstream to a computer readable medium.

26. A method, device or system described herein.

In the solutions described herein, the encoder can comply with the format rule by yielding a coded representation according to the format rule. In the solutions described herein, a decoder may use the format rule to parse syntax elements in a coded representation with knowledge of the presence and absence of syntax elements according to the format rule to yield a decoded video. .

In this document, the term “video processing” may refer to video encoding, video decoding, video compression or video decompression. For example, video compression algorithms may be applied during conversion from a pixel representation of a video to a corresponding bitstream representation or vice versa. A bitstream representation of the current video block may correspond to bits that are co-located or distributed at different locations within the bitstream, eg, as defined by syntax. For example, a macroblock can be encoded in terms of transformed and coded error residual values and also using bits in headers and other fields in the bitstream. Moreover, during transformation, the decoder can parse the bitstream with the knowledge that some fields may or may not be present based on the decision, as described in the solutions above. Similarly, an encoder can determine whether certain syntax fields are to be included, and can create a coded representation accordingly by including or excluding syntax fields from a coded representation.

The disclosed and other solutions, examples, embodiments, modules and functional operations described herein may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structure disclosed herein and structural equivalents thereof, or a combination of one or more of them. The disclosed and other embodiments may be implemented as one or more computer program products, ie, as one or more modules of computer program instructions encoded on a computer readable medium for execution by or for controlling the operation of a data processing apparatus. there is. The computer readable medium may be a machine readable storage device, a machine readable storage substrate, a memory device, a composition of matter realizing a machine readable propagated signal, or a combination of one or more of these. The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the device may include code that creates an execution environment for the computer program in question, such as code that makes up a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these. there is. A propagated signal is an artificially generated signal, for example a machine generated electrical, optical or electromagnetic signal generated to encode information for transmission to a suitable receiver device.

A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and may be written as a stand-alone program or as a module, component, or subroutine. or as other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be part of a file holding other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple consolidated files (e.g. For example, files that store one or more modules, subprograms, or code sections). A computer program may be distributed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may also be implemented by a special purpose logic circuit, eg, a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from read only memory or random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or operate to receive data from or transmit data to, one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks. possibly be combined, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as built-in hard disks or removable disks; magneto-optical disk; and all forms of non-volatile memory, media and memory devices, including CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or integrated with special purpose logic circuitry.

Although this patent document contains many specifics, they should not be construed as limitations on the scope of any subject matter or of what may be claimed, but rather on features that may be specific to particular embodiments of particular technologies. should be interpreted as an explanation of Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, while features are described above as functioning in particular combinations and may even be initially claimed as such, one or more features from a claimed combination may in some cases be removed from that combination, and a claimed combination may be a sub-combination. Or it may be about a variation of a sub-combination.

Similarly, while actions are depicted in a particular order in the figures, this requires that either those acts be performed in the particular order shown or in a sequential order, or that all illustrated acts be performed, in order to achieve a desired result. It should not be understood as Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

Only a few implementations and examples are described and other implementations, enhancements and variations may be made based on what is described and illustrated in this patent document.

The first component is directly coupled to the second component in the absence of intermediate components, except when there is a line, trace, or other medium between the first component and the second component. When there are intermediate components other than lines, traces, or other media between the first component and the second component, the first component is indirectly coupled to the second component. The term “coupled” and variations thereof include both directly coupled and indirectly coupled. Use of the term “about” refers to a range inclusive of ±10% of the following number unless otherwise stated.

Although several embodiments have been provided in this disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit and scope of the disclosure. These examples are to be regarded as illustrative rather than limiting and are not intended to be limiting to the details given herein. For example, various elements or components may be combined or integrated into another system, and certain features may be omitted or not implemented.

Further, the techniques, systems, subsystems, and methods described and illustrated as individually or separately in various embodiments may be combined with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. may be combined with or integrated with. Other items shown or discussed as coupled may be directly connected or indirectly coupled or communicate through some interface, device, or intermediate component, whether electrically, mechanically, or otherwise. Other examples of alterations, substitutions, and modifications may occur to those skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims (22)

As a method for processing video data,
Determining a description of cross random access point reference (CRR) samples in a visual media data file in an International Organization for Standardization (ISO) based media file format (ISOBMFF) ; and
performing conversion between the visual media data file and the visual media data based on the CRR sample group.
According to claim 1,
Wherein the description of the CRR samples is included within a CRR sample group.
According to claim 1,
wherein the description of the CRR samples is included within a dependent random access point (DRAP) sample group.
According to claim 1,
Wherein the description of the CRR samples is included within a type 2 DRAP sample group.
According to claim 1,
wherein the description of the CRR samples is included within an Enhanced Dependent Random Access Point (EDRAP) sample group.
According to any one of claims 1 to 5,
Wherein the description of the CRR samples is included in a SampleToGroupBox.
According to any one of claims 1 to 6,
Wherein the description of the CRR samples is contained within a CompactSampleToGroupBox.
According to any one of claims 1 to 6,
Wherein the description of the CRR samples is included in a group type parameter (group_type_parameter).
According to any one of claims 1 to 8,
Wherein the CRR samples are denoted as type 2 DRAP samples.
According to any one of claims 1 to 8,
Wherein the CRR samples are denoted as Enhanced Dependent Random Access Point (EDRAP) samples.
According to any one of claims 1 to 10,
Each sample comprises a picture.
According to any one of claims 1 to 6,
Wherein the description of the CRR samples includes one or more sample identifiers for identifying samples belonging to a certain sample group.
According to any one of claims 1 to 12,
Wherein the description of the CRR samples includes identifiers of reference pictures for the CRR samples.
According to any one of claims 1 to 13,
Wherein the description of CRR samples includes a number of samples needed as a reference to decode a current sample.
According to any one of claims 1 to 14,
wherein the description of the CRR samples is included within a sample entry within a sample group.
According to any one of claims 1 to 15,
When a current sample refers to only the nearest preceding initial sample, one or more CRR samples preceding the current sample in decoding order, or a combination thereof, the current sample is one of the CRR samples,
According to any one of claims 1 to 16,
When decoding starts from a current sample, if the current sample and all samples after the current sample in decoding order and output order can be correctly decoded, the current sample is one of the CRR samples,
According to any one of claims 1 to 17,
After decoding the nearest preceding initial sample, one or more CRR samples preceding the current sample in decoding order, or a combination thereof, the current sample and all samples after the current sample are correctly decoded.
According to any one of claims 1 to 18,
wherein the converting comprises generating the visual media data file according to the visual media data.
According to any one of claims 1 to 18,
wherein the transforming comprises parsing the visual media data file to obtain the visual media data.
An apparatus for processing video data, comprising:
a processor and non-transitory memory that stores instructions;
wherein the instructions, when executed by the processor, cause the processor to carry out the method of any of claims 1-20.
As a non-transitory computer readable medium,
a computer program product to be used by a video coding device;
The computer program product comprises computer-executable instructions stored on the non-transitory computer-readable medium that, when executed by a processor, cause the video coding device to perform the method of any of claims 1-20. Including, medium.

Images