WO2020228835A1 - Adaptive color-format conversion in video coding - Google Patents

Adaptive color-format conversion in video coding Download PDF

Info

Publication number
WO2020228835A1
WO2020228835A1 PCT/CN2020/090801 CN2020090801W WO2020228835A1 WO 2020228835 A1 WO2020228835 A1 WO 2020228835A1 CN 2020090801 W CN2020090801 W CN 2020090801W WO 2020228835 A1 WO2020228835 A1 WO 2020228835A1
Authority
WO
WIPO (PCT)
Prior art keywords
picture
video
reference picture
adaptive
color format
Prior art date
Application number
PCT/CN2020/090801
Other languages
English (en)
French (fr)
Inventor
Kai Zhang
Li Zhang
Hongbin Liu
Yue Wang
Original Assignee
Beijing Bytedance Network Technology Co., Ltd.
Bytedance Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Bytedance Network Technology Co., Ltd., Bytedance Inc. filed Critical Beijing Bytedance Network Technology Co., Ltd.
Priority to CN202080036227.5A priority Critical patent/CN113875232A/zh
Publication of WO2020228835A1 publication Critical patent/WO2020228835A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/117—Filters, e.g. for pre-processing or post-processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution

Definitions

  • This patent document relates to video coding techniques, devices and systems.
  • Devices, systems and methods related to digital video coding, and specifically, to bit-depth and color format conversions for video coding may be applied to both the existing video coding standards (e.g., High Efficiency Video Coding (HEVC) ) and future video coding standards or video codecs.
  • HEVC High Efficiency Video Coding
  • a method for video processing comprising: applying an adaptive color format conversion (ACC) process to a video region within a current picture, wherein a set of color formats is applicable to one or more color components of the video region, and the set of color formats is signaled in a specific video unit; and performing a conversion between the video region and a bitstream representation of the current video region based on the adaptive color format conversion (ACC) .
  • ACC adaptive color format conversion
  • a method for video processing comprising: determining, for a video region within a current picture, a relationship between a color format of a reference picture to which the video region refers and that of the current picture; performing, in response to the relationship, a specific operation for samples within the reference picture during an adaptive color format conversion (ACC) process in which a set of color formats is applicable; and performing a conversion between a bitstream representation of the video region and the video region based on the specific operation.
  • ACC adaptive color format conversion
  • a method for video processing comprising: applying a first adaptive conversion process for a video region within a current picture; applying a second adaptive conversion process for the video region; and performing a conversion between a bitstream representation of the video region and the video region based on the first adaptive conversion process and the second adaptive conversion process; wherein the first adaptive conversion process is one of an adaptive color format conversion (ACC) process and an adaptive resolution change (ARC) process, and the second adaptive conversion process is the other of the adaptive color format conversion (ACC) process and the adaptive resolution change (ARC) process.
  • ACC adaptive color format conversion
  • ARC adaptive resolution change
  • a method for video processing comprising: applying a first adaptive conversion process for a video region within a current picture; applying a second adaptive conversion process for the video region; and performing a conversion between a bitstream representation of the video region and the video region based on the first adaptive conversion process and the second adaptive conversion process; wherein the first adaptive conversion process is one of an adaptive color format conversion (ACC) process and an adaptive bit-depth conversion (ABC) process, and the second adaptive conversion process is the other of the adaptive color format conversion (ACC) process and the adaptive bit-depth conversion (ABC) process.
  • ACC adaptive color format conversion
  • ABSC adaptive bit-depth conversion
  • a method for video processing comprising: determining a plurality of ALF parameters; and applying, based on the plurality of ALF parameters, an adaptive loop filtering (ALF) process to samples within a current picture; wherein a plurality of ALF parameters is applicable to the samples based on a corresponding color format to which the plurality of ALF parameters is associated.
  • ALF adaptive loop filtering
  • a method for video processing comprising: determining a plurality of LMCS parameters; and applying, based on the plurality of LMCS parameters, a luma mapping with chroma scaling (LMCS) process to samples within a current picture; wherein the plurality of LMCS parameters is applicable to the samples based on a corresponding color format to which the plurality of LMCS parameters is associated.
  • LMCS luma mapping with chroma scaling
  • a method for video processing comprising: determining whether a chroma residue scaling process is applied in a LMCS process based on a color format associated with a current picture; and applying the luma mapping with chroma scaling (LMCS) process to samples within a current picture based on the determining.
  • LMCS luma mapping with chroma scaling
  • a method for video processing comprising: determining whether and/or how to perform a specific coding tool for a video block within a current picture based on a relationship between a color format of at least one of a plurality of reference pictures to which the video block refers and that of the current picture; and performing a conversion, based on the specific coding tool, between a bitstream representation of the video block and the video block.
  • a method for video processing comprising: performing a conversion between a bitstream representation of a current video block and the current video block, wherein if two reference pictures to which the current video block refers have different color formats, a bi-prediction from the two reference pictures is disabled for the video block during the conversion.
  • the above-described method is embodied in the form of processor-executable code and stored in a computer-readable program medium.
  • a device that is configured or operable to perform the above-described method.
  • the device may include a processor that is programmed to implement this method.
  • a video decoder apparatus may implement a method as described herein.
  • FIG. 1 shows an example of adaptive stream of two representations of the same content coded at different resolutions.
  • FIG. 2 shows another example of adaptive stream of two representations of the same content coded at different resolutions, where segments use either closed Group of Pictures (GOP) or open GOP prediction structures.
  • GOP closed Group of Pictures
  • FIG. 3 shows an example of open GOP prediction structures of the two representations.
  • FIG. 4 shows an example of representation switching at an open GOP position.
  • FIG. 5 shows an example of using resampled reference pictures from another bitstream as a reference for decoding Random Access Skipped Leading (RASL) pictures.
  • RASL Random Access Skipped Leading
  • FIGS. 6A-6C show examples of motion-constrained tile set (MCTS) -based region-wise mixed-resolution (RWMR) viewport-dependent 360 streaming.
  • MCTS motion-constrained tile set
  • RWMR region-wise mixed-resolution
  • FIG. 7 shows an example of collocated sub-picture representations of different intra random access point (IRAP) intervals and different sizes.
  • IIRAP intra random access point
  • FIG. 8 shows an example of segments received when a viewing orientation change causes a resolution change at the start of a segment.
  • FIG. 9 shows an example of a viewing orientation change.
  • FIG. 10 shows an example of sub-picture representations for two sub-picture locations.
  • FIG. 11 shows an example of encoder modifications for adaptive resolution conversion (ARC) .
  • FIG. 12 shows an example of decoder modifications for ARC.
  • FIG. 13 shows an example of tile group based resampling for ARC.
  • FIG. 14 shows an example of an ARC process.
  • FIG. 15 shows an example of alternative temporal motion vector prediction (ATMVP) for a coding unit.
  • ATMVP alternative temporal motion vector prediction
  • FIGS. 16A-16B show an example of a simplified affine motion model.
  • FIG. 17 shows an example of an affine motion vector field (MVF) per sub-block.
  • FIGS. 18A and 18B show an example of the 4-parameter affine model and the 6-parameter affine model, respectively.
  • FIG. 19 shows an example of a motion vector prediction (MVP) for AF_INTER for inherited affine candidates.
  • MVP motion vector prediction
  • FIG. 20 shows an example of an MVP for AF_INTER for constructed affine candidates.
  • FIGS. 21A and 21B show examples of candidates for AF_MERGE.
  • FIG. 22 shows an example of candidate positions for affine merge mode.
  • FIG. 23 is a block diagram of an example of a hardware platform for implementing a visual media decoding or a visual media encoding technique described in the present document.
  • FIG. 24 shows a flowchart of an example method for video processing.
  • FIG. 25 shows a flowchart of another example method for video processing.
  • FIG. 26 shows a flowchart of an example method for video processing.
  • FIG. 27 shows a flowchart of another example method for video processing.
  • FIG. 28 shows a flowchart of an example method for video processing.
  • FIG. 29 shows a flowchart of another example method for video processing.
  • FIG. 30 shows a flowchart of an example method for video processing.
  • FIG. 31 shows a flowchart of another example method for video processing.
  • FIG. 32 shows a flowchart of another example method for video processing.
  • Embodiments of the disclosed technology may be applied to existing video coding standards (e.g., HEVC, H. 265) and future standards to improve compression performance. Section headings are used in the present document to improve readability of the description and do not in any way limit the discussion or the embodiments (and/or implementations) to the respective sections only.
  • Video codecs typically include an electronic circuit or software that compresses or decompresses digital video, and are continually being improved to provide higher coding efficiency.
  • a video codec converts uncompressed video to a compressed format or vice versa.
  • the compressed format usually conforms to a standard video compression specification, e.g., the High Efficiency Video Coding (HEVC) standard (also known as H. 265 or MPEG-H Part 2) , the Versatile Video Coding standard to be finalized, or other current and/or future video coding standards.
  • HEVC High Efficiency Video Coding
  • MPEG-H Part 2 MPEG-H Part 2
  • Video coding standards have evolved primarily through the development of the well-known ITU-T and ISO/IEC standards.
  • the ITU-T produced H. 261 and H. 263, ISO/IEC produced MPEG-1 and MPEG-4 Visual, and the two organizations jointly produced the H. 262/MPEG-2 Video and H. 264/MPEG-4 Advanced Video Coding (AVC) and H. 265/HEVC standards.
  • AVC H. 264/MPEG-4 Advanced Video Coding
  • H. 265/HEVC High Efficiency Video Coding
  • the video coding standards are based on the hybrid video coding structure wherein temporal prediction plus transform coding are utilized.
  • Joint Video Exploration Team JVET was founded by VCEG and MPEG jointly in 2015.
  • JVET Joint Exploration Model
  • AVC and HEVC does not have the ability to change resolution without having to introduce an IDR or intra random access point (IRAP) picture; such ability can be referred to as adaptive resolution change (ARC) .
  • ARC adaptive resolution change
  • Rate adaption in video telephony and conferencing For adapting the coded video to the changing network conditions, when the network condition gets worse so that available bandwidth becomes lower, the encoder may adapt to it by encoding smaller resolution pictures.
  • changing picture resolution can be done only after an IRAP picture; this has several issues.
  • An IRAP picture at reasonable quality will be much larger than an inter-coded picture and will be correspondingly more complex to decode: this costs time and resource. This is a problem if the resolution change is requested by the decoder for loading reasons. It can also break low-latency buffer conditions, forcing an audio re-sync, and the end-to-end delay of the stream will increase, at least temporarily. This can give a poor user experience.
  • Active speaker changes in multi-party video conferencing For multi-party video conferencing, it is common that the active speaker is shown in bigger video size than the video for the rest of conference participants. When the active speaker changes, picture resolution for each participant may also need to be adjusted. The need to have ARC feature becomes more important when such change in active speaker happens frequently.
  • the Dynamic Adaptive Streaming over HTTP (DASH) specification includes a feature named @mediaStreamStructureId. This enables switching between different representations at open-GOP random access points with non-decodable leading pictures, e.g., CRA pictures with associated RASL pictures in HEVC.
  • CRA pictures with associated RASL pictures in HEVC.
  • switching between the two representations at a CRA picture with associated RASL pictures can be performed, and the RASL pictures associated with the switching-at CRA pictures can be decoded with acceptable quality hence enabling seamless switching.
  • the @mediaStreamStructureId feature would also be usable for switching between DASH representations with different spatial resolutions.
  • ARC is also known as Dynamic resolution conversion.
  • ARC may also be regarded as a special case of Reference Picture Resampling (RPR) such as H. 263 Annex P.
  • RPR Reference Picture Resampling
  • This mode describes an algorithm to warp the reference picture prior to its use for prediction. It can be useful for resampling a reference picture having a different source format than the picture being predicted. It can also be used for global motion estimation, or estimation of rotating motion, by warping the shape, size, and location of the reference picture.
  • the syntax includes warping parameters to be used as well as a resampling algorithm.
  • the simplest level of operation for the reference picture resampling mode is an implicit factor of 4 resampling as only an FIR filter needs to be applied for the upsampling and downsampling processes. In this case, no additional signaling overhead is required as its use is understood when the size of a new picture (indicated in the picture header) is different from that of the previous picture.
  • the spatial resolution may differ from the nominal resolution by a factor 0.5, applied to both dimensions.
  • the spatial resolution may increase or decrease, yielding scaling ratios of 0.5 and 2.0.
  • the cropping areas are scaled in proportion to spatial resolutions.
  • the down-sampling points are at even sample positions and are co-sited.
  • the same filter is used for luma and chroma.
  • the combined up-and down-sampling will not change phase or the position of chroma sampling points.
  • pic_width_in_luma_samples specifies the width of each decoded picture in units of luma samples. pic_width_in_luma_samples shall not be equal to 0 and shall be an integer multiple of MinCbSizeY.
  • pic_height_in_luma_samples specifies the height of each decoded picture in units of luma samples. pic_height_in_luma_samples shall not be equal to 0 and shall be an integer multiple of MinCbSizeY. ] ]
  • num_pic_size_in_luma_samples_minus1 plus 1 specifies the number of picture sizes (width and height) in units of luma samples that may be present in the coded video sequence.
  • pic_width_in_luma_samples [i] specifies the i-th width of decoded pictures in units of luma samples that may be present in the coded video sequence.
  • pic_width_in_luma_samples [i] shall not be equal to 0 and shall be an integer multiple of MinCbSizeY.
  • pic_height_in_luma_samples [i] specifies the i-th height of decoded pictures in units of luma samples that may be present in the coded video sequence.
  • pic_height_in_luma_samples [i] shall not be equal to 0 and shall be an integer multiple of MinCbSizeY.
  • pic_size_idx specifies the index to the i-th picture size in the sequence parameter set.
  • the width of pictures that refer to the picture parameter set is pic_width_in_luma_samples [pic_size_idx] in luma samples.
  • the height of pictures that refer to the picture parameter set is pic_height_in_luma_samples [pic_size_idx] in luma samples.
  • sub-picture track is defined as follows in Omnidirectional Media Format (OMAF) : track that is with spatial relationships to other track (s) and that represents that represents a spatial subset of the original video content, which has been split into spatial subsets before video encoding at the content production side.
  • OMAF Omnidirectional Media Format
  • a sub-picture track for HEVC can be constructed by rewriting the parameter sets and slice segment headers for a motion-constrained tile set so that it becomes a self-standing HEVC bitstream.
  • a sub-picture Representation can be defined as a DASH Representation that carries a sub-picture track.
  • JVET-M0261 used the term sub-picture as a spatial partitioning unit for VVC, summarized as follows:
  • Pictures are divided into sub-pictures, tile groups and tiles.
  • a sub-picture is a rectangular set of tile groups that starts with a tile group that has tile_group_address equal to 0.
  • Each sub-picture may refer to its own PPS and may hence have its own tile partitioning.
  • Sub-pictures are treated like pictures in the decoding process.
  • the reference pictures for decoding the sub-picture are generated by extracting the area collocating with the current sub-picture from the reference pictures in the decoded picture buffer.
  • the extracted area shall be a decoded sub-picture, i.e. inter prediction takes place between sub-pictures of the same size and the same location within the picture.
  • a tile group is a sequence of tiles in tile raster scan of a sub-picture.
  • sub-picture as defined in JVET-M0261.
  • a track that encapsulates a sub-picture sequence as defined in JVET-M0261 has very similar properties as a sub-picture track defined in OMAF, the examples given below apply in both cases.
  • Section 5.13 ( "Support for Adaptive Streaming" ) of MPEG N17074 includes the following requirement for VVC:
  • the standard shall support fast representation switching in the case of adaptive streaming services that offer multiple representations of the same content, each having different properties (e.g. spatial resolution or sample bit depth) .
  • the standard shall enable the use of efficient prediction structures (e.g. so-called open groups of pictures) without compromising from the fast and seamless representation switching capability between representations of different properties, such as different spatial resolutions.
  • Content generation for adaptive bitrate streaming includes generations of different Representations, which can have different spatial resolutions.
  • the client requests Segments from the Representations and can hence decide at which resolution and bitrate the content is received.
  • the Segments of different Representations are concatenated, decoded, and played.
  • the client should be able to achieve seamless playout with one decoder instance. Closed GOP structures (starting with an IDR picture) are conventionally used as illustrated in FIG. 1.
  • Open GOP prediction structures starting with CRA pictures
  • Open GOP prediction structures reportedly also reduce subjectively visible quality pumping.
  • a challenge in the use of open GOPs in streaming is that RASL pictures cannot be decoded with correct reference pictures after switching Representations. We describe this challenge below in relation to the Representations presented in FIG. 2.
  • the Segments starting with a CRA picture contain RASL pictures for which at least one reference picture is in the previous Segment. This is illustrated in FIG. 3, where picture 0 in both bitstreams resides in the previous Segment and is used as reference for predicting the RASL pictures.
  • FIG. 4 The Representation switching marked with a dashed rectangle in FIG. 2 is illustrated in FIG. 4. It can be observed that the reference picture ( "picture 0" ) for RASL pictures has not been decoded. Consequently, RASL pictures are not decodable and there will be a gap in the playout of the video.
  • RWMR 360° streaming offers an increased effective spatial resolution on the viewport.
  • Schemes where tiles covering the viewport originate from a 6K (6144 ⁇ 3072) ERP picture or an equivalent CMP resolution, illustrated in FIG. 6, with "4K" decoding capacity (HEVC Level 5.1) were included in clauses D. 6.3 and D. 6.4 of OMAF and also adopted in the VR Industry Forum Guidelines. Such resolutions are asserted to be suitable for head-mounted displays using quad-HD (2560 ⁇ 1440) display panel.
  • Encoding The content is encoded at two spatial resolutions with cube face size 1536 ⁇ 1536 and 768 ⁇ 768, respectively. In both bitstreams a 6 ⁇ 4 tile grid is used and a motion-constrained tile set (MCTS) is coded for each tile position.
  • MCTS motion-constrained tile set
  • Each MCTS sequence is encapsulated as a sub-picture track and made available as a sub-picture Representation in DASH.
  • Merging MCTSs to a bitstream to be decoded The received MCTSs of a single time instance are merged into a coded picture of 1920 ⁇ 4608, which conforms to HEVC Level 5.1. Another option for the merged picture is to have 4 tile columns of width 768, two tile columns of width 384, and three tile rows of height 768 luma samples, resulting into a picture of 3840 ⁇ 2304 luma samples.
  • Sub-picture Representations are merged to coded pictures for decoding, and hence the VCL NAL unit types are aligned in all selected sub-picture Representations.
  • multiple versions of the content can be coded at different IRAP intervals. This is illustrated in FIG. 7 for one set of collocated sub-picture Representations for encoding presented in FIG. 6.
  • FIG. 8 presents an example where the sub-picture location is first selected to be received at the lower resolution (384 ⁇ 384) .
  • a change in the viewing orientation causes a new selection of the sub-picture locations to be received at the higher resolution (768 ⁇ 768) .
  • the viewing orientation change happens so that Segment 4 is received from the short-IRAP-interval sub-picture Representations. After that, the viewing orientation is stable and thus, the long-IRAP-interval version can be used starting from Segment 5 onwards.
  • FIG. 9 illustrates a viewing orientation change from FIG. 6 slightly upwards and towards the right cube face. Cube face partitions that have a different resolution as earlier are indicated with "C" . It can be observed that the resolution changed in 6 out of 24 cube face partitions. However, as discussed above, Segments starting with an IRAP picture need to be received for all 24 cube face partitions in response to the viewing orientation change. Updating all sub-picture locations with Segments starting with an IRAP picture is inefficient in terms of streaming rate-distortion performance.
  • the ability to use open GOP prediction structures with sub-picture Representations of RWMR 360° streaming is desirable to improve rate-distortion performance and to avoid visible picture quality pumping caused by closed GOP prediction structures.
  • the VVC design should allow merging of a sub-picture originating from a random-access picture and another sub-picture originating from a non-random-access picture into the same coded picture conforming to VVC.
  • the VVC design should enable the use of open GOP prediction structure in sub-picture representations without compromising from the fast and seamless representation switching capability between sub-picture representations of different properties, such as different spatial resolutions, while enabling merging of sub-picture representations into a single VVC bitstream.
  • FIG. 10 The design goals can be illustrated with FIG. 10, in which sub-picture Representations for two sub-picture locations are presented. For both sub-picture locations, a separate version of the content is coded for each combination among two resolutions and two random access intervals. Some of the Segments start with an open GOP prediction structure. A viewing orientation change causes the resolution of sub-picture location 1 to be switched at the start of Segment 4. Since Segment 4 starts with a CRA picture, which is associated with RASL pictures, those reference pictures of the RASL pictures that are in Segment 3 need to be resampled. It is remarked that this resampling applies to sub-picture location 1 while decoded sub-pictures of some other sub-picture locations are not resampled.
  • the viewing orientation change does not cause changes in the resolution of sub-picture location 2 and thus decoded sub-pictures of sub-picture location 2 are not resampled.
  • the Segment for sub-picture location 1 contains a sub-picture originating from a CRA picture
  • the Segment for sub-picture location 2 contains a sub-picture originating from a non-random-access picture. It is suggested that merging of these sub-pictures into a coded picture is allowed in VVC.
  • JCTVC-F158 proposed adaptive resolution change mainly for video conferencing.
  • the following sub-sections are copied from JCTVC-F158 and present the use cases where adaptive resolution change is asserted to be useful.
  • the IDR is typically sent at low quality, using a similar number of bits to a P frame, and it takes a significant time to return to full quality for the given resolution.
  • the quality can be very low indeed and there is often a visible blurring before the image is “refocused” .
  • the Intra frame is doing very little useful work in compression terms: it is just a method of re-starting the stream.
  • Video conferences also often have a feature whereby the person speaking is shown full-screen and other participants are shown in smaller resolution windows. To support this efficiently, often the smaller pictures are sent at lower resolution. This resolution is then increased when the participant becomes the speaker and is full-screened. Sending an intra frame at this point causes an unpleasant hiccup in the video stream. This effect can be quite noticeable and unpleasant if speakers alternate rapidly.
  • VVC version 1 The following is high-level design choices are proposed for VVC version 1:
  • VVC version 1 1. It is proposed to include a reference picture resampling process in VVC version 1 for the following use cases:
  • VVC design is proposed to allow merging of a sub-picture originating from a random-access picture and another sub-picture originating from a non-random-access picture into the same coded picture conforming to VVC. This is asserted to enable efficient handling of viewing orientation changes in mixed-quality and mixed-resolution viewport-adaptive 360° streaming.
  • VVC version 1 It is proposed to include sub-picture-wise resampling process in VVC version 1. This is asserted to enable efficient prediction structure for more efficient handling of viewing orientation changes in mixed-resolution viewport-adaptive 360° streaming.
  • Section 5.13 ( "Support for Adaptive Streaming" ) of MPEG N17074 includes the following requirement for VVC:
  • the standard shall support fast representation switching in the case of adaptive streaming services that offer multiple representations of the same content, each having different properties (e.g. spatial resolution or sample bit depth) .
  • the standard shall enable the use of efficient prediction structures (e.g. so-called open groups of pictures) without compromising from the fast and seamless representation switching capability between representations of different properties, such as different spatial resolutions.
  • JVET-M0259 discusses how to meet this requirement by resampling of reference pictures of leading pictures.
  • JVET-M0259 discusses how to address this use case by resampling certain independently coded picture regions of reference pictures of leading pictures.
  • sps_max_rpr specifies the maximum number of active reference pictures in reference picture list 0 or 1 for any tile group in the CVS that have pic_width_in_luma_samples and pic_height_in_luma_samples not equal to pic_width_in_luma_samples and pic_height_in_luma_samples, respectively, of the current picture.
  • max_width_in_luma_samples specifies that it is a requirement of bitstream conformance that pic_width_in_luma_samples in any active PPS for any picture of a CVS for which this SPS is active is less than or equal to max_width_in_luma_samples.
  • max_height_in_luma_samples specifies that it is a requirement of bitstream conformance that pic_height_in_luma_samples in any active PPS for any picture of a CVS for which this SPS is active is less than or equal to max_height_in_luma_samples.
  • the decoding process operates as follows for the current picture CurrPic:
  • Variables and functions relating to picture order count are derived as specified in clause 8.3.1. This needs to be invoked only for the first tile group of a picture.
  • the reference picture used as input to the resampling process is marked as "unused for reference” .
  • the current decoded picture is marked as "used for short-term reference" .
  • SHVC resampling process (HEVC clause H. 8.1.4.2) is proposed with the following additions:
  • Adaptive resolution change as a concept in video compression standards, has been around since at least 1996; in particular H. 263+ related proposals towards reference picture resampling (RPR, Annex P) and Reduced Resolution Update (Annex Q) . It has recently gained a certain prominence, first with proposals by Cisco during the JCT-VC time, then in the context of VP9 (where it is moderately widely deployed nowadays) , and more recently in the context of VVC.
  • ARC allows reducing the number of samples required to be coded for a given picture, and upsampling the resulting reference picture to a higher resolution when such is desirable.
  • Intra coded pictures such as IDR pictures are often considerably larger than inter pictures. Downsampling pictures intended to be intra coded, regardless of reason, may provide a better input for future prediction. It’s also clearly advantageous from a rate control viewpoint, at least in low delay applications.
  • ARC may become handy even for non-intra coded pictures, such as in scene transitions without a hard transition point.
  • ARC can be implemented as reference picture resampling.
  • Implementing reference picture resampling has two major aspects: the resampling filters, and the signaling of the resampling information in the bitstream. This document focusses on the latter and touches the former only to the extent we have implementation experience. More study of suitable filter design is encouraged.
  • FIGS. 11 and 12 illustrate an existing ARC en-/decoder implementation, respectively.
  • the input image data is down-sampled to the selected picture size for the current picture encoding.
  • the decoded picture is stored in the decoded picture buffer (DPB) .
  • the reference picture (s) in the DPB is/are up-/down-scaled according the spatial ratio between the picture size of the reference and the current picture size.
  • the decoded picture is stored in the DPB without resampling.
  • the reference picture in the DPB is up-/down-scaled in relation to the spatial ratio between the currently decoded picture and the reference, when used for motion compensation.
  • the decoded picture is up-sampled to the original picture size or the desired output picture size when bumped out for display.
  • motion vectors are scaled in relation to picture size ratio as well as picture order count difference.
  • ARC parameters is used herein as a combination of any parameters required to make ARC work.
  • TGs may have different ARC parameters
  • the appropriate place for ARC parameters would be either in the TG header or in a parameter set with the scope of a TG, and referenced by the TG header-the Adaptation Parameter Set in the current VVC draft, or a more detailed reference (an index) into a table in a higher parameter set.
  • the use of the PPS for the reference is counter-indicated if, as we do, the per tile group signaling of ARC parameters is a design criterion.
  • Down-sampling per tile group is preferred to allow for picture composition/extraction. However, it is not critical from a signaling viewpoint. If the group were making the unwise decision of allowing ARC only at picture granularity, we can always include a requirement for bitstream conformance that all TGs use the same ARC parameters.
  • max_pic_width_in_luma_samples specifies the maximum width of decoded pictures in units of luma samples in the bitstream.
  • max_pic_width_in_luma_samples shall not be equal to 0 and shall be an integer multiple of MinCbSizeY.
  • the value of dec_pic_width_in_luma_samples [i] cannot be greater than the value of max_pic_width_in_luma_samples.
  • max_pic_height_in_luma_samples specifies the maximum height of decoded pictures in units of luma samples.
  • max_pic_height_in_luma_samples shall not be equal to 0 and shall be an integer multiple of MinCbSizeY.
  • the value of dec_pic_height_in_luma_samples [i] cannot be greater than the value of max_pic_height_in_luma_samples.
  • adaptive_pic_resolution_change_flag 1 specifies that an output picture size (output_pic_width_in_luma_samples, output_pic_height_in_luma_samples) , an indication of the number of decoded picture sizes (num_dec_pic_size_in_luma_samples_minus1) and at least one decoded picture size (dec_pic_width_in_luma_samples [i] , dec_pic_height_in_luma_samples [i] ) are present in the SPS.
  • a reference picture size reference_pic_width_in_luma_samples, reference_pic_height_in_luma_samples is present conditioned on the value of reference_pic_size_present_flag.
  • output_pic_width_in_luma_samples specifies the width of the output picture in units of luma samples. output_pic_width_in_luma_samples shall not be equal to 0.
  • output_pic_height_in_luma_samples specifies the height of the output picture in units of luma samples. output_pic_height_in_luma_samples shall not be equal to 0.
  • reference_pic_size_present_flag 1 specifies that reference_pic_width_in_luma_samples and reference_pic_height_in_luma_samples are present.
  • reference_pic_width_in_luma_samples specifies the width of the reference picture in units of luma samples. output_pic_width_in_luma_samples shall not be equal to 0. If not present, the value of reference_pic_width_in_luma_samples is inferred to be equal to dec_pic_width_in_luma_samples [i] .
  • reference_pic_height_in_luma_samples specifies the height of the reference picture in units of luma samples. output_pic_height_in_luma_samples shall not be equal to 0. If not present, the value of reference_pic_height_in_luma_samples is inferred to be equal to dec_pic_height_in_luma_samples [i] .
  • the size of the output picture shall be equal to the values of output_pic_width_in_luma_samples and output_pic_height_in_luma_samples.
  • the size of the reference picture shall be equal to the values of reference_pic_width_in_luma_samples and _pic_height_in_luma_samples, when the reference picture is used for motion compensation.
  • num_dec_pic_size_in_luma_samples_minus1 plus 1 specifies the number of the decoded picture size (dec_pic_width_in_luma_samples [i] , dec_pic_height_in_luma_samples [i] ) in units of luma samples in the coded video sequence.
  • dec_pic_width_in_luma_samples [i] specifies the i-th width of the decoded picture sizes in units of luma samples in the coded video sequence.
  • dec_pic_width_in_luma_samples [i] shall not be equal to 0 and shall be an integer multiple of MinCbSizeY.
  • dec_pic_height_in_luma_samples [i] specifies the i-th height of the decoded picture sizes in units of luma samples in the coded video sequence.
  • dec_pic_height_in_luma_samples [i] shall not be equal to 0 and shall be an integer multiple of MinCbSizeY.
  • the i-th decoded picture size (dec_pic_width_in_luma_samples [i] , dec_pic_height_in_luma_samples [i] ) may be equal to the decoded picture size of the decoded picture in the coded video sequence.
  • dec_pic_size_idx specifies that the width of the decoded picture shall be equal to pic_width_in_luma_samples [dec_pic_size_idx] and the height of the decoded picture shall be equal to pic_height _in_luma_samples [dec_pic_size_idx] .
  • the proposed design conceptually includes four different filter sets: down-sampling filter from the original picture to the input picture, up-/down-sampling filters to rescale reference pictures for motion estimation/compensation, and up-sampling filter from the decoded picture to the output picture.
  • the first and last ones can be left as non-normative matters.
  • up-/down-sampling filters need to be explicitly signaled in an appropriate parameter set, or pre-defined.
  • SHVC SHM ver. 12.4
  • SHVC SHM ver. 12.4
  • 2D separable filter a 12-tap and 2D separable filter
  • the phase of the down-sampling filter is set equal to zero by default.
  • 8-tap interpolation filters are used, with 16-phases, to shift the phase and align the luma and chroma pixel positions to the original positions.
  • Table 3 provides the 12-tap filter coefficients for down-sampling process. The same filter coefficients are used for both luma and chroma for down-sampling.
  • the parameter dec_pic_size_idx can be moved into whatever header that starts a sub-picture. Our current feeling is that most likely that will continue to be a tile group header.
  • FIG. 13 is made up of four sub-pictures (expressed perhaps as four rectangular tile groups in the bitstream syntax) . To the left, the bottom right TG is subsampled to half the size. What do we do with the samples outside the relevant area, marked as “Half” ?
  • a list of picture resolutions is signalled in the SPS, and an index to the list is signalled in the PPS to specify the size of an individual picture.
  • the decoded picture before resampling is cropped (as necessary) and outputted, i.e., a resampled picture is not for output, only for inter prediction reference.
  • both the resampled version and the original, resampled version of the reference picture are stored in the DPB, and thus both would affect the DPB fullness.
  • a resampled reference picture is marked as "unused for reference” when the corresponding un-resampled reference picture is marked as "unused for reference” .
  • the RPL signalling syntax is kept unchanged, while the RPL construction process is modified as follows: When a reference picture needs to be included into a RPL entry, and a version of that reference picture with the same resolution as the current picture is not in the DPB, the picture resampling process is invoked and the resampled version of that reference picture is included into the RPL entry.
  • the number of resampled reference pictures that may be present in the DPB should be limited, e.g., to be less than or equal to 2.
  • Another option is that, when resampling and quarter-pel interpolation need to be done, the two filters are combined and the operation is applied at once.
  • temporal motion vector scaling is applied as needed.
  • the ARC software was implemented on top of VTM-4.0.1, with the following changes:
  • the spatial resolution signalling was moved from SPS to PPS.
  • a picture-based resampling scheme was implemented for resampling reference pictures. After a picture is decoded, the reconstructed picture may be resampled to a different spatial resolution. The original reconstructed picture and the resampled reconstructed picture are both stored in the DPB and are available for reference by future pictures in decoding order.
  • the up-sampling filter 4-tap +/-quarter-phase DCTIF with taps (-4, 54, 16, -2) /64
  • the down-sampling filter the h11 filter with taps (1, 0, -3, 0, 10, 16, 10, 0, -3, 0, 1) /32
  • the reference picture lists of the current picture i.e., L0 and L1
  • the reference pictures may be available in both their original sizes or the resampled sizes.
  • TMVP and ATVMP may be enabled; however, when the original coding resolutions of the current picture and a reference picture are different, TMVP and ATMVP are disabled for that reference picture.
  • the decoder when outputting a picture, the decoder outputs the highest available resolution.
  • the decoded picture before resampling is cropped (as necessary) and outputted, i.e., a resampled picture is not for output, only for inter prediction reference.
  • the ARC resampling filters should be designed to optimize the use of the resampled pictures for inter prediction, and such filters may not be optimal for picture outputting/displaying purpose, while video terminal devices usually have optimized output zooming/scaling functionalities already implemented.
  • Resampling of a decoded picture can be either picture-based or block-based.
  • block-based resampling over picture-based resampling.
  • JVET makes a decision on which of these two should be specified for ARC support in VVC.
  • a reference picture may need to be resampled multiple times since multiple pictures may refer to the same reference picture, and .
  • both the resampled version and the original, resampled version of the reference picture are stored in the DPB, and thus both would affect the DPB fullness.
  • a resampled reference picture is marked as "unused for reference” when the corresponding un-resampled reference picture is marked as "unused for reference” .
  • the reference picture lists (RPLs) of each tile group contain reference pictures that have the same resolution as the current picture. While there is no need for a change to the RPL signalling syntax, the RPL construction process is modified to ensure what is said in the previous sentence, as follows: When a reference picture needs to be included into a RPL entry while a version of that reference picture with the same resolution as the current picture is not yet available, the picture resampling process is invoked and the resampled version of that reference picture is included.
  • the number of resampled reference pictures that may be present in the DPB should be limited, e.g., to be less than or equal to 2.
  • temporal MV usage e.g. merge mode and ATMVP
  • scaling temporal MV to the current resolution as needed.
  • a reference block is resampled whenever needed, and no resampled picture is stored in the DPB.
  • the main issue here is the additional decoder complexity. This is because a block in a reference picture may be referred to multiple times by multiple blocks in another picture and by blocks in multiple pictures.
  • the reference block is resampled by invocation of the interpolation filter such that the reference block has the integer- pel resolution.
  • the interpolation process is invoked again to obtain the resampled reference block in the quarter-pel resolution. Therefore, for each motion compensation operation for the current block from a reference block involving different resolutions, up to two, instead of one, interpolation filtering operations are needed. Without ARC support, up to only one interpolation filter operation (i.e., for generation of the reference block in the quarter-pel resolution) is needed.
  • block blkA in the current picture picA refers to a reference block blkB in a reference picture picB
  • block blkA shall be a uni-predicted block.
  • the worst-case number of interpolation operations needed to decode a block is limited to two. If a block refers to a block from a different-resolution picture, the number of interpolation operations needed is two as discussed above. This is the same as in the case when the block refers to a reference block from a same-resolution picture and coded as a bi-predicted block since the number of interpolation operations is also two (i.e., one for getting the quarter-pel resolution for each reference block) .
  • the corresponding positions of every pixel of predictors are calculated first, and then the interpolation is applied only one time. That is, two interpolation operations (i.e. one for resampling and one for quarter-pel interpolation) are combined into only one interpolation operation.
  • the sub-pel interpolation filters in the current VVC can be reused, but, in this case, the granularity of interpolation should be enlarged but the interpolation operation times are reduced from two to one.
  • temporal MV usage e.g. merge mode and ATMVP
  • scaling temporal MV to the current resolution as needed.
  • the DPB may contains decoded pictures of different spatial resolutions within the same CVS.
  • counting DPB size and fullness in units decoded picture no longer works.
  • PicSizeInSamplesY pic_width_in_luma_samples *pic_height_in_luma_samples
  • MaxDpbSize maximum number of reference picture that may present in the DPB
  • MinPicSizeInSampleY (Width of the smallest picture resolution in the bitstream) * (Height of the smallest resolution in the bitstream)
  • MaxDpbSize is modified as follows (based on the HEVC equation) :
  • MaxDpbSize Min (4 *maxDpbPicBuf, 16)
  • MaxDpbSize Min (2 *maxDpbPicBuf, 16)
  • MaxDpbSize Min ( (4 *maxDpbPicBuf) /3, 16)
  • PictureSizeUnit is an integer value that specifies how big a decoded picture size is relative to the MinPicSizeInSampleY.
  • the definition of PictureSizeUnit depends on what resampling ratios are supported for ARC in VVC.
  • the PictureSizeUnit is defined as follows:
  • Decoded pictures having the resolution that is 2 by 2 of the smallest resolution in the bitstream is associated with PictureSizeUnit of 4 (i.e., 1 *4) .
  • the PictureSizeUnit is defined as follows:
  • Decoded pictures having the resolution that is 1.5 by 1.5 of the smallest resolution in the bitstream is associated with PictureSizeUnit of 9 (i.e., 2.25 *4) .
  • Decoded pictures having the resolution that is 2 by 2 of the smallest resolution in the bitstream is associated with PictureSizeUnit of 16 (i.e., 4 *4) .
  • MinPictureSizeUnit be the smallest possible value of PictureSizeUnit. That is, if ARC supports only resampling ratio of 2, MinPictureSizeUnit is 1; if ARC supports resampling ratios of 1.5 and 2, MinPictureSizeUnit is 4; likewise, the same principle is used to determine the value of MinPictureSizeUnit.
  • the value range of sps_max_dec_pic_buffering_minus1 [i] is specified to range from 0 to (MinPictureSizeUnit * (MaxDpbSize –1) ) .
  • MinPictureSizeUnit is the smallest possible value of PictureSizeUnit.
  • the DPB fullness operation is specified based on PictureSizeUnit as follows:
  • the HRD is initialized at decoding unit 0, with both the CPB and the DPB being set to be empty (the DPB fullness is set equal to 0) .
  • the DPB fullness is set equal to 0.
  • the DPB fullness is decrement by the value of PictureSizeUnit associated with the removed picture.
  • the DPB fullness is increment by the value of PictureSizeUnit associated with the inserted picture.
  • the implemented resampling filters were simply taken from previously available filters described in JCTVC-H0234. Other resampling filters should be tested and used if they provide better performance and/or lower complexity. We propose that various resampling filters to be tested to strike a trade-off between complexity and performance. Such tests can be done in a CE.
  • the standard shall support fast representation switching in the case of adaptive streaming services that offer multiple representations of the same content, each having different properties (e.g. spatial resolution or sample bit depth) . ”
  • allowing resolution change within a coded video sequence without inserting an I picture can not only adapt the video data to dynamic channel conditions or user preference seamlessly, but also remove the beating effect caused by I pictures.
  • FIG. 14 A hypothetical example of adaptive resolution change is shown in FIG. 14 where the current picture is predicted from reference pictures of different sizes.
  • This contribution proposes high level syntax to signal adaptive resolution change as well as modifications to the current motion compensated prediction process in the VTM. These modifications are limited to motion vector scaling and subpel location derivations with no changes in the existing motion compensation interpolators. This would allow the existing motion compensation interpolators to be reused and not require new processing blocks to support adaptive resolution change which would introduce additional cost.
  • pic_width_in_luma_samples specifies the width of each decoded picture in units of luma samples. pic_width_in_luma_samples shall not be equal to 0 and shall be an integer multiple of MinCbSizeY. ] ]
  • pic_height_in_luma_samples specifies the height of each decoded picture in units of luma samples. pic_height_in_luma_samples shall not be equal to 0 and shall be an integer multiple of MinCbSizeY. ] ]
  • max_pic_width_in_luma_samples specifies the maximum width of decoded pictures referring to the SPS in units of luma samples.
  • max_pic_width_in_luma_samples shall not be equal to 0 and shall be an integer multiple of MinCbSizeY.
  • max_pic_height_in_luma_samples specifies the maximum height of decoded pictures referring to the SPS in units of luma samples.
  • max_pic_height_in_luma_samples shall not be equal to 0 and shall be an integer multiple of MinCbSizeY.
  • pic_size_different_from_max_flag 1 specifies that the PPS signals different picture width or picture height from the max_pic_width_in_luma_samples and max_pic_height_in_luma_sample in the referred SPS.
  • pic_size_different_from_max_flag 0 specifies that pic_width_in_luma_samples and pic_height_in_luma_sample are the same as max_pic_width_in_luma_samples and max_pic_height_in_luma_sample in the referred SPS.
  • pic_width_in_luma_samples specifies the width of each decoded picture in units of luma samples.
  • pic_width_in_luma_samples shall not be equal to 0 and shall be an integer multiple of MinCbSizeY.
  • pic_width_in_luma_samples is not present, it is inferred to be equal to max_pic_width_in_luma_samples
  • pic_height_in_luma_samples specifies the height of each decoded picture in units of luma samples.
  • pic_height_in_luma_samples shall not be equal to 0 and shall be an integer multiple of MinCbSizeY.
  • pic_height_in_luma_samples is not present, it is inferred to be equal to max_pic_height_in_luma_samples.
  • horizontal and vertical scaling ratios shall be in the range of 1/8 to 2, inclusive for every active reference picture.
  • the scaling ratios are defined as follows:
  • horizontal_scaling_ratio ( (reference_pic_width_in_luma_samples ⁇ 14) + (pic_width_in_luma_samples/2) ) /pic_width_in_luma_samples
  • vertical_scaling_ratio ( (reference_pic_height_in_luma_samples ⁇ 14) + (pic_height_in_luma_samples/2) ) /pic_height_in_luma_samples
  • a picture When there is a resolution change within a CVS, a picture may have a different size from one or more of its reference pictures.
  • This proposal normalizes all motion vectors to the current picture grid instead of their corresponding reference picture grids. This is asserted to be beneficial to keep the design consistent and make resolution changes transparent to the motion vector prediction process. Otherwise, neighboring motion vectors pointing to reference pictures with different sizes cannot be used directly for spatial motion vector prediction due to the different scale.
  • the scaling range is limited to [1/8, 2] , i.e. the upscaling is limited to 1: 8 and downscaling is limited to 2: 1. Note that upscaling refers to the case where the reference picture is smaller than the current picture, while downscaling refers to the case where the reference picture is larger than the current picture. In the following sections, the scaling process is described in more detail.
  • the scaling process includes two parts:
  • the subpel location (x′, y′) in the reference picture pointed to by a motion vector (mvX, mvY) in units of 1/16 th pel is specified as follows:
  • x′ ( (x ⁇ 4) +mvX) ⁇ hori_scale_fp, (3)
  • the reference location of the upper left corner pixel of the current block is at (x′, y′) .
  • the other reference subple/pel locations are calculated relative to (x′, y′) with horizontal and vertical step sizes. Those step sizes are derived with 1/1024-pel accuracy from the above horizontal and vertical scaling factors as follows:
  • x′ i and y′ j have to be broken up into full-pel parts and fractional-pel parts:
  • the existing motion compensation interpolators can be used without any additional changes.
  • the full-pel location will be used to fetch the reference block patch from the reference picture and the fractional-pel location will be used to select the proper interpolation filter.
  • chroma motion vectors When the chroma format is 4: 2: 0, chroma motion vectors have 1/32-pel accuracy.
  • the scaling process of chroma motion vectors and chroma reference blocks is almost the same as for luma blocks except a chroma format related adjustment.
  • x c ′ ( (x c ⁇ 5) +mvX) ⁇ hori_scale_fp, (1)
  • mvX and mvY are the original luma motion vector but now should be examined with 1/32-pel accuracy.
  • x c ′ and y c ′ are further scaled down to keep 1/1024 pel accuracy
  • tile_group_temporal_mvp_enabled_flag When tile_group_temporal_mvp_enabled_flag is equal to 1, the current picture and its collocated picture shall have the same size.
  • decoder motion vector refinement shall be turned off.
  • sps_bdof_enabled_flag shall be equal to 0.
  • CTB Coding Tree Block
  • ALF Adaptive Loop Filter
  • Adaptive parameter set was adopted in VTM4.
  • Each APS contains one set of signalled ALF filters, up to 32 APSs are supported.
  • slice-level temporal filter is tested.
  • a tile group can re-use the ALF information from an APS to reduce the overhead.
  • the APSs are updated as a first-in-first-out (FIFO) buffer.
  • For luma component when ALF is applied to a luma CTB, the choice among 16 fixed, 5 temporal or 1 signaled filter sets is indicated. Only the filter set index is signalled. For one slice, only one new set of 25 filters can be signaled. If a new set is signalled for a slice, all the luma CTBs in the same slice share that set. Fixed filter sets can be used to predict the new slice-level filter set and can be used as candidate filter sets for a luma CTB as well. The number of filters is 64 in total.
  • chroma component when ALF is applied to a chroma CTB, if a new filter is signalled for a slice, the CTB used the new filter, otherwise, the most recent temporal chroma filter satisfying the temporal scalability constrain is applied.
  • the APSs are updated as a first-in-first-out (FIFO) buffer.
  • the motion vectors temporal motion vector prediction is modified by fetching multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU.
  • the sub-CUs are square N ⁇ N blocks (N is set to 8 by default) .
  • ATMVP predicts the motion vectors of the sub-CUs within a CU in two steps. The first step is to identify the corresponding block in a reference picture with a so-called temporal vector. The reference picture is called the motion source picture. The second step is to split the current CU into sub-CUs and obtain the motion vectors as well as the reference indices of each sub-CU from the block corresponding to each sub-CU, as shown in FIG. 15.
  • a reference picture and the corresponding block is determined by the motion information of the spatial neighbouring blocks of the current CU.
  • the merge candidate from block A0 (the left block) in the merge candidate list of the current CU is used.
  • the first available motion vector from block A0 referring to the collocated reference picture are set to be the temporal vector. This way, in ATMVP, the corresponding block may be more accurately identified, compared with TMVP, wherein the corresponding block (sometimes called collocated block) is always in a bottom-right or center position relative to the current CU.
  • a corresponding block of the sub-CU is identified by the temporal vector in the motion source picture, by adding to the coordinate of the current CU the temporal vector.
  • the motion information of its corresponding block (the smallest motion grid that covers the center sample) is used to derive the motion information for the sub-CU.
  • the motion information of a corresponding N ⁇ N block is identified, it is converted to the motion vectors and reference indices of the current sub-CU, in the same way as TMVP of HEVC, wherein motion scaling and other procedures apply.
  • HEVC high definition motion model
  • MCP motion compensation prediction
  • a simplified affine transform motion compensation prediction is applied with 4-parameter affine model and 6-parameter affine model.
  • FIGS. 16A-16B the affine motion field of the block is described by two control point motion vectors (CPMVs) for the 4-parameter affine model and 3 CPMVs for the 6-parameter affine model.
  • CPMVs control point motion vectors
  • the motion vector field (MVF) of a block is described by the following equations with the 4-parameter affine model (wherein the 4-parameter are defined as the variables a, b, e and f) in equation (1) and 6-parameter affine model (wherein the 4-parameter are defined as the variables a, b, c, d, e and f) in equation (2) respectively:
  • control point motion vectors (CPMV)
  • (x, y) represents the coordinate of a representative point relative to the top-left sample within current block
  • (mv h (x, y) , mv v (x, y) ) is the motion vector derived for a sample located at (x, y) .
  • the CP motion vectors may be signaled (like in the affine AMVP mode) or derived on-the-fly (like in the affine merge mode) .
  • w and h are the width and height of the current block.
  • the division is implemented by right-shift with a rounding operation.
  • the representative point is defined to be the center position of a sub-block, e.g., when the coordinate of the left-top corner of a sub-block relative to the top-left sample within current block is (xs, ys) , the coordinate of the representative point is defined to be (xs+2, ys+2) .
  • the representative point is utilized to derive the motion vector for the whole sub-block.
  • sub-block based affine transform prediction is applied.
  • the motion vector of the center sample of each sub-block is calculated according to Equation (1) and (2) , and rounded to 1/16 fraction accuracy.
  • the motion compensation interpolation filters for 1/16-pel are applied to generate the prediction of each sub-block with derived motion vector.
  • the interpolation filters for 1/16-pel are introduced by the affine mode.
  • the high accuracy motion vector of each sub-block is rounded and saved as the same accuracy as the normal motion vector.
  • AFFINE_INTER Similar to the translational motion model, there are also two modes for signaling the side information due affine prediction. They are AFFINE_INTER and AFFINE_MERGE modes.
  • AF_INTER mode can be applied.
  • An affine flag in CU level is signalled in the bitstream to indicate whether AF_INTER mode is used.
  • an affine AMVP candidate list is constructed with three types of affine motion predictors in the following order, wherein each candidate includes the estimated CPMVs of the current block.
  • the differences of the best CPMVs found at the encoder side (such as mv 0 mv 1 mv 2 in FIG. 18) and the estimated CPMVs are signalled.
  • the index of affine AMVP candidate from which the estimated CPMVs are derived is further signalled.
  • the checking order is similar to that of spatial MVPs in HEVC AMVP list construction.
  • a left inherited affine motion predictor is derived from the first block in ⁇ A1, A0 ⁇ that is affine coded and has the same reference picture as in current block.
  • an above inherited affine motion predictor is derived from the first block in ⁇ B1, B0, B2 ⁇ that is affine coded and has the same reference picture as in current block.
  • the five blocks A1, A0, B1, B0, B2 are depicted in FIG. 19.
  • the CPMVs of the coding unit covering the neighboring block are used to derive predictors of CPMVs of current block. For example, if A1 is coded with non-affine mode and A0 is coded with 4-parameter affine mode, the left inherited affine MV predictor will be derived from A0. In this case, the CPMVs of a CU covering A0, as denoted by for the top-left CPMV and for the top-right CPMV in FIG.
  • 21B are utilized to derive the estimated CPMVs of current block, denoted by for the top-left (with coordinate (x0, y0) ) , top-right (with coordinate (x1, y1) ) and bottom-right positions (with coordinate (x2, y2) ) of current block.
  • a constructed affine motion predictor consists of control-point motion vectors (CPMVs) that are derived from neighboring inter coded blocks, as shown in FIG. 20 that have the same reference picture.
  • CPMVs control-point motion vectors
  • the number of CPMVs is 2, otherwise if the current affine motion model is 6-parameter affine, the number of CPMVs is 3.
  • the top-left CPMV is derived by the MV at the first block in the group ⁇ A, B, C ⁇ that is inter coded and has the same reference picture as in current block.
  • the top-right CPMV is derived by the MV at the first block in the group ⁇ D, E ⁇ that is inter coded and has the same reference picture as in current block.
  • the bottom-left CPMV is derived by the MV at the first block in the group ⁇ F, G ⁇ that is inter coded and has the same reference picture as in current block.
  • a constructed affine motion predictor is inserted into the candidate list only if both and are founded, that is, and are used as the estimated CPMVs for top-left (with coordinate (x0, y0) ) , top-right (with coordinate (x1, y1) ) positions of current block.
  • a constructed affine motion predictor is inserted into the candidate list only if and are all founded, that is, and are used as the estimated CPMVs for top-left (with coordinate (x0, y0) ) , top-right (with coordinate (x1, y1) ) and bottom-right (with coordinate (x2, y2) ) positions of current block.
  • MVD In AF_INTER mode, when 4/6-parameter affine mode is used, 2/3 control points are required, and therefore 2/3 MVD needs to be coded for these control points, as shown in FIGS. 18A and 18B.
  • JVET-K0337 it is proposed to derive the MV as follows, i.e., mvd 1 and mvd 2 are predicted from mvd 0 .
  • two motion vectors e.g., mvA (xA, yA) and mvB (xB, yB)
  • newMV mvA + mvB and the two components of newMV is set to (xA + xB) and (yA + yB) , respectively.
  • a CU When a CU is applied in AF_MERGE mode, it gets the first block coded with affine mode from the valid neighbour reconstructed blocks. And the selection order for the candidate block is from left, above, above right, left bottom to above left as shown in FIG. 21A (denoted by A, B, C, D, E in order) .
  • the neighbour left bottom block is coded in affine mode as denoted by A0 in FIG. 21B
  • the Control Point (CP) motion vectors mv 0 N , mv 1 N and mv 2 N of the top left corner, above right corner and left bottom corner of the neighbouring CU/PU which contains the block A are fetched.
  • the motion vector mv 0 C , mv 1 C and mv 2 C (which is only used for the 6-parameter affine model) of the top left corner/top right/bottom left on the current CU/PU is calculated based on mv 0 N , mv 1 N and mv 2 N .
  • sub-block e.g. 4 ⁇ 4 block in VTM located at the top-left corner stores mv0
  • the sub-block located at the top-right corner stores mv1 if the current block is affine coded.
  • the sub-block located at the bottom-left corner stores mv2; otherwise (with the 4-parameter affine model) , LB stores mv2’.
  • Other sub-blocks stores the MVs used for MC.
  • the MVF of the current CU is generated.
  • an affine flag is signalled in the bitstream when there is at least one neighbour block is coded in affine mode.
  • an affine merge candidate list is constructed with following steps:
  • Inherited affine candidate means that the candidate is derived from the affine motion model of its valid neighbor affine coded block.
  • the maximum two inherited affine candidates are derived from affine motion model of the neighboring blocks and inserted into the candidate list.
  • the scan order is ⁇ A0, A1 ⁇ ; for the above predictor, the scan order is ⁇ B0, B1, B2 ⁇ .
  • Constructed affine candidate means the candidate is constructed by combining the neighbor motion information of each control point.
  • T is temporal position for predicting CP4.
  • the coordinates of CP1, CP2, CP3 and CP4 is (0, 0) , (W, 0) , (H, 0) and (W, H) , respectively, where W and H are the width and height of current block.
  • the motion information of each control point is obtained according to the following priority order:
  • the checking priority is B2->B3->A2.
  • B2 is used if it is available. Otherwise, if B2 is available, B3 is used. If both B2 and B3 are unavailable, A2 is used. If all the three candidates are unavailable, the motion information of CP1 cannot be obtained.
  • the checking priority is B1->B0.
  • the checking priority is A1->A0.
  • T is used.
  • the three control points can be selected from one of the following four combinations ( ⁇ CP1, CP2, CP4 ⁇ , ⁇ CP1, CP2, CP3 ⁇ , ⁇ CP2, CP3, CP4 ⁇ , ⁇ CP1, CP3, CP4 ⁇ ) .
  • Combinations ⁇ CP1, CP2, CP3 ⁇ , ⁇ CP2, CP3, CP4 ⁇ , ⁇ CP1, CP3, CP4 ⁇ will be converted to a 6-parameter motion model represented by top-left, top-right and bottom-left control points.
  • Motion information of two control points are needed to construct a 4-parameter affine candidate.
  • the two control points can be selected from one of the two combinations ( ⁇ CP1, CP2 ⁇ , ⁇ CP1, CP3 ⁇ ) .
  • the two combinations will be converted to a 4-parameter motion model represented by top-left and top-right control points.
  • the reference indices of list X for each CP are checked, if they are all the same, then this combination has valid CPMVs for list X. If the combination does not have valid CPMVs for both list 0 and list 1, then this combination is marked as invalid. Otherwise, it is valid, and the CPMVs are put into the sub-block merge list.
  • ARC When applied in VVC, ARC may have the following problems:
  • bit-depth and color format such as 4: 0: 0, 4: 2: 0 or 4: 4: 4) may also be changed from one picture to another picture in one sequence.
  • Shift (x, n) (x+ offset0) >>n.
  • offset0 and/or offset1 are set to (1 ⁇ n) >>1 or (1 ⁇ (n-1) ) . In another example, offset0 and/or offset1 are set to 0.
  • Clip3 (min, max, x) is defined as
  • Floor (x) is defined as the largest integer less than or equal to x.
  • Ceil (x) the smallest integer greater than or equal to x.
  • Log2 (x) is defined as the base-2 logarithm of x.
  • Whether to enable DMVR/BIO or other kinds of motion derivation/refinement at the decoder side may depend on whether the two reference pictures are in the same resolution or not.
  • motion derivation/refinement such as DMVR/BIO is disabled.
  • how to apply motion derivation/refinement at the decoder side may depend on whether the two reference pictures are in the same resolution or not.
  • the allowed MVDs associated with each reference picture may be scaled, e.g., according to resolutions.
  • all reference pictures stored in the decoded picture buffer are in the same resolution (denoted as a first resolution) , such as the maximumly/minimally allowed resolution.
  • the samples in a decoded picture may be firstly modified, (e.g., via up-sampling or down-sampling) , before being stored in the decoded picture buffer.
  • the modification may be according to the first resolution.
  • the modification may be according to the resolution for the reference picture, and that for the current picture.
  • all reference pictures stored in the decoded picture buffer may be in the resolution that picture has been coded with.
  • conversion of reference samples may be firstly applied (e.g., via up-sampling or down-sampling) , before invoking the motion compensation process.
  • the motion compensation (MC) may be done directly using reference samples in the reference pictures.
  • the prediction block generated from the MC process may be further modified (e.g., via up-sampling or down-sampling) , and the final prediction block for current block may depend on the modified prediction block.
  • ABS Adaptive Bit-depth Conversion
  • one or multiple sets of sample bit-depths for one or multiple components may be signaled in a video unit such as DPS, VPS, SPS, PPS, APS, picture header, slice header, tile group header.
  • one or multiple sets of sample bit-depths for one or multiple components may be signaled in a Supplemental Enhancement Information (SEI) message.
  • SEI Supplemental Enhancement Information
  • one or multiple sets of sample bit-depths for one or multiple components may be signaled in an individual video unit for Adaptive bit-depth conversion.
  • a set of sample bit-depths for one or multiple components may be coupled the dimensions of the picture.
  • one or multiple combinations of sample bit-depths for one or multiple components and the corresponding dimensions/down sampling ratios/up sampling ratios of the picture may be signaled in the same video unit.
  • indication of the ABCMaxBD and/or ABCMinBD may be signaled.
  • differences of other bit-depth values compared to ABCMaxBD and/or ABCMinBD may be signaled.
  • all reference pictures stored in the decoded picture buffer are in the same bit-depth (denoted as the a first bit-depth) , such as ABCMaxBD [i] or ABCMinBD [i] (with i being 0.. 2 indicating the color component indices) .
  • the samples in a decoded picture may be firstly modified, via left-shift or right-shift, before being stored in the decoded picture buffer.
  • the modification may be according to the first bit-depth.
  • the modification may be according to the defined bit-depth for reference picture, and that for current picture.
  • all reference pictures stored in the decoded picture buffer are in the bit-depth of what has been coded with.
  • conversion of reference samples may be firstly applied, before invoking the motion compensation process.
  • the motion compensation is done directly using reference samples.
  • the prediction block generated from the MC process may be further modified (e.g., via shifting) , and the final prediction block for current block may depend on the modified prediction block.
  • S’ S ⁇ (BD0-BD1) + (1 ⁇ (BD0-BD1-1) ) .
  • the reference picture with samples not converted may be removed after a reference picture is converted from it.
  • the reference picture with samples not converted may be kept but marked as unavailable after a reference picture is converted from it.
  • the reference picture with samples not converted may be put in the reference picture list.
  • the reference samples are converted when they are used to in the inter-prediction.
  • ARC is conducted first, then ABC is conducted.
  • samples are up-sampled/down-sampled according to different picture dimensions first, following by left-shifted/right-shifted according to different bit-depths in the ARC+ABC conversion.
  • ABC is conducted first, then ARC is conducted.
  • samples are left-shifted/right-shifted according to different bit-depths first, following by up-sampled/down-sampled according to different picture dimensions in the ABC+ARC conversion.
  • a merge candidate referring to a reference picture with a higher sample bit-depth may have a higher priority than a merge candidate referring to a reference picture with a lower bit-depth.
  • a merge candidate referring to a reference picture with a higher sample bit-depth may be put before a merge candidate referring to a reference picture with a lower sample bit-depth in the merge candidate list.
  • a motion vector referring to a reference picture with a sample bit-depth lower than the sample bit-depth of the current picture cannot be in the merge candidate list.
  • ALF parameters signaled in a video unit such as APS may be associated with one or multiple sample bit-depths.
  • a video unit such as APS signaling ALF parameters may be associated with one or multiple sample bit-depths.
  • a picture may only apply ALF parameters signaled in a video unit such as APS, associated with the same sample bit-depth.
  • a picture may use ALF parameters associated with a different sample bit-depth.
  • ALF parameters associated with a first corresponding sample bit-depth may inherit or be predicted from ALF parameters associated with a second corresponding sample bit-depth.
  • the first corresponding sample bit-depth must be the same as the second corresponding sample bit-depth.
  • the first corresponding sample bit-depth may be different to the second corresponding sample bit-depth.
  • LMCS parameters signaled in a video unit such as APS may be associated with one or multiple sample bit-depths.
  • a video unit such as APS signaling LMCS parameters may be associated with one or multiple sample bit-depths.
  • a picture may only apply LMCS parameters signaled in a video unit such as APS, associated with the same sample bit-depth.
  • LMCS parameters associated with a first corresponding sample bit-depth may inherit or be predicted from LMCS parameters associated with a second corresponding sample bit-depth.
  • the first corresponding sample bit-depth must be the same with the second corresponding sample bit-depth.
  • the first corresponding sample bit-depth may be different to the second corresponding sample bit-depth.
  • coding tool X may be disabled for a block if the block refers to at least one reference picture with a different sample bit-depth to the current picture.
  • the information related to the coding tool X may not be signaled.
  • the block cannot refer to a reference picture with a different sample bit-depth to the current picture.
  • a merge candidate referring to a reference picture with a different sample bit-depth to the current picture may be skipped or not put into the merge candidate list.
  • the reference index corresponds to a reference picture with a different sample bit-depth to the current picture may be skipped or not allowed to be signaled.
  • the coding tool X may be anyone below.
  • color format may refer to 4: 4: 4, 4: 2: 2, 4: 2: 0 or 4: 0: 0,
  • color format may refer to YCbCr or RGB
  • a video unit such as DPS, VPS, SPS, PPS, APS, picture header, slice header, tile group header.
  • one or multiple color formats may be signaled in a Supplemental Enhancement Information (SEI) message.
  • SEI Supplemental Enhancement Information
  • one or multiple color formats may be signaled in an individual video unit for ACC.
  • a color format may be coupled the dimensions and/or sample bit-depth of the picture.
  • one or multiple combinations of color formats, and/or sample bit-depth for one or multiple components and/or the corresponding dimensions of the picture may be signaled in the same video unit.
  • pictures with different color format are disallowed to be put in a reference picture list for a block in current picture.
  • the reference samples may be converted accordingly.
  • samples of chroma components in the reference picture may be up-sampled vertically by a ratio of 1: 2.
  • samples of chroma components in the reference picture may be up-sampled horizontally by a ratio of 1: 2.
  • samples of chroma components in the reference picture may be up-sampled vertically by a ratio of 1: 2 and up-sampled horizontally by a ratio of 1: 2.
  • samples of chroma components in the reference picture may be down-sampled vertically by a ratio of 1: 2.
  • samples of chroma components in the reference picture may be down-sampled horizontally by a ratio of 1: 2.
  • samples of chroma components in the reference picture may be down-sampled vertically by a ratio of 1: 2 and down-sampled horizontally by a ratio of 1: 2.
  • samples of the luma component in the reference picture may be used to perform inter-prediction to the current picture.
  • samples of the luma component in the reference picture may be used to perform inter-prediction to the current picture.
  • the reference picture with samples not converted may be removed after a reference picture is converted from it.
  • the reference picture with samples not converted may be kept but marked as unavailable after a reference picture is converted from it.
  • the reference picture with samples not converted may be put in the reference picture list.
  • the reference samples are converted when they are used to in the inter-prediction.
  • ARC is conducted first, then ACC is conducted.
  • the samples are first down-sampled/up-sampled according to the different picture dimensions, following by down-sampled/up-sampled according to the different color formats in the ARC+ACC conversion.
  • ACC is conducted first, then ARC is conducted.
  • the samples are first down-sampled/up-sampled according to the different color formats, following by down-sampled/up-sampled according to the different picture dimensions in the ARC+ACC conversion.
  • ACC and ARC may be conducted together.
  • the samples are down-sampled/up-sampled according to the scaling ratio derived from different color formats and different picture dimensions in the ARC+ACC or ACC+ARC conversion.
  • ACC is conducted first, then ABC is conducted.
  • samples are up-sampled/down-sampled according to the different color formats first, following by left-shifted/right-shifted according to different bit-depths in the ACC+ABC conversion.
  • ABC is conducted first, then ACC is conducted.
  • samples are left-shifted/right-shifted according to different bit-depths first, following by up-sampled/down-sampled according to the different color formats in the ABC+ACC conversion.
  • ALF parameters signaled in a video unit such as APS may be associated with one or multiple color formats.
  • a video unit such as APS signaling ALF parameters may be associated with one or multiple color formats.
  • a picture may only apply ALF parameters signaled in a video unit such as APS, associated with the same color format.
  • ALF parameters associated with a first corresponding color format may inherit or be predicted from ALF parameters associated with a second corresponding color format.
  • the first corresponding color format must be the same as the second corresponding color format.
  • the first corresponding color format may be different to the second corresponding color format.
  • different default ALF parameters may be designed for YCbCr and RGB format.
  • different default ALF parameters may be designed for 4: 4: 4, 4: 2: 2, 4: 2: 0, 4: 0: 0 format.
  • LMCS parameters signaled in a video unit such as APS may be associated with one or multiple color formats.
  • a video unit such as APS signaling LMCS parameters may be associated with one or multiple color formats.
  • a picture may only apply LMCS parameters signaled in a video unit such as APS, associated with the same color format.
  • LMCS parameters associated with a first corresponding sample color format may inherit or be predicted from LMCS parameters associated with a second corresponding color format.
  • the first corresponding color format must be the same with the second corresponding color format.
  • the first corresponding color format may be different to the second corresponding color format.
  • the indication of whether the chroma residue scaling is applied may not be signaled and inferred to be “not used” if the color format is 4: 0: 0.
  • the indication of whether the chroma residue scaling is applied must be “not used” in a conformance bit-stream if the color format is 4: 0: 0.
  • the signaled indication of whether the chroma residue scaling is applied is ignored and set to be “not used” by the decoder if the color format is 4: 0: 0.
  • coding tool X may be disabled for a block if the block refers to at least one reference picture with a different color format to the current picture.
  • the information related to the coding tool X may not be signaled.
  • the block cannot refer to a reference picture with a different color format to the current picture.
  • a merge candidate referring to a reference picture with a different color format to the current picture may be skipped or not put into the merge candidate list.
  • the reference index corresponds to a reference picture with a different color format to the current picture may be skipped or not allowed to be signaled.
  • the coding tool X may be anyone below.
  • methods 2400-3100 may be implemented at a video decoder or a video encoder.
  • FIG. 24 shows a flowchart of an exemplary method for video processing.
  • the method 2400 includes, at step 2402, applying an adaptive color format conversion (ACC) process to a video region within a current picture, wherein a set of color formats is applicable to one or more color components of the video region, and the set of color formats is signaled in a specific video unit; and at step 2404, performing a conversion between the video region and a bitstream representation of the current video region based on the adaptive color format conversion (ACC) .
  • ACC adaptive color format conversion
  • FIG. 25 shows a flowchart of an exemplary method for video processing.
  • the method 2500 includes, at step 2502, determining, for a video region within a current picture, a relationship between a color format of a reference picture to which the video region refers and that of the current picture; at step 2504, performing, in response to the relationship, a specific operation for samples within the reference picture during an adaptive color format conversion (ACC) process in which a set of color formats is applicable; and at step 2506, performing a conversion between a bitstream representation of the video region and the video region based on the specific operation.
  • ACC adaptive color format conversion
  • FIG. 26 shows a flowchart of an exemplary method for video processing.
  • the method 2600 includes, at step 2602, applying a first adaptive conversion process for a video region within a current picture; at step 2604, applying a second adaptive conversion process for the video region; and at step 2606, performing a conversion between a bitstream representation of the video region and the video region based on the first adaptive conversion process and the second adaptive conversion process; wherein the first adaptive conversion process is one of an adaptive color format conversion (ACC) process and an adaptive resolution change (ARC) process, and the second adaptive conversion process is the other of the adaptive color format conversion (ACC) process and the adaptive resolution change (ARC) process.
  • ACC adaptive color format conversion
  • ARC adaptive resolution change
  • FIG. 27 shows a flowchart of an exemplary method for video processing.
  • the method 2700 includes, at step 2702, applying a first adaptive conversion process for a video region within a current picture; at step 2704, applying a second adaptive conversion process for the video region; and at step 2706, performing a conversion between a bitstream representation of the video region and the video region based on the first adaptive conversion process and the second adaptive conversion process; wherein the first adaptive conversion process is one of an adaptive color format conversion (ACC) process and an adaptive bit-depth conversion (ABC) process, and the second adaptive conversion process is the other of the adaptive color format conversion (ACC) process and the adaptive bit-depth conversion (ABC) process.
  • ACC adaptive color format conversion
  • ABSC adaptive bit-depth conversion
  • FIG. 28 shows a flowchart of an exemplary method for video processing.
  • the method 2800 includes, at step 2802, determining a plurality of ALF parameters; and at step 2804, applying, based on the plurality of ALF parameters, an adaptive loop filtering (ALF) process to samples within a current picture; wherein a plurality of ALF parameters is applicable to the samples based on a corresponding color format to which the plurality of ALF parameters is associated.
  • ALF adaptive loop filtering
  • FIG. 29 shows a flowchart of an exemplary method for video processing.
  • the method 2900 includes, at step 2902, determining a plurality of LMCS parameters; and at step 2904, applying, based on the plurality of LMCS parameters, a luma mapping with chroma scaling (LMCS) process to samples within a current picture; wherein the plurality of LMCS parameters is applicable to the samples based on a corresponding color format to which the plurality of LMCS parameters is associated.
  • LMCS luma mapping with chroma scaling
  • FIG. 30 shows a flowchart of an exemplary method for video processing.
  • the method 3000 includes, at step 3002, determining whether a chroma residue scaling process is applied in a LMCS process based on a color format associated with a current picture; and at step 3004, applying the luma mapping with chroma scaling (LMCS) process to samples within a current picture based on the determining.
  • LMCS luma mapping with chroma scaling
  • FIG. 31 shows a flowchart of an exemplary method for video processing.
  • the method 3100 includes, at step 3102, determining whether and/or how to perform a specific coding tool for a video block within a current picture based on a relationship between a color format of at least one of a plurality of reference pictures to which the video block refers and that of the current picture; and at step 3104, performing a conversion, based on the specific coding tool, between a bitstream representation of the video block and the video block.
  • FIG. 32 shows a flowchart of an exemplary method for video processing.
  • the method 3200 includes, at step 3202, performing a conversion between a bitstream representation of a current video block and the current video block, wherein if two reference pictures to which the current video block refers have different color formats, a bi-prediction from the two reference pictures is disabled for the video block during the conversion.
  • a method for video processing comprising: applying an adaptive color format conversion (ACC) process to a video region within a current picture, wherein a set of color formats is applicable to one or more color components of the video region, and the set of color formats is signaled in a specific video unit; and performing a conversion between the video region and a bitstream representation of the current video region based on the adaptive color format conversion (ACC) .
  • ACC adaptive color format conversion
  • the specific video unit comprises at least one of a decoder parameter set (DPS) , a video parameter set (VPS) , a sequence parameter set (SPS) , a picture parameter set (PPS) , an adaptive parameter set (APS) , a picture header, a slice header and a tile group header.
  • DPS decoder parameter set
  • VPS video parameter set
  • SPS sequence parameter set
  • PPS picture parameter set
  • APS adaptive parameter set
  • the specific video unit comprises a Supplemental Enhancement Information (SEI) message.
  • SEI Supplemental Enhancement Information
  • the specific video unit comprises an individual video unit for the adaptive color format conversion (ACC) .
  • ACC adaptive color format conversion
  • the set of color formats is coupled to at least one of corresponding dimensional information of the current picture and a set of sampling bit-depths for one or more color components of the current picture.
  • At least one of corresponding dimensional information of the current picture and the set of sampling bit-depths for one or more color components is signaled in a same specific video unit as the set of color formats.
  • the set of color formats comprises at least two of 4: 4: 4, 4: 2: 2, 4: 2: 0 and 4: 0: 0 color formats.
  • the set of color formats comprises YCbCr and RGB color formats.
  • more than one combinations are configured from the set of color formats, the set of sampling bit-depths and corresponding dimensional information, and any two of the more than one combinations are different.
  • the video region is a video block.
  • a method for video processing comprising:
  • the reference picture is not allowed for a motion prediction for the video region.
  • the reference picture is excluded from a reference picture list for the video region.
  • the specific operation comprises at least one of an up-sampling conversion and a down-sampling conversion on chroma samples within the reference picture based on different color formats between the reference picture and the current picture to generate a converted reference picture.
  • the chroma samples within the reference picture are up-sampled vertically by a ratio of 1: 2.
  • the chroma samples within the reference picture are up-sampled horizontally by a ratio of 1: 2.
  • the chroma samples within the reference picture are up-sampled vertically and horizontally by a ratio of 1: 2 respectively.
  • the chroma samples within the reference picture are down-sampled vertically by a ratio of 1: 2.
  • the chroma samples within the reference picture are down-sampled horizontally by a ratio of 1: 2.
  • the chroma samples within the reference picture are down-sampled vertically and horizontally by a ratio of 1: 2 respectively.
  • the specific operation comprises determining whether luma samples within the reference picture are allowed for performing an inter-prediction for the video region.
  • luma samples within the reference picture are allowed for performing the inter-prediction.
  • the reference picture is removed.
  • the reference picture is kept and marked as unavailable.
  • the specific operation comprises:
  • the video region is a video block.
  • a method for video processing comprising:
  • first adaptive conversion process is one of an adaptive color format conversion (ACC) process and an adaptive resolution change (ARC) process
  • second adaptive conversion process is the other of the adaptive color format conversion (ACC) process and the adaptive resolution change (ARC) process.
  • the adaptive resolution change (ARC) process is performed before or after the adaptive color format conversion (ACC) process.
  • the adaptive resolution change (ARC) process comprises:
  • the adaptive color format conversion (ACC) process comprises:
  • the modification comprises at least one of an up-sampling or a down-sampling.
  • the video region is a video block.
  • a method for video processing comprising:
  • the first adaptive conversion process is one of an adaptive color format conversion (ACC) process and an adaptive bit-depth conversion (ABC) process
  • the second adaptive conversion process is the other of the adaptive color format conversion (ACC) process and the adaptive bit-depth conversion (ABC) process.
  • the adaptive color format conversion (ACC) process is performed before or after the adaptive bit-depth conversion (ABC) process.
  • the adaptive bit-depth conversion (ABC) process comprises:
  • the modification comprises at least one of a left-shift or a right-shift.
  • the adaptive color format conversion (ACC) process comprises:
  • the modification comprises at least one of an up-sampling or a down-sampling.
  • a method for video processing comprising:
  • ALF adaptive loop filtering
  • the plurality of ALF parameters is signaled in a specific video unit.
  • the specific video unit comprises a adaptive parameter set (APS) .
  • APS adaptive parameter set
  • the specific video unit is associated with one or more color formats.
  • the plurality of ALF parameters is associated with one or more color formats.
  • ones from the plurality of ALF parameters which are associated with different color formats are applicable to the samples.
  • the plurality of ALF parameters is divided into at least a first subset and a second subset which are associated with a first color format and a second color format respectively, and the first subset of the ALF parameters are inherited or predicted from the second subset of the ALF parameters.
  • the first color format is identical to the second color format.
  • the first color format is different from the second color format.
  • the plurality of ALF parameters comprises default ALF parameters, wherein different default ALF parameters are designed for different color formats.
  • the different color formats comprise YCbCr and RGB color formats.
  • the different color formats comprise at least two of 4: 4: 4, 4: 2: 2, 4: 2: 0, 4: 0: 0 color formats.
  • a method for video processing comprising:
  • LMCS luma mapping with chroma scaling
  • the plurality of LMCS parameters is signaled in a specific video unit.
  • the specific video unit comprises a adaptive parameter set (APS) .
  • APS adaptive parameter set
  • the specific video unit is associated with one or more color formats.
  • the plurality of LMCS parameters is associated with one or more color formats.
  • the plurality of LMCS parameters is divided into at least a first subset and a second subset which are associated with a first color format and a second color format respectively, and the first subset of the LMCS parameters are inherited or predicted from the second subset of the LMCS parameters.
  • the first color format is identical to the second color format.
  • the first color format is different from the second color format.
  • a method for video processing comprising:
  • LMCS luma mapping with chroma scaling
  • no indication is signaled for indicating that whether the chroma residual scaling process is applied.
  • an indication is signaled for indicating that no chroma residual scaling process is applied.
  • an indication which is signaled for indicating that whether the chroma residual scaling process is applied, is ignored and is inferred that no chroma residual scaling process is applied.
  • a method for video processing comprising:
  • the specific coding tool is disable for the video block if it is determined that the color format of the at least one of the plurality of reference pictures to which the video block refers is different from that of the current picture.
  • information on the specific coding tool is not signaled.
  • the specific coding tool is enabled for the video block if the plurality of reference pictures to which the video block refers excludes a reference picture which has a different color format from that of the current picture.
  • a reference index which indicates the reference picture which has a different color format from that of the current picture, is not signaled or skipped for the video block.
  • the conversion between the bitstream representation of the video block and the video block is based on at least one of a plurality of merge candidates arranged in a merge candidate list, and the merge candidate list excludes a merge candidate which refers to a reference picture whose color format is different from that of the current picture.
  • the conversion between the bitstream representation of the video block and the video block is based on at least one of a plurality of merge candidates arranged in a merge candidate list, and any merge candidate in the merge candidate list, which refers to a reference picture whose color format is different from that of the current picture, is skipped for the video block.
  • the specific coding tool is selected from a group comprising at least one of: an adaptive loop filtering (ALF) tool, a luma mapping with chroma scaling (LMCS) tool, a decoder-side motion vector refinement (DMVR) tool, a bi-directional optical flow (BDOF) tool, an affine prediction tool, a triangular partitioning mode (TPM) tool, a symmetric motion vector difference (SMVD) tool, a merge with motion vector difference (MMVD) tool, an inter-inter prediction tool in versatile visual coding (VVC) , a local illumination compensation (LIC) tool, a history-based motion vector prediction (HMVP) tool, a multiple transform set (MTS) tool and a sub-block transform (SBT) tool.
  • ALF adaptive loop filtering
  • LMCS luma mapping with chroma scaling
  • DMVR decoder-side motion vector refinement
  • BDOF bi-directional optical flow
  • affine prediction tool affine prediction tool
  • TPM
  • a method for video processing comprising:
  • the conversion includes encoding the video block into the bitstream representation of the video region and decoding the video block from the bitstream representation of the video region.
  • an apparatus in a video system comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to implement the method as described above.
  • a non-transitory computer readable media having program code stored thereupon, the program code, when executed, causing a processor to implement the the method as described above.
  • FIG. 23 is a block diagram of a video processing apparatus 2300.
  • the apparatus 2300 may be used to implement one or more of the methods described herein.
  • the apparatus 2300 may be embodied in a smartphone, tablet, computer, Internet of Things (IoT) receiver, and so on.
  • the apparatus 2300 may include one or more processors 2302, one or more memories 2304 and video processing hardware 2306.
  • the processor (s) 2302 may be configured to implement one or more methods (including, but not limited to, method 2300) described in the present document.
  • the memory (memories) 2304 may be used for storing data and code used for implementing the methods and techniques described herein.
  • the video processing hardware 2306 may be used to implement, in hardware circuitry, some techniques described in the present document.
  • the video coding methods may be implemented using an apparatus that is implemented on a hardware platform as described with respect to FIG. 23.
  • Implementations of the subject matter and the functional operations described in this patent document can be implemented in various systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer readable medium for execution by, or to control the operation of, data processing apparatus.
  • the computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them.
  • data processing unit or “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.
  • the apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
  • a computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
  • a computer program does not necessarily correspond to a file in a file system.
  • a program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document) , in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code) .
  • a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
  • the processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output.
  • the processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit) .
  • processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer.
  • a processor will receive instructions and data from a read only memory or a random access memory or both.
  • the essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data.
  • a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • a computer need not have such devices.
  • Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices.
  • semiconductor memory devices e.g., EPROM, EEPROM, and flash memory devices.
  • the processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
PCT/CN2020/090801 2019-05-16 2020-05-18 Adaptive color-format conversion in video coding WO2020228835A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202080036227.5A CN113875232A (zh) 2019-05-16 2020-05-18 视频编解码中的自适应颜色格式转换

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN2019087209 2019-05-16
CNPCT/CN2019/087209 2019-05-16

Publications (1)

Publication Number Publication Date
WO2020228835A1 true WO2020228835A1 (en) 2020-11-19

Family

ID=73289350

Family Applications (3)

Application Number Title Priority Date Filing Date
PCT/CN2020/090799 WO2020228833A1 (en) 2019-05-16 2020-05-18 Adaptive resolution change in video coding
PCT/CN2020/090801 WO2020228835A1 (en) 2019-05-16 2020-05-18 Adaptive color-format conversion in video coding
PCT/CN2020/090800 WO2020228834A1 (en) 2019-05-16 2020-05-18 Adaptive bit-depth conversion in video coding

Family Applications Before (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/090799 WO2020228833A1 (en) 2019-05-16 2020-05-18 Adaptive resolution change in video coding

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/090800 WO2020228834A1 (en) 2019-05-16 2020-05-18 Adaptive bit-depth conversion in video coding

Country Status (2)

Country Link
CN (3) CN113875232A (zh)
WO (3) WO2020228833A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022268627A1 (en) * 2021-06-24 2022-12-29 Interdigital Vc Holdings France, Sas Methods and apparatuses for encoding/decoding a video
WO2023168166A1 (en) * 2022-03-03 2023-09-07 Qualcomm Incorporated Bandwidth efficient image processing
WO2023171940A1 (ko) * 2022-03-08 2023-09-14 현대자동차주식회사 적응적 크로마컨버전을 이용하는 비디오 코딩을 위한 방법과 장치

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220182643A1 (en) * 2020-12-04 2022-06-09 Ofinno, Llc No Reference Image Quality Assessment Based Decoder Side Inter Prediction
CA3142044A1 (en) * 2020-12-14 2022-06-14 Comcast Cable Communications, Llc Methods and systems for improved content encoding
US20240155114A1 (en) * 2021-02-26 2024-05-09 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Video coding concept allowing for limitation of drift
EP4087238A1 (en) * 2021-05-07 2022-11-09 Panasonic Intellectual Property Corporation of America Encoder, decoder, encoding method, and decoding method

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105264888A (zh) * 2014-03-04 2016-01-20 微软技术许可有限责任公司 用于对色彩空间、色彩采样率和/或比特深度自适应切换的编码策略
CN105308960A (zh) * 2013-05-30 2016-02-03 苹果公司 自适应色彩空间变换编码
CN106105203A (zh) * 2014-03-14 2016-11-09 高通股份有限公司 块自适应颜色空间转换译码
CN108370444A (zh) * 2015-12-21 2018-08-03 汤姆逊许可公司 组合自适应分辨率和内部比特深度增加编码的方法和装置

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9510016B2 (en) * 2008-06-12 2016-11-29 Thomson Licensing Methods and apparatus for video coding and decoding with reduced bit-depth update mode and reduced chroma sampling update mode
KR101356613B1 (ko) * 2009-08-21 2014-02-06 에스케이텔레콤 주식회사 적응적 움직임 벡터 해상도를 이용한 영상 부호화/복호화 방법 및 장치
DK2882190T3 (en) * 2011-04-21 2019-02-18 Hfi Innovation Inc Method and apparatus for improved ring-structured filtration
JP6190103B2 (ja) * 2012-10-29 2017-08-30 キヤノン株式会社 動画像符号化装置、動画像符号化方法およびプログラム
CA2897152C (en) * 2013-01-07 2019-03-05 Kemal Ugur Inter-layer video encoding and decoding with adaptive resolution change at indicated switching points
CN105284116B (zh) * 2013-04-19 2019-05-21 麦克赛尔株式会社 摄像方法和摄像装置
KR20160085891A (ko) * 2013-11-24 2016-07-18 엘지전자 주식회사 적응적 샘플링을 이용한 비디오 신호의 인코딩, 디코딩 방법 및 그 장치
KR101789954B1 (ko) * 2013-12-27 2017-10-25 인텔 코포레이션 차세대 비디오 코딩을 위한 콘텐츠 적응적 이득 보상된 예측
CA2939434C (en) * 2014-03-04 2021-02-16 Microsoft Techology Licensing, Llc Adaptive switching of color spaces, color sampling rates and/or bit depths
JP6368795B2 (ja) * 2014-03-14 2018-08-01 ヴィド スケール インコーポレイテッド Ｒｇｂビデオコーディングエンハンスメントのためのシステムおよび方法
US10448029B2 (en) * 2014-04-17 2019-10-15 Qualcomm Incorporated Signaling bit depth values for 3D color prediction for color gamut scalability
RU2017101574A (ru) * 2014-06-19 2018-07-19 Вид Скейл, Инк. Системы и способы для оптимизации параметра модели в основанном на трехмерном представлении отображении цветов
US10687069B2 (en) * 2014-10-08 2020-06-16 Microsoft Technology Licensing, Llc Adjustments to encoding and decoding when switching color spaces
EP4020983A1 (en) * 2014-12-31 2022-06-29 Nokia Technologies Oy An apparatus, a method and a computer program for video coding and decoding
KR101770300B1 (ko) * 2015-06-09 2017-08-22 삼성전자주식회사 비디오 부호화 방법 및 그 장치, 비디오 복호화 방법 및 그 장치
WO2017052250A1 (ko) * 2015-09-23 2017-03-30 엘지전자(주) 영상의 부호화/복호화 방법 및 이를 위한 장치
US20170105014A1 (en) * 2015-10-08 2017-04-13 Qualcomm Incorporated Luma-driven chroma scaling for high dynamic range and wide color gamut contents
US11750832B2 (en) * 2017-11-02 2023-09-05 Hfi Innovation Inc. Method and apparatus for video coding

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105308960A (zh) * 2013-05-30 2016-02-03 苹果公司 自适应色彩空间变换编码
CN105264888A (zh) * 2014-03-04 2016-01-20 微软技术许可有限责任公司 用于对色彩空间、色彩采样率和/或比特深度自适应切换的编码策略
CN106105203A (zh) * 2014-03-14 2016-11-09 高通股份有限公司 块自适应颜色空间转换译码
CN108370444A (zh) * 2015-12-21 2018-08-03 汤姆逊许可公司 组合自适应分辨率和内部比特深度增加编码的方法和装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
STEPHAN WENGER , BYEONGDOO CHOI , SHAN LIU: "[AHG19] On Signaling of Adaptive Resolution Change", JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/WG 11, no. JVET-N0052, 27 March 2019 (2019-03-27), Geneva, CH, pages 1 - 11, XP030202788 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022268627A1 (en) * 2021-06-24 2022-12-29 Interdigital Vc Holdings France, Sas Methods and apparatuses for encoding/decoding a video
WO2023168166A1 (en) * 2022-03-03 2023-09-07 Qualcomm Incorporated Bandwidth efficient image processing
WO2023171940A1 (ko) * 2022-03-08 2023-09-14 현대자동차주식회사 적응적 크로마컨버전을 이용하는 비디오 코딩을 위한 방법과 장치

Also Published As

Publication number Publication date
CN113826382A (zh) 2021-12-21
WO2020228833A1 (en) 2020-11-19
CN113875232A (zh) 2021-12-31
CN113841395B (zh) 2022-10-25
CN113826382B (zh) 2023-06-20
CN113841395A (zh) 2021-12-24
WO2020228834A1 (en) 2020-11-19

Similar Documents

Publication Publication Date Title
US11575887B2 (en) Selective use of coding tools in video processing
US11671602B2 (en) Signaling for reference picture resampling
WO2020228835A1 (en) Adaptive color-format conversion in video coding
US11689747B2 (en) High level syntax for video coding tools
KR102637881B1 (ko) 정제 비디오 코딩 툴의 사용 및 시그널링
CN114641992B (zh) 参考图片重采样的信令
KR20220070437A (ko) 비디오 코딩 툴의 레벨 기반 시그널링
KR20220061108A (ko) 비디오 코딩에서 레퍼런스 샘플 위치 유도

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20805127

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20805127

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 22/03/2022)