WO2020086421A1 - Video encoding and decoding using block-based in-loop reshaping - Google Patents

Video encoding and decoding using block-based in-loop reshaping Download PDF

Info

Publication number
WO2020086421A1
WO2020086421A1 PCT/US2019/057128 US2019057128W WO2020086421A1 WO 2020086421 A1 WO2020086421 A1 WO 2020086421A1 US 2019057128 W US2019057128 W US 2019057128W WO 2020086421 A1 WO2020086421 A1 WO 2020086421A1
Authority
WO
WIPO (PCT)
Prior art keywords
block
values
samples
mapped
mapping
Prior art date
Application number
PCT/US2019/057128
Other languages
French (fr)
Inventor
Edouard Francois
Christophe Chevance
Franck Hiron
Original Assignee
Interdigital Vc Holdings, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Interdigital Vc Holdings, Inc. filed Critical Interdigital Vc Holdings, Inc.
Publication of WO2020086421A1 publication Critical patent/WO2020086421A1/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/182Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/117Filters, e.g. for pre-processing or post-processing
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136Incoming video signal characteristics or properties
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/184Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/80Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
    • H04N19/82Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation involving filtering within a prediction loop
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/90Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
    • H04N19/98Adaptive-dynamic-range coding [ADRC]

Abstract

Different implementations are described, particularly implementations for video encoding and decoding using a block-adaptive mapping for reshaping. The method for encoding and/or decoding a block of a picture comprises: determining one or more values characterizing samples of a block; modifying the determined values using a mapping function; deriving from the modified values one or more block-dependent parameters of a parametric function that approximates the mapping function; and mapping samples of the block using the parametric function and the derived one or more block-dependent parameters.

Description

VIDEO ENCODING AND DECODING USING BLOCK-BASED IN-LOOP RESHAPING
TECHNICAL FIELD
The disclosure is in the field of video compression, and at least one embodiment relates more specifically to a reshaping method using a block-adaptive mapping applied to a block of a picture.
BACKGROUND
Reshaping, also known as mapping, is a technique that can be used to improve the compression efficiency by adapting the effective bit depth of a video so that areas with higher importance use a higher effective bit depth to preserve the details of a video while areas with lower importance use a lower effective bit depth, potentially loosing some of the details of the video.
SUMMARY
According to an aspect of the present disclosure, a method for encoding and/or decoding a block of a picture is disclosed. Such a method comprises determining one or more values characterizing samples of a block; modifying the determined values using a mapping function; deriving from the modified values one or more block-dependent parameters of a parametric function that approximates the mapping function; and mapping samples of the block using the parametric function and the derived one or more block-dependent parameters.
According to another aspect of the present disclosure, an apparatus for encoding and/or decoding a block of a picture is disclosed. Such an apparatus comprises means for determining one or more values characterizing samples of a block; means for modifying the determined values using a mapping function; means for deriving from the modified values one or more block-dependent parameters of a parametric function that approximates the mapping function; and means for mapping samples of the block using the parametric function and the derived one or more block-dependent parameters. The present disclosure also provides a signal comprising video data generated according to the described method or apparatus. The present embodiments also provide a computer program product including instructions, which, when executed by a computer, cause the computer to carry out the methods described.
The above presents a simplified summary of the subject matter in order to provide a basic understanding of some aspects of subject matter embodiments. This summary is not an extensive overview of the subject matter. It is not intended to identify key/critical elements of the embodiments or to delineate the scope of the subject matter. Its sole purpose is to present some concepts of the subject matter in a simplified form as a prelude to the more detailed description that is presented later.
Additional features and advantages of the present disclosure will be made apparent from the following detailed description of illustrative embodiments which proceeds with reference to the accompanying figures
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 illustrates a block diagram of an exemplary video encoder.
FIG. 2 illustrates a block diagram of an exemplary video decoder.
FIG. 3 illustrates a block diagram of the block processing.
FIG. 4 illustrates in-loop mapping of prediction signal in an encoder.
FIG. 5 illustrates in-loop mapping of prediction signal in a decoder.
FIG. 6 illustrates examples of positions in or outside the block.
FIG. 7 illustrates a block diagram of a linear-based filtering process using scales signaled in the stream.
FIG. 8 illustrates a piece-wise-linear model of the mapping or inverse mapping function.
FIG. 9 illustrates an embodiment of the third variant using an average value.
FIG. 10 illustrates an embodiment of the third variant using an average value with multiple linear models.
FIG11 illustrates a block diagram of an exemplary system in which various aspects of the exemplary embodiments may be implemented. It should be understood that the drawings are for purposes of illustrating examples of various aspects and embodiments and are not necessarily the only possible configurations. Throughout the various figures, like reference designators refer to the same or similar features.
DESCRIPTION OF EMBODIMENTS
For clarity of description, the following description will describe aspects with reference to embodiments involving video compression technology such as, for example, HEVC, JEM and/or H.266. However, the described aspects are applicable to other video processing technologies and standards.
This application describes a variety of aspects, including tools, features, embodiments, models, approaches, etc. Many of these aspects are described with specificity and, at least to show the individual characteristics, are often described in a manner that may sound limiting. However, this is for purposes of clarity in description, and does not limit the application or scope of those aspects. Indeed, all of the different aspects can be combined and interchanged to provide further aspects. Moreover, the aspects can be combined and interchanged with aspects described in earlier filings as well.
The aspects described and contemplated in this application can be implemented in many different forms. Figures 1 and 2 below provide some embodiments, but other embodiments are contemplated and the discussion of figures 1 and 2 does not limit the breadth of the implementations. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded. These and other aspects can be implemented as a method, an apparatus, a computer readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods described, and/or a computer readable storage medium having stored thereon a bitstream generated according to any of the methods described.
In the present application, the terms“reconstructed” and“decoded” may be used interchangeably, the terms“pixel” and“sample” may be used interchangeably, the terms “image,”“picture” and“frame” may be used interchangeably. Usually, but not necessarily, the term“reconstructed” is used at the encoder side while“decoded” is used at the decoder side. Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and/or use of specific steps and/or actions may be modified or combined.
Various methods and other aspects described in this application can be used to modify modules of a video encoder 100 and decoder 200 as shown in figures 1 and 2. Moreover, the present aspects are not limited to VVC or HEVC, and can be applied, for example, to other standards and recommendations, whether pre-existing or future-developed, and extensions of any such standards and recommendations (including VVC and HEVC). Unless indicated otherwise, or technically precluded, the aspects described in this application can be used individually or in combination. In case of numeric values used in the present application the specific values are for example purposes and the aspects described are not limited to these specific values.
Figure 1 illustrates an encoder 100. Variations of this encoder 100 are contemplated, but the encoder 100 is described below for purposes of clarity without describing all expected variations. Before being encoded, the video sequence may go through pre-encoding processing (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to get a signal distribution more resilient to compression (for instance using a histogram equalization of one of the color components). Metadata can be associated with the pre processing, and attached to the bitstream.
In the encoder 100, a picture is encoded by the encoder elements as described below. The picture to be encoded is partitioned (102) and processed in units of, for example, CUs. Each unit is encoded using, for example, either an intra or inter mode. When a unit is encoded in an intra mode, it performs intra prediction (160). In an inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which one of the intra mode or inter mode to use for encoding the unit, and indicates the intra/inter decision by, for example, a prediction mode flag. Prediction residuals are calculated, for example, by subtracting (110) the predicted block from the original image block. The prediction residuals are then transformed (125) and quantized (130). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the non-transformed residual signal. The encoder can bypass both transform and quantization, i.e., the residual is coded directly without the application of the transform or quantization processes.
The encoder decodes an encoded block to provide a reference for further predictions. The quantized transform coefficients are de-quantized (140) and inverse transformed (150) to decode prediction residuals. Combining (155) the decoded prediction residuals and the predicted block, an image block is reconstructed. In-loop filters (165) are applied to the reconstructed picture to perform, for example, deblocking/SAO (Sample Adaptive Offset) filtering to reduce encoding artifacts. The filtered image is stored at a reference picture buffer (180).
Figure 2 illustrates a block diagram of a video decoder 200. In the decoder 200, a bitstream is decoded by the decoder elements as described below. Video decoder 200 generally performs a decoding pass reciprocal to the encoding pass as described in Figure 1. The encoder 100 also generally performs video decoding as part of encoding video data. In particular, the input of the decoder includes a video bitstream, which can be generated by video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. The picture partition information indicates how the picture is partitioned. The decoder may therefore divide (235) the picture according to the decoded picture partitioning information. The transform coefficients are de-quantized (240) and inverse transformed (250) to decode the prediction residuals. Combining (255) the decoded prediction residuals and the predicted block, an image block is reconstructed. The predicted block can be obtained (270) from intra prediction (260) or motion-compensated prediction (i.e., inter prediction) (275). In-loop filters (265) are applied to the reconstructed image. The filtered image is stored at a reference picture buffer (280).
The decoded picture can further go through post-decoding processing (285), for example, an inverse color transform (e.g. conversion from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping performing the inverse of the remapping process performed in the pre- encoding processing (101). The post-decoding processing can use metadata derived in the pre encoding processing and signaled in the bitstream.
The encoder 100 of Figure 1 and decoder 200 of Figure 2 are adapted to implement at least one of the embodiments described below for which it is proposed to operate in-loop mapping at block level at the encoder and/or decoder. The process of mapping or inverse mapping is performed using a block-adaptive simplified version of the global mapping or global inverse mapping functions, applied to each sample of the block to process in the coding loop. For each block for which mapping or inverse mapping applies, a limited number of values derived from the samples in the block are derived (for instance the average sample value in the block, or the minimum and maximum sample values in the block). These values are then modified by the global mapping or inverse mapping function f[.]. A parametric model is then deduced from these values and from their modified versions, this model corresponding to a simplified approximation of the function f[.] in the block. Finally, the parametric model is used to modify all the samples in the block, using the same simple direct formula for each sample. This principle is used both for mapping / inverse mapping at the encoder and at the decoder.
Performing the mapping / inverse mapping at an image block level rather than at pixel level allows to improve the quality of the video for a same bitrate or to decrease the bitrate for a same quality, compared to the coding without any mapping / inverse mapping.
It is considered that a block of samples S(p), of size N, i.e. that is comprising N samples, is processed, p corresponding to a location in the block, according to a mapping or inverse mapping global function, noted f[.]. The global function f[.] is given, for instance determined from a piece-wise-linear model, or from a look-up-table of size corresponding to the range of the samples to be processed, for instance 1024 in case of lO-bit samples. This PWL or LUT can be coded in the bitstream, for instance for each random-access point, or defined by default.
In at least one embodiment, a mapping process reproduces a behavior similar to the process of mapping the samples by the function f[.], corresponding to the formula of equation 1 below, but uses a simplified parametric approximation of f[.], noted g_P[.], P corresponding to local parameters used to model f[.] by a parametric model inside the block.
M(p) = f[ S(p) ] (equation 1) The block diagram of Figure 3 provides the different steps of the proposed process, which is made of the four following steps. In a first step 401 a limited set of n values V_i are determined (for i=0..n- 1 , n being small compared to N), these values characterizing the samples of the block. In a second step 402 mapped versions M_i of values V_i (for i=0..n-l) are computed using the global function f[.]:
M_i = f[ V_i ] for i=0 to n-l (equation 2)
A third step 403 includes deriving p parameters P_k of a parametric model, p <= n, from the values V_i and M_i, i=0 to n-l, the parametric model corresponding to a function g_P[.] that approximates the global function f[.] in a simplified way. In other words, g_P[ V i ] is expected to be close to M_i = f[ V_i ] for most of the i indexes in 0 to n-l. A simplified way means that the implementation of g_P[.] can be achieved with less operations (comparisons, multiplications, additions, accesses to lookup tables (LUTs)) than the implementation of function f[.]. Parameters P_k are local because they are block-dependent. In a fourth step 404, all samples of the block are modified using the parametric function g_P[.] and parameters P_k. The modified samples S_out(p), for p in the block, are the output of the overall process. It is expected that S_out(p) = g_P[ S(p) ] is close to f[ S(p) ] for most of the locations p.
S_out(p) = g_P[ S(p) ] for all locations p in the block (equation 3)
Contrarily to the function f[.] that is a function global to the picture, or to a set of blocks, g_P[.] is a block-adaptive function, that is varying per block, i.e. its parameters P_k vary per block. The block on which the local function g_P applies can be the entire Coding Unit (CU), Tree Unit (TU) or blocks of pre-defmed size (e.g. block of 4x4 samples). If the CU or TU is larger than the pre-defmed size, it is divided in sub-blocks having the pre-defmed size.
As illustrated in figures 4 and 5, additional blocks of locally adaptive mapping and inverse mapping are added to the conventional encoder and decoder block diagrams. Furthermore, the mapping proposed in this application can be done on values representative of the luma of the video, on values representative of the chroma of the video, or on both values. In particular, figure 4 depicts the in-loop mapping of the prediction signal in an encoder according to the invention. Besides the locally adaptive mapping and inverse mapping, the processing is performed as described above for encoder 100 illustrated in figure 1.
In block 2001, a locally adapted forward mapping is applied to the non-compressed input samples. Prediction residuals are calculated, by subtracting (2002) a predicted block determined in prediction block 2008 from the original image block. The prediction residuals are transformed and quantized (2003) and finally entropy coded (2010) to output a bitstream. In the prediction loop of the encoder, the quantized transform coefficients are de-quantized and inverse transformed (2004). Locally adaptive inverse mapping (2005) is applied to the reconstructed block before prediction (2008) and locally adaptive mapping (2009) after prediction. In addition, as mentioned for the encoder of figure 1, in-loop filters (2006) are applied to reduce encoding artifacts and a reference picture buffer (2007) is provided for storing the filtered blocks of the picture.
Correspondingly, figure 5 illustrates the in-loop mapping of the prediction signal in a decoder according to the invention. Similarly to the encoder, besides the mapping the rest of the decoder processing corresponds to the processing of the video decoder 200 illustrated in figure 2.
In the decoder, a video bitstream is entropy decoded (2101) to obtain transform coefficients, which are de-quantized and inverse transformed (2102) to decode the prediction residuals. Combining (2103) the decoded prediction residuals and a predicted block, an image block is reconstructed. Locally adaptive inverse mapping (2104) is applied to the reconstructed image block. After in-loop filtering (2105) the filtered image block is output as well as stored at a reference picture buffer (2106). Prediction (2107) is performed based on the pictures stored in the reference picture buffer and locally adaptive mapping (2108) is applied to the predicted blocks after prediction.
Multiple variants of the above embodiment are described hereafter. First variant embodiment using a block-adaptive additive offset model
In a first variant embodiment, the parametric model (and therefore the function g_P) corresponds to an additive offset, and one single value V_0 is derived.
The process is implemented as follows. The first step (401 in figure 3) includes obtaining one value V_0, which is determined based on all samples in the currently processed block or one or more samples positioned as shown in figure 6, such as positions A, B, C, D and H in the currently processed block or positions E, F, G and I outside the currently processed block. The depicted positions A to I are exemplarily only, other positions could be used as well.
For instance, V_0 can be determined according to one of the following options:
- V_0 is the average of the samples S(p) in the block,
- V_0 is the average of a sub-sampled version of the block
- V O is the minimum of the samples S(p) in the block
- V O is the maximum of the samples S(p) in the block
- V_0 is the median value of the samples S(p) in the block
- V_0 is the value of one specific sample in the block, as the top left sample (A in Figure 6), or one sample close to the center of the block (e.g. H in Figure 6)
- V_0 is the value of one specific external sample close to the block, as the external top left sample (I in Figure 6)
- V_0 is the average of the 4 comers samples S(p) in the block (A, B, C, D in Figure 6)
In the second step (402 in figure 3) the mapped version M_0 is computed from value V_0, using function f[.]:
M_0 = f[ V_0 ] (equation 4)
In the third step (403 in figure 3) an offset parameter P_0 is determined from the values V_0 and M_0 as follows: P_0 = M_0 - V_0, P_0 being an offset value. The fourth step (403 in figure 3) includes modifying all samples of the block using the parametric function g_P[.] and the offset parameter P_0. The modified samples S_out(p), for p in the block, are the output of the overall process: S_out(p) = g_P[ S(p) ] = S(p) + P_0 (equation 5)
Thus, with this variant embodiment, only a single addition per sample is required. The process using an additive offset leads to the following number of operations:
mapping of V_0 requires 16 checks in average, 1 access to the piece wise linear (PWL) LUT, plus 1 multiplication and 2 additions,
- No multiplication per sample,
1 addition per sample
Figure imgf000012_0001
The mapping of samples using g_P[.] is much simpler than using a linear interpolation based on a piece-wise linear model requiring for each sample to identify the piece it belongs to.
Second variant embodiment using a block-adaptive scaling model
In a second variant embodiment, the parametric model (and therefore the function g_P) corresponds to a scaling and uses the same principles as the first variant, replacing the offset by a scaling. In this case: P_0 = M_0 / V_0. As a result, the modified samples are determined as follows:
S_out(p) = g_P[ S(p) ] = S(p) x P_0 = S(p) x M_0 / V_0 (equation 6) In order to avoid the division, a shifted based formula can be used:
S_out(p) = g_P[ S(p) ] = ( S(p) x Q_0 + OffO ) » shift (equation 7) where Q_0 = Int( ( M O « shift ) / V O ) and Int(x) returns the integer part of x, and OffO is a given offset, for instance equal to ( 1 « ( shift - 1 ) ).
Third variant embodiment using a block-adaptive linear based model
In a third variant embodiment, the parametric model, and therefore the function g_P, corresponds to a linear function, parametrized by 2 parameters, and at least 2 values V_0 and V_l are derived, and used to compute the parameters of the linear model. The process is implemented as follows. The first step (401 of figure 3) includes deriving at least 2 values V_0 and V_l. For instance, these values can correspond to:
The values of 2 samples located in 2 comers inside the block (among A, B, C, D in Figure 6)
2 average values of 2 different samples in the block (e.g. V_0 is the average of A and B, V_l of C and D, cf Figure 6)
The values of 2 samples located in 2 comers outside the block (E or F and G in Figure 6)
- V O is the minimum of the samples S(p) in the block, and V_l is the maximum of the samples S(p) in the block
The second step (402 in figure 3) includes computing the mapped version M_i of value V_i, i=0,l, using f[.]:
M i = f[ V i ] (equation 8)
The third step (403 in figure 3) includes deriving the 2 parameters of the linear model from the values V_i and M_i, such that M_i = P_0 + P_l x V_i for i=0, 1 The fourth step (404 in figure 3) includes modifying all samples of the block using the parametric function g_P[.] and linear parameters P i, i=0,l. The modified samples S_out(p), for p in the block, are the output of the overall process. S_out(p) = g_P[ S(p) ] = P_0 + P_l x S(p) (equation 9)
This requires 1 multiplication and 1 addition per sample. In case of integer implementations, a few shifts and one more addition may be required. The process using a linear model leads to the following number of operations:
Mapping of V_0 and V_l requires 2x16 checks, 2 accesses to the PWL LUT, plus 2 multiplications and 2 additions
- N multiplications (1 per sample)
2xN additions (2 per sample)
Figure imgf000014_0001
The mapping of samples remains much simpler than using a linear interpolation based on a piece-wise linear model requiring for each sample to identify the piece it belongs to. Fourth variant embodiment using a block-adaptive linear model and scales signaled in the stream
In a fourth variant embodiment, a linear model is used at the block level, and steps 402 and 403 used to derive the linear parameters at block level are removed. Instead of these steps, a simple way of getting the linear parameters is used. In this embodiment, linear parameters, related to piece-wise-linear models, are pre-computed in an initialization phase achieved prior to decoding the blocks. When processing a block, the actual parameters used for processing the block are picked-up from the parameters computed in the initialization phase.
The block diagram of this embodiment is provided in Figure 7. This variant embodiment uses two main steps. An initialization step (501) is used to derive linear model parameters for a given set of blocks, e.g. the entire slice, or picture, or tile. A block-level process (504) is the step, which performs the mapping or inverse mapping of the samples in the block.
The initialization step (501) comprises two steps. Step 502 includes decoding data related to the piece-wise-linear model of the mapping or inverse mapping function to be used in the further block-level steps. The data is carried by the bitstream received by a decoder or by the media read by a player. In particular, the number of scales values (NbPieces=6 in illustrative Figure 8), plus the scale values themselves of each piece of the model (“scale_i” in illustrative Figure 8), are decoded, as well as the sample range to which they apply (“S ini” in illustrative Figure 8). In at least one embodiment, equidistant ranges are used so that the ranges do not need to be coded, in which case S ini = i * interval applies, where the parameter “interval” is equal to the signal range divided by NbPieces. Step 503 builds the complete PWL model from the scale values and range values. In particular, pivot points values (“S outi” in illustrative Figure 8) are derived, for instance as follows:
S_outi+i = S_outi + (S_im+i - S_im) * scale_i (equation 10)
The initial offset S_outo is either coded in the bitstream, or set to 0 in case of full range video, or set to (16 * 2B 8 ) where B is the bitdepth of the samples, in case of limited (or standard) range video. The information about full range video / limited range video can be signaled in the stream
The block level process (504) applies the following steps:
In step 505, a value V_0 is determined, V_0 being characterizing the block samples (similar to step 401 of figure 3).
In step 506, the offset and scale values corresponding to the PWL piece which V_0 belongs to is determined. In step 507 the linear model is applied to all samples in the block.
S_out(p) = g_P[ S(p) ] = scale * S(p) + offset (equation 11)
Fifth variant embodiment combining mapping and inverse mapping in a single
In a fifth variant embodiment, the 3 steps 2108-2103-2104 of Figure 5 are applied into one single step. If we consider that both mapping and inverse mapping functions are modeled by piece-wise linear models, the following can be observed. Let us consider a sample value L (here the value will be considered as a value characterizing the samples of the block, for instance the average sample value over the block). Let am and bm be the linear parameters of the mapping PWL piece which L belongs to. These parameters are considered as applying to all samples in the block. The mapping M of L, as achieved in step 2108, is obtained as follows:
M = map[ L ] = am x L + bm (equation 12)
Let us consider a residual value R added to M, according to step 2103. This results in:
Ml = M + R = am x L + bm + R (equation 13)
Let aim and him be the linear parameters of the inverse mapping PWL piece which Ml belongs to. These parameters are considered as applying to all samples in the block. Step 2104 of inverse mapping applied to Ml corresponds to:
IM = invmapl Ml ] = aim x Ml + him (equation 14)
= aim X ( am X L + bm + R ) + him
= aim X am X L + aim X R + ( aim X bm + him )
= A x L + B x R + C
As a result, the mapping and inverse mapping functions can be combined together in one single step. In such embodiment, steps 505 to 507 of Figure 7 are implemented as follows. In step 505 a value V_0 characterizing the block samples is derived, typically from the prediction block. The step 506 comprises the following steps:
identifying the pivot point (offset, corresponding to bm) and scale (corresponding to am) values corresponding to the mapping PWL piece which V_0 belongs to.
mapping of V_0 into M_0 according to the offset and scale values derived in previous step (this is equivalent to mapping V O by the mapping function map[.]):
M_0 = am x V_0 + bm (equation 15) computing a residual signal value Res avg characterizing the residual sample values Res(p) in the block. For instance, Res avg is the average of the Res(p) over the block. Res_avg is added to M_0:
R_0 = M_0 + Res_avg (equation 16) identifying the pivot point (offset, corresponding to him) and scale (corresponding to aim) values corresponding to the inverse mapping PWL piece which R_0 belongs to.
In one further step, values A, B, C are derived as
A = aim x am (equation 17)
B = aim (equation 18)
C = aim x bm + him (equation 19)
The last step 507 allows performing the 3 steps 2108-2103-2104 of figure 6 into one single operation. It consists in applying the linear model to all prediction and residual samples in the block using the derived linear parameters A, B, C (common to all samples in the block):
S_out(p) = A * S(p) + B * Res(p) + C (equation 20) where S(p) is the sample value, Res(p) is the residual signal value, and S_out(p) the output sample value, at location p in the block.
The concepts discussed herein have been described for an intra-component mapping: the function f[.] applies to a sample of a given component and provides a modified sample version of this component. The concepts described herein can be extended to the cross component case, where f[.] modifies the sample of one component using the sample values of this component, and of another component. For instance, cross-component modification of a chroma sample C(p) based on co-located luma sample Y(p) can be considered:
MC(p) = f[C(p), Y(p)] (equation 21) where MC(p) is a modified version of the chroma sample C(p).
A typical example is a cross-component scaling of the chroma sample:
MC(p) = ( C(p) - 2L(B-1) ) * sc[Y(p)] + 2L(B-1) (equation 22) where B is the bit-depth of the chroma component, sc[.] is a luma dependent scaling function, and the operator“aAb” corresponds to a power of b (also noted ab).
Fixed-point implementation of the third variant embodiment using a block- adaptive linear model, based on the minimum and maximum sample values in the block
In at least an embodiment, a fixed-point implementation allows to perform the computations in the integer computation space rather than floating-point computation space. This provides more efficient computations and easier implementation in hardware. Furthermore, this embodiment, referring to the main embodiment described above, uses only a set of n values V i limited to two values: the minimal value and the maximal value of a block.
In step 401, V O is computed as the minimum of the samples S(p) in the block, and V_l is computed as the maximum of the samples S(p) in the block.
In step 402, the mapped values of V_0 and V_l, M_0 = f[ V_0 ] and M_l = f[ V_l ] are derived.
In step 403, the linear model parameters a, b are derived as follows. a is initialized to a = (1 « shiftAlgol), where shiftAlgol is a pre-determined value. One example of predefined value for shiftAlgol is 16. Other examples are 8 or 32.
If V_0 is not equal to VI, a is set as follows: a= ( (M_l - M_0) * (1« shiftAlgol) + (1 «(shift Algol -1)) ) / (V_l - V_0) b is set as follows:
b = (M_0« shiftAlgol) + (1 « (shiftAlgol -1)) - V_0 * a
In step 404, the model is applied to all samples of the block as follows.
S_out(p) = ( S(p) * a + b ) » shiftAlgol
A clipping of S_out(p) is applied to stay in the range [ f[ 0 ] - f[ 2BD-l ] ] (for a signal of bit-depth BD).
Fixed-point implementation of the third variant embodiment using a block- adaptive linear model, based on the values surrounding the average sample value in the block
In an embodiment, the set of n values V_i is limited to a single value: the average value of a block (avgY) and uses a pre-determined value (delta) that specifies an operation range around this average value. Like the previous embodiment, a fixed-point implementation is used to allow computations in the integer computation space rather than floating-point computation space.
In this implementation, a linear model as shown in figure 9 is derived from two values V_0 and V_l surrounding the average sample value of the samples S(p) in the block.
In step 401, firstly the average sample value avgY in the block is computed. V_0 is set to avgY - delta, where delta is a pre-determined value. One example of predefined value for delta is 32. Other examples are 16 or 64. If V_0 is lower than 0, V_0 is set to 0. Otherwise, if (avgY + delta) is greater than (2BD-l), V_0 is set to (2BD-l - 2*delta). These operations allow to maximise the operating range when the average is too close of the minimal and maximal values. Then, V_l is set to ( V_0 + 2*delta ).
In step 402, the mapped values of V_0 and V_l, M_0 = f[ V_0 ] and M_l = f[ V_l ] are derived.
In step 403, the linear model parameters a, b are derived as follows.
a = M_l - M_0
b = f[ avgY ] * 2 * delta + delta - avgY * a In step 404, the model is applied to all samples of the block as follows.
S_out(p) = (S(p) * a + b) / (2*delta)
The division by (2* delta) can be implemented using a right shift operation if delta is a power of 2. A clipping of S_out(p) is applied to stay in the range [ f[ 0 ] - f[ 2BD-l ] ], or alternatively [ 0 - 2BD-l] (for a signal of bit-depth BD).
Fixed-point implementation of the third variant embodiment using several linear models, based on the values surrounding the average sample value in the block
In an embodiment, two linear models are derived from two values V O and V_l surrounding the average sample value of the samples S(p) in the block as shown in figure 10.
In step 401, the following applies.
First the average sample value avgY in the block is computed.
A parameter avgY2 is derived as follows.
avgY2 = Max(delta, Min(2BD-l - delta, avgY))
V_0 and V_l are derived as follows.
V_0 = avgY2 - delta
V_l = avgY2 + delta
In step 402, the mapped values of V_0 and V_l, M_0 = f[ V_0 ] and M_l = f[ V_l ] are derived.
In step 403, the linear model parameters a[0], b[0], a[l], b[l], are derived as follows.
A parameter aa is derived as aa = (M_l - M_0)
A parameter a[0] is derived as a[0] = ( (f[avgY] - M_0)*2 * wl + aa * w2 )
A parameter a[l] is derived as a[l] = ( (M_l - f[avgY])*2 * wl + aa * w2 ) where wl and w2 are pre-defmed values (e.g. both set to 1)
A parameter b[0] is set to b[0] = f[avgY] * 4 * delta + 2 * delta - avgY * a[0]
A parameter b[l] is set to b[l] = fjavgY] * 4 * delta + 2 * delta - avgY * a[l]
In step 404, the mapping is applied to all samples of the block as follows. An index idx is derived as idx = 0 if S(p) <= avgY, idx = 1 otherwise.
S_out(p) = ( S(p) * a[idx] + b[idx] ) / ( (wl+w2) * 2 * delta)
The division by ( (wl+w2) * 2 * delta ) can be implemented using a right shift operation if ((wl+w2) * 2 * delta) is power of 2.
A clipping of S_out(p) is applied to stay in the range [ f[ 0 ] - f[ 2BD-l ] ], or alternatively [ 0 - 2BD-l] (for a signal of bit-depth BD).
Implementation example of the fifth variant embodiment combining mapping and inverse mapping in a single step
A possible implementation of the fifth variant is described below. The implementation is considered to be in floating point. Its version for fixed-point can be easily derived using the same principles as illustrated above for the third variant.
Derivation of Linear parameters al. bl
The average sample value avgP of the prediction samples Pred(p) in the block is computed.
Parameters V_0 and V_l are computed as follows.
V_0 is set to (avgP-delta)
If V_0 is lower than 0, V_0 is set to 0
Otherwise, if (avgP + delta) is greater than (2BD-l), V_0 is set to (2BD-l - 2*delta)
V_l is set to ( V_0 + 2* delta )
The mapped values of V_0 and V_l, M_0 = f[ V_0 ] and M_l = f[ V_l ] are derived. Linear parameters al, bl are computed as follows
al = ( M_l - M_0 ) / (2* delta)
bl = f[ avgP ]
Derivation of Linear parameters a2. b2
The average sample value avgR of the residual samples Res(p) in the block is computed. The parameter avgT is derived as avgT = al*avgP + bl + avgR
Parameters V’_0 and V’_l are computed as follows.
V’_0 is set to (avgT - delta) If V’_0 is lower than 0, it is set to 0
Otherwise, if (avgT + delta) is greater than (2BD-l), V’_0 is set to (2BD-l - 2*delta) V’_l is set to ( V’_0 + 2*delta )
The mapped values of V’_0 and V’_l, M’_0 = f[ V’_0 ] and M’_l = f[ V’_l ] are derived.
Linear parameters a2, b2 are computed as follows.
a2 = ( M’_l - M’_0 ) / (2* delta)
b2 = f[ avgT ]
Derivation of parameters A. B. C
Parameters A, B, C are derived as follows.
A = al * a2
B = a2
C = a2 * bl + b2
Mapping of samples in the block
The samples in the block are derived as follows.
S_out(p) = A * Pred(p) + B * Res(p) + C
A clipping of S_out(p) is applied to stay in the range [ f[ 0 ] - f[ 2BD-l ] ], or alternatively [ 0 - 2BD-l] (for a signal of bit-depth BD).
Figure 11 illustrates a block diagram of an example of a system in which various aspects and embodiments are implemented. System 1000 can be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000, singly or in combination, can be embodied in a single integrated circuit (IC), multiple ICs, and/or discrete components. For example, in at least one embodiment, the processing and encoder/decoder elements of system 1000 are distributed across multiple ICs and/or discrete components. In various embodiments, the system 1000 is communicatively coupled to one or more other systems, or other electronic devices, via, for example, a communications bus or through dedicated input and/or output ports. In various embodiments, the system 1000 is configured to implement one or more of the aspects described in this document.
The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this document. Processor 1010 can include embedded memory, input output interface, and various other circuitries as known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device, and/or a non-volatile memory device). System 1000 includes a storage device 1040, which can include non-volatile memory and/or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read- Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and/or optical disk drive. The storage device 1040 can include an internal storage device, an attached storage device (including detachable and non- detachable storage devices), and/or a network accessible storage device, as non-limiting examples.
System 1000 includes an encoder/decoder module 1030 configured, for example, to process data to provide an encoded video or decoded video, and the encoder/decoder module 1030 can include its own processor and memory. The encoder/decoder module 1030 represents module(s) that can be included in a device to perform the encoding and/or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, encoder/decoder module 1030 can be implemented as a separate element of system 1000 or can be incorporated within processor 1010 as a combination of hardware and software as known to those skilled in the art.
Program code to be loaded onto processor 1010 or encoder/decoder 1030 to perform the various aspects described in this document can be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. In accordance with various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder/decoder module 1030 can store one or more of various items during the performance of the processes described in this document. Such stored items can include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.
In some embodiments, memory inside of the processor 1010 and/or the encoder/decoder module 1030 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device can be either the processor 1010 or the encoder/decoder module 1030) is used for one or more of these functions. The external memory can be the memory 1020 and/or the storage device 1040, for example, a dynamic volatile memory and/or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of, for example, a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations, such as for MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also referred to as ISO/IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).
The input to the elements of system 1000 can be provided through various input devices as indicated in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and/or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples, not shown in Figure 11, include composite video.
In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF portion can be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs various of these functions, including, for example, downconverting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and/or add other elements performing similar or different functions. Adding elements can include inserting elements in between existing elements, such as, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna. Additionally, the USB and/or HD MI terminals can include respective interface processors for connecting system 1000 to other electronic devices across USB and/or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, can be implemented, for example, within a separate input processing IC or within processor 1010 as necessary. Similarly, aspects of USB or HDMI interface processing can be implemented within separate interface ICs or within processor 1010 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 1010, and encoder/decoder 1030 operating in combination with the memory and storage elements to process the datastream as necessary for presentation on an output device.
Various elements of system 1000 can be provided within an integrated housing. Within the integrated housing, the various elements can be interconnected and transmit data therebetween using suitable connection arrangement, for example, an internal bus as known in the art, including the Inter-IC (I2C) bus, wiring, and printed circuit boards.
The system 1000 includes communication interface 1050 that enables communication with other devices via communication channel 1060. The communication interface 1050 can include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 1060. The communication interface 1050 can include, but is not limited to, a modem or network card and the communication channel 1060 can be implemented, for example, within a wired and/or a wireless medium.
Data is streamed, or otherwise provided, to the system 1000, in various embodiments, using a wireless network such as a Wi-Fi network, for example IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal of these embodiments is received over the communications channel 1060 and the communications interface 1050 which are adapted for Wi-Fi communications. The communications channel 1060 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet for allowing streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 1000 using a set-top box that delivers the data over the HDMI connection of the input block 1130. Still other embodiments provide streamed data to the system 1000 using the RF connection of the input block 1130. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, for example a cellular network or a Bluetooth network.
The system 1000 can provide an output signal to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes one or more of, for example, a touchscreen display, an organic light- emitting diode (OLED) display, a curved display, and/or a foldable display. The display 1100 can be for a television, a tablet, a laptop, a cell phone (mobile phone), or other device. The display 1100 can also be integrated with other components (for example, as in a smart phone), or separate (for example, an external monitor for a laptop). The other peripheral devices 1120 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disk player, a stereo system, and/or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide a function based on the output of the system 1000. For example, a disk player performs the function of playing the output of the system 1000.
In various embodiments, control signals are communicated between the system 1000 and the display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communications protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to system 1000 using the communications channel 1060 via the communications interface 1050. The display 1100 and speakers 1110 can be integrated in a single unit with the other components of system 1000 in an electronic device such as, for example, a television. In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (T Con) chip.
The display 1100 and speaker 1110 can alternatively be separate from one or more of the other components, for example, if the RF portion of input 1130 is part of a separate set-top box. In various embodiments in which the display 1100 and speakers 1110 are external components, the output signal can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.
The embodiments can be carried out by computer software implemented by the processor 1010 or by hardware, or by a combination of hardware and software. As a non- limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 1020 can be of any type appropriate to the technical environment and can be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 1010 can be of any type appropriate to the technical environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.
Various implementations involve decoding.“Decoding”, as used in this application, can encompass all or part of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding. In various embodiments, such processes also, or alternatively, include processes performed by a decoder of various implementations described in this application, for example, the processes presented in figures 3 and 7. As further examples, in one embodiment“decoding” refers only to entropy decoding, in another embodiment“decoding” refers only to differential decoding, and in another embodiment“decoding” refers to a combination of entropy decoding and differential decoding. Whether the phrase“decoding process” is intended to refer specifically to a subset of operations or generally to the broader decoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.
Various implementations involve encoding. In an analogous way to the above discussion about“decoding”,“encoding” as used in this application can encompass all or part of the processes performed, for example, on an input video sequence in order to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, for example, partitioning, differential encoding, transformation, quantization, and entropy encoding. In various embodiments, such processes also, or alternatively, include processes performed by an encoder of various implementations described in this application, for example, the process of figures 3 and 7.
As further examples, in one embodiment“encoding” refers only to entropy encoding, in another embodiment“encoding” refers only to differential encoding, and in another embodiment“encoding” refers to a combination of differential encoding and entropy encoding. Whether the phrase“encoding process” is intended to refer specifically to a subset of operations or generally to the broader encoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.
Note that the syntax elements as used herein are descriptive terms. As such, they do not preclude the use of other syntax element names.
When a figure is presented as a flow diagram, it should be understood that it also provides a block diagram of a corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow diagram of a corresponding method/process.
The implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable/personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.
Reference to“one embodiment” or“an embodiment” or“one implementation” or“an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase“in one embodiment” or“in an embodiment” or“in one implementation” or“in an implementation”, as well any other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment.
Additionally, this application may refer to“determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.
Further, this application may refer to“accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.
Additionally, this application may refer to“receiving” various pieces of information. Receiving is, as with“accessing”, intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further,“receiving” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.
It is to be appreciated that the use of any of the following“/”,“and/or”, and“at least one of’, for example, in the cases of“A/B”,“A and/or B” and“at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of“A, B, and/or C” and“at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.
Also, as used herein, the word“signal” refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments the encoder signals a particular mode. In this way, in an embodiment the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as others, then signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, a bit savings is realized in various embodiments. It is to be appreciated that signaling can be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word“signal”, the word“signal” can also be used herein as a noun.
As will be evident to one of ordinary skill in the art, implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry the bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.
We describe a number of embodiments. Features of these embodiments can be provided alone or in any combination. Further, embodiments can include one or more of the following features, devices, or aspects, alone or in any combination, across various claim categories and types:
A mapping function operates at block level in a prediction loop of a video encoding/decoding method. A mapping function operating at block level uses a parametric model corresponding to a simplified approximation of a global mapping function f[.] in the block. An inverse mapping function operating at block level uses a parametric model corresponding to a simplified approximation of a global inverse mapping function f[.] in the block. The simplified approximation uses an offset model, a scaling model or a linear based model.
A video encoding method uses a mapping function operating at block level and using a parametric model corresponding to a simplified approximation of a global mapping function.
A video decoding method uses an inverse mapping function operating at block level and using a parametric model corresponding to a simplified approximation of a global inverse mapping function. A video decoding method uses a mapping and inverse mapping functions modeled by piece-wise linear models, wherein the steps of mapping and an inverse mapping are combined together in a single step.
A video encoder, transcoder or other electronic device performs video encoding using a mapping function operating at block level and using a parametric model corresponding to a simplified approximation of a global mapping function. A TV, set-top box, cell phone, tablet, or other electronic device performs video decoding using an inverse mapping function operating at block level and using a parametric model corresponding to a simplified approximation of a global inverse mapping function. An electronic device performs video decoding using a mapping and inverse mapping functions modeled by piece-wise linear models, wherein the steps of mapping and an inverse mapping are combined together in a single step.

Claims

1. A method for encoding and/or decoding a block of a picture, comprising:
- determining one or more values characterizing samples of a block;
- modifying the determined values using a mapping function;
- deriving from the modified values one or more block-dependent parameters of a parametric function that approximates the mapping function; and
- mapping samples of the block using the parametric function and the derived one or more block-dependent parameters.
2. The method according to claim 1, wherein
- for forward mapping the determined values characterizing samples of the block are modified using a global mapping function and a first parametric function is used, which corresponds to an approximation of the global mapping function; and
- for inverse mapping the determined values characterizing samples of the block are modified using a global inverse mapping function and a second parametric function is used, which corresponds to an approximation of the global inverse mapping function.
3. The method for encoding according to claim 2, further comprising
- obtaining source values of input samples;
- determining mapped source values by applying the first parametric function to the input samples;
- determining in a prediction loop mapped prediction values, wherein before prediction the second parametric function is applied and after prediction the first parametric function is applied;
- determining mapped residual values of said samples by subtracting the mapped prediction values from the mapped source values; and
- encoding the mapped residual values of said samples into a bitstream.
4. The method for decoding according to claim 2, further comprising
- decoding a bitstream to obtain mapped residual values for samples of a block;
- obtaining mapped prediction values;
- determining mapped reconstructed values for the samples by adding the obtained mapped residual values and mapped prediction values; wherein the second parametric function is applied to the determined mapped reconstructed values to obtain reconstructed values for the samples and wherein in a prediction loop after prediction the first parametric function is applied to obtain the mapped prediction values.
5. The method according to any of claims 2 to 4, wherein
- a single characterizing value is determined;
- the determined value is modified using the mapping function;
- an offset parameter is derived by subtracting the single characterizing value from the modified value; and
- the samples of the block are mapped by adding the derived offset parameter to the sample values.
6. The method according to any of claims 2 to 4, wherein
- a single characterizing value is determined;
- the determined value is modified using the mapping function;
- a scaling parameter is derived by a division algorithm determining the quotient of the modified value divided by the single characterizing value; and
- the samples of the block are mapped by a multiplication algorithm determining the product of the sample values with the derived scaling parameter.
7. The method according to claim 5 or 6, wherein the single characterizing value is one of the following:
- the average of sample values of the samples in the block,
- the average of sample values of a sub-sampled version of the block,
- the minimum of sample values of the samples in the block,
- the maximum of sample values of the samples in the block,
- the median value of sample values of the samples in the block,
- the sample value of one specific sample in the block,
- the sample value of one specific external sample close to the block, or
- the average of sample values of the four comer samples in the block.
8. The method according to any of claims 2 to 4, wherein
- two characterizing values are determined; - the two determined values are modified using the mapping function;
- the parametric function corresponds to a linear function, parametrized by two linear parameters, which are derived from the two determined values and the two modified values; and
- the samples of the block are mapped using the linear function and the linear parameters.
9. The method according to claim 8, wherein the two characterizing values are one of the following:
- the sample values of two samples located in two comers inside the block,
- a first average value of sample values of two samples in the block and a second average value of sample values of two different samples in the block,
- the sample values of two samples located in two comers outside the block, or
- the minimum of the sample values of the samples in the block and the maximum of the sample values of the samples in the block.
10. The method according to any of claims 2 to 4, wherein in an initialization phase linear model parameters are pre-computed for a plurality of blocks, and the pre-computed parameters are used at block-level for mapping or inverse mapping of the samples of an individual block.
11. The method according to claim 10, wherein
- the initialization phase comprises decoding offset and/or scale value data related to a piece-wise-linear model of the mapping or inverse mapping function to be used in the processing of the individual blocks at block-level;
- in the processing at block level
- a value characterizing the block samples is determined;
- offset and/or scale values are determined, which correspond to the part of the piece-wise-linear model to which the determined value belongs; and
- the piece-wise-linear model is applied to all samples in the block using the determined offset and/or scale values.
12. The method according to any of claims 2 to 11, wherein both the mapping and inverse mapping are combined in a single processing step.
13. The method according to any of the preceding claims, wherein the blocks have a pre defined size and processing units with block structures larger than the pre-defmed size are divided into sub-blocks having the pre-defmed size.
14. The method according to claim 13, wherein the blocks have a pre-defmed size of 4x4 samples.
15. An apparatus for encoding and/or decoding a block of a picture, comprising:
- means for determining one or more values characterizing samples of a block;
- means for modifying the determined values using a mapping function;
- means for deriving from the modified values one or more block-dependent parameters of a parametric function that approximates the mapping function; and
- means for mapping samples of the block using the parametric function and the derived one or more block-dependent parameters.
16. The apparatus according to claim 15, wherein
- for forward mapping the determined values characterizing samples of the block are modified using a global mapping function and a first parametric function is used, which corresponds to an approximation of the global mapping function; and
- for inverse mapping the determined values characterizing samples of the block are modified using a global inverse mapping function and a second parametric function is used, which corresponds to an approximation of the global inverse mapping function.
17. The apparatus for encoding according to claim 16, further comprising
- means for obtaining source values of input samples;
- means for determining mapped source values by applying the first parametric function to the input samples;
- means for determining in a prediction loop mapped prediction values, wherein before prediction the second parametric function is applied and after prediction the first parametric function is applied;
- means for determining mapped residual values of said samples by subtracting the mapped prediction values from the mapped source values; and
- means for encoding the mapped residual values of said samples into a bitstream.
18. The apparatus for decoding according to claim 16, further comprising
- means for decoding a bitstream to obtain mapped residual values for samples of a block;
- means for obtaining mapped prediction values;
- means for determining mapped reconstructed values for the samples by adding the obtained mapped residual values and mapped prediction values;
wherein the second parametric function is applied to the determined mapped reconstructed values to obtain reconstructed values for the samples and wherein in a prediction loop after prediction the first parametric function is applied to obtain the mapped prediction values.
19. A signal comprising data generated according to the method of any of claims 1 to 3, and 5 to 14 or by the apparatus of any of claims 15 to 17.
20. A computer program product including instructions, which, when executed by a computer, cause the computer to carry out the methods for encoding or decoding data according to any of claims 1 to 14.
PCT/US2019/057128 2018-10-25 2019-10-21 Video encoding and decoding using block-based in-loop reshaping WO2020086421A1 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
EP18306393.2 2018-10-25
EP18306393 2018-10-25
EP18306573.9 2018-11-27
EP18306573 2018-11-27

Publications (1)

Publication Number Publication Date
WO2020086421A1 true WO2020086421A1 (en) 2020-04-30

Family

ID=68503216

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2019/057128 WO2020086421A1 (en) 2018-10-25 2019-10-21 Video encoding and decoding using block-based in-loop reshaping

Country Status (1)

Country Link
WO (1) WO2020086421A1 (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112351264A (en) * 2020-11-04 2021-02-09 深圳Tcl数字技术有限公司 Display precision adjusting method and device, terminal equipment and storage medium
US11343505B2 (en) 2019-02-01 2022-05-24 Beijing Bytedance Network Technology Co., Ltd. Configuring luma-dependent chroma residue scaling for video coding
US11375188B2 (en) 2019-03-23 2022-06-28 Beijing Bytedance Network Technology Co., Ltd. Restrictions on adaptive-loop filtering parameter sets
US11388406B2 (en) * 2019-05-20 2022-07-12 Dolby Laboratories Licensing Corporation Piece-wise-linear mapping for in-loop reshaping in image coding
US20220239932A1 (en) 2019-03-14 2022-07-28 Beijing Bytedance Network Technology Co., Ltd. Signaling and syntax for in-loop reshaping information
US11553203B2 (en) 2019-02-01 2023-01-10 Beijing Bytedance Network Technology Co., Ltd. Signaling of in-loop reshaping information using parameter sets

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016164235A1 (en) * 2015-04-06 2016-10-13 Dolby Laboratories Licensing Corporation In-loop block-based image reshaping in high dynamic range video coding
EP3386198A1 (en) * 2017-04-07 2018-10-10 Thomson Licensing Method and device for predictive picture encoding and decoding

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016164235A1 (en) * 2015-04-06 2016-10-13 Dolby Laboratories Licensing Corporation In-loop block-based image reshaping in high dynamic range video coding
EP3386198A1 (en) * 2017-04-07 2018-10-10 Thomson Licensing Method and device for predictive picture encoding and decoding

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
CHUJOH (TOSHIBA) T ET AL: "Video coding technology proposal by Toshiba", 1. JCT-VC MEETING; 15-4-2010 - 23-4-2010; DRESDEN; (JOINTCOLLABORATIVE TEAM ON VIDEO CODING OF ISO/IEC JTC1/SC29/WG11 AND ITU-TSG.16 ); URL: HTTP://WFTP3.ITU.INT/AV-ARCH/JCTVC-SITE/,, no. JCTVC-A117, 18 April 2010 (2010-04-18), XP030007559 *
FANGJUN PU (DOLBY) ET AL: "CE12-4: SDR In-loop Reshaping", no. m44267, 25 September 2018 (2018-09-25), XP030190982, Retrieved from the Internet <URL:http://phenix.int-evry.fr/mpeg/doc_end_user/documents/124_Macao/wg11/m44267-JVET-L0246-v1-JVET-L0246_SDR_inloop_reshaping.zip JVET-L0246_CE12-4_SDR_inloop_reshaping_v1.docx> [retrieved on 20180925] *
FRANCOIS (TECHNICOLOR) E ET AL: "CE12-related: block-based in-loop reshaping", no. JVET-M0109, 2 January 2019 (2019-01-02), XP030200178, Retrieved from the Internet <URL:http://phenix.int-evry.fr/jvet/doc_end_user/documents/13_Marrakech/wg11/JVET-M0109-v1.zip JVET-M0109.docx> [retrieved on 20190102] *
LU (DOLBY) T ET AL: "CE12: HDR In-loop Reshaping (CE12-5, 12-6, 12-7 and 12-8)", no. JVET-K0308, 15 July 2018 (2018-07-15), XP030199853, Retrieved from the Internet <URL:http://phenix.int-evry.fr/jvet/doc_end_user/documents/11_Ljubljana/wg11/JVET-K0308-v2.zip JVET-K0308_reshaping.docx> [retrieved on 20180715] *

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11343505B2 (en) 2019-02-01 2022-05-24 Beijing Bytedance Network Technology Co., Ltd. Configuring luma-dependent chroma residue scaling for video coding
US11394995B2 (en) * 2019-02-01 2022-07-19 Beijing Bytedance Network Technology Co., Ltd. Interactions between in-loop reshaping and inter coding tools
US11553203B2 (en) 2019-02-01 2023-01-10 Beijing Bytedance Network Technology Co., Ltd. Signaling of in-loop reshaping information using parameter sets
US11558636B2 (en) 2019-02-01 2023-01-17 Beijing Bytedance Network Technology Co., Ltd. Configuring luma-dependent chroma residue scaling for video coding
US11627333B2 (en) 2019-02-01 2023-04-11 Beijing Bytedance Network Technology Co., Ltd. Interactions between in-loop reshaping and palette mode
US20220239932A1 (en) 2019-03-14 2022-07-28 Beijing Bytedance Network Technology Co., Ltd. Signaling and syntax for in-loop reshaping information
US11412238B2 (en) 2019-03-14 2022-08-09 Beijing Bytedance Network Technology Co., Ltd. Signaling and syntax for in-loop reshaping information
US11375188B2 (en) 2019-03-23 2022-06-28 Beijing Bytedance Network Technology Co., Ltd. Restrictions on adaptive-loop filtering parameter sets
US11388406B2 (en) * 2019-05-20 2022-07-12 Dolby Laboratories Licensing Corporation Piece-wise-linear mapping for in-loop reshaping in image coding
CN112351264A (en) * 2020-11-04 2021-02-09 深圳Tcl数字技术有限公司 Display precision adjusting method and device, terminal equipment and storage medium
CN112351264B (en) * 2020-11-04 2023-05-23 深圳Tcl数字技术有限公司 Display precision adjusting method, device, terminal equipment and storage medium

Similar Documents

Publication Publication Date Title
WO2020086421A1 (en) Video encoding and decoding using block-based in-loop reshaping
US20220078405A1 (en) Simplifications of coding modes based on neighboring samples dependent parametric models
US20220109871A1 (en) Method and apparatus for video encoding and decoding with bi-directional optical flow adapted to weighted prediction
US20230232037A1 (en) Unified process and syntax for generalized prediction in video coding/decoding
US20220046244A1 (en) Directions for wide angle intra prediction
WO2022063729A1 (en) Template matching prediction for versatile video coding
EP3939322A1 (en) In-loop reshaping adaptive reshaper direction
WO2020112437A1 (en) Triangle and multi-hypothesis combination for video coding and decoding
WO2021130025A1 (en) Estimating weighted-prediction parameters
WO2020061319A1 (en) Harmonization of intra transform coding and wide angle intra prediction
US20230096533A1 (en) High-level constraint flag for local chroma quantization parameter control
US20240031607A1 (en) Scaling list control in video coding
US20220272356A1 (en) Luma to chroma quantization parameter table signaling
US20220368912A1 (en) Derivation of quantization matrices for joint cb-br coding
US20220224902A1 (en) Quantization matrices selection for separate color plane mode
US20210344962A1 (en) Method and apparatus for video encoding and decoding with signaling of coding type or coding tree type
EP4035367A1 (en) Video encoding and decoding using block area based quantization matrices
EP4070547A1 (en) Scaling process for joint chroma coded blocks
WO2021048077A1 (en) Transform size interactions with coding tools
EP4320862A1 (en) Geometric partitions with switchable interpolation filter
EP4035392A1 (en) Most probable mode signaling with multiple reference line intra prediction
WO2021001215A1 (en) Chroma format dependent quantization matrices for video encoding and decoding
WO2020117628A1 (en) Method and device for picture encoding and decoding

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19801451

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19801451

Country of ref document: EP

Kind code of ref document: A1