EA040654B1 - TRANSFORMATIONS FOR SIGNAL ENHANCEMENT CODING - Google Patents

Description

Technical field

The disclosure relates to methods and devices for encoding and/or decoding signals.

More specifically, the present disclosure relates to the encoding and decoding of video and image signals, but can be extended to any other type of data to be compressed and decompressed.

State of the art

The methods and apparatuses described herein are based on a common algorithm that builds on an existing encoding and/or decoding algorithm (which operates as a base layer for an enhanced layer) and that operates according to another encoding and/or decoding algorithm. Examples of existing encoding and/or decoding algorithms include, but are not limited to, MPEG standards such as AVC/H.264, HEVC/H.265, etc., and non-standard algorithms such as VP9, AV1, and others.

Brief summary of the invention

Various measures (eg, methods and devices for encoding and decoding) provided in accordance with the present disclosure are defined in the appended claims.

Additional features and advantages will become apparent from the following description, by way of example only, which is made with reference to the accompanying drawings.

Brief description of graphic materials

In FIG. 1 is a diagram showing an example of LOQ-1 residual coding;

in fig. 2 is a diagram showing an example of LOQ-0 residual coding.

in fig. 3 is a diagram showing an example of a decoding process and system;

in fig. 4 is a diagram showing an example of a data structure for the converted residuals;

in fig. 5 is a diagram showing an example of calculating a compression error;

in fig. 6 is a diagram showing an example of a mean value decomposition process;

in fig. 7 is a diagram showing an example of a horizontal decomposition process;

in fig. 8 is a diagram showing an example of a vertical decomposition process;

in fig. 9 is a diagram showing an example of a diagonal decomposition process.

in fig. 10 is a diagram showing an example of computing a decomposition of the average value of LoQ-0;

in fig. 11 is a diagram showing an example of an inverse directional decomposition (DD) (LOQ-1) transformation process; and in FIG. 12 is a diagram showing an example of how to reconstruct the average component in the decoder starting from the encoded average, A _enc .

Detailed Description of the Invention

The overall algorithm described herein hierarchically encodes and/or decodes a video frame, as opposed to using the block-based approaches used in the MPEG family of algorithms. Techniques for hierarchical frame coding that are described herein include creating residuals for a full frame and then a decimated frame, and so on. Different levels in the hierarchy may refer to different resolutions, referred to here as quality levels - LOQ - and residual data may be generated for different levels. In the examples, video compression residual data for a full video frame may be referred to as LOQ-0 (eg, 1920x1080 for a HD video frame), while decimated frame data may be referred to as LOQx. In these cases, x denotes the number of hierarchical decimations. In some of the examples described here, the variable x has a maximum value of one, and therefore there are exactly two hierarchical levels for which compression residuals will be generated (for example, x=0 and x=1).

In FIG. 1 shows an example of how encoded data for one quality level, LOQ-1, is generated in an encoder.

The general algorithm and methods are described using the AVC/H.264 encoding/decoding algorithm as an example of a basic algorithm. However, other encoding/decoding algorithms can be used as base algorithms without any impact on the way the whole algorithm works. In FIG. 1 shows a process 100 for generating entropy-coded residuals for the LOQ-1 hierarchy level.

The first step 101 is to decimate the incoming uncompressed video by a factor of two. This may include downsampling an input frame 102 (labeled Input Frame in Figure 1) having a height H and a width W to create a sparse frame 103 (labeled Half-2D size in Figure 1) having a height H/2 and a width W/2 . The downsampling process involves halving each axis, and is efficiently accomplished by using 2x2 grid blocks. Downsampling can be performed in a variety of ways, examples of which include, but are not limited to, Lanczos averaging and resampling.

The decimated frame 103 then passes through a base coding algorithm (in this example AVC/H.264 coding algorithm), where the entropy-coded reference frame 105 (labeled Base Half Size in FIG. 1) has a height of H/2 and width W/2, then generated by object 104 labeled H.264 Encode in FIG. 1 and stored as H.264 entropy encoded data. Object 104 may contain an encoding component of a base encoder/decoder, such as a base codec or a base encoding/decoding algorithm. The base-encoded data stream may be output as an entropy-encoded reference frame 105, where the base-encoded data stream has a lower resolution than the input data stream that input frame 102 provides.

In this example, the encoder then simulates decoding the output of object 104. A decoded version of the encoded reference frame 105 is then generated by object 106 labeled H.264 Decode in FIG. 1. Object 106 may include decoding a base codec component. The decodable version of the encoded reference frame 105 may be a variant of the decimated frame 103 that will be produced by the decoder upon receipt of the entropy-coded reference frame 105.

In the example in FIG. 1, the difference between the decoded reference frame output by object 106 and the decimated frame 103 is calculated. This difference is referred to here as LOQ-1 residuals. The difference forms the input data for the transform block 107 .

The transformation (in this example, the Hadamard-based transformation) used by the transformation block 107 transforms the difference into four components.

Transform block 107 may perform guided (or directed) decomposition to obtain a set of coefficients or components that relate to various aspects of the set of residues. In FIG. 1, the conversion unit 107 generates the coefficients A (average), H (horizontal), V (vertical), and D (diagonal). Transform block 107 in this case uses directional correlation between LOQ-1 residuals, which has proven to be surprisingly efficient in addition to or as an alternative to performing a transform operation for a higher quality level of LOQ-0. level. The LOQ-0 transformation is described in more detail below. In particular, it has been found that, in addition to using directional correlation in LOQ-0, directional correlation can also be present and surprisingly effectively used in LOQ-1 to provide more efficient coding than using directional correlation in LOQ-0 alone, or not at all. directional correlation on LOQ-0.

The coefficients (A, H, V and D) generated by the transformation block 107 are then quantized by the quantization block 108 . Quantization may be performed using variables referred to as step widths (also referred to as step sizes) to obtain quantized transformed residuals 109. Each quantized transformed residual 109 has a height of H/4 and a width of W/4. For example, if a 4x4 block of an input frame is taken as a reference, each quantized transform residue 109 may have a height and width of one pixel. Quantization involves reducing the decomposition components (A, H, V, and D) by a predetermined factor (step width). The reduction may be performed by dividing, eg dividing the coefficient values by the step width, eg representing the bin width for quantization. The quantization may generate a set of coefficient values having a range of values that is smaller than the range of values input to quantizer 108 (eg, converted values in the range of 0 to 21 may be reduced using a step width of 7 to a value range of 0 to 3). In a hardware implementation, the reciprocal of a set of step width values may be precomputed and used to perform reduction by multiplication, which may be faster than division (eg, multiplying by the reciprocal of the step width value).

The quantized residuals 109 are then entropy encoded to remove any redundant information. Entropy encoding may include, for example, data transmission through a run-length encoder (RLE) encoder 110 followed by a Huffman encoder 111 .

The quantized, encoded components (Ae, He, Ve, and De) are then placed on the serial stream with definition packets inserted at the beginning of the stream. Definition packages may also be referred to as header information. Definition packets can be inserted into each frame. This final step may be performed using file serialization procedure 112 . The definition packet data may include information such as the specification of the Huffman encoder 111, the type of upsampling applied, whether the A and D coefficients are discarded, and other information to enable a decoder to decode the streams. Thus, the output of the 113 residues is entropy encoded and serialized.

Both reference data 105 (half-size frame, base entropy-coded frame) and entropy-coded LOQ-1 residual data 113 are generated for decoding by the decoder during the reconstruction process. In one case, reference data 105 and entropy-encoded LOQ-1 residue data 113 may be stored and/or buffered. Reference data 105 and entropy-encoded LOQ-1 residue data 113 may be transmitted to a decoder for decoding.

In the example in FIG. 1 performs a series of additional operations to produce a set of residues

- 2 040654 at another (for example, higher) quality level - LOQ-0. In FIG. 1, the encoder simulates a number of decoder operations for the LOQ-1 stream.

First, the quantized output signal 109 is branched and inverse quantization 114 (or dequantization) is performed. This generates a representation of the coefficient values output by the transform block 107 . However, the representation output by the dequantization block 109 will be different from the output of the transform block 107 because errors will be introduced due to the quantization process. For example, multiple values in the range 7 to 14 can be replaced by a single quantized value of 1 if the step width is 7. During dequantization, this single value of 1 can be dequantized by multiplying by the step width to generate a value of 7. Therefore, any value in the range from 8 to 14 will have an error at the output of the dequantization block 109. Because the higher quality LOQ-0 is generated using the dequantized values (eg, including decoder simulation), the LOQ-0 residuals can also encode quantization/dequantization error correction.

Second, the inverse transform block 115 is applied to the dequantized coefficient values output by the dequantization block 114 . The inverse transform block 115 applies a transform that is the inverse of the transform performed by the transform block 107 . In this example, transform block 115 performs an inverse Hadamard transform, although other transforms may be used. An inverse transform block 115 converts the dequantized coefficient values (eg, values for A, H, V, and D in a coding block or unit) back to corresponding residual values (eg, representing a reconstructed version of the input to transform block 107). The output of the demapper 115 is a set of recovered LOQ-1 residues (eg, representing the output of the decoding process of the LOQ-1 decoder). The recovered LOQ-1 residuals are added to the decoded reference data (eg, the output of decoding entity 106) to generate a reconstructed video frame 116 (labeled Half-2D size Recon (To LOQ-0) in FIG. 1). having a height of H / 2 and a width of W / 2. The reconstructed video frame 116 is very similar to the originally decimated input frame 103 in that it is reconstructed from the output of decode object 106, but with the recovered LoQ-1 residuals added. The reconstructed video frame 116 is an intermediate output for the LOQ-0 mechanism. This process mimics the decoding process, and hence the original decimated frame 103 is not used. Adding the recovered LOQ-1 residuals to the decoded base stream, i.e., the output of decoding entity 106, allows the LOQ-0 residuals to also correct errors that are introduced into the LOQ-1 stream through quantization (and in some cases transform), for example, a See also errors related to downsampling and upsampling.

In FIG. 2 shows an example of how LOQ-0 is generated by 200 on the encoder.

To derive LOQ-0 residuals, a LOQ-1 size reconstructed frame 216 (labeled Half 2D Recon size (from LOQ-1) in FIG. 2) is obtained as described above with reference to FIG. 1. For example, reconstructed frame 216 of size LOQ-1 contains reconstructed video frame 116.

The next step is to upsample the reconstructed frame 216 to its full size, WxH. In this example, the zoom is doubled. At this point, various algorithms may be used to improve the upsampling process, examples of which include, but are not limited to, nearest, bilinear, sharp, or cubic algorithms. The reconstructed full size frame 217 is designated as the predicted frame in FIG. 2 because it is a prediction of a frame having full width and height decoded by the decoder. The reconstructed full frame 217, having a height H and a width W, is then subtracted from the original uncompressed video input 202, which creates a set of residuals, referred to herein as LOQ-0 residuals. LOQ-0 residuals are generated at a quality level (eg, resolution) that is higher than LOQ-1 residuals.

Similar to the LOQ-1 process described above, the LOQ-0 residues are converted by transform block 218 . This may include using a directional decomposition, such as a Hadamard transform, to obtain the A, H, V, and D coefficients or components. The output of transform block 218 is then quantized by quantizer 219 . This can be done based on the determined step width as described for the first quality level (LOQ-1). The output of quantizer 219 is a set of quantized coefficients, and in FIG. 2, they are then entropy encoded 220, 221 and serialized in file 222. Again, entropy encoding may include applying sequential encoding 220 and Huffman encoding 221. The result of the entropy encoding is a set of entropy encoding output residues 223 . They form the LOQ-0 stream, which can be output by the encoder, as well as the LOQ-1 stream (ie 113) and the main stream (ie 105). Streams may be stored and/or buffered prior to subsequent decoding by a decoder.

As can be seen in FIG. 2, a predicted mean component 224 (described in more detail below and labeled Ae _nc below) can be obtained using data from the reconstructed video frame 116 (LOQ-1) prior to the upsampling process. This can be used in place of the A (middle) component in transform block 218 to further improve the efficiency of the coding algorithm.

In FIG. 3 schematically shows an example of how the decoding process 300 is performed. This decoding process 300 may be performed by a decoder.

The decoding process 300 starts with three input streams. Thus, the decoder input consists of entropy encoded data 305, entropy encoded LOQ-1 residual data 313, and entropy encoded LOQ-0 residual data 323 (represented in FIG. 3 as data serialized as files). Entropy encoded data 305 includes a reduced size encoded base such as data 105 as output in FIG. 1. The entropy encoded data 305 is, for example, half the size with dimensions W/2 and H/2 of a full frame having dimensions W and H.

The entropy encoded data 305 is decoded by the core decoder 306 using a decoding algorithm corresponding to the algorithm that was used to encode the data (in this example, the AVC/H.264 decoding algorithm). This may correspond to decode object 106 in FIG. 1. At the end of this step, a reduced size (eg, half size) decoded video frame 325 (referred to as AVC/H.264 video in this example) is created. This can be thought of as a standard definition video stream.

In parallel, LOQ-1 entropy encoded residue data 313 are decoded. As explained above, LOQ-1 residuals are encoded into four components (A, V, H and D) which, as shown in Figure 3, are one-quarter the size of a full frame, namely W/4 and H/4. This is because, as also described below and in previous patent applications US 13/893669 and PCT/EP2013/059847, the contents of which are incorporated herein by reference, the four components contain all the information associated with a particular direction in untransformed residues. , (for example, the components are defined relative to the block of untransformed residues). As described above, the four components can be generated by applying a 2x2 transformation kernel to the residuals whose dimensions for LOQ-1 will be W/2 and H/2, in other words, the same dimension as the entropy-coded reduced size data 305. In progress 300 decoding as shown in FIG. 4, the four components are entropy-decoded in entropy decoder 326, then dequantized in dequantizer 314 before an inverse transform is applied via inverse transform 315 to generate a representation of the original LOQ-1 residuals (e.g., input to transform 107 in FIG. 1). The inverse transform may comprise an inverse Hadamard transform, such as applied to a 2x2 block of residual data. Dequantizer 314 is the inverse of quantizer 108 described above with reference to FIG. 1. In this step, the quantized values (ie, the output of entropy decoder 326) are multiplied by a step width factor (ie, step size) to generate reconstructed transformed residuals (ie, components or coefficients). It can be seen that blocks 114 and 115 in FIG. 1 reflect blocks 314 and 315 in FIG. 3.

The decoded LOQ-1 residuals, such as the output of demapper 315, are then added to the decoded video frame, such as the output of base decoder 306, to create a reduced size (in this example, half size) reconstructed video frame 316, indicated in FIG. . 3 as Half-2D size Recon. This reconstructed video frame 316 is then upsampled to bring it to full resolution (eg, quality level ⁰ from quality level ¹ ) using an upsampling filter such as bilinear, bicubic, sharp, etc. In this example, the reconstructed video frame 316 is upsampled from half-width (W/2) and half-height (H/2) to full width (W) and full height (H)).

The reconstructed upsampled video frame 317 will be a predicted frame in LOQ-0 (full size, WxH), to which the decoded LOQ-0 residuals are then added.

In FIG. 3, the residual data 323 encoded with LOQ-0 is decoded using an entropy decoder 327, a dequantizer 328, and an inverse transform 329. As described above, LOQ-0 residue data 323 is encoded using four components (ie, converted to A, V, H, and D components), which, as shown in FIG. 3 have a size equal to half the size of a full frame, namely W/2 and H/2. This is because, as described here and in previous patent applications US 13/893669 and PCT/EP2013/059847, the contents of which are incorporated herein by reference, the four components contain all information related to residues and are generated by applying 2x2 kernel-to-remainder conversions, the dimension of which for LOQ-0 will be W and H, in other words, the same dimension of the full frame. The four components are entropy decoded by an entropy decoder 327, then dequantized by a dequantizer 328, and finally converted 329 back to the original LOQ-0 residues by a transform by an inverse transform 329 (e.g., in this example, a 2x2 inverse Hadamard transform).

- 4 040654

The decoded LOQ-0 residuals are then added to the predicted frame 317 to obtain a reconstructed complete video frame 330. Frame 330 is an output frame having a height H and a width W. Therefore, the decoding process 300 in FIG. 3 allows two user data items to be output: the main decoded video stream 325 at a first quality level (e.g., half-resolution stream on LOQ-1) and a full resolution or higher resolution video stream 330 at the highest quality level (e.g., full resolution stream on LOQ -0).

The above description has been made with reference to specific dimensions and underlying algorithms. However, the above methods are applicable to other sizes and/or underlying algorithms. The above description is given only as an example of the more general concepts described in this document.

In FIG. 4 shows a representation of an exemplary residue data structure 400.

In the encoding/decoding algorithm described above, there are usually three planes (e.g. YUV or RGB) with two levels of quality (LOQ), which are described as LOQ-0 (or high level, full resolution) and LOQ-1 (or lower level, reduced size resolution such as half resolution) in each plane. Each plane may refer to a different color component of the video data. Each LOQ contains four components, namely A, H, V and D. In some examples, they can be considered as different levels in each plane. Thus, a frame of video data with a given quality level can be defined by a set of planes, where each plane has a set of layers. In the examples in Figures 1-3, there are 2x3x4 = 24 surfaces in total, that is, 2 quality levels, 3 color components, and 4 component or coefficient layers. As described with reference to FIG. 1 and 2, 12 of these surfaces are full size (eg W x H for LOQ-0) and 12 are reduced (eg W/2 x H/2 for LOQ-1).

As described above, a Directional Decomposition (DD) transform can be used to decompose the error component (i.e., difference or residuals) between the downsampled input frame 103 and the decoded reduced size base version of the same frame, (e.g., as output by the object 106 decoding) into four separate components; medium (A), horizontal (H), vertical (V) and diagonal (D). This operation can be performed on a 2x2 block grid. Each grid is independent of its neighbors. Therefore, it is suitable for efficient implementation, such as fully parallel operation. However, since all the operations used for decomposition are linear, it is possible to perform this operation using the just-in-time (JIT) (on the fly) processing paradigm.

In FIG. 5 shows an example of a compression error calculation that can be performed for a first quality level (LOQ-1) 500.

In FIG. 5, the LOQ-1 residual set is computed as the difference between the base reference decoded frame 531 (eg, output of decode object 106 in FIG. 1) and the downsampled input frame 503 (eg, downsampled frame 103 in FIG. 1. FIG. 5, the base decoded frame 531 is an H.264-based frame that is subtracted from the downsampled frame 503 to obtain a set of LOQ-1 residuals 532. The set of LOQ-1 residuals can be considered a compression error because they represent the difference between the input to the base encoder and the decoded output of the same base encoder, i.e. the differences between data 503 and 531 can be considered as a result of the encoding and decoding process for the main stream, and these differences are usually the result of the lossy coding algorithms applied by the base encoder.In addition to compression errors , LOQ-1 residuals 532 may also represent other artifacts generated by the base encoder process. decoding and basic decoding, which may include motion correction artifacts, blocking artifacts, quantization artifacts, symbol encoding artifacts, etc.

In FIG. 6 shows an example of an average decomposition process 600 that can be used for a first quality level (LOQ-1). For example, the mean decomposition process 600 can be used to determine the Mean A component, as described above. In FIG. 6, the average decomposition is calculated as the average of all compression error pixels (residuals) in the current 2x2 grid of compression error frame 632. The average decomposition may be repeated for multiple 2x2 grids in compression error frame 632, such as in FIG. 6 shows the first and last 2x2 grid or coding block for a frame. The average decomposition may be performed in a manner similar to the downsampling shown by arrow 101 in FIG. 1.

In FIG. 7 shows an example of a horizontal decomposition process 700 that can be used for a first quality level (LoQ-1). For example, the horizontal decomposition process 700 may be used to determine the horizontal H component as described above. In FIG. 7, the operation calculates a normalized horizontal difference between the residuals in the 2x2 grid of compression error frame 732. For example, as shown in FIG. 7, the residuals in the first column of the 2x2 grid (D ₀₀ and D ₁₀ ) are added, and then the residuals in the second column of the 2x2 grid (D ₀₁ and Du) are subtracted. from the sum. In FIG. 7, a normalizing division by 4 is used to create the horizontal H component. The operation of obtaining a horizontal decomposition is a linear process and, therefore, if necessary, can be performed on the fly. Visually, the result of this operation will look mostly vertical in nature if there are any obvious errors (residues) across the entire plane. Ideally, similar errors (residuals) will exist in the plane (no added entropy) and therefore result in a reduction in the amount of data compressed by the entropy encoder.

In FIG. 8 shows an example of a vertical decomposition process 800 that can be used for a first quality level (LoQ-1). For example, the vertical decomposition process 800 can be used to determine the vertical V component as described above. This operation calculates the normalized difference in the vertical plane between the residuals in the 2x2 grid of the compression error frame 832. For example, as shown in FIG. 8, the residuals in the first row of the 2x2 grid (D ₀₀ and D ₀₁ ) are added, and then the residuals in the second row of the 2x2 grid (D ₁₀ and Du) are subtracted from the sum. In FIG. 8, a normalizing division by 4 is used to create the vertical V-component. The operation to obtain a vertical decomposition is a linear process and therefore can be performed on the fly if desired, for example when 2x2 residual sets are received in a transformation block such as 107.

In FIG. 9 shows an example of a diagonal decomposition process 900 that can be used for a first quality level (LoQ-1). For example, the diagonal decomposition process 900 can be used to determine the -D diagonal component as described above. This operation calculates the normalized difference in the diagonal plane between the residuals in the 2x2 grid of frame 932 of the compression error. For example, as shown in FIG. 9, the difference of the residuals in the first row of the 2x2 grid (D ₀₀ and D ₀₁ ) is determined, and then the difference of the residuals in the second row of the 2x2 grid (D ₁₀ and Du). In FIG. 9, a normalizing division by 4 is used to create the diagonal component of D.

In FIG. 10 shows an example of a directed decomposition (DD) transform process 1000 that can be performed at a high quality level (LoQ-0). In FIG. 10, the reconstructed LoQ-1 frame 1016 is upsampled. As described with reference to FIG. 1 and 2, the reconstructed frame 1016 may comprise the sum of the decoded LOQ-1 residuals (eg, as output of inverse transform block 115) and the decoded base encoded frame (eg, as output of decoding entity 106). In this case, upsampling takes the single reconstructed frame pixel R ₀ and transforms it into an upsampled 2x2 LOQ-1 prediction block (see above). This can be done for all of the reconstructed frame pixel values in the reconstructed LoQ-1 frame 1016. In FIG. 10, the upsampled LOQ-1 predictions are then subtracted from the full size input frame 1002 to generate residuals 1033 for the LoQ-0 processing step. The horizontal, vertical and diagonal expansions for LoQ-0 can be calculated in the same way as described above for LOQ-1. However, the average decomposition can be calculated differently to reduce the entropy of the quantized LOQ-0 transformed residuals to be encoded.

In FIG. 10, the reconstructed pixel value R0 from the previous step (LoQ-1) is used in the calculation of the mean component 1024 for the transformed LOQ-0 residual data, rather than in recalculating the mean from the reconstruction error 1033. For example, in transform block 218 in FIG. 2, the average A may not be determined using the inputs from the sum of 217 and 202 as for the H, V, and D components. Using the configuration shown in FIG. 10, the calculation may produce fewer errors (residuals) if pre-upsampled data 1016 is used. This effectively eliminates any additional errors due to upsampling filters and hence will result in lower entropy at the input of entropy encoders.

In particular, R0 is the reconstructed element at the LOQ-1 level, obtained by adding the reduced size decoded frame to the LOQ-1 residuals as described above. A single upsampling R0 element will result in four elements in upsampling LOQ-1 prediction frame 1017, namely H ₀₀ , H ₀₁ , H ₁₀ and H11, assuming upsampling from half size to full size. In FIG. 10, the reconstructed element R ₀ is subtracted from the average of the four elements in the original image 1002, namely I ₀₀ , I ₀₁ , I ₁₀ and I11, corresponding in position to the four elements in the upsampled predictive LoQ-1 frame 1017, H ₀₀ , H ₀₁ . H ₁₀ and H11. For example, the average value in block 1030 may be generated similar to the calculation in FIG. 6. The resulting average value, denoted A _enc , 1024 is then quantized and encoded for transmission to the decoder, for example, instead of the average - A - the output signal of the LOQ-0 transform block 1018. If the average of the four elements in the original image 1002, namely I00, I ₀₁ , I ₁₀ and I11, is denoted by A _b then A _enc = A! - R0. The resulting average 1024 encoding may be generated for multiple blocks or 2x2 grids throughout the frame.

Using A _enc instead of the standard mean A (which would be the mean of the reconstruction errors from D ₀₀ to D11 in the 2x2 block shown in 1033) is efficient because the entropy content of Aenc is lower than that of the mean (A) and hence it results in more efficient coding. This is because, if R ₀ was recovered correctly (for example, the error introduced by the encoder and decoder was corrected properly by the LOQ-1 residuals), then the difference between R ₀ and the average of the four original elements of input frame 1002 in should be zero in most cases. On the other hand, the standard mean (A) will contain significantly fewer zeros, since the effects of upsampling and downsampling will be taken into account.

In FIG. 11 shows an example of an inverse DD transform 1100. For example, this can be used to perform an inverse transform in one of the blocks 115, 314 in FIG. 1 and 3.

The goal of this process 1100 is to convert the (directed) decomposed values back to the original residuals. The residuals are values that were obtained by subtracting the reconstructed video frame from the ideal input (or downsampled) frame. The inverse transform DD 1100 shown in FIG. 11 is the LoQ-1 inverse transform performed in LoQ-1. The inverse transformation of LoQ-0 performed to LoQ-0 may be different when a predictive mean is used. One example of another implementation is described below with reference to FIG. 12.

In FIG. 12 shows an example of a LoQ-0 inverse transform 1200 that may use the average coding value 1024 described with reference to FIG. 10.

Due to the method used to average the decomposition, there is a difference in calculating the inverse transform for LoQ-0 compared to LoQ-1. In particular, an extra step is used to transform the mean component so that it conforms to the input minus the predicted remainder format. This is used to perform back calculations.

As described above, the component Ae _nc corresponds to the average component (A) calculated by subtracting R0 (LOQ-1 recovery element) from the average value of the corresponding elements in the original input frame (T ₀₀ to Tc), which can be expressed as Ae _nc = A! - R0. If the 2x2 grid average of the reconstructed upsampled LoQ-1 frame is denoted as Au, then A=A| - Au, since the mean (A) in LoQ-O is based on the difference between the elements in the original input frame and the elements in the LoQ-1 frame after upsampling and reconstruction. This can be rewritten as A = {AI - R ₀ } + {R ₀ - AU} or, using the above equation for A _enc , A = A _enc + {R ₀ - AU}. The transformed mean (transformed A) therefore consists in adding a LOQ-1 reconstruction element, R0,, to the decoded Ae _nc (referred to as A _invvQ in Fig. 12 - i.e. Ae _nc , i.e. entropy decoded and dequantized ) and subtracting the 2x2 grid average for the upsampled LoQ-1 frame and reconstructed frame, AU. Thus, on the decoder side A, _{the converted} = A _invvQ + R0 - AU. As can be seen from the formula shown in Fig. 12, the resulting equation corresponds to what should have been the actual average (A) component to be encoded in the encoder. Thereafter, the same steps as described in Fig. 11 are executed (with A in the inverse transformation, replaced by the transformed A) in order to generate the remainders of D from ₀₀ dots D.

The present disclosure describes a method for encoding and decoding a signal, in particular a video signal and/or an image signal.

A method for encoding a signal is described, the method comprising the steps of receiving an input frame and processing the input frame to generate at least one first set of residual data, wherein the residual data allows the decoder to reconstruct the original frame from the reference reconstructed frame.

In some examples, the method includes deriving a reconstructed frame from a decoded frame received from a decoding module, the decoding module being configured to generate said decoded frame by decoding a first encoded frame that was encoded according to the first encoding method. The method further comprises downsampling an input frame to obtain a downsampled frame and passing said downsampled frame to an encoding module configured to encode said downsampled frame in accordance with a first encoding method to generate a first encoded frame. Receiving the reconstructed frame may further comprise upsampling the decoded frame to generate the reconstructed frame.

In some examples, the method includes deriving a reconstructed frame from a combination of a second set of residual data and a decoded frame obtained from a decoding module, the decoding module being configured to generate said decoded frame by decoding a first encoded frame that was encoded according to the first encoding method. The method further includes downsampling the input frame to obtain a downsampling frame and passing said downsampling frame to an encoding module configured to encode said downsampled frame in accordance with the first encoding method to generate a first encoded frame. The method further comprises generating said second set of residual data by taking the difference between the decoded frame and the downsampled frame. The method further comprises encoding said second set of residual data to generate a first set of encoded residual data. The encoding of said second set of residual data may be performed according to the second encoding method. The second way to encode

Claims (17)

The conversion process involves converting the second residual data set to the transformed second residual data set. Transforming the second residual data set includes selecting a subset of the second residual data set and applying a transformation to said subset to create a corresponding subset of the transformed second residual data set. One of a subset of the transformed second residual data set may be obtained by averaging a subset of the second residual data set. Obtaining the reconstructed frame may further comprise upsampling the combination of the second set of residual data and the decoded frame to generate the reconstructed frame. In some examples, creating at least one set of residual data includes obtaining a difference between a reference reconstructed frame and an input frame. The method further includes encoding said first set of residual data to generate a first set of encoded residual data. Encoding of said first set of residual data may be performed according to a third encoding method. The third encoding method includes converting the first residual data set into a transformed first residual data set. Transforming the first residual data set includes selecting a subset from the first residual data set and applying a transformation to said subset to generate a corresponding subset from the transformed first residual data set. One of the subsets of the transformed first residual data set may be obtained by the difference between the subset mean of the input frame and the corresponding element of the combination of the second residual data set and the decoded frame. A signal decoding method is also described, comprising receiving an encoded frame and at least one set of encoded residual data. The first encoded frame may be encoded using the first encoding method. At least one set of residual data may be encoded using the second and/or third encoding method. The method further includes transmitting the first encoded frame to the decoding module, wherein the decoding module is configured to generate a decoded frame by decoding an encoded frame that has been encoded according to the first encoding method. The method further includes decoding at least one set of encoded residual data according to a respective encoding method used to encode it. In some examples, the first set of encoded residual data is decoded by applying a second decoding method corresponding to said second encoding method to obtain a first set of decoded residual data. The method further comprises combining the first set of residual data with the decoded frame to obtain a combined frame. The method further comprises upsampling the combined frame to obtain a decoded frame reference. The method further comprises decoding the second set of coded residual data by applying a third decoding method corresponding to said third encoding method to obtain a second set of decoded residual data. The method further comprises combining the second set of decoded residual data with a reference decoded frame to obtain a reconstructed frame. In some embodiments, the method includes upsampling a decoded frame to obtain a decoded frame reference. The method further includes decoding the encoded residual data set by applying a second or third decoding method corresponding to said second or third encoding method to obtain a decoded residual data set. The method further includes combining the set of decoded residual data with the reference decoded frame to obtain a reconstructed frame. The above embodiments should be understood as illustrative examples. Other embodiments are contemplated. It should be understood that any feature described with respect to one embodiment may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or in any combination of any other of implementation options. In addition, equivalents and modifications not described above may also be used without departing from the scope of the invention as defined in the accompanying claims. CLAIM 1. Signal encoding method, including receiving an input frame; downsampling the input frame to obtain a downsampled frame; - 8 040654 transmitting a downsampled frame to an encoding module, wherein the encoding module is configured to encode the downsampled frame to generate an encoded frame; receiving a decoded frame from a decoding module, the decoding module being configured to generate a decoded frame by decoding the encoded frame; generating a first set of residual data by taking the difference between the decoded frame and the downsampled frame; upsampling the combination of the first set of residual data and the decoded frame to generate a reference reconstructed frame; generating a second set of residual data by taking the difference between the reference reconstructed frame and the input frame; encoding the first and second residual data sets to generate the first and second encoded residual data sets, respectively, the encoding comprising converting the first and second residual data sets to the transformed first and second residual data sets, respectively; and outputting the encoded frame and the first and second sets of encoded residual data to the decoder, wherein the encoded frame and the first and second sets of residual data allow the decoder to reconstruct the input frame. 2. The method of claim 1, wherein the coding unit is configured to downsample the frame according to the first coding method to generate the encoded frame, and wherein the encoding of the first residual data set is performed according to the second coding method, the second coding method being different from the first encoding method; and/or encoding of the second set of residual data is performed according to a third encoding method, wherein the third encoding method is different from the first encoding method. 3. The method of claim 1 or 2, wherein the encoding and decoding modules are configured to use a block-based encoding algorithm. 4. The method according to any one of claims 1 to 3, wherein the first set of residual data represents a compression error associated with the encoding module and/or the decoding module. 5. The method according to any one of claims 1-4, wherein the second set of residual data represents the recovery error associated with downsampling and upsampling. 6. A method according to any one of claims 1 to 5, wherein the method is performed in a hierarchical coding system having exactly two hierarchical levels. 7. The method of claim 6, wherein one of the two hierarchical levels corresponds to the base level and the other of the two hierarchical levels corresponds to the enhancement level. 8. The method according to any one of claims 1 to 7, wherein the transform uses a Hadamard transform. 9. A method according to any one of claims 1 to 8, wherein the transformation uses a 2x2 transformation kernel. 10. The method according to any one of paragraphs.1-9, in which the transformation of the first set of residual data includes selecting a subset of the first set of residual data; and applying a transform to a subset of the first residual data set to create a corresponding subset of the transformed first residual data set. 11. The method of claim 10, wherein one of a subset of the transformed first residual data set is obtained by averaging a subset of the first residual data set. 12. The method according to any one of paragraphs.1-11, characterized in that the transformation of the second set of residual data includes selecting a subset of the second set of residual data; and applying a transform to a subset of the second residual data set to generate a corresponding subset of the transformed second residual data set. 13. The method of claim 12, wherein one of the subset of the transformed second residual data set is obtained by a difference between the mean of the input frame subset and the corresponding combination element of the first residual data set and the decoded frame. 14. The method of any one of claims 1-13, wherein the encoding comprises quantizing at least some of the transformed first and second residual data sets, respectively, into quantized first and second residual data sets, respectively. 15. The method of claim 14, wherein the encoding comprises entropy encoding the quantized first and second residual data sets, respectively, into the entropy encoded first and second residual data sets. 16. The method according to any one of claims 1 to 15, wherein the signal comprises a video signal and/or an image signal. 17. A method for encoding a signal, the method comprising: generating first and second sets of residual data; and encoding the first and second sets of residual data to generate the first and second sets of encoded residual data, respectively, the encoding comprising transforming