Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/12—Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/154—Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/189—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
  • H04N19/19—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding using optimisation based on Lagrange multipliers

Definitions

• the present embodiments generally relate to pre-processing and encoding of pixels in a picture, and in particular to such pre-processing and encoding that improves luminance and/or chrominance values of pixels in a computational efficient way.
• a non-linear transfer function converts linear samples to non-linear samples with the purpose to mimic human vision.
• a simple example of a non-linear transfer function is ⁇ TM), where gamma is 2.2.
• An example of another transfer function is the one used in Society of Motion Picture & Television Engineers (SMPTE) specification ST 2084 [1].
• SMPTE Society of Motion Picture & Television Engineers
• xa amma can be used to go back to linear samples.
• One example is the way to carry out conversion from RGB 4:4:4 to Y'CbCr 4:2:0 that is described in [2], which we will refer to as the "anchor" way of processing in this document.
• RGB 4:4:4 is the color format typically used by cameras to capture video and by displays to present the video. To compress the video with less perceptual artifacts, RGB 4:4:4 is typically converted to Y'CbCr 4:2:0 before compression. In this case, RGB 4:4:4 is transferred by a non-linear transfer function to R'G'B' 4:4:4 which then is converted to Y'CbCr 4:4:4 by a linear color transform. Finally, the chroma samples Cb and Cr are subsampled, by a factor two in both vertical and horizontal directions, to quarter resolution resulting in Y'CbCr 4:2:0. As described in Annex B, the anchor way of processing gives rise to situations where changes between two colors of similar luminance can result in a reconstructed picture or image with very different luminances.
• An aspect of the embodiments relates to a method of pre-processing a pixel in a picture.
• the method comprises determining, based on a minimum color component value for the pixel, whether a default processing chain is used to derive a luma component value, a first subsampled chroma component value and a second subsampled chroma component value or whether an auxiliary processing chain is used to derive at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the device comprises a determining unit for determining, based on a minimum color component value for the pixel, whether a default processing chain is used to derive a luma component value, a first subsampled chroma component value and a second subsampled chroma component value or whether an auxiliary processing chain is used to derive at least one of the luma component value, the first subsampled chroma component value and the second subsample dchroma component value.
• the device also comprises a deriver for deriving the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to default processing chain or the least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to the auxiliary processing chain.
• the device comprises a processor and a memory comprising instructions executable by the processor.
• the processor is operative to determine, based on the minimum color component value, whether the default processing chain is used to derive the luma component value, the first subsampled chroma component value and the second chroma component value or whether the auxiliary processing chain is used to derive the at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the processor is also operative to derive the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to default processing chain or the least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to the auxiliary processing chain.
• the processor is further operative to encode the luma component value, the first subsampled chroma component value and the second chroma component value.
• a further aspect of the embodiments relates to a device for encoding a pixel in a picture.
• the device comprises a determining unit for determining, based on the minimum color component value, whether the default processing chain is used to derive the luma component value, the first subsampled chroma component value and the second chroma component value or whether the auxiliary processing chain is used to derive the at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the device also comprises a deriver for deriving the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to default processing chain or the least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to the auxiliary processing chain.
• the device further comprises an encoder for encoding the luma component value, the first subsampled chroma component value and the second chroma component value.
• Another aspect of the embodiments relates to a computer program comprising instructions, which when executed by a processor, cause the processor to determine, based on a minimum color component value for a pixel in a picture, whether a default processing chain is used to derive a luma component value, a first subsampled chroma component value and a second subsampled chroma component value or whether an auxiliary processing chain is used to derive at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• a related aspect of the embodiments defines a carrier comprising a computer program according to above.
• the carrier is one of an electric signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal, or a computer-readable storage medium.
• the present embodiments provide a pixel pre-processing and encoding that combats artifacts that otherwise may occur due to usage of a non-linear transfer function in combination with chroma subsampling. The quality improvement in luminance and/or chrominance is achieved at a low cost with regard to processing time.
• Fig. 1 is a flow chart illustrating a method of pre-processing a pixel in a picture according to an embodiment
• Fig. 2 is a flow chart illustrating additional, optional steps of the method shown in Fig. 1 according to an embodiment
• Fig. 3 is a flow chart illustrating a method of pre-processing a pixel in a picture according to another embodiment
• Fig. 4 is a flow chart illustrating a method of pre-processing a pixel in a picture according to a further embodiment
• Fig. 5 is a flow chart illustrating a method of pre-processing a pixel in a picture according to yet another embodiment
• Fig. 6 is a flow chart illustrating a method of pre-processing a pixel in a picture according to an additional embodiment
• Fig. 7 is a flow chart illustrating steps of a default ("anchor") processing chain according to an embodiment
• Fig. 8 is a flow chart illustrating steps of a first auxiliary ("Ajusty") processing chain according to an embodiment
• Fig. 9 is a flow chart illustrating steps of a second auxiliary ("Ajustc") processing chain according to an embodiment
• Fig. 10 is a flow chart illustrating an additional step of the method shown in Fig. 1 to form a method of encoding a pixel according to an embodiment
• Fig. 11 illustrates a flow chart of a method according to one embodiment
• Figs. 12A and 12B schematically illustrate performing the Ajusty processing chain in sequential passes
• Fig. 13A illustrate differences between deriving luma component values according to the anchor processing chain and the Ajusty processing chain
• Fig. 13B schematically illustrate the quotient of maxRGB/(minRGB + s) for the picture shown in Fig. 13A;
• Fig. 14 is a schematic illustration of hardware implementations of a device according to the embodiments;
• Fig. 15 is a schematic illustration of an implementation of a device according to the embodiments with a processor and a memory;
• Fig. 16 is a schematic illustration of a user equipment according to an embodiment
• Fig. 17 is a schematic illustration of an implementation of a device according to the embodiments with function modules
• Fig. 18 schematically illustrate a distributed implementation of the embodiments among multiple network devices
• Fig. 19 is a schematic illustration of an example of a wireless communication system with one or more cloud-based network devices according to an embodiment
• Fig. 20 illustrates an embodiment of deriving the corrected Y'
• Fig. 21 is a diagram illustrating that there can be different linearizations in different color areas
• Fig. 22 illustrates Barten's curve for contrast sensitivity
• Fig. 23 illustrates a comparison between Rec709 and BT.2020 color gamuts
• Fig. 24 illustrates an embodiment of obtaining Cb and Cr by subsampling in linear RGB and obtaining Y' by using Ajusty
• Fig. 25 illustrates an embodiment of creating references with chroma upsampling in a representation invariant to intensity
• Fig. 26 illustrates an embodiment of iterative refinement of Cb and Cr.
• the present embodiments generally relate to pre-processing and encoding of pixels in a picture, and in particular to such pre-processing and encoding that improves luminance and/or chrominance values of pixels.
• a traditional compression chain also denoted default processing chain or anchor processing chain herein, involves feeding pixels of incoming linear light, typically ranging from 0 to 10,000 cd/m 2 , to an inverse transfer function, which results in new pixel values between 0 and 1. After this, the pixels undergo color transform resulting in a luma component and two chroma components. Then the two chroma components are subsampled, such as to 4:2:0 or 4:2:2. The pixels may then be subject to encoding or compression. After decoding or decompression, the 4:2:0 or 4:2:2 sequences are upsampled to 4:4:4, inverse color transformed and finally a transfer function gives back pixels of linear light that can be output on a monitor.
• a combination of a highly non-linear transfer function, chroma subsampling and non-constant luminance ordering gives rise to severe artifacts to the video data, in particular for saturated colors.
• the trouble comes from the fact that the chroma components are interpolated, whereas the luma component is not.
• the chroma components cannot follow since they are interpolated.
• the result is a pixel of completely wrong intensity, which is clearly visible as an artifact.
• the pre-processing of pixels according to the embodiments can be used to combat or at least reduce the impact of artifacts, thereby resulting in a color that is closer to the incoming "true" color of a pixel.
• Ajusty method or processing chain One way to get around the problem is described in Annex A, a method that is referred to as Ajusty method or processing chain herein.
• the luma component value ( ⁇ ') of a pixel is adjusted so that the resulting linear luminance Y is closer to its correct value (Yo). It is therefore possible to compensate for the fact that some of the luminance information is also carried in the chroma components Cb and Cr.
• the Ajusty processing chain improves the luminance of the pixels but at a cost of somewhat increased processing time.
• the conversion from RGB 4:4:4 to Y'CbCr 4:2:0 takes x seconds using the anchor processing chain mentioned above.
• compression using the High- Efficiency Video Coding (HEVC) reference encoder HM typically takes 10x seconds, resulting in a total processing time of 11x for the anchor processing chain.
• HEVC High- Efficiency Video Coding
• Annex C describes a method that is referred to as Ajustc herein.
• the Ajustc method or processing chain at least one of the chroma component values (Cb, Cr) is derived in a new processing chain that improves the chrominance of pixels.
• this improvement in chrominance comes at a cost of somewhat increased processing time as compared to the anchor processing chain.
• the present embodiments provide a trade-off between improved quality of pixels, such as in the form of improved luminance, improved chrominance or improved luminance and chrominance, and increased processing time and complexity. Accordingly, embodiments are provided that determine for which pixels a default processing chain, such as the anchor processing chain, can be used and for which pixels an auxiliary processing chain, such as the Ajusty processing chain, the Ajustc processing chain or both the Ajusty and Ajustc processing chains, can be used.
• the auxiliary processing chain that may improve the quality of pixels in a picture should preferably only be applied to those pixels that would benefit from the improvement in quality, i.e. for which the auxiliary processing chain leads to a visible quality improvement as compared to the default processing chain.
• the present embodiments are thereby directed towards improving the luminance and/or chrominance of pixels in a computationally efficient way.
• a color space, color domain or color format is the type and number of colors that originate from the combinations of color components of a color model.
• a color model is an abstract configuration describing the way colors can be represented as tuples of numbers, i.e. color components.
• the color components have several distinguishing features such as the component type, e.g. hue, and its unit, e.g. degrees or percentage, or the type of scale, e.g. linear or non-linear, and its intended number of values referred to as the color depth or bit depth.
• Non-limiting, but illustrative, color spaces that are commonly used for pixels in pictures and videos include the red, green, blue (RGB) color space, the luma, chroma blue and chroma red (YCbCr, sometimes denoted Y'CbCr, Y'Cb'Cr 1 , YCBCR, Y'CBCR Y'CB'CR' or YUV, Yuv, or DVD'CBD'CR or EVE'CBE'CR) color space and the luminance and chrominances (XYZ) color space.
• RGB red, green, blue
• YCbCr sometimes denoted Y'CbCr, Y'Cb'Cr 1 , YCBCR, Y'CBCR Y'CB'CR' or YUV, Yuv, or DVD'CBD'CR or EVE'CBE'CR
• XYZ luminance and chrominances
• Fig. 1 is a flow chart illustrating a method of pre-processing a pixel in a picture.
• the method comprises determining, in step S1 , based on a minimum color component value for the pixel, whether a default processing chain is used to derive a luma component value, a first subsampled chroma component value and a second subsampled chroma component value or whether an auxiliary processing chain is used to derive at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the method step S1 in Fig. 1 is preferably applied to at least one pixel in the picture, and more preferably to multiple, i.e. at least two, pixels in the picture, such as all pixels in the picture, which is schematically illustrated by the line L1 in Fig. 1.
• the pre-processing of Fig. 1 comprises determining a luma component value, a first subsampled chroma component value and a second subsampled chroma component value for the pixel based on the input or original color component values of the pixel.
• the following processing involves compressing or encoding the picture of the video sequence to form a bitstream of encoded pixel data representing the encoded video sequence or stream.
• Another example of following processing is the transfer of pictures, such as of a video sequence, over an interface, such as high-definition multimedia interface (HDMI), DisplayPort or Thunderbolt.
• HDMI 2.0a the only way to convey 4K resolution at 50 or 60 frames per second (fps) is to use 4:2:0 or 4:2:2 Y'CbCr subsampling. If the video data is in full chroma sampling format (4:4:4) then a subsampling pre-processing step must be applied before sending the video data over the HDMI cable. The 4:2:0 or 4:2:2 video data is then transferred over the interface without further compression.
• the particular color space and format output from the pre-processing of the embodiments is a luma component value and two chroma component values, i.e. the Y'CbCr color space.
• the chroma component values are subsampled, whereas the luma component is not, so the color space and format from the pre-processing is preferably Y'CbCr 4:2:2 or Y'CbCr 4:2:0.
• the decision as to whether use the default processing chain or the auxiliary processing chain in order to derive at least one of the luma component value ( ⁇ '), the first subsampled chroma component value (Cb) and the second subsampled chroma component value (Cr) is made at least partly based on the smallest color component value of the pixel.
• the minimum or smallest color component value of a pixel is used to determine whether the luminance and/or the chrominance of the pixel benefits from quality improvement by using the auxiliary processing chain or whether the auxiliary processing chain does not improve the luminance and/or chrominance for the particular pixel or any luminance and/or chrominance improvement achieved by the auxiliary processing chain is too small in relation to the additional processing complexity and time when using the auxiliary processing chain.
• the color component values of the pixel that are assessed to determine whether to use the default processing chain or the auxiliary processing chain are preferably of a color space different from luminance + chrominance.
• auxiliary processing chain could result in an improvement as seen in luminance (Y) or chrominance, preferably in the XYZ color space, but the decision of which processing chain to use is made in a color space different from the luminance + chrominance color space, i.e. preferably a color space different from the XYZ color space.
• the assessment of the color component value is made in the color space of the pixel as input to the pre-processing of the embodiments.
• This initial color space is typical the RGB color space.
• the decision of whether to use the default processing chain and the auxiliary processing chain is made based on a minimum color component value of a red color component value (R), a green color component value (G) and a blue color component value (B) of the pixel, i.e. min(R, G, B).
• step S1 of Fig. 1 comprises selecting the auxiliary processing chain if the minimum color component value is below a threshold value and otherwise selecting the default processing chain.
• the auxiliary processing chain is selected in step S1 if min(R, G, B)
• T or more generally if function( min(R, G, B) ) ⁇ T for some function of the minimum of the color component value of the pixel, wherein T represents the threshold value.
• step S1 could comprise selecting the auxiliary processing chain if R ⁇ TR or G ⁇ TG or B
• step S1 of Fig. 1 comprises determining, based on the minimum color component value and a maximum color component value for the pixel, whether the default processing chain is used to derive the luma component value, the first subsampled chroma component value and the second subsampled chroma component value or whether the auxiliary processing chain is used to derive the at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the decision whether to use the default or auxiliary processing chain for a pixel in the picture is made based on the smallest color component value of the pixel and the largest color component value of the pixel.
• decision is made based on a function of these two color component values, preferably function( min(R, G, B), max(R, G, B) ).
• the function is a quotient between the two color component values.
• the quotient could be between the maximum color component value or a weighted version thereof and the minimum color component value or a weighted version thereof.
• the quotient is between the maximum color component value or a weighted version thereof and a sum of the minimum color component value or a weighted version thereof and a constant.
• the weights wi, W2, w 3 could be used to weight the color components differently, such as weight one of the color components, for instance G, more than the other color components.
• wi + W2 +w 3 1.
• step S1 comprises determining, based on a quotient between the maximum color component value or a weighted version thereof and 1) the minimum color component value or a weighted version thereof, or 2) a sum of the minimum color component value or the weighted version thereof and a constant, whether the default processing chain is used to derive the luma component value, the first subsampled chroma component value and the second subsampled chroma component value or whether the auxiliary processing chain is used to derive at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• step S1 comprises selecting the auxiliary processing chain if the quotient exceeds a threshold value and otherwise selecting the default processing chain.
• Fig. 2 schematically illustrates such an embodiment.
• the method starts in step S10, which comprises calculating a function value, preferably as max / ( min + s), wherein max represents the maximum color component value or the weighted version thereof, min represents the minimum color component value or the weighted version thereof and s is a constant.
• step S10 comprises calculating a function value, preferably as max / ( min + s), wherein max represents the maximum color component value or the weighted version thereof, min represents the minimum color component value or the weighted version thereof and s is a constant.
• step S11 comprises comparing the function value with a threshold value.
• step S1 in Fig. 1 which comprises, in this embodiment, selecting the auxiliary processing chain if the function value exceeds the threshold value and otherwise selecting the default processing chain.
• the constant s is equal to 0.1 and the threshold value is equal to 2, i.e. selecting the auxiliary processing chain if max(R, G, B)/(min(R, G, B)+0.1) > 2.
• the quotient or function value is calculated for a current pixel to be pre-processed, i.e. for which the luma value and the two subsampled chroma component values are to be determined.
• the calculations can be made in advance to form a lookup table (LUT).
• the color component values for the pixel are input into the LUT and the output is then an indication of whether to pre-process the according to the default or auxiliary processing chain.
• the indication is a one-bit value (Obin or 1 bin).
• step S20 comprises inputting color component values for the pixel in a LUT configured to output a first value if the quotient exceeds a threshold value and otherwise output a second value.
• step S21 comprises selecting the auxiliary processing chain if a value output from the LUT is equal to the first value (exemplified by 1 bin in the figure) and selecting the default processing chain if the value output form the LUT is equal to the second value (exemplified by O in in the figure).
• the default and auxiliary processing chains will be further described with reference to Figs. 4 to 11.
• the subsampled chroma component values are derived for the pixel according to the default processing chain.
• the selection between the default and auxiliary processing chain is then made as described herein to decide whether to determine the luma component value for the pixel according to the default processing chain or the auxiliary processing chain.
• the auxiliary processing chain is preferably the Ajusty processing chain described in more detail in Annex A.
• Fig. 4 illustrates a method of pre-processing a pixel in a picture according to this embodiment.
• the method starts in step S30, which comprises deriving the first subsampled chroma component value and the second subsampled chroma component value using the default processing chain.
• step S31 which comprises determining, based on the minimum color component value, whether the default processing chain is used to derive the luma component value or whether a first auxiliary processing chain is used to derive the luma component value.
• the luma component value is derived for the pixel according to the default processing chain.
• the selection between the default and auxiliary processing is then made as described herein to decide whether to determine the subsampled chroma component values according to the default processing chain or the auxiliary processing chain.
• the auxiliary processing chain is preferably the Ajustc processing chain described in more detail in Annex C.
• Fig. 5 illustrates a method of pre-processing a pixel in a picture according to this embodiment.
• the method starts in step S40, which comprises deriving the luma component value using the default processing chain.
• step S41 comprises determining, based on the minimum color component value, whether the default processing chain is used to derive the first subsampled chroma component value and the second subsampled chroma component value or whether a second auxiliary processing chain is used to derive the first subsampled chroma component value and the second subsampled chroma component value.
• the subsampling of the first and second chroma component values implies that each chroma component value influences several pixels during decoding, i.e. when the subsampled chroma component values are upsampled back to Y'CbCr 4:4:4 format. For instance, if nearest neighbor upsampling is used, each first and second chroma component values is used for four neighboring pixels. If other types of upsamplings are used, involving longer filters, then more than four pixels may be influenced by the upsampling, such as pixels in a 7x7 pixel area. Pixels near the center of this pixel area will be influenced much more and pixels near the edge or perimeter of the pixel area will be influenced much less.
• the determination in step S41 may thereby be performed based on the minimum color component value of at least one pixel within a pixel area comprising multiple pixels in the picture.
• the pixel area is defined as a portion of the picture enclosing pixels influenced or affected by upsampling of the first subsampled chroma component value and the second subsampled chroma component value.
• a respective function value, max / ( min + s ) is calculated for each pixel in the pixel area or for a subset of the pixels in the pixel area. Examples of such a subset indue 2x1 pixels, 1 x2 pixels and 2x2 pixels.
• the auxiliary processing chain is selected if at least one of the function values exceeds the threshold value and otherwise selecting the default processing chain. In another embodiment, the auxiliary processing chain is selected if the function values for all pixels in the subset of pixels in the pixel area exceeds the threshold value and otherwise selecting the default processing chain. Further embodiments include selecting the auxiliary processing chain if the average of the function values for all pixels in the subset of pixels in the pixel area or the average of the function values for all pixels in the pixel area exceeds the threshold value and otherwise selecting the default processing chain.
• the selection between the default and auxiliary processing chain is made as described herein to decide whether to determine the luma component value and the subsampled chroma component values according to the default processing chain or whether to determine the luma component value according to a first auxiliary processing chain and determine the subsampled chroma component values according to a second auxiliary processing chain.
• the first auxiliary processing chain is preferably the Ajusty processing chain described in more detail in Annex A and the second auxiliary processing chain is preferably the Ajustc processing chain described in more detail in Annex C.
• Fig. 6 illustrates a method of pre-processing a pixel in a picture according to this embodiment.
• the method comprises determining, in step S50 and based on the minimum color component value, whether the default processing chain is used to derive the luma component value, the first subsampled chroma component value and the second subsampled chroma component value or whether a first auxiliary processing chain is used to derive the luma component value and a second auxiliary processing chain is used to derive the first subsampled chroma component value and the second chromap component value.
• the decision whether to use the default processing chain or the first auxiliary processing chain (Fig. 4), the second auxiliary processing chain (Fig. 5), the first and second auxiliary processing chains (Fig. 6) can be made as previously described herein, i.e. based on the minimum color component value or based on the minimum and maximum color component value.
• Figs. 4 to 6 are preferably applied to multiple pixels in a picture, such as to all pixels in the picture, which is schematically illustrated by the lines L2, L3 and L4 in Figs. 4, 5 and 6, respectively.
• Fig. 7 is a flow chart illustrating method steps performed according to default processing chain, also referred to as the anchor processing chain.
• the method continues from step S31 , S41 or S50 in Figs. 4 to 6.
• the method comprises applying, in step S60, a first transfer function (TF) to a linear color in a first color space to obtain a non-linear color in the first color space.
• a next step S61 comprises applying a first color transform (CT) to the non-linear color in the first color space to obtain the luma component value and two chroma component values in a second color space.
• the following step S62 comprises subsampling the two chroma component values in the second color space to obtain the first subsampled chroma component value and the second subsampled chroma component value in the second color space.
• CT first color transform
• a first transfer function such as the inverse of the transfer function (equation A1) in Annex A, is applied to the initial or input RoGoBo color of the pixel, i.e. the original color of the pixel, to get a non-linear color R'G'B' in the RGB color space.
• This R'G'B' color is then color transformed from the RGB color space to the YCbCr color space using a first color transform, such as the color transform:
• step S62 subsamples the two chroma components Cb and Cr to get a color in the 4:2:0 or 4:2:2 format, i.e. with subsampled nonlinear chroma components Cb and Cr.
• Subsampling in step S62 can be performed according to known subsampling techniques. For instance, a filtering operation or a nearest neighbor operation can be used. An example of subsampling technique that can be used according to the embodiments is disclosed in section B.1.5.5 Chroma downsampling from 4:4:4 to 4:2:0 in document [5].
• the default processing chain i.e. the anchor processing chain, can be summarized according to below:
• RoGoBo 4:4:4 ⁇ (first transfer function) ⁇ R'G'B' 4:4:4 ⁇ (first color transform) ⁇ Y'CbCr 4:4:4 ⁇ (subsampling of Cb and Cr) ⁇ Y'CbCr 4:2:0
• Fig. 8 is a flow chart illustrating method steps performed according to the first auxiliary processing chain, also referred to as the Ajusty processing chain.
• the method continues from step S31 or S50 in Figs. 4 or 6.
• the method comprises obtaining, in step S70, an original linear luminace component value of the pixel in a third color space determined based on a linear color of the pixel in a first color space.
• the method also comprises deriving the luma component value in a second color space based on two chroma component values in the second color space and the original linear luminance component value in the third color space.
• the third color space mentioned above is the XYZ color space. Accordingly, the linear luminance component value of the pixel in the third color space is, in this embodiment, a Y component value.
• obtaining the original linear luminance component comprises determining the original linear luminance component value in the third color space based on the linear color of the pixel in the first color space.
• This original linear luminance component value preferably reflects the true luminance of the pixel, i.e. the original luminance of the pixel prior to any color transformation, application of transfer functions and subsampling.
• This original linear luminance component value is determined based on the linear color of the pixel in the first color space.
• this linear color of the pixel in the first color space is the original incoming color of the pixel.
• this original incoming color is a RoGoBo herein and the original linear luminance component is denoted Yo.
• the original linear luminance is, thus, the target luminance which could be calculated in many different ways.
• This original linear luminance does not necessarily have to correspond directly to the actual luminance of the sensor in the camera taking a picture or recording a video or in the panel of the display.
• the original linear luminance component value in the third color space could be obtained to the pre- processing and encoding functionality as an original linear luminance component value, preferably Yo value, in a pre-determined or pre-calculated form. This means that the determination of the original linear luminance component value based on the linear color in the first color space has already taken place and only the result of the determination is provided to the pre-processing and encoding functionality.
• the pre-processing of the embodiments comprises determination or calculation of the original linear luminance component value as described above.
• Yo g( Ro, Go, Bo ), see equation 5 or 6 for examples of the function g(.).
• equation 5 and 6 only the second line need to be calculated to obtain the original linear luminance component value.
• the luma component value in the second color space is then derived in step S71 based on the two chroma component values in the second color space and the original linear luminance component value in the third color space.
• step S71 of Fig. 8 comprises deriving a luma component value in the second color space that minimizes a difference between the original linear luminance component value in the third color space and a linear luminance component value in the third color space determined based on the luma component value in the second color space, the first subsampled chroma component value in the second color space and the second subsampled chroma component value in the second color space.
• step S71 involves finding the luma component value ( ⁇ ') in the second color space (YCbCr) that minimizes the difference between the original linear luminance component value (Yo) and the linear luminance component value (Y) in the third color space (XYZ).
• This linear luminance component value (Y) in the third color space (XYZ) is in turn obtained based on the luma component value ( ⁇ ') and the two subsampled chroma component values (Cb, Cr) in the second color space (YCbCr).
• this embodiment involves finding the Y' component value that minimizes the difference
• or (Yo - Y) 2 , wherein Y h( Y', Cb, Cr) and h( . ) defines that Y is determined based on Y', Cb and Cr.
• step S71 involves deriving a luma component value in the second color space that minimizes a difference between a function of the original luminance component value in the third color space and a function of a linear luminance component value in the third color space.
• This linear luminance component value in the third color space is determined based on the luma component value in the second color space, the first subsampled chroma component value in the second color space and the second subsampled chroma component value in the second color space.
• this embodiment involves finding the Y' component value that minimizes the difference
• or (k( Yo ) - k( Y )) 2 , wherein Y h( Y', Cb, Cr).
• the function (k( . )) is preferably an inverse transfer function, such as the inverse of the transfer function in equation A1 as shown in Annex A.
• the two subsampled chroma component values in the second color space preferably as obtained from the default or anchor processing chain as previously described herein, are upsampled to obtain an upsampled first chroma component value in the second color space and an upsampled second chroma component value in the second color space.
• a second color transform is then applied to a candidate luma component value in the second color space, the upsampled first chroma component value in the second color space and the upsampled second chroma component value in the second color space to obtain a non-linear color in the first color space.
• a second transfer function is applied to the non-linear color in the second color space to obtain a linear color in the first color space.
• a third color transform is applied to the linear color in the first color space to obtain a linear luminance component value in the third color space.
• the luma component value is derived in step S71 based on a comparison of the original linear luminance component value in the third color space and the linear luminance component value in the third color space.
• the subsampled Cb and Cr component values in 4:2:0 or 4:2:2 format are first upsampled to the 4:4:4 format.
• Upsampling can be performed according to known upsampling techniques. For instance, upsampling could be performed by using bilinear or longer filters.
• An example of upsampling technique that can be used according to the embodiments is disclosed in section B.1.5.6 Chroma upsampling from 4:2:0 to 4:4:4 (Y'CbCr domain) in document [5].
• These two upsampled Cb and Cr component values are then input together with a candidate Y' component value into a second color transform to get a non-linear R'G'B' color, such as the color transform:
• This R'G'B' color is then input into a second transfer function, such as the transfer function (equation A1) in Annex A, to get a linear RGB color.
• This RGB color is then transformed from the RGB color space to the XYZ color space using a third color transform, such as the color transform in equation 5 when RGB originates from BT.2020 or equation 6 when RGB originates from BT.709:
• the linear luminance component Y value output form the third color transform is then compared to the original linear luminance component Yo value of the pixel.
• step S71 of Fig. 8 comprises selecting a candidate luma component value in the second color space that reduces a difference between the original linear luminance component value in the third color space and the linear luminance component value in the third color space.
• step S71 preferably comprises selecting a candidate luma component value in the second color space that leads to at least a reduction in the difference between the original linear luminance component value and the linear luminance component value.
• step S71 comprises selecting a candidate luma component value in the second color space that minimizes the difference between the original luminance component value and the linear luminance component value in the third color space.
• This difference could, as mentioned in the foregoing, be represented as
• step S71 involves selecting a candidate luma component value in the second color space that reduces or, preferably, minimizes a difference between a function of the original luminance component value in the third color space and a function of the linear luminance component value in the third color space, i.e. selecting the candidate Y' component value that minimizes the difference
• different candidate luma component values in the second color space are tested.
• step S71 preferably comprises selecting the candidate luma component value among the different candidate luma component values in the second color space that results in a smallest difference between the original linear luminance component value in the third color space and the linear luminance component value in the third color space or a smallest difference between a function of the original linear luminance component value in the third color space and a function of the linear luminance component value in the third color space.
• the following embodiments are described in more detail with regard to a difference between the original linear luminance component value in the third color space and the linear luminance component value in the third color space. These embodiments also encompass a difference between a function of the original linear luminance component value in the third color space and a function of the linear luminance component value in the third color space.
• the function is preferably, as previously mentioned herein, the inverse of a transfer function, such as an inverse of the transfer function in equation A1 in Annex A.
• the selection of the optimal candidate Y' component value among multiple candidate Y' component values can be performed according to various embodiments as described further herein.
• a first embodiment involves performing a binary search.
• the method comprises performing a binary search to select a candidate luma component value in the second color space that minimizes a difference between the original linear luminance component value in the third color space and the linear luminance component value in the third color space.
• a binary search is an efficient technique that can be used to find the optimal candidate luma component value.
• a binary search algorithm begins by comparing the original luminance component value in the third color space to the linear luminance component value in the third color space obtained using the middle element of a sorted array of possible candidate luma component values in the second color space. If the linear luminance component value in the third color space is equal to the original luminance component value in the third color space or differs from the original luminance component value in the third color space with not more than a defined amount, then the position of the middle element is returned and the search is finished.
• the selection of a search interval generally involves selecting a search interval having approximately half the size as compared to the search interval used above. For instance, if the search interval contains the values 100, 101 , 102, 103 then one could choose either 101 or 102 as the "middle value", resulting in a "halved" search interval of [100, 101] (a true halving of the search interval) or [101 , 103] (an approximate halving of the search interval) or a "halved" search interval of [100, 102] (an approximate halving of the search interval) or [102, 103] (a true halving of the search interval).
• Gradient descent can be slow, so a quicker way may be to use a second-order optimization algorithm that calculates or approximates the second order derivatives d 2 E/dY' 2 .
• Gauss-Newton is an example of such an algorithm.
• a further embodiment involves using a LUT when selecting the luma component value.
• a LUT may, for instance, comprise the best Y' component value for every possible combination of Cb, Cr and Yo component values. Assume, for instance, that the Cb and Cr components are quantized to 10 bits and that the Yo component is also quantized to 10 bits. Then the LUT should contain 2 1 ° ⁇ 2 1 ° ⁇ 2 10 different Y' component values. This is equivalent to 2 30 Y' component values. If each such Y' component value is two bytes, the LUT will have a size of 2 31 bytes, or 2 Gb. It may also be possible to use a smaller LUT. For instance, it may be possible to quantize the Yo, Cb and Cr components to a smaller size, say 6 bits. Then the LUT would be 2 18 Y' component values, or 2 19 bytes, which is equal to 512 kb.
• the Yo component is linear. Accordingly, it may be inefficient to just quantize it. It may instead be better to use a function of Yo together with the Cb and Cr as input to the LUT to get the optimal Y' component.
• the function preferably outputs a non-linear representation of the Yo component and may, for instance, be an inverse transfer function (TF-1 ( . )), such as the inverse of the transfer function in equation A1 in Annex A.
• Yo 4:4:4 is the luminance component of XYZ 4:4:4 and CbCr 4:4:4 are the upsampled chroma components of Y'CbCr.
• the target luminance Yo 4:4:4 is found by first converting RoGoBo 4:4:4 to XoYoZo 4:4:4 using the third color transform and then using Yo 4:4:4.
• Fig. 9 is a flow chart illustrating method steps performed according to the second auxiliary processing chain, also referred to as the Ajustc processing chain. The method continues from step S41 or S50 in Figs. 5 or 6. The method comprises subsampling, in step S80 a linear color in a first color space to obtain a subsampled linear color in the first color space.
• a next step S81 comprises applying a first transfer function to the subsampled linear color in the first color space to obtain a subsampled non- linear color in the first color space.
• the method continues to step S82, which comprises applying a first color transform to the subsampled non-linear color in the first color space to obtain the first subsampled chroma component value and the second subsampled chroma component value in a second color space.
• the Ajustc processing chain involves subsampling of a linear color, i.e. subsampling is performed in a linear color domain. The generation of a non-linear color then takes place following the subsampling, i.e.
• the first transfer function is applied to a subsampled linear color.
• the default processing chain as shown in Fig. 7 instead first applies a transfer function to a linear color to obtain a non-linear color.
• a color transform is then applied to the non-linear color followed by subsampling.
• the first color space is an RGB color space and the linear color is thereby an RGB color, denoted RoGoBo herein.
• the initial color is in unsampled or original format, i.e. 4:4:4 format.
• This RoGoBo 4:4:4 color is then preferably subsampled in step S80 to get a subsampled RGB color, i.e. RGB 2:2:2 color.
• Step S81 comprises applying a first transfer function to the RGB 2:2:2 color to obtain a subsampled non-linear color R'G'B' 2:2:2.
• the " ' " is used to indicate that the red, green and blue color components are non-linear color components.
• the first transfer function is the inverse of the transfer function in equation A1 shown in Annex A.
• the resulting subsampled non-linear color R'G'B' 2:2:2 is then color transformed from the RGB color space to the second color space using a first color transform.
• This second color space is preferably the YCbCr color space and the first color transform is preferably the color transform specified in any of equations 1 to 3.
• the first color transform results in a Y'CbCr 2:2:2 color.
• the subsampled luma component value Y', the first subsampled chroma component value Cb and the second subsampled chroma component value Cr, all in the YCbCr color space, are obtained in step S83.
• only the subsampled chroma component values Cb, Cr are calculated in step S82 thereby basically omitting the first line relating to the luma component Y in equations 1-3.
• step S83 comprises applying the first color transform to the subsampled non-linear color in the first color space to obtain the first subsampled non-linear chroma component value and the second subsampled chroma component value in the second color space.
• the following pre-processing chain is used in the Ajustc processing chain:
• RoGoBo 4:4:4 ⁇ (subsampling of RGB) ⁇ RGB 2:2:2 ⁇ (first transfer function) ⁇ R'G'B' 2:2:2 ⁇ (first color transform) ⁇ Y'CbCr 2:2:2
• step S80 comprises subsampling the linear color in the first color space in both vertical and horizontal direction to obtain the subsampled linear color in the first color space.
• step S80 comprises subsampling the linear color in the first color space in only one direction, such as the horizontal direction or the vertical direction, to obtain the subsampled linear color in the first color space.
• step S80 instead results in, with the full resolution luma component in the second color format, a Y'CbCr color in the 4:2:2 format.
• Subsampling in step S80 can be performed according to known subsampling techniques.
• a filtering operation or a nearest neighbor operation can be used.
• An example of subsampling technique that can be used according to the embodiments is disclosed in section B.1.5.5 Chroma downsampling from 4:4:4 to 4:2:0 in document [5].
• the pre-processing of pixels according to the embodiments can be used in different applications, in particular different video applications, including video encoding.
• An embodiment therefore relates to a method of encoding a pixel in a picture.
• the method comprises pre-processing the pixel according to any of the embodiments as disclosed herein to derive the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the method also comprises, in step S2 as shown in Fig. 10, encoding the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the pre-processing of pixels can be used as an additional processing during encoding pixels of pictures, such as in connection with encoding pictures of a video sequence.
• the output of the pre-processing i.e. Y'CbCr, such as in the 4:2:0 or 4:2:2 format, is then input to a video encoder, such as a H.264 video encoder, a HEVC or H.265 video encoder, or another video encoder.
• the encoding as performed by the video encoder can include traditional video encoding steps, such as inter prediction, intra prediction, mode decision, transformation of residual, quantization and encoding of the transformed and quantized residuals, such as in an entropy encoder, e.g. a context-adaptive binary arithmetic coding (CABAC) encoder.
• CABAC context-adaptive binary arithmetic coding
• the pre-processing of the embodiments is that it can be applied to any video or picture encoding process without the need for performing any adjustments in the video or picture encoder or any adjustments in the corresponding video or picture decoder.
• the pre-processing can be seen as an additional processing, i.e. pre-processing, that is applied to the input pictures instead of the traditional conversion of original colors, such as RoGoBo colors, of pixels in pictures to be encoded into Y'CbCr colors that involves application of transfer function, application of color transform and chroma subsampling.
• the pre-processing of pixels in pictures, such as of a video sequence may also be used in other applications besides video encoding as previously mentioned herein. Such other applications include transferring video over an interface, such as HDMI, DisplayPort or Thunderbolt.
• a method is provided as illustrated in Fig. 1 1.
• the method can be performed in an encoder or in a pre-process to the encoder.
• it is determined for which pixels the anchor processing chain can be used and for which pixels the Ajusty processing chain can be used.
• the determination can be done according to any of the embodiments below or a combination thereof.
• a further advantage is that at least some of the embodiments provide a conversion from RGB 4:4:4 to Y'CbCr 4:2:0 that uses less memory than previous art.
• a yet further advantage is that at least some of the embodiments provide a conversion from RGB 4:4:4 to Y'CbCr 4:2:0 that is of a higher quality than previous art.
• the correction needed in Y' is not so big.
• the Y' produced by the anchor processing chain i.e. default processing chain, from hereon called Y'anchor
• the Y' produced by the Ajusty processing chain i.e. the auxiliary processing chain, from hereon called Y' ajusty
• the number of pixels where the Y'anchor and Y'ajusty differ by more than two steps is very small, typically less than 5 %.
• the proposed solution is to detect these pixels and apply the Ajusty processing chain only on them.
• a pixel with colors (R, G, B) is selected for the Ajusty processing chain if a function h(R, G, B) exceeds a threshold.
• a function h(R, G, B) max(R, G, B) / (min(R, G, B) + s), where min(a,b,c) calculates the minimum of the three inputs a, b, c and max(a, b, c) calculates the maximum of the three inputs a, b, c, and s is a constant, such as 0.1.
• h(.) max(w1*R, w2*G, w3*B) / (min(w1 *R, w3*G, w3*B) + s).
• a pixel with the Y'CbCr components ( ⁇ ', Cb, Cr) is selected for the Ajusty processing chain if a function g(Y', Cb, Cr) exceeds a threshold.
• a function g(Y', Cb, Cr) h( RGBptoRGB( YpCbCrToRGBp( Y', Cb, Cr) ), where h(.) is the function above, RGBptoRGB(.) converts a color from R'G'B' to RGB using a transfer function TF(.), i.e. R -TF(R), G -TF(G), and B -TF(B), and where YCbCrToRGBp(.) converts a color from Y'CbCr to R'G'B', for instance using:
• a transfer function such as PQ used in Annex A (see equation A1 ) can be used.
• a LUT such as a 1-bit LUT
• the 1-bit lookup-table can be created by trying all possible combinations of R'1 Obit, G'l Obit, ⁇ Obit, and see which ones would likely result in a big difference between Y'anchor and Y'ajusty.
• the table can calculate RGB from R'G'B' using
• TP 1 is the inverse to the transfer function TF(.).
• PQ described in Annex A (equation A1) is an example of a transfer function.
• the function h(R, G, B) described above can be used. If h(R, G, B) is larger than a threshold, the bit is set to 1 for that table entry. If it is smaller than the threshold, the bit is set to 0 for that table entry.
• the LUT uses R, G, B instead of R' G' B'.
• the LUT uses Y', Cb, Cr instead of R' G' B'.
• RGBmax max(R'10bit, G'l Obit, B'l Obit)
• the two-dimensional 1-bit lookup table can be created by trying all possible combinations of R'1 Obit, G'1 Obit and B'l Obit and see which ones would likely result in a big difference between Y'anchor and Y' ajusty. Assume we have such a function called MkelyBigDifference(R'1 Obit, G'1 Obit, B'l Obit) that returns 1 if the color (R'1 Obit, G'1 Obit B'l Obit) is likely to result in a big difference between Y'anchor and Y' ajusty and 0 otherwise. The two-dimensional 1-bit lookup table can then be created using
• RGBmax max(R'10bit, G'l Obit, B'l Obit)
• RGBmin min(R'10bit, G'l Obit, B'l Obit)
• RGBmax/(RGBmin + s) > t where s is a constant, such as 0.1 , and t is a threshold.
• a first round of processing may select some pixels as being marked for the Ajusty processing chain, with any of the methods presented herein.
• a second round of processing one may select pixels that are neighbors to pixels that were marked in the first round. The Ajusty processing chain will then be carried out for pixels selected in either the first or the second round.
• a more exact value for Y' is obtained than is possible with the binary search version of the Ajusty processing chain described in Annex A.
• a binary search works by reducing the size of an interval, typically by cutting it in half.
• the interval is set to cover all possible values of Y', for instance [64, 940], or [0, 1023].
• [a, b] performBinarySearch(a, b, Yo, Cbl Obit, CrI Obit, 10)
• the Ajusty processing chain is only calculated if the R, G, and B color components all fall within a certain criteria defined by a set of functions. These functions could be derived and illustrated by plotting the samples for which Y' would need to be calculated using the Ajusty processing chain in a cube where R, G and B are the axes of the 3D plot. The functions would then be planes, of the form aR+bG-3 ⁇ 4B > d, or non-linear surfaces through the cube dividing the samples that would not need to be calculated with Ajusty processing chain from the rest of the samples. In other versions of this embodiment other color representations are used, e.g. R'G'B' or Y'CbCr.
• a 12th embodiment we take advantage of the fact that the ideal Y' for a color with chrominance Cb Cr and desired linear luminance Yo may not be so different from the ideal Y' of a color with a neighboring chrominance Cb+d1 , Cr-nj2 and neighboring desired luminance ⁇ - ⁇ 3.
• startLUT is used to find the start of the interval to search
• stopLUT is used to find the stop of the interval to search
• startLUT(a, b, c) 1023 for all values a, b, c
• stopLUT(a, b, c) 0 for all values a, b ,c for all Cbl Obit from 0 to 1023
• stopLUT(Y_6bits, Cbjbits, Crjbits) Y'
• [a, b] performBinarySearch(a, b, Yo, Cbl Obit, Cr1 Obit, ceil(log(b-a)/log(2)))
• the estimation of the Ajusty processing chain is based on knowledge of the optimal Y' from a previous picture. For example the range of the possible Y' values can be estimated from a previous picture and used as in the current picture. If some of the end points is selected the search can be refined for the range below or the range above pending on the position of the best Y' from the search in the first range.
• the estimation of the Ajusty processing chain is based on knowledge of the optimal Y' from one or more spatially neighboring samples. For instance, the average of the optimal Y' for one or more of the already processed neighboring samples could be used to determine a local starting range to perform a binary search for the current sample.
• a starting range [avg-2 L , avg+2 L +1] for the binary search could then be selected. If some of the end points is selected the search can be refined for the range below or the range above pending on the position of the best Y' from the search in the first range. L is preferably chosen to minimize the average number of iterations needed to find the optimal Y' for a current sample.
• the estimation of the Ajusty processing chain is performed in a first pass for a limited number of samples for example for every second sample horizontally and every second sample vertically, see Fig. 12A.
• the Ajusty processing chain is performed in a second pass for remaining samples inbetween the samples of the first pass based on knowledge of optimal Y' from one or several neighboring samples from the first pass and one or several neighboring samples from already determined optimal Y' in the second pass, see Fig. 12B.
• the first sample to be processed in the second pass can use Y'max and Y'min to determine the range for the search of the optimal Y' for that sample. If some of the end points is selected the search can be refined below or above the best Y' value pending on the position of the best Y'. Then the Y'max and Y'min is updated based on optimal Y' before next sample in the second pass is determined. This repeats until all second pass samples have been determined. In a 17th embodiment, only the samples with linear luminance error, when the Ajusty processing chain is not used, larger than a threshold are adjusted by the Ajusty processing chain.
• the threshold can be based on knowledge from a previous picture.
• only the samples with linear luminance error larger than a threshold are adjusted by the Ajusty processing chain, where the threshold is based on the average luminance error obtained by omitting the subsampling and upsampling of chroma components.
• a 19th embodiment only the samples with linear luminance error larger than a threshold are adjusted by the Ajusty processing chain, where the threshold is based on the average luminance error for a previous picture after the Ajusty processing chain has been applied to that picture.
• a 20th embodiment only the samples with a linear luminance larger than a threshold are adjusted by the Ajusty processing chain. If the pixels are too dark, the difference is simply not visible and there is no need to improve the quality using the Ajusty processing chain.
• a computation cache is used.
• the correct Y' is then calculated using the Ajusty processing chain.
• the result is stored in a cache, and a hash-value is created, for example by performing bit-wise XOR on the last eight bits of the three values:
• a person skilled in the art will realize that it is possible to combine one or more of the above described embodiments, even for cases where it says "only the samples with York. For example, it would be possible to apply one threshold related to the linear luminance in combination with a threshold related to luminance error obtained by omitting the subsampling and upsampling of chroma components.
• 2:2:2 we have used the term 2:2:2 to indicate that all three samples are at half resolution in both the x- and y-dimension. In this way, we do not get a full-resolution Y' component, since the Y' component in the last step is 2:2:2, i.e. half resolution in both directions. This problem is overcome by the following processing steps:
• RGB 4:4:4 --> XYZ 4:4:4 --> Y 4:4:4 H— > Y' AJUSTY(Y 4:4:4, CbCr 4:4:4)
• the target luminance Y 4:4:4 is found by converting RGB 4:4:4 to XYZ 4:4:4 and then use Y and the chroma from Y'CbCr 2:2:2 after upsampling CbCr to same resolution as Y, with the Ajusty processing chain to find the best Y'.
• the anchor processing chain could instead be used to derive the luma component Y'.
• Fig. 13A depicts a case where we have calculated both Y'anchor and Y' ajusty, and calculated the difference between them. Gray equals zero difference, whereas white or black equals a great difference. As can be seen only a few pixels are far from zero. Only 4.4 % of the pixels have a greater difference between Y'anchor and Y' ajusty than two steps.
• the white areas in Fig. 13B i.e. large value of maxRGB/(minRGB + 0.1), correspond well with the areas where Y'anchor differ from Y' ajusty in Fig. 13A.
• performing the decision between the default and auxiliary processing chain based on the minimum color component value for a pixel and in particular a quotient between the maximum and minimum color component value identifies those pixels in a picture that would benefit from using the auxiliary processing chain and those pixels that the auxiliary processing chain does not lead to any significant visual improvements over the anchor processing chain.
• a further aspect of the embodiments relates to a device for pre-processing a pixel in a picture.
• the device is configured to determine, based on a minimum color component value for the pixel, whether a default processing chain is used to derive a luma component value, a first subsampled chroma component value and a second subsampled chroma component value or whether an auxiliary processing chain is used to derive at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the device is configured to select the auxiliary processing chain if the minimum color component value is below a threshold value and otherwise selecting the default processing chain.
• the device is configured to determine, based on the minimum color component value and a maximum color component value for the pixel, whether the default processing chain is used to derive the luma component value, the first subsampled chroma component value and the second subsampled chroma component value or whether the auxiliary processing chain is used to derive the at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the device is configured to determine, based on a quotient between the maximum color component value or a weighted version thereof and 1) the minimum color component value or a weighted version thereof, or 2) a sum of the minimum color component value or the weighted version thereof and a constant, whether the default processing chain is used to derive the luma component value, the first subsampled chroma component value and the second subsampled chroma component value or whether the auxiliary processing chain is used to derive the at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the device is configured to select the auxiliary processing chain if the quotient exceeds a threshold value and otherwise selecting the default processing chain.
• the device is configured to calculate a function value as min+s , wherein max represents the maximum color component value or the weighted version thereof, min represents the minimum color component value or the weighted version thereof and s is the constant.
• the device is also configured to compare the function value with the threshold value.
• the device is configured to input color component values for the pixel in a look-up table configured to output a first value if the quotient exceeds a threshold value and otherwise output a second value.
• the device is also configured to select the auxiliary processing chain if a value output from the look-up table is equal to the first value.
• the device is further configured to select the auxiliary processing chain if the value output from the look-up table is equal to the second value.
• the device is configured to derive the first subsampled chroma component value and the second subsampled chroma component value using the default processing chain.
• the device is configured to determine, based on the minimum color component value, whether the default processing chain is used to derive the luma component value or whether a first auxiliary processing chain is used to derive the luma component value.
• the device is configured to derive the luma component value using the default processing chain.
• the device is configured to determine, based on the minimum color component value, whether the default processing chain is used to derive the first subsampled chroma component value and the second subsampled chroma component value or whether a second auxiliary processing chain is used to derive the first subsampled chroma component value and the second subsampled chroma component value.
• the device is configured to determine, based on the minimum color component value, whether the default processing chain is used to derive the luma component value, the first subsampled chroma component value and the second subsampled chroma component value or whether a first auxiliary processing chain is used to derive the luma component value and a second auxiliary processing chain is used to derive the first subsampled chroma component value and the second subsampled chroma component value.
• the device is configured, according to the default processing chain, to apply a first transfer function to a linear color in a first color space to obtain a non-linear color in the first color space.
• the device is also configured, according to the default processing chain, to apply a first color transform to the non-linear color in the first color space to obtain the luma component value and two chroma component values in a second color space.
• the device is further configured, according to the default processing chain, to subsample the two chroma component values in the second color space to obtain the first subsampled chroma component value and the second subsampled chroma component value in the second color space.
• This embodiment corresponds to pre-processing the pixel according to the default processing chain.
• the device is configured, according to the first auxiliary processing chain to obtain an original linear luminance component value of the pixel in a third color space determined based on a linear color of the pixel in a first color space.
• the device is also configured, according to the first auxiliary processing chain, to derive the luma component value in a second color space based on two chroma component values in the second color space and the original linear luminance component value in the third color space.
• This embodiment corresponds to pre-processing the pixel according the Ajusty processing chain.
• the device is configured, according to the second auxiliary processing chain, to subsample a linear color in a first color space to obtain a subsampled linear color in the first color space.
• the device is also configured, according to the second auxiliary processing chain, to apply a first transfer function to the subsampled linear color in the first color space to obtain a subsampled nonlinear color in the first color space.
• the device is further configured, according to the second auxiliary processing chain, to apply a first color transform to the subsampled non-linear color in the first color space to obtain the first subsampled chroma component value and the second subsampled chroma component value in a second color space.
• This embodiment corresponds to pre-processing the pixel according to the Ajustc processing chain.
• a unit such as a pre-processor or an encoder.
• the unit is configured to determine for which pixels the anchor processing chain can be used and for which pixels the Ajusty processing chain can be used.
• the unit is configured to perform the determination according to any of the embodiments below or a combination thereof.
• Fig. 14 illustrates a particular hardware implementation of a device 100 according to the embodiments.
• the device 100 comprises a determining unit 101 configured to determine for which pixels the anchor processing chain can be used and for which pixels the auxiliary processing chain, i.e. Ajusty and/or Ajustc processing chain, can be used.
• the determining unit 101 is configured to do the determination according to any of the disclosed embodiments or a combination thereof.
• the device 100 also comprises a deriver 102 configured to derive a corrected Y' according to the Ajusty processing chain and/or a corrected CbCr according to the Ajustc processing chain.
• the device 100 further comprises a video encoder 103 configured to encode a picture.
• the device 100 also comprises an input unit 104 configured to receive the prictures of a video sequence to be encoded and an output unit 105 configured to output an encoded bitstream.
• the input unit 104 could be in the form of a general input unit, in particular in the case of a wired connection to external devices.
• the input unit 104 could be in the form of a receiver or transceiver, in particular in the case or a wireless connection to external devices.
• the output unit 105 could be in the form of a general output unit, in particular in the case of a wired connection to external devices.
• the output unit 105 could be in the form of a transmitter or transceiver, in particular in the case or a wireless connection to external devices
• the input unit 104 is preferably connected to the determining unit 101 , the deriver 102 and the video encoder 103 to forward the video to be encoded thereto.
• the determining unit 101 is preferably connected to the deriver 102 and the video encoder 103.
• the video encoder 103 is preferably connected to the output unit 105 to forward the encoded bitstream to a decoder.
• at least some of the steps, functions, procedures, modules and/or blocks described herein may be implemented in software such as a computer program for execution by suitable processing circuitry such as one or more processors or processing units.
• processing circuitry includes, but is not limited to, one or more microprocessors, one or more Digital Signal Processors (DSPs), one or more Central Processing Units (CPUs), video acceleration hardware, and/or any suitable programmable logic circuitry such as one or more Field Programmable Gate Arrays (FPGAs), or one or more Programmable Logic Controllers (PLCs).
• DSPs Digital Signal Processors
• CPUs Central Processing Units
• FPGAs Field Programmable Gate Arrays
• PLCs Programmable Logic Controllers
• the device 110 for pro-processing a pixel in a picture comprises a processor 111 and a memory 112 comprising instructions executable by the processor 111.
• the processor 111 is operative to determine, based on the minimum color component value, whether the default processing chain is used to derive the luma component value, the first subsampled chroma component value and the second chroma component value or whether the auxiliary processing chain is used to derive the at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the device 110 comprises a processor 111 and a memory 112 comprising instructions executable by the processor 111.
• the processor 111 is operative to determine for which pixels the anchor processing chain can be used and for which pixels the Ajusty and/or Ajustc processing chain can be used according to any of the disclosed embodiments or a combination thereof.
• the processor 111 may be operative to derive a corrected Y' according to the Ajusty processing chain, a corrected CbCr according to the Ajustc processing chain and to encode a picture.
• the device 110 also comprises an input unit 113 configured to receive the video with pictures to be encoded.
• the processor 111 is operative to receive the video to be encoded from the input unit 113.
• the device 110 also comprises an output unit 114 configured to output encoded bitstream as received from the processor 111.
• the processor 111 is operative, when executing the instructions stored in the memory 112 to perform the above described operations.
• the processor 111 is thereby interconnected to the memory 112 to enable normal software execution.
• the device 110 comprises a processor 111 and a memory 112 comprising instructions executable by the processor 111.
• the processor 111 is operative to determine, based on the minimum color component value, whether the default processing chain is used to derive the luma component value, the first subsampled chroma component value and the second chroma component value or whether the auxiliary processing chain is used to derive the at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the processor 111 is also operative to derive the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to default processing chain or the least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to the auxiliary processing chain.
• the processor 111 is further operative to encode the luma component value, the first subsampled chroma component value and the second chroma component value.
• device 110 may comprise multiple different physical components that make up a single illustrated component, e.g. input unit 113 may comprise terminals for coupling wires for a wired connection and a radio transceiver for a wireless connection.
• device 110 may be composed of multiple physically separate components which may each have their own respective processor, memory, and interface components.
• one or more of the separate components may be shared among several devices.
• a single memory unit 112 may be shared by multiple devices 110.
• the processor 111 may be a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and/or encoded logic operable to provide, either alone or in conjunction with other device components, such as memory 112.
• the processor 111 may execute instructions stored in the memory 112.
• Such functionality may include providing various encoding or decoding features and/or any of the other features or benefits disclosed herein.
• the memory 112 may comprise any form of volatile or non-volatile computer readable memory including, without limitation, persistent memory, solid state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), removable media, or any other suitable local or remote memory component.
• the memory 112 may store any suitable instructions, data or information, including software and encoded logic, utilized by the device 110.
• the memory 112 may be used to store any calculations made by the processor 111 and/or any data received via the I/O interfaces 113, 114.
• the device 110 also comprises an input unit 113 and an output unit 114 (I/O interfaces), which may be used in the wired or wireless communication of video and/or data to and from the device 110.
• the I/O interfaces may include a radio transmitter and/or receiver that may be coupled to or a part of an antenna.
• the radio may receive video that is to be encoded or decoded.
• Fig. 16 is a schematic block diagram illustrating an example of a user equipment (UE) 200 comprising a processor 210, an associated memory 220 and a communication circuitry 230.
• UE user equipment
• a computer program 240 which is loaded into the memory 220 for execution by a processing circuitry including one or more processors 210.
• the processor 210 and memory 220 are interconnected to each other to enable normal software execution.
• a communication circuitry 230 is also interconnected to the processor 210 and/or the memory 220 to enable input and/or output of video data and encoded video data.
• the user equipment 200 can be any device or apparatus that can receive and process video data.
• the user equipment 200 could be a computer, either stationary or portable, such as laptop, a smart phone, a tablet, a set-top box, etc.
• processor' should be interpreted in a general sense as any system or device capable of executing program code or computer program instructions to perform a particular processing, determining or computing task.
• the processing circuitry including one or more processors is thus configured to perform, when executing the computer program, well-defined processing tasks such as those described herein.
• the processing circuitry does not have to be dedicated to only execute the above-described steps, functions, procedure and/or blocks, but may also execute other tasks.
• a further aspect of the embodiments relates to a computer program 240 comprising instructions, which when executed by a processor 210, cause the processor 210 to determine, based on a minimum color component value for a pixel in a picture, whether a default processing chain is used to derive a luma component value, a first subsampled chroma component value and a second subsampled chroma component value or whether an auxiliary processing chain is used to derive at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the computer program 240 further comprises instructions, which when executed by the processor 210, cause the processor 210 to derive the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to default processing chain or the least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to the auxiliary processing chain.
• the processor 210 is also caused to encode the luma component value, the first subsampled chroma component value and the second chroma component value.
• the computer program 240 comprises instructions, which when executed by the processor 210, cause the processor 210 to determine for which pixels the anchor processing chain can be used and for which pixels the Ajusty processing chain can be used.
• the processor is configured to do the determination according to any of the embodiments below or a combination thereof.
• the proposed technology also provides a carrier 250 comprising a computer program 240.
• the carrier 250 is one of an electronic signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal, or a computer-readable storage medium 250.
• the software or computer program 240 may be realized as a computer program product, which is normally carried or stored on a computer-readable medium 240, preferably nonvolatile computer-readable storage medium 250.
• the computer-readable medium 250 may include one or more removable or non-removable memory devices including, but not limited to a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc (CD), a Digital Versatile Disc (DVD), a Blue-ray disc, a Universal Serial Bus (USB) memory, a Hard Disk Drive (HDD) storage device, a flash memory, a magnetic tape, or any other conventional memory device.
• ROM Read-Only Memory
• RAM Random Access Memory
• CD Compact Disc
• DVD Digital Versatile Disc
• USB Universal Serial Bus
• HDD Hard Disk Drive
• the computer program 240 may thus be loaded into the operating memory of a computer or equivalent processing device, represented by the user equipment 200 in Fig. 16, for execution by the processor 210 thereof.
• the flow diagram or diagrams presented herein may therefore be regarded as a computer flow diagram or diagrams, when performed by one or more processors.
• a corresponding device may be defined as a group of function modules, where each step performed by the processor corresponds to a function module.
• the function modules are implemented as a computer program running on the processor.
• the device may alternatively be defined as a group of function modules, where the function modules are implemented as a computer program running on at least one processor.
• the computer program residing in memory may thus be organized as appropriate function modules configured to perform, when executed by the processor, at least part of the steps and/or tasks described herein.
• An example of such function modules is illustrated in Fig. 17.
• Fig. 17 is a schematic block diagram of a device 120 with function modules.
• the device 120 for pre-processing a pixel in a picture comprises a determining unit 121 for determining, based on a minimum color component value for the pixel, whether a default processing chain is used to derive a luma component value, a first subsampled chroma component value and a second subsampled chroma component value or whether an auxiliary processing chain is used to derive at least one of the luma component value, the first subsampled chroma component value and the second subsample dchroma component value.
• the device 120 also comprises a deriver 122 for deriving the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to default processing chain or the least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to the auxiliary processing chain.
• the device 120 comprises a determining unit 121 configured to determine for which pixels the anchor processing chain can be used and for which pixels the Ajusty processing chain can be used.
• the determining unit 121 is configured to do the determination according to any of the embodiments disclosed herein.
• the device 120 also comprises a deriver 122 configured to derive a corrected Y'
• the device 120 further comprises a video encoder 123 for encoding the bitstream using the corrected Y'.
• the device 120 comprises a determining unit 121 for determining, based on the minimum color component value, whether the default processing chain is used to derive the luma component value, the first subsampled chroma component value and the second chroma component value or whether the auxiliary processing chain is used to derive the at least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value.
• the device 120 also comprises a deriver 122 for deriving the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to default 5 processing chain or the least one of the luma component value, the first subsampled chroma component value and the second subsampled chroma component value according to the auxiliary processing chain.
• the device 120 further comprises an encoder 123 for encoding the luma component value, the first subsampled chroma component value and the second chroma component value.
• a further aspect of the embodiments relates to a user equipment comprising a device according to any of the embodiments disclosed herein and, for instance illustrated in Figs. 14, 15 and 17.
• the user equipment is selected from a group consisting of a computer, a laptop, a smart phone, a tablet and a set-top box.
• any appropriate steps, methods, or functions may be performed through a computer program product that may, for example, be executed by the components and equipment illustrated in the attached figures.
• the memory 112 in Fig. 15 may comprise computer readable means on which a computer program can be stored.
• the computer program may include instructions which cause the processor 111 , and any operatively coupled entities and devices, such as input unit 113, output unit
• the computer program and/or computer program product may thus provide means for performing any steps herein disclosed.
• Each functional module may comprise software, computer programs, sub-routines, libraries, source code, or any other form of executable instructions that are executed by, for example, a processor.
• each functional module may be implemented in hardware and/or in software.
• one or more or all functional modules may be implemented by a processor 111 , see Fig. 15, possibly in cooperation with a memory 112.
• the processor 111 and the memory 112 may, thus, be arranged to allow the processor 111 to fetch instructions from the memory 112 and execute the fetched instructions to allow the respective functional module to perform any steps or functions disclosed herein.
• computing services such as hardware and/or software
• network devices such as network nodes and/or servers
• functionality can be distributed or re-located to one or more separate physical nodes or servers.
• the functionality may be re-located or distributed to one or more jointly acting physical and/or virtual machines that can be positioned in separate physical node(s), i.e. in the so-called cloud.
• cloud computing is a model for enabling ubiquitous on-demand network access to a pool of configurable computing resources such as networks, servers, storage, applications and general or customized services.
• FIG. 18 is a schematic diagram illustrating an example of how functionality can be distributed or partitioned between different network devices 300, 301 , 302 in a general case.
• there are at least two individual, but interconnected network devices 300, 301 which may have different functionalities, or parts of the same functionality, partitioned between the network devices 300, 301.
• the network devices 300, 301 , 302 may be part of the same wireless communication system, or one or more of the network devices may be so-called cloud-based network devices located outside of the wireless communication system.
• Fig. 19 is a schematic diagram illustrating an example of a wireless communication system, including an access network 1 and/or a core network 2 and/or an Operations and Support System (OSS) 3 in cooperation with one or more cloud-based network devices 300.
• the figure also illustrates a network node 4 of the access network 1 and a user equipment 5 according to the embodiments.
• OSS Operations and Support System
• the present Annex A comprises a description of the Ajusty processing chain that can be used according to the embodiments in order to derive a non-linear luma component value in the second color space.
• a combination of a highly non-linear transfer function, 4:2:0 or 4:2:2 subsampling and non-constant luminance ordering gives rise to severe artifacts in saturated colors.
• An example is described in Annex B, where changes between two colors of similar luminance can result in a reconstructed image with very different luminances.
• each component of a sample i.e. pixel, is represented by an integer or floating point value.
• a display such as screen, TV or monitor, that renders the video omits optical lights based on the digital values of the video signal.
• the function that translates the digital value V to optical light Y is the Electro-Optical-Transfer-Function (EOTF).
• EOTF Electro-Optical-Transfer-Function
• n X - « 0.15930176
• This transfer function is more non-linear than the gamma function in the sense that the maximum value of its first derivative over the range from 0 to 1 is larger than that of the gamma function.
• Chroma subsampling is typically done before compression as an initial step to reduce the amount of data.
• the chroma signal is reduced to half the resolution in the vertical direction.
• the chroma signal is reduced to half the resolution in both the vertical direction and the horizontal direction. This is typically done with some filtering operation to get a good quality signal but can also be done using nearest neighbor.
• a decoder In order to display a 4:2:0 or 4:2:2 video, a decoder performs upsampling of the chroma signal, which can be done using bilinear filters or longer filters.
• an Ajusty method is provided.
• the method can be performed in an encoder or in a pre-process to the encoder.
• a corrected Y' component is derived to compensate for the errors in the Cb' and/or the Cr' components.
• a unit such as a pre-processor or an encoder.
• the unit is configured to determine that the Cb' and/or Cr' components include errors, and when it has determined that the Cb' and/or Ci ⁇ components include errors, it is configured to derive a corrected Y' component to compensate for the errors in the Cb' and the Cr' components.
• the corrected Y' component can derived according to different embodiments as described below. Hence the corrected Y' component, Cb' and Cr' are then compressed resulting in that the image or video is perceived with a higher quality.
• the eye is much more sensitive to luminance changes than to chrominance changes, so the first rule must always be to make sure that the luminance does not deviate too much from the original value.
• the non-linear luminance Y' is adjusted prior to compression so that the linear luminance Y of the pixel is closer to its correct value. This is further described below.
• the Y value is the linear luminance. That is, the Y value of the original is the value you get when you take the original linear light RGB (2142, 4, 138) and convert it to XYZ. For example, if RGB is in the BT.2020 color space you can convert to XYZ using
• This Y component is the luminance that the eye is most sensitive to. It should not be confused with the Y' component mentioned above, which depends nonlinearly on R, G and B.
• the Y value is grossly wrong for pixel 97 when using RGB 4:2:0 subsampling and upsampling. Note that in this example, no compression has taken place, only quantization to 10 bits, and yet the Y value has a relative error of 85 %. Using Barten's model that predicts how large differences the eye can see, we see that this error is 195 Barten steps, or 195 times greater than what would be just noticeable.
• the Cb' component If we look at the Cb' component, it has the value 607, which is about halfway between 650, before the discontinuity, i.e. pixel 94, and 575, after the discontinuity, but the correct value is 575.
• the trouble is that the error not only affects the chrominance but also the luminance of the pixel, which becomes way too big.
• Our idea is to compensate this by changing the Y' component of the pixel. According to the embodiments we do not let the Y' component be 422 for the pixel, but selects the value 363 instead. In Table A3 we show the result.
• Table A3 - Pixel 97 is much more similar before and after color subsampling
• the error is now instead in the chrominance, but given the fact that the human visual system is less sensitive to errors in chrominance than in luminance, this is not much of a problem.
• the change in chrominance may not even be noticeable - the only difference is that the green component is 0.7008 cd/m 2 instead of 3.9750 cd/m 2 but that is most likely dwarfed by the red component anyway, since it is 2145, almost a thousand times larger.
• the value Y' is not optimized for a specific value of Cb' and Cr'. Instead the Y' value is selected so that it minimizes the luminance error for some different values of Cb' and Cr' or for a range of Cb' and Cr' values. This can be done in scenarios where the chroma upsampling method is unknown.
• One variant is to perform a specific number of chroma upsampling methods and then select the Y' value that minimizes the average squared error for the different upsampling methods.
• the Y' value is selected so that the worst case, i.e. largest error, is minimized.
• Another variant is to use two or more neighboring values of Cb' and Cr' and use them directly to calculate the range of possible Cb' and C values.
• Y' we will go through a number of them.
• Fig. 20 One way is shown in Fig. 20.
• the original pixel Ro, Go, Bo is transformed from RGB to XYZ as mentioned above.
• Xo, Yo, Zo of which we are only interested in Yo.
• Yo we do not need to calculate Xo and Zo.
• This Yo is the original luminance in linear light, and it is this that the eye is sensitive to.
• Yo is equal to 573.5991, see Table A1.
• %%% L is output luminance in nits
• n 0.1593017578125
• an embodiment is to do a binary search to find the best Y' value.
• the maximum Y' value is tried, for instance 1023.
• the minimum Y' value is tried, for instance 0.
• a value in the middle is tried, for instance 512. If the Y value resulting from Y -512 is larger than Yo, we should search in the interval [0, 512]. If the Y-value is smaller than Yo, we should instead search the interval [512, 1023].
• We then proceed by calculating a new value in the middle of the selected interval, and proceeds until the interval only contains a single number, such as [363, 363], or [363, 364]. This is guaranteed to only take log2(N) steps, where N is the number of possible values, which in this case is 1024. Hence only Iog2(1024) 10 steps are sufficient.
• X, Yo and Z are converted with a XYZ to RGB conversion to produce new values R1 , G1 and
• R1, G1 and B1 are converted with an inverse transfer function to produce R1', G1' and B1'.
• - R1', G1' and B1' are converted with an inverse color transform to produce Y'
• the Y' values of all pixels in an image or picture are corrected.
• only pixels that are at risk of having visible luminance errors are corrected. This may include pixels that are close to the color gamut edge, but exclude pixels closer to the middle of the color gamut triangle.
• Y' ⁇ Y'k (Yo - (ml + m2 + m3)) / (k1 +k2+k3).
• Another way may be to have linearizations that will be used close to the red primary, another close to the green primary, and a third that is used close to the blue primary. Furthermore it is possible to have several linearizations along the line of the gamut, as shown in Fig. 21. Thus, there can be different linearizations in different areas. If the color is inside the circle with a solid line, we are close to the green primary and we can use one linearization. If we are further along towards the red primary, i.e. inside the dashed box, we can use a second linearization. If we are close to the edge of the gamut and roughly half way between the red and the green primary, i.e. inside the dotted box, a third linearization can be used.
• a fourth linearization can be used. If we are inside the dotted circle, i.e. close to the red primary, a fifth linearization can be used. The same partitioning of the other two lines in the triangle can also be used. Finally the area inside the triangle but in no box or circle can be partitioned into one or more areas, each area having a different linearization. Another way is to create a look-up table (LUT). As can be seen from the above formulation, if we have Cb', Cr' and the wanted Y-value, it is possible to find the best Y' using any of the iterative techniques mentioned above.
• Ynonlinear TF(Y)
• Y' LUT(Cb', Cr', Ynonlinear). It may also be possible to use a smaller LUT. For instance, it may be possible to quantize Y (or Ynonlinear), Cb' and Cr' to, say, 6 bits.
• Y' (LUT(Cb'6bit, Cr ⁇ bit, Y6bit)+ (LUT(Cb'6bit+1 , Cr'6bit, Y6bit))/2
• a look-up table is used for deriving the value of Y'.
• the look-up table contains every possible value of Yo, Cb' and Cr ⁇ For 10 bit video that will result in 1024x1024x1024 entries and in some applications this size is acceptable.
• the look-up table (LUT) is pruned, for example through rounding one or more of Yo, Cb' and Cr', e.g. to 8 bits. If a pruned look-up table is used the algorithm can be extended with a refinement step that finds the best Y' starting from the Y' that was retrieved from the LUT.
• the LUT is only used when the Cb' value and/or the Cr' value is such that Y' can become too different from Yo, i.e. when the initially calculated Y' gives a linear Y value that is too different from Yo, and thereby the size of the LUT can be substantially reduced since many values of Cb' and Cr' will not cause any problems.
• a function of Yo, Cb' and Cr' e.g. a linear combination, polynomial function, exponential function, logarithmic function, trigonometric function, etc.
• the function can be defined such that for any value of Yo, Cb' and Cr' the difference between Y and Yo is below a certain threshold or the difference between the derived Y' and the optimal Y' is below a certain threshold.
• Yo, Cb' and Cr' are defined for deriving Y'.
• the selection of which function to use is based on the value of Yo, Cb' and Cr'.
• This Annex B investigates color artifacts due to 4:2:0 subsampling.
• Table B1 The data presented in Table B1 was for BT.2020 primaries. If the input data is with Rec709 primaries, but the container is BT.2020, it will not be possible to reach the color gamut boundary. This is due to the fact that the Rec709 gamut triangle is inside the BT.2020 triangle, not touching the gamut boundary, as can be seen in Fig. 23. It is therefore reasonable to assume that the relative error will be smaller.
• Table B6 data for the "worst" color for 4:2:0 subsampling if input is Rec709 and container format is
• Table B7 data for the "worst" color for 4:2:0 subsampling if input is P3 and container format is
• the present Annex C comprises a description of the Ajustc method that can be used according to the embodiments in order to derive non-linear chroma component value(s) in the second color space.
• RGB Linear RGB values, where each value is proportional to the cd/m 2 ("number of photons").
• XYZ Linear XYZ values, where each value is a linear combination of RGB. Y is called “luminance” and loosely speaking reflects well what the eye perceives as "brightness.
• pq(Y) A non-linear representation where the non-linear function pq(Y) has been applied to the linear luminance Y. pq(Y) is not to be confused with Y'. Since pq(.) resembles Barten's curve, pq(Y) a small step in pq(Y) is equivalent to a small step in perceived luminance.
• R'G'B' Non-linear RGB values.
• An example of a non-linear function is the PQ transfer function.
• 'Cb'C A non-linear representation where each value is a linear combination of R', G' and B'.
• Y' is called "luma”
• Cb' and Cr' are collectively called "chroma”. This is to distinguish Y' from luminance, since Y' also contains some chrominance, and Cb' and Cr' also contains some luminance.
• xy is a good measure of chrominance.
• u'v' A non-linear representation of chrominance, that is a non-linear function of xy. It is supposed to be more perceptually uniform, meaning that a small step in u'v' will be equally perceptible regardless of which chrominance we are at.
• pq(Y)xy A representation of color where pq(Y) contains all the luminance and xy all the chrominance. From pq(Y) it is possible to extract Y, and from Y, x, and y it is possible to extract XYZ which can be transformed to RGB.
• the problem with using the anchor processing chain is that apart from getting inaccurate luminance, we may also get inaccurate chrominance. This is due to the fact that the chroma samples Cb' and Cr' are subsampled in the Y'Cb'Cr' space. There is a problem with this, namely that the non-linearity of the Y'Cb'Cr' space will favor dark colors. This is not a desired outcome. This means that the chroma samples Cb' and Cr' will be inaccurate to start with.
• the HDR pixel is quite dark - the highest coefficient is 3.41 out of 4000, so the darkest exposure is the most relevant here.
• the top left pixel is reddish and the surrounding pixels look black. Only in the brighter exposures is it possible to see that the dark pixels are actually a bit greenish.
• the problem here is that the redness of the top left pixel has disappeared and has been replaced with a gray/white pixel.
• the reason is that averaging in the non-linear Y'Cb'Cr' domain favors dark colors, which will make the resulting pixel unduly green.
• the result will be yet greener.
• the result is not very similar to the original.
• the result of the solution proposed in this Annex can be seen below.
• the colors are closer to the original, especially in the relevant first exposure and in the important top left pixel, which is now more reddish. Note that in general it is not possible to get a perfect color in all four pixels since the chrominance is subsampled, hence even the proposed Ajustc method will not generate a perfect color. Nevertheless, the color fidelity is greatly improved.
• the embodiments relate to video coding and processing of at least one pixel of a picture of a video sequence.
• the processing is a pre-process to encoding or part of the encoding step.
• the embodiments propose a novel way of arriving at the subsampled chroma samples Cb' and Cr' when processing a pixel of the video sequence.
• RGB 4:4:4 -subsampling-of-RGB-> RGB 2:2:2 -> R'G'B' 2:2:2 -> YCb'Cf 2:2:2 where the subsampling instead is the first part of the chain and is performed in the linear domain.
• 2:2:2 we have used the term 2:2:2 to indicate that all three samples are at half resolution in both the x- and y- dimension. In this way, we do not get a full-resolution Y' component, since the Y' component in the last step is 2:2:2, i.e., half resolution in both directions.
• This problem is overcome by the following processing steps:
• RGB 4:4:4 --> XYZ 4:4:4 --> Y 4:4:4 H— > Y' Ajusty(Y4:4:4, Cb'Cr'4:4:4)
• the Y' component is found in (G) and the Cb'Cr' components are found in (D), it is possible to further improve it.
• the reason is that the each Cb' value, not being full resolution, will be used for several Y' values. However, due to the non-linear nature of the 'Cb'C space, changing the Y' component will affect not only the luminance of the pixel, but also the chrominance.
• the Cb' and Cr' samples are further refined to counteract the shifting that happens when the low-resolution Y' component is replaced by a high resolution Y' component. This further improves the color fidelity.
• the Y' samples after the Ajusty method are modified towards the Y' values before the ajusty method if that reduces the error in another metric than the luminance without causing any visual degradation of the luminance.
• Figs. 24-26 The input is a high resolution image in linear RGB.
• a real image is of course much bigger.
• the embodiments described below can be used in combination or separately.
• First embodiment - find good values for Y' and Cb'Cr'
• the first step is to subsample the RGB image (A) to obtain (B), subsampling in both the x- and y-dimension.
• Second embodiment - improve values for Y' and Cb'Cr'
• the Cb'Cr' from the first embodiment are updated by the difference between the Cb'Cr' derived from upsampling in a representation invariant to intensity and the upsampled Cb'Cr' from the first embodiment.
• the updated Cb'Cr' is then used together with the linear luminance Y to derive new Y' values using ADJUSTY. All steps are shown below and are also indicated in Figs. 24-26.
• the next step is to convert from RGB ( ⁇ ') to XYZ and then to xyz, throwing away z and keeping xy (C).
• Each Yxy value (D') is then converted back to XYZ, then to RGB, then to R'G'B' and lastly to Y'Cb'Cr' ( ⁇ '). This is the second reference.
• Second step iterating.
• the difference signal in (K) is then subsampled (L), and then added to the Cb'C signal in (H), resulting in an updated version of Cb'Cr' (M).
• the "find good values for Y' and Cb' Cr'" is replaced by the anchor processing chain; RGB 4:4:4 -> R'G'B' 4:4:4 -> Y'Cb'Cr' 4:4:4 -> subsampling Y'Cb'Cr' 4:2:0. Then a new reference Y must be created each iteration by taking Y'Cb'Cr' 4:2:0-upsample-Cb'-and-Cr'-> Y'Cb'Cr' 4:4:4 -> R'G'B' 4:4:4 -> RGB 4:4:4 -> XYZ 4:4:4 -> Y 4:4:4.
• the subsampling in above embodiment consists of an adaptive filter which is optimized to reduce the error in a representation space in accordance with human vision.
• Cb' and Cr' are modified to get a better representation in the original representation space (or a representation space in accordance with human vision).
• the modification can be performed by iterative testing to modify a Cb' or Cr' value +1 or -1 and then upsample the modified value and unmodified values of Cb' and Cr', conversion to R'G'B', application of non-linear transfer function and converting to a representation space in accordance with human vision. If the modified Cb'/Cr' gives less absolute or squared error for a spatial region in a chroma specific color component representation space (not luminance) similar to Z for Cb' and X for Cr", the modification is selected.
• the spatial region is related to the size of the upsampling filter(s). If bi-linear filters are used for the upsampling of 4:2:0 Cb' or Cr' refinements to 4:4:4 the size of the region is 3 horizontally and 4 vertically. If 4-taps filters are used for the upsampling the size of the region is 7 horizontally and 8 vertically.
• the error can also be weighted according to how a modified value can influence each of the 3x4 or 7x8 neighboring samples in 4:4:4 or just simply include the central part (3x2) of the region.
• the magnitude of a Cb' or Cr' refinement is initialized to the magnitude of the average value of ajusty in the central 3x2 region around the sample to be modified.
• the sign of the Cb' or Cr' refinement is derived by taking the sign from the average of the error before the refinement in a chroma specific color component representation space (not luminance) similar to Z for Cb' and X for Cr ⁇
• initial value of the Cb' refinement is Cb'-3 ⁇ 4ign(error)xabs(average(Y' before adjust - Y' after adjust)/N). Where N is about 2.
• the modification is selected. Similar applies if the method is applied for Cr'.ln another embodiment the magnitude of a Cb' or Cr' refinement is initialized to the average value of Cb' or Cr' in 4:4:4 in the central 3x2 region around the sample to be modified. Thus init value of the Cb' refinement is average(Cb' in 4:4:4). If the modified Cb' gives less absolute or squared error for a spatial region in a chroma specific color component representation space (not luminance) similar to Z for Cb', the modification is selected. Similar applies if the method is applied for Cr ⁇ Y' refinement
• the adjustment of Y' to Ymod' by the Ajusty method is modified by one step towards Y' if that reduces the error in a chroma specific color component representation space similar to Z in XYZ (a metric other than luminance) while not causing any visual degradation in luminance Y.
• This can besides improving performance in a chroma specific color representation space also in some cases make it easier to encode Y'.
• Certain embodiments in this Annex describe the conversion from RGB 4:4:4 to Y'Cb'Cr' 4:2:0 where the chroma components are subsampled in both vertical and horizontal direction.
• the methods of the embodiments would of course work equally well when converting from RGB 4:4:4 to another subsampled Y'Cb'Cr format, e.g. Y'Cb'Cr' 4:2:2 where the chroma components have been subsampled in the horizontal direction only.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• Discrete Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Processing Of Color Television Signals (AREA)
• Color Television Systems (AREA)
• Color Image Communication Systems (AREA)
• Compression Of Band Width Or Redundancy In Fax (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=57441338

Family Applications (1)

Country Status (5)

Cited By (2)

Families Citing this family (1)

Citations (2)

Family Cites Families (7)

• 2016
  • 2016-05-02 WO PCT/SE2016/050383 patent/WO2016195567A1/en unknown
  • 2016-05-02 CN CN201680045853.4A patent/CN107925778B/zh active Active
  • 2016-05-02 BR BR112017025604-5A patent/BR112017025604A2/pt unknown
  • 2016-05-02 US US15/579,053 patent/US10674182B2/en active Active
  • 2016-05-02 EP EP16803847.9A patent/EP3304913A4/en not_active Ceased

Patent Citations (2)

Non-Patent Citations (4)

Also Published As

Similar Documents

Legal Events