Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/14—Picture signal circuitry for video frequency region
  • H04N5/20—Circuitry for controlling amplitude response
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
  • G09G5/02—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the way in which colour is displayed
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
  • G09G5/10—Intensity circuits
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/117—Filters, e.g. for pre-processing or post-processing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/182—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/80—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
  • H04N19/86—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving reduction of coding artifacts, e.g. of blockiness
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
  • H04N19/98—Adaptive-dynamic-range coding [ADRC]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/64—Circuits for processing colour signals
  • H04N9/67—Circuits for processing colour signals for matrixing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/64—Circuits for processing colour signals
  • H04N9/68—Circuits for processing colour signals for controlling the amplitude of colour signals, e.g. automatic chroma control circuits
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G2340/00—Aspects of display data processing
  • G09G2340/04—Changes in size, position or resolution of an image
  • G09G2340/0407—Resolution change, inclusive of the use of different resolutions for different screen areas
  • G09G2340/0428—Gradation resolution change

Definitions

• the invention relates to methods and apparatuses for encoding respectively decoding a temporally successive set of high dynamic range images, called herein a HDR video.
• LDR low dynamic range
• SDR standard dynamic range
• the black of such a SDR display may typically be 0.1 nit in good circumstances, yet 1 nit or even several nits in worst circumstances, so the SDR display dynamic range (the brightest white divided by the darkest viewable black) would be 1000: 1 at best, or worse, which corresponds nicely to such uniform illuminated scenes, and an 8 bit coding for all the required to be rendered pixel grey values or brightnesses, having a gamma of approximately 2.0, or encoding inverse gamma 0.5. Rec. 709 was the typically used SDR video coding.
• a 10 bit coding system may encode either a HDR video, or an SDR video, determined on the type of encoding, and in particular the electro-optical transfer function EOTF of the reference display associated with the coding, i.e. defining the relationship between the luma codes [0, 1023] and the corresponding luminances of the pixels, as they need to be rendered on a display.
• EOTF electro-optical transfer function
• a HDR image or video when a HDR image or video is mentioned, it has a corresponding peak brightness or maximum luminance for the highest luma code (or equivalently highest R', G', B' values in case of an RGB coding e.g. rather than an YCbCr encoding) which is higher than the SDR value of 100 nit, typically at least 4x higher, i.e. the to be rendered maximum display luminance for having the HDR image look optimal may be e.g.
• a high dynamic range coding of a high dynamic range image is capable of encoding images with to be rendered luminances of e.g. up to 1000 nit, to be able to display- render good quality HDR, with e.g. bright explosions compared to the surrounding rendered scene, or sparkling shiny metal surfaces, etc.
• very high dynamic range e.g. an indoors capturing with objects as dark as 1 nit, whilst simultaneously seeing through the window outside sunlit objects with luminances above 10, 000 nit, giving a 10000: 1 dynamic range, which is lOx larger than a 1000:1 DR, and even 100 times larger than a 100: 1 dynamic range, and e.g.
• TV viewing may have a DR of less than 30: 1 in some typical situations, e.g. daylight viewing). Since displays are becoming better (a couple of times brighter PB than 100 nit, with 1000 nit currently appearing, and several thousands of nits PB being envisaged), a goal is to be able to render these images beautifully, and although not exactly identical to the original because of such factor like different viewing conditions, at least very natural, or at least pleasing. And this needs what was missing in the SDR video coding era: a good pragmatic HDR video coding technology.
• Fig. 1 shows a couple of illustrative examples of the many possible HDR scenes a HDR system of the future (e.g. connected to a 1000 nit PB display) may need to be able to correctly handle, i.e. by rendering the appropriate luminances for all objects/pixels in the image.
• a HDR system of the future e.g. connected to a 1000 nit PB display
• ImSCNl is a sunny outdoors image from a western movie
• ImSCN2 is a nighttime image.
• Fig. 1 So on the left axis of Fig. 1 are object luminances as one would like to see them in a 5000 nit PB master HDR grading for a 5000 nit PB display. If one wants to convey not just an illusion, but a real sense of the cowboy being in a bright sunlit environment, one must specify and render those pixel luminances sufficiently bright (though also not too bright), around e.g. 500 nit. For the night scene one wants mostly dark luminances, but the main character on the motorcycle should be well-recognizable i.e. not too dark (e.g. around 5 nit), and at the same time there can be pixels of quite high luminance, e.g. of the street lights, e.g.
• the third example ImSCN3 shows what is now also possible on HDR displays: one can simultaneously render both very bright and very dark pixels.
• a dark cave with a small opening through which we see the sunny outside.
• a color grader may want to optimally coordinate the luminances of all objects, so that nothing looks inappropriately dark or bright and the contrast are good, e.g. the person standing in the dark in this cave may be coded in the master HDR graded image around 0.05 nit (assuming HDR renderings will not only be able to render bright highlights, but also dark regions).
• Applicant has designed a coding system, which not only can handle the communication (encoding) of merely a single standardized HDR video, for a typical single kind of display in the field (with every end viewer having e.g. a 1000 nit PB display), but which can at the same time communicate and handle the videos which have an optimal look for various possible other display types with various other peak brightnesses in the field, in particular the SDR image for a 100 nit PB SDR display.
• Applicant however designed a system which communicates HDR images as LDR images, i.e. actually LDR (or SDR, i.e. referred to a 100 nit PB reference display, and often optimally color graded on such a reference display) images are communicated, which then can already immediately be used for rendering the correctly looking SDR look on legacy 100 nit PB SDR displays.
• LDR or SDR, i.e. referred to a 100 nit PB reference display, and often optimally color graded on such a reference display
• F_ct a set of appropriate reversible color transformation functions
• These functions may be defined by a human color grader, to get a reasonably looking SDR image (Im_LDR) corresponding to the HDR master image MAST_HDR, whilst at the same time ensuring that by using the inverse functions IF_ct the original master HDR (MAST_HDR) image can be reconstructed with sufficient accuracy as a reconstructed HDR image (Im_RHDR), or, automatic analysis algorithms may be used at the content creation side for determining suitable such color transformation functions F_ct. Note that instead of relying on a receiving side to invert the functions F_ct into IF_ct, one can also send already the needed functions for calculating Im_RHDR from the received and decoded SDR image Im_RLDR.
• a typical coding chain as shown in Fig. 2 works as follows.
• Some image source 201 which may e.g. be a grading computer giving an optimally graded image, or a camera giving a HDR output image, delivers a master HDR image MAST_HDR, to be color transformed and encoded.
• a determined color transformation e.g. a concave bending function
• more complex luminance mapping functions may be employed, provided that they are sufficiently reversible, i.e. the Im_RHDR image has negligible or acceptable banding.
• an output image Im_LDR results.
• This image or set of images is encoded with a legacy LDR image encoder, which may potentially be modified somewhat, e.g. the quantization tables for the DCT-ed transformations of the prediction differences may have been optimized to be better suited for images with HDR characteristics (although the color transformations may typically already make the statistics of the Im_LDR look much more like a typical LDR image than a typical HDR image, which HDR image typically has relatively many pixels with relatively dark luminances, as the upper part of the range may often contain small lamps etc.).
• a MPEG-type encoder may be used like HEVC (H265), yielding an encoded SDR image Im_COD.
• This video encoder 203 pretends it gets a normal SDR image, although it also gets the functions F_ct which allow the reconstruction of the master HDR image, i.e. effectively making this a dual co-encoding of both an SDR and a HDR look, and their corresponding set of images (Im_RLDR, respectively Im_RHDR).
• This metadata comprising all the information of the functions F_ct, e.g. they may be communicated as SEI messages.
• a transmission formatter 204 applies all the necessary transformations to format the data to go over some transmission medium 205 according to some standard, e.g. a satellite or cable or internet transmission, e.g.
• a receiver 206 which may be incorporated in various physical apparatuses like e.g. a settopbox, television or computer, undoes the channel encoding by applying unformatting and channel decoding.
• a video decoder 207 applies e.g. HEVC decoding, to yield a decoded LDR image Im_RLDR.
• a color transformer 208 is arranged to transform the SDR image to an image of any non-LDR dynamic range.
• a display tuning unit 209 may be comprised which transforms the SDR image Im_RLDR to a different dynamic range, e.g. Im3000 nit being optimally graded in case display 210 is a 3000 nit PB display, or a 1500 nit or 1000 nit PB image, etc.
• Fig. 3 shows how one can design such a chromaticity-preserving luminance re-calculation, taken from WO2014056679.
• This processing when seen in the gamut normalized to 1.0 maximum relative luminance for both the SDR and the HDR image (i.e. assuming that the SDR and HDR have the same e.g. Rec. 2020 primaries, they have then exactly the same tent-shaped gamut; as shown in fig. 1 of WO2014056679). If one were to drive any display with e.g. the cowboy having in the driving image a luma code corresponding to a luminance of 10% of peak brightness of the display, then that cowboy would render brighter the higher the PB of the display is.
• Downgrading in this text means changing the luma codes of the pixels (or their corresponding to be rendered luminances) from a representation of higher peak brightness (i.e. for rendering on a higher PB display, e.g. of 1000 nit PB) to the lumas of an image of the same scene in a lower PB image for rendering on a lower PB display, e.g.
• the appropriate g-value is calculated for applying the desired color transformation which transforms Im_RLDR into Im_RHDR (or in an appropriately scaled manner into any other graded image, like Im3000nit), when luminance mapper 307 gets some SDR-luminance to HDR_luminance mapping function, e.g. a parametrically specified loggamma function or sigmoid, or a multilinear curve received as a LUT.
• a parametrically specified loggamma function or sigmoid or a multilinear curve received as a LUT.
• the components of the exemplary embodiment circuit are: 305: maximum calculator, outputting the maximum one (maxRGB) of the R,G, and B values of a pixel color being processed; 301: luminance convertor, calculating the luminance of a color according to some color definition standard with which the system currently works, e.g. Rec.
• the closer approximating artifact-free reconstruction of the master HDR images can be obtained by agreeing to make some modifications to the decoder and corresponding encoder, and in particular by having a video decoder (220) arranged to decode and output a set of temporally successive high dynamic range images (Im_RHDR) from coded input images which are a set of encoded low dynamic range images (Im_COD) which are in a format wherein pixel colors are represented by luma values (Y'4L) and chrominance values (C'bOL, C'rOL) , the chrominance values (C'bOL, C'rOL) being sub-sampled spatially compared to the luma values (Y'4L), the video decoder comprising:
• a video decoder (207) arranged to decode the encoded low dynamic range images into uncoded low dynamic range images (Im_RLDR);
• a first and second spatial upscaler (501, 502) arranged to upscale chrominance values of the low dynamic range images (Im_RLDR) to a resolution of the luma values;
• a color converter (503) arranged to, for each pixel of the low dynamic range images (Im_RLDR), convert the luma value and chrominance values into a non-linear R'G'B' representation (R'4, G'4, B'4) comprising three non-linear R'G'B' color values;
• a maximum calculation unit (504) arranged to output, for each pixel, a maximum (m) of the three non-linear R'G'B' color values;
• a luminance mapping unit (505) arranged to receive a luminance mapping function (TM) which defines a relationship between luminances or lumas of one of the low dynamic range input images (Im_RLDR) being processed to luminances respectively lumas of the corresponding high dynamic range output image (Im_RHDR) to be obtained, and arranged to, for each pixel, apply the luminance mapping function to the maximum (m), yielding an output maximum (m*);
• TM luminance mapping function
• Im_RLDR low dynamic range input images
• Im_RHDR high dynamic range output image
• a gain factor calculator (506) arranged to, for each pixel obtain a gain value (g) by dividing the output maximum (m*) by the maximum (m);
• first and second spatial downscaler (507, 508) arranged to spatially downscale the gain values to a resolution of the chrominance values of the low dynamic range images;
• the uncoded images will typically be in a known format, namely Y'CbCr, in which Y' is a luma which can be calculated from the non-linear R', G', B' coefficients, which without losing generality we assume to be the square root of the linear RGB
• the upscaler preferably does a good quality upscaling, but more importantly, an upscaling that at this place before establishing and performing of the color transformation to convert to the other dynamic range image, is the same in the encoder as the decoder uses. That guarantees that the functional color transforms generating the Im_LDR for the MAST_HDR, and the inverse one can be the same (though inverse), i.e. the MAST_HDR can be closely reconstructed as Im_RHDR. It will lead to the correct gain values in a 2D gain image to be applied, rather than potentially at some spatial locations in some scenarios incorrect gain values.
• the appropriate brightness scaling can happen on any power function of the linear red, green and blue coefficients, e.g. a square root, or an additive or subtractive combination of two such components with the same power, but, to allow the property of staying within gamut, the evaluation of where the color resides in the gamut (i.e. e.g. safely halfway, or close to the top of the 3D color gamut, i.e. the maximally achievable brightness for that chromaticity) is in this technology done by working on the maximal one of the three non-linear red, green and blue color components (typically preferably defined with an OETF or code allocation power value of 0.5, as a pure power, i.e.
• R' power(R,0.5) and the same for the other color components, in a full range normalized [0.0,1.0] representation of both R and R'). Instead of 0.5 e.g. 1/(2.4) or other values can be used too.
• the color transformation can then be formulated as a multiplicative strategy with a factor which we call gain factor, whereby the skilled reader should understand that if the gain factor is smaller than 1.0, e.g. 0.3, then we actually have a darkening of the relative luminance in the normalized color gamut.
• gain factor is smaller than 1.0, e.g. 0.3, then we actually have a darkening of the relative luminance in the normalized color gamut.
• the decoder now establishes the correct gain for each incoming Y'4L, C'bOL, C'rOL triplet, as it comes in.
• the decoding of the encoded low dynamic range images may provide uncoded low dynamic range images in a format wherein pixel colors are represented by luma values and chrominance values, the chrominance values being sub-sampled spatially compared to the luma values.
• a pixel color may specifically be represented by one luma value and two chrominance values.
• the resolution of the luma values is typically the same as the pixel resolution of the low dynamic range images.
• the resolution of the chrominance values is lower than the resolution of the low dynamic range images, and thus typically lower than the pixel resolution of the low dynamic range images.
• a chrominance value may typically be common for a plurality of luma values/ pixels of the low dynamic range images.
• the high dynamic range images may similarly be in a format wherein pixel colors are represented by luma values and chrominance values.
• a maximum m, output maximum m*, and gain value may be determined for each pixel of the (uncoded) low dynamic range images, and each pixel of the (uncoded) low dynamic range images may be multiplied by the corresponding gain value.
• the low dynamic range images may specifically refer to the uncoded low dynamic range images.
• the respective gain value for a given luma or chrominance value may be a gain value determined for the position of the value from the values determined by the gain factor calculator. For a luma value for a pixel, the respective gain value is typically the gain value determined for that pixel.
• the respective gain may be a gain determined for the position of the chrominance value by the first or second spatial downscaler.
• the first and second spatial upscaler (501, 502) are programmable to apply at least two different upscaling algorithms, and arranged to receive the upscaling algorithm to be used (INTSTR1) for decoding the presently input set of encoded low dynamic range images (Im_COD).
• Im_COD encoded low dynamic range images
• the particular upscaling algorithm (and preferably also the used downscaling algorithm for the gain values in case that is not fixed) can be forced into exactly the same one used at the encoder, because as said that is needed for accurate dynamic range conversion.
• the encoder need not actually communicate the algorithm as code, but typically it will in a rescaling strategy type indicator INTSTR1 indicate which type of interpolation is used, e.g. the intermediate missing values of the chrominances are copied from their neighbor, or equated as a weighed combination from their neighbours, etc., and then
• the two upscaling or downscaling algorithms could be different for the two chrominances, but typically both chrominance components are defined similarly, i.e. their upscaling respectively downscaling algorithms will be the same, and only one INTSTR1 may be needed. In the simpler cases the downscaling may simply be a dropping in case of co-sited chrominance coefficients co-sited with luma samples, i.e.
• the video decoder (220) comprises a color converter (512) arranged to convert the output set of high dynamic range images (Im_RHDR) from a Y'CbCr representation defined with an inverse gamma of 0.5 (or 1/(2.4)) into a high dynamic range RGB color representation.
• a color converter 512
• This display may accept pixel colors which are defined in e.g. HDR10, or any HDR-OETF-based e.g. 10 or 12 bit representation of the additive primary RGB color channels needed for establishing the local color contributions as they can be realized e.g. with LCD valves, or OLED sub-pixels, etc. This does not intend to limit that both any combination of high dynamic range images (Im_RHDR) from a Y'CbCr representation defined with an inverse gamma of 0.5 (or 1/(2.4)) into a high dynamic range RGB color
• the decoder may reside e.g. directly in a television or mobile device, without the need of an intermediate apparatus like a settopbox, blu-ray player, computer, professional cinema HDR video receiver for supplying a HDR cinema projector, image pre-analysis system in a video surveillance system, video telecommunication or telepresence system, medical system with HDR output, etc.
• an intermediate apparatus like a settopbox, blu-ray player, computer, professional cinema HDR video receiver for supplying a HDR cinema projector, image pre-analysis system in a video surveillance system, video telecommunication or telepresence system, medical system with HDR output, etc.
• the video decoder (220) may output as final output red, green and blue color components (R", G", B") of the high dynamic range image represented with RGB pixel colors which are defined with an opto-electrical transfer function (OETF) as defined in SMPTE 2084.
• OETF opto-electrical transfer function
• one has a method of video decoding arranged to decode and output a set of temporally successive high dynamic range images (Im_RHDR) from coded input images which are a set of encoded low dynamic range images (Im_COD) which are in a format wherein pixel colors are represented by luma values (Y'4L) and chrominance values (C'bOL, C'rOL), the chrominance values (C'bOL, C'rOL) being sub-sampled spatially compared to the luma values (Y'4L), the method comprising:
• TM luminance mapping function
• the upscaling is programmable and can apply at least two different upscaling algorithms, and applies a received upscaling algorithm to be used (INTSTR1) for decoding the presently input set of encoded low dynamic range images (Im_COD).
• C'b4, C'b4 of one of the high dynamic range images into sub-sampled chrominance values (C'bO, C'rO) of the pixel colors of the one of the high dynamic range images (MAST_HDR);
• a first and second spatial upscaler (406, 407) arranged to upscale the sub- sampled chrominance values (C'bO, C'rO) to a resolution of the luma values (Y'4) of the one of the high dynamic range images (MAST_HDR);
• a color converter (408) arranged to, for each pixel of the one of the high dynamic range images (MAST_HDR), convert the luma values and the upscaled
• a maximum calculation unit (409) arranged to, for each pixel, output a maximum (m) of the three non-linear R'G'B' color values;
• a luminance mapping unit (410) arranged to receive a luminance mapping function (TM) which defines a relationship between luminances or lumas of a corresponding one of the low dynamic range input images (Im_LDR) to be encoded as an encoded low dynamic range image (Im_COD) and the luminances respectively lumas of the high dynamic range output image (MAST_HDR) being encoded, and arranged to, for each pixel, apply the luminance mapping function to the maximum (m), yielding an output maximum (m*); - a gain factor calculator (411) arranged to, for each pixel, obtain a gain value (g) by dividing the output maximum (m*) by the maximum (m);
• TM luminance mapping function
• a first and second spatial downscaler (412, 413) arranged to spatially downscale the calculated gain values to a resolution of the sub- sampled chrominance values (C'bO, C'rO) of the pixel colors of the one of the high dynamic range images (MAST_HDR); and
• - multipliers (414, 415, 416) arranged to, for each pixel, multiply the luma values and sub-sampled chrominance values (C'bO, C'rO) with the determined respective gain value (g) , the respective gain values for the sub- sampled chrominance values being downscaled gain values.
• the high dynamic range images may be in a format wherein pixel colors are represented by luma values and chrominance values.
• a pixel color may specifically be represented by one luma value and two chrominance values, i.e. the pixel colors may be represented by one luma channel and two chrominance channels.
• the resolution of the luma values and the chrominance values (before sub-sampling) for the high dynamic range images is typically the same as the pixel resolution of the high dynamic range images.
• the resolution of the chrominance values of the high dynamic range images following sub-sampling/ downscaling is lower than the pixel resolution of the high dynamic range images, and thus typically lower than the pixel resolution of the high dynamic range images.
• a sub- sampled chrominance value may typically be common for a plurality of luma values/ pixels of the low dynamic range images.
• a maximum m, output maximum m*, and gain value may be determined for each pixel of the high dynamic range images, and each pixel of the high dynamic range images may be multiplied by the corresponding gain value (it will be appreciated that for sub-sampled chrominance values being common for a plurality of pixels, only one multiplication may be performed, i.e. a single common multiplication may implement the multiplication of chrominance values and the gain values for a plurality of pixels).
• the respective gain value for a given luma or chrominance value may be a gain value determined for the position of the value from the values determined by the gain factor calculator. For a luma value for a pixel, the respective gain value is typically the gain value determined for that pixel.
• the respective gain may be a gain determined for the position of the chrominance value by the first or second spatial downscaler. It may be advantageous to have a video encoder wherein the first and second spatial upscaler are determined to apply a fixed upscaling algorithm which is pre-agreed to be used by all decoders to be supplied with the set of encoded low dynamic range images.
• the first and second spatial upscaler (406, 407) are determined to apply a fixed upscaling algorithm which is pre-agreed to be used by all decoders to be supplied with the set of encoded low dynamic range images (Im_COD).
• one has a method of video encoding arranged to encode a set of temporally successive high dynamic range images (MAST_HDR) into encoded images which are a set of encoded low dynamic range images (Im_COD) which are in a format wherein pixel colors are represented by luma values (Y'4L) and chrominance values (C'bOL, C'rOL), the chrominance values being sub-sampled spatially compared to luma values (Y'4L), the method comprising:
• C'b4, C'b4 - spatially sub-sampling chrominance values (C'b4, C'b4) of one of the high dynamic range images (MAST_HDR) being encoded into sub-sampled chrominance values (C'bO, C'rO) of the pixel colors of the one of the high dynamic range images (MAST_HDR);
• R'4, G'4, B'4 comprising three non-linear R'G'B' color values
• TM luminance mapping function
• Im_LDR low dynamic range input images
• Im_COD encoded low dynamic range image
• MAST_HDR high dynamic range output image
• the present new technical ideas may be embodied in various forms, such as connected systems, partial services on remote locations which may be communicated over generic or dedicated networks, a computer program product comprising code which when run on a processor enables the processor to perform all methods steps of one of the above method claims, any video signal codification comprising the codification INTSTR1 uniquely specifying the upscaling algorithm which must be used at both the encoder and decoder, and if applicable a codification INTSTR1* equating the downscaling algorithm for the gain values, etc.
• Fig. 1 schematically illustrates a number of typical color transformations which occur when one optimally maps a high dynamic range image to a corresponding optimally color graded similarly looking (as similar as desired and feasible given the differences in the first and second dynamic ranges DR_1 resp. DR_2), which in case of reversibility would also correspond to a mapping of an LDR image of the HDR scene, to a HDR image of that scene;
• Fig. 2 schematically illustrates a technology to encode high dynamic range images, i.e. images capable of having luminances of at least 400 nit typically or more, which applicant recently developed, which actually communicates the HDR image(s) as an LDR image plus metadata encoding color transformation functions comprising at least an appropriate determined luminance transformation for the pixel colors, to be used by the decoder to convert the received LDR image(s) into HDR images(s) which are a faithful reconstruction of the original master HDR image(s) created at the image creation side;
• Fig. 3 schematically illustrates a first version technology of applicant which can be used to enable a within-gamut color (i.e. chromaticity) -preserving luminance transformation for changing the brightnesses of image objects or pixels for making them more conforming to what is needed in an image of a dynamic range which is different and specifically larger than the dynamic range of the input image, which works well for particular types of situations;
• a within-gamut color i.e. chromaticity
• Fig. 4 schematically shows the newly invented colorimetric core part (400) for calculating the dynamic range transform of a video decoder
• Fig. 5 schematically shows the newly invented colorimetric core part (500) for calculating the dynamic range transform to the low dynamic range image corresponding to the master HDR image of a encoder, which is suitable to operate with installed legacy video compression technologies like HEVC, and the further video communication technologies based thereupon, or future similar video compression technologies; and
• Fig 6 schematically shows two possible algorithms for upsampling sub- sampled chromaticities to the resolution of the lumas.
• Fig. 4 shows a typical exemplary embodiment of how to build the new encoder. It is assumed that the master HDR image is defined in an R"G"B" color representation according to a SMPTE ST2084 OETF. Three color component transformation units (401, 402, 403) apply the function which redefines those values in a square root (in this elucidation, or alternatively e.g. defined by a 1/(2.4) power) OETF representation (i.e. one can interpret this as first transforming the R"G"B" values into linear color components RGB, and then in square root versions R'G'B', but this can be done by applying one function). The color transformer 404 transforms this in a Y'CbCr format.
• this Y'CbCr representation of the pixel colors is in spatial subscaler 405 sub-sampled to e.g. 4:2:0, i.e. dropping 75% of the chrominance values, yielding the sub-sampled image representation of the HDR master image: Y'4, C'bO, C'rO.
• the pixel colors are following the spatial subscaler 405 represented in a non-linear format comprising one luma value and typically two spatially subsampled chrominance values.
• this signal needs to be upsampled again, to get the right resolution for the dynamic range transform (remember that the lumas are on the original, full resolution). It is important however that this is done in a consistent manner, and exactly the same manner as the decoder will apply it. Whatever the "real" values of the high resolution Y'CbCr values are is then not so much important, as it is important that the encoder and decoder do the same dynamic range transformation, so that at any
• decoder/receiver side it is fully reversible (at least up to some errors like banding from word length, and DCT errors and the like).
• This allows the original master HDR image to be be reconstructed with high accuracy, yet, whilst having all the benefits of this HDR encoding technology, such as inter alia, a good LDR image quality (in particular with no color desaturation as would occur with typical non-chromaticity preserving dynamic range changing color processing), staying within the achievable color gamut, which would again avoid color errors due to clipping, etc.
• a particular advantage of the described approach is that it allows the processing (upscaling) to be performed in both the encoder and the decoder based on the same values (assuming ideal coding/ decoding).
• the degradation associated with downsampling in the encoder and upsampling in the encoder can be avoided and the exact same processing can be implemented in both units.
• the correctly, coordinatedly upsampled YCbCr colors are then converted into non-linear R'G'B' colors by color converter 408.
• the YCbCr representation is converted into an RGB format which is non-linear.
• a maximum calculation unit determines the largest/ maximal one of those three color components as maximum m, of the original or input colors.
• TM() which is a good or reasonable manner to transform the object or pixel luminances of the HDR image representing the scene into the LDR image luminances. I.e. this is typically a function which defines for each possible input luminance [0.0, 1.0] an output luminance.
• luminance mapping or transformation function TM() which is a good or reasonable manner to transform the object or pixel luminances of the HDR image representing the scene into the LDR image luminances.
• this is typically a function which defines for each possible input luminance [0.0, 1.0] an output luminance.
• luminance transformations i.e. giving the same luminance e.g. 0.45 in an upper region being the sky a different output luminance then the one it would have if that 0.45 luminance occurred for a pixel in the lower, ground part of the image, but for this elucidation we will assume global luminance mappings solely.
• the color conversions of the dynamic range transformation may also be comprised in the total set of functions (F_ct) also for various reasons one or more chromatic color transformation such as a saturation change (e.g., one of the images may be in a different color space, such as the HDR image in Rec. 2020 and the LDR image in Rec. 709 RGB primaries, or one may additionally boost the saturations of colors which need to become dark in the LDR image compared to in the HDR image, to increase their vividness), but for simplicity of elucidation of the present new concept we will assume that the color transformation F_ct consists solely of a single luminance transformation function.
• a saturation change e.g., one of the images may be in a different color space, such as the HDR image in Rec. 2020 and the LDR image in Rec. 709 RGB primaries, or one may additionally boost the saturations of colors which need to become dark in the LDR image compared to in the HDR image, to increase their vividness
• this function may actually be defined in the luma domain similarly, we will without loss of generality assume that it is a function which specifies how each HDR relative luminance (i.e. in [0.0, 1.0]) should map to its corresponding relative LDR luminance, and to avoid further complications we assume float numbers, since the amount of bits for each representation can also be determined at a later stage, e.g. when formatting the Y'4L, Cb'OL, and Cr'OL for entering the HEVC encoder.
• TM luminance transformation function
• the encoder is connected to a direct video supply, e.g. in a real-life broadcast, a good luminance mapping may have been established previously in a coarse manner, which is sufficiently good for the entire program, e.g. even when the camera moves between brighter and darker parts of the scene, or turns towards a brighter sub-area etc.
• This simple function may also have been (partly) determined by a human, or even automatically, e.g. based on the statistics of the luminances in a number of captured typical images, etc.
• the details of how the luminance transformation function TM() has been established is not important for the elucidation of this invention or some of its embodiments, as it suffices there is one.
• This luminance transformation function TM() (between the domains [0.0, 1.0] and [0.0, 1.0]) is now by a luminance mapping unit 401 applied to the maximum m, which by definition also falls within [0.0, 1.0], yielding a corresponding output maximum color component, the output maximum m*.
• downscalers 412 and 413 process the determined gain values to perform the appropriate downscaling of the gain values.
• the gain values may be determined as a 2-dimensional set of calculated gain values at the resolution of the luma values, and this may be downscaled by the downscalers 412 and 413 to the resolution of the chrominance values.
• a full resolution "gain image" can be downscaled to the resolution of the chrominance values (a downscaling by a factor of four in the specific example).
• the resulting gains may be applied to multipliers 414, 415, 415 which multiply the luma and downsampled chrominance values from the spatial subscaler 405.
• appropriate gain values are determined for the respective multipliers 414, 415, and 416, which multiply the color components as they come in, ultimately realizing the required dynamic range transformation to the LDR image.
• the resulting LDR image may then be encoded with potentially some legacy
• DCT-based video compression strategy in an image representation which is already in a suitable format (i.e. YCbCr, and with sub-sampled chrominance components).
• YCbCr a suitable format
• Cb'OL, and C'rOL the luma, and blue respectively red chrominance
• a real LDR image in fact a legacy SDR image; defined in a Rec. 709 OETF-based format, or the good approximation thereof being a square-root-non-linear RGB-based format
• an LDR image which corresponds to the HDR image, in that it is not only a directly viewable LDR image (i.e.
• Fig. 6 Two examples of possible upscaling algorithms are shown in Fig. 6.
• the circles give pixel positions (in the highest resolution format, which we will call 4:4:4) where two chrominance values exist in addition to of course a luma value (i.e. whether in the decoder or the encoder preparing the dynamic range transform), i.e. at some pixel places there will not be chrominances available, but only lumas (the squares).
• Fig. 6 A elucidates an algorithm in which the required chrominances for pixel positions in which they are not available are determined by copying the values from an adjacent neighbor, which neighbor is indicated with the arrows.
• Fig. 5 shows the corresponding color transformation part for establishing the luminance dynamic range transformation in a decoder (the luminance dynamic range transformation now being from the received low dynamic range image, Im_COD
• Im_RLDR decompressed into Im_RLDR, to the high dynamic range image as it is needed, e.g. the 5000 nit master original, or a 3000 nit equivalent automatic re-grading thereof in case a 3000 nit display is connected, which can also be realized by using the information in the grading functions encoding how different dynamic range images should ideally look).
• the generated SDR signal from the circuit of FIG. 4 may be encoded in e.g. an HEVC encoder and transmitted to a receiver/ decoder.
• This may comprise an HEVC decoder which delivering pixels with pixel colors represented by luma values and spatially sub-sampled chrominance values, and specifically it provides Y'4L, C'bOL, C'rOL (LDR or in fact SDR) color values/ components.
• the chrominance values are spatially upscaled by upscalers 501 and 502, which as said employ exactly the same algorithm as the encoder has used for establishing the dynamic range downgrading.
• the upscalers may be programmed (e.g. under control of some CPU unit) to employ that algorithm which corresponds to the received INTSTR1 algorithm type indication for this video (e.g. type #3 being algorithm nr. 3; or the skilled person understands how one could communicate this in various manners).
• the exact upscaling algorithm employed by the upscalers 501 and 502 may be determined from data received with the images such that it can be made identical to the upscaling algorithm used in the encoder.
• the upscaled full resolution YCbCr colors are then as they come in pixel per pixel converted to non-linear R'G'B' colors by color converter 503, and then the maximum m is calculated by maximum calculation unit 505. Then the luminance mapping/function TM() is applied to the maximum by luminance mapping unit 505.
• this is of course the inverse function, or the inverse set of functions of the encoder's downgrading to SDR, and if the function is embodied in a LUT, then swapping of the axes is sufficient.
• the gain calculation unit 506 determines the appropriate gain value corresponding to this input color, in particular its luminance, given this luminance transformation strategy as defined by the luminance mapping TM.
• gain values are correctly downsampled as needed, by downsamplers 507 and 508, in some embodiments programmed to different algorithms depending on the received indication INTSTR1* of which algorithm should be used.
• multipliers 509, 510, and 511 calculate the final color transformation with the appropriate gain values (g for the lumas and gs for the chrominances, although in some embodiments these will have the same value), which effectively determines the HDR pixel colors for the received LDR image, being the inverse of the downgrading at the encoding side.
• the approach of the decoder may correspond closely to the approach of the encoder but with the luminance transformation TM() being different, and in many cases with these being the inverse of each other.
• the HDR pixel colors may in this embodiment by calculated in a YCbCr color representation
• there may again be any color transformer 512 for color conversion to any particular needed color representation e.g. for communicating the image to any connected display 520, e.g. over a HDMI connection, etc.
• the color transformations F_ct, and in particular the luminance transformation TM come in some manner from the content creation side, e.g. a CPU may read them from metadata associated with the encoded LDR images, e.g. co-stored on an optical disk or other memory, co- communicated over a network, etc.
• the described systems may specifically provide an approach for effectively generating and communicating signals that may effectively represent both HDR and SDR images.
• the approach may include a color preserving approach to adapting between HDR and SDR signals.
• the dynamic range transformations may be applied directly to pixel colors based on a non-linear representation including a luma value and (at least) two spatially subsampled chrominance values.
• the current approach does not rely on a representation of images by pixel colors being linear RGB values at full resolution.
• MPEG encoders typically operate on non-linear color-subsampled signals, such as e.g. Y C bC r 4:2:0. Conventionally, this requires conversion from this format to full resolution linear RGB values. However, such a conversion is complex and has a high computational burden. It also tends to be suboptimal and introduce image quality
• the sub-sampling of the chrominance components can result in the generated RGB values at the decoder side being different from those that were used at the encoder side, especially near edges in the video signal (where also the MPEG encoder tends to make somewhat larger compression errors, hence amplifying the difference). This difference can lead to visible differences (artefacts) in the reconstructed HDR signal.
• the conversion applied in the gain determining path can be based on the same values that are present in the encoder and can use the exact same algorithms as used in the decoder. Thus, improved quality can be achieved.
• the approach is based on the insight that some of the specific functions that are conventionally applied in the linear RGB domain can be ported to a non-linear subsampled luma/chrominance domain by integrating functions and changing the order in which various functions are performed. It also reflects the insight that the dynamic range conversion can in this way be applied on the sub- sampled chrominance signals, and that doing so can remove/ reduce the mismatch between the encoder and decoder sides (since the same sub-sampled chrominance data is available at both the encoder and decoder side (ignoring differences, i.e. compression errors, caused by the MPEG compression itself)).
• the encoder approach would require an HDR signal to be in a linear full resolution RGB format with the HDR to SDR conversion first being applied to this. This would generate an RGB SDR signal which would then be converted to an YCbCr SDR signal (with chrominance downscaling).
• HDR to SDR conversion is performed in the RGB domain and with full resolution and this is followed by a separate operation in which the resulting RGB SDR signal is converted into an YCbCr SDR signal for encoding.
• the complementary operation would traditionally be performed in the decoder.
• the received decoded image will be a chrominance downsampled YCbCr SDR image that is first converted to a full scale RGB SDR signal followed by a subsequent and independent SDR to HDR conversion.
• the received chrominance downsampled YCbCr SDR signal is first converted to a full scale RGB SDR signal. Only then is the SDR to HDR conversion performed, and thus the dynamic range conversion is entirely performed in the full resolution RGB domain.
• the current approach is fundamentally different. Indeed, in the current approach the SDR/HDR conversions are performed in the non-linear downscaled luma chrominance domain and not in the full scale and linear RGB domain. Indeed, gain values are determined and then applied directly to the luma/chrominance values.
• the received YCbCr values are directly fed to the multipliers 509, 510, 511 where they are multiplied by gain values to produce a dynamic range adjusted YCbCr output image.
• upscaling and conversion to an RGB format is in this approach performed as part of the SDR to HDR conversion but with this being performed only in the gain determination path and with no such functions being performed anywhere in the signal path to generate the HDR image (but of course the YCbCr to RGB is in the example subsequently performed in the HDR domain by block 512).
• the processes of color representation conversion, resolution change, and dynamic range conversion are closely integrated in a single process rather than being separate and independent sequential processes.
• the dynamic range conversion is performed in the luma and chrominance domain rather than in an RGB domain.
• the dynamic range conversion is performed in the downscaled domain, and indeed is typically performed at different resolutions for the luma values and the chrominance values.
• the algorithmic components disclosed in this text may (entirely or in part) be realized in practice as hardware (e.g. parts of an application specific IC) or as software running on a special digital signal processor, or a generic processor, etc.
• the computer program product denotation should be understood to encompass any physical realization of a collection of commands enabling a generic or special purpose processor, after a series of loading steps (which may include intermediate conversion steps, such as translation to an intermediate language, and a final processor language) to enter the commands into the processor, and to execute any of the characteristic functions of an invention.
• the computer program product may be realized as data on a carrier such as e.g. a disk or tape, data present in a memory, data travelling via a network connection -wired or wireless- , or program code on paper.
• characteristic data required for the program may also be embodied as a computer program product.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• Computer Hardware Design (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Image Processing (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Processing Of Color Television Signals (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=55299246

Family Applications (1)

Country Status (5)

Cited By (2)

Families Citing this family (22)

Citations (5)

Family Cites Families (6)

• 2017
  • 2017-01-27 WO PCT/EP2017/051852 patent/WO2017129793A1/en not_active Ceased
  • 2017-01-27 CN CN201780008780.6A patent/CN108605125B/zh active Active
  • 2017-01-27 JP JP2018538891A patent/JP6619888B2/ja active Active
  • 2017-01-27 EP EP17701544.3A patent/EP3409015B1/en active Active
  • 2017-01-27 US US16/071,929 patent/US10728560B2/en active Active

Patent Citations (5)

Non-Patent Citations (6)

Also Published As

Similar Documents

Legal Events