WO2019071045A1 - Correction de colorisation à plage dynamique élevée - Google Patents

Correction de colorisation à plage dynamique élevée Download PDF

Info

Publication number
WO2019071045A1
WO2019071045A1 PCT/US2018/054465 US2018054465W WO2019071045A1 WO 2019071045 A1 WO2019071045 A1 WO 2019071045A1 US 2018054465 W US2018054465 W US 2018054465W WO 2019071045 A1 WO2019071045 A1 WO 2019071045A1
Authority
WO
WIPO (PCT)
Prior art keywords
value
color
component value
values
color space
Prior art date
Application number
PCT/US2018/054465
Other languages
English (en)
Inventor
Andrey Norkin
Original Assignee
Netflix, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/725,266 external-priority patent/US10715772B2/en
Application filed by Netflix, Inc. filed Critical Netflix, Inc.
Publication of WO2019071045A1 publication Critical patent/WO2019071045A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/46—Colour picture communication systems
    • H04N1/64—Systems for the transmission or the storage of the colour picture signal; Details therefor, e.g. coding or decoding means therefor
    • H04N1/646—Transmitting or storing colour television type signals, e.g. PAL, Lab; Their conversion into additive or subtractive colour signals or vice versa therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/46—Colour picture communication systems
    • H04N1/64—Systems for the transmission or the storage of the colour picture signal; Details therefor, e.g. coding or decoding means therefor
    • H04N1/648—Transmitting or storing the primary (additive or subtractive) colour signals; Compression thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/117—Filters, e.g. for pre-processing or post-processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/46—Embedding additional information in the video signal during the compression process
    • H04N19/463—Embedding additional information in the video signal during the compression process by compressing encoding parameters before transmission
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • H04N19/86—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving reduction of coding artifacts, e.g. of blockiness
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/77—Circuits for processing the brightness signal and the chrominance signal relative to each other, e.g. adjusting the phase of the brightness signal relative to the colour signal, correcting differential gain or differential phase
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/46—Colour picture communication systems
    • H04N1/56—Processing of colour picture signals
    • H04N1/60—Colour correction or control
    • H04N1/6058—Reduction of colour to a range of reproducible colours, e.g. to ink- reproducible colour gamut

Definitions

  • Embodiments of the present invention relate generally to audiovisual processing and, more particularly, to high dynamic range (HDR) color conversion correction.
  • HDR high dynamic range
  • High Dynamic Range (HDR) imaging is a technique that enables a wider range of luminance values to be represented than is typically possible via conventional imaging techniques.
  • conventional imaging equipment is capable of capturing and reproducing only a limited luminance range, commonly resulting in a loss of detail in the luminance ranges associated with shadows and/or highlights.
  • HDR techniques are capable of capturing and representing a luminance range that is closer to the range perceived by the human visual system.
  • HDR techniques are expected to significantly improve the visual quality of many types of multimedia content.
  • luminance values associated with HDR content e.g., luminance values of approximately 0 to 10,000 cd/m 2
  • moderate bit depths e.g., 10 to 12 bits per sample
  • Bit depth generally refers to the number of bits used to represent one image component sample (e.g., a luma or chroma value).
  • a nonlinear transfer function is commonly applied to linear color values (e.g., RGB values) associated with HDR content in order to allocate a greater number of values to the lower end of the luminance range. Allocating more values to the lower end of the luminance range allows quantization to be achieved at moderate bit depths without any perceptible artifacts from the perspective of the viewer.
  • a nonlinear transfer function is applied to HDR content
  • the resulting nonlinear values are converted to a desired color space and further processed so that the HDR content can be transmitted and broadcast more efficiently.
  • Those additional processing operations typically include chroma subsampling and video compression, each of which are performed to reduce the bitrate of the HDR content.
  • chroma subsampling and video compression each of which are performed to reduce the bitrate of the HDR content.
  • a ST.2084 nonlinear transfer function is applied to linear RGB values.
  • the nonlinear R'G'B' values are then converted into the BT.2020 color space, and the resulting Y'CbCr 4:4:4 values are subsampled to generate Y'CbCr 4:2:0 values.
  • the Y'CbCr 4:2:0 values are then compressed via High
  • HEVC Efficiency Video Coding
  • the reconstructed color values may be significantly different than the color values present in the original HDR content.
  • These types of differences are most noticeable with color values that fall within regions of the nonlinear opto-electrical transfer function that have a steep slope (e.g., color components with low luminance values), since, in these regions, small variations in the value of a color component can have a significant impact on Y', Cb, and Cr values.
  • Another drawback of conventional HDR conversion processes is that, when the subsampled color values (e.g., Y'CbCr 4:2:0 color values) are later upsampled (e.g., to Y'CbCr 4:4:4 color values) for display, the type of upsampling filter that is
  • the color values may be different than the type of downsampling filter that was implemented by the encoder. Consequently, the color values may be
  • subsampled color values that are near the boundaries of a range of acceptable color values may be reconstructed to generate color values that are outside of the acceptable range (e.g., below 0 nits or above 10,000 nits). As a result, the reconstructed color values may not be properly displayed.
  • One embodiment of the present invention sets forth a method for correcting color values.
  • the method includes downsampling first color space values to generate downsampled color space values, upsampling the downsampled color space values to generate second color space values, and determining a first new value for at least one component value included in the downsampled color space values based on a first component value included in the first color space values, a second component value included in the second color space values, and an approximation of a nonlinear transfer function.
  • the method further includes determining that a first color
  • component value associated with the first new value is outside of a color space range, and determining a second new value for the at least one component value, where the first color component associated with the second new value is within the color space range.
  • At least one advantage of the disclosed techniques is that the resulting reconstructed HDR images are more accurate relative to the original HDR images than images generated via conventional chroma downsampling. Additionally, the complexity of the disclosed techniques is significantly lower than that of the
  • an optimal component value e.g., a luma value
  • the receiving device is able to account for the filter type and generate images that are more accurate relative to the original HDR images.
  • Figure 1 is a conceptual illustration of a computing device configured to implement one or more aspects of the present invention
  • Figure 2 illustrates a nonlinear opto-electrical transfer function (OETF) that can be implemented to convert linear light/color values to nonlinear light/color values, according to various embodiments of the present invention
  • Figure 3 illustrates a process for converting, compressing, and reconstructing high dynamic range (HDR) content, according to various embodiments of the present invention
  • Figure 4 illustrates a mapping of R' component values to linear light in the ST.2084 transfer function and non-constant luminance, according to various embodiments of the present invention
  • Figures 5A and 6A are original HDR images
  • Figures 5B and 6B are images processed according to conventional techniques
  • Figures 5C and 6C are images processed according to one or more embodiments of the present invention.
  • Figure 7 illustrates a flow diagram of method steps for performing HDR color conversion correction on an image, according to various embodiments of the present invention.
  • FIG. 1 is a conceptual illustration of a computing device 100 configured to implement one or more aspects of the present invention.
  • the color conversion system includes a computing device 100.
  • the computing device 100 includes a processor 102, input/output (I/O) devices 104, and a memory 110.
  • the memory 110 includes a color conversion application 112 configured to interact with a database 114.
  • the processor 102 may be any technically feasible form of processing device configured to process data and execute program code.
  • the processor 102 could be, for example, and without limitation, a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field-programmable gate array (FPGA), and so forth.
  • CPU central processing unit
  • GPU graphics processing unit
  • ASIC application-specific integrated circuit
  • DSP digital signal processor
  • FPGA field-programmable gate array
  • Memory 110 may include a memory module or a collection of memory modules.
  • the color conversion application 112 within memory 110 is executed by the processor 102 to implement the overall functionality of the color conversion system .
  • multimedia content e.g., images, video, etc.
  • multimedia content received by the color conversion system may be processed by the color conversion application 112 to apply linear and/or nonlinear transfer functions to the multimedia content, to convert the multimedia content between one or more color spaces, to perform downsampling, subsampling, upsampling, etc. on the multimedia content, and/or to apply one or more compression algorithms to the multimedia content.
  • Database 114 within memory 110 may store images, video, algorithms, parameters, lookup tables, and/or other types of data associated with transfer functions, color spaces, lossy and/or lossless codecs, etc.
  • I/O devices 104 may include input devices, output devices, and devices capable of both receiving input and providing output.
  • I/O devices 104 could include wired and/or wireless communication devices that send data to and/or receive data from a camera, a display screen, a media player, a storage device, speakers, a microphone, a networking device, and/or another computing device.
  • computing device 100 is configured to coordinate the overall operation of the color conversion system.
  • the computing device 100 may be coupled to, but separate from other components of the color conversion system.
  • the embodiments disclosed herein contemplate any technically feasible system configured to implement the functionality of the color conversion system.
  • FIG. 2 illustrates a nonlinear opto-electrical transfer function (OETF) that can be implemented to convert linear light/color values to nonlinear light/color values, according to various embodiments of the present invention.
  • OETF nonlinear opto-electrical transfer function
  • the slope of the transfer function is steep in the low-luminance range and shallow in the high luminance range, since the human visual system is more sensitive to differences in luminance values when the luminance values are low.
  • the nonlinear transfer function allocates more codewords and uses smaller quantization steps in the low luminance range, reducing the likelihood that quantization will produce visible artifacts in an image.
  • any other type of nonlinear transfer function having any shape may be implemented to convert and reconstruct linear color values.
  • FIG. 3 illustrates a process for converting, compressing, and reconstructing HDR content, according to various embodiments of the present invention.
  • a nonlinear transfer function is applied to the linear RGB values to generate nonlinear R'G'B' values.
  • the nonlinear R'G'B' values are then converted to a desired color space, and the resulting color values are processed so that the HDR content can be transmitted and broadcast more efficiently.
  • nonlinear R'G'B' values are generated by applying the OETF perceptual quantizer (PQ) to each linear light R, G, and B component separately. Then, the nonlinear R'G'B' values are converted to the non- constant luminance BT.2020 color space according to Equation 1 , shown below. Next, further processing may be performed on the Y'CbCr 4:4:4 color values, for example, to reduce the bitrate for ease of transmission.
  • PQ OETF perceptual quantizer
  • the color values are further processed via subsampling, such as chroma subsampling.
  • subsampling color values luma and/or chroma information is averaged over adjacent pixels in order to reduce the bitrate of the processed images.
  • the chroma components of the image are subsampled while retaining the original resolution of the luma components, since the human visual system is able to detect differences in luminance more readily than differences in color.
  • color values specified in a Y'CbCr 4:4:4 format may be subsampled to generate Y'CbCr 4:2:2 values or Y'CbCr 4:2:0 values.
  • the resulting color values are then optionally compressed via a codec, such as HEVC, H.264, etc.
  • one drawback of these conventional HDR conversion processes is that reconstructing the subsampled color values can produce artifacts.
  • an inverse of the color space transform and an inverse of the nonlinear transform function e.g., an EOTF PQ
  • an inverse of the nonlinear transform function e.g., an EOTF PQ
  • the color conversion application 112 adjusts one or more of the downsampled component values such that the values, when upsampled and reconstructed, produce color values that are similar to the color values present in the original HDR content. More specifically, the color conversion application 112 downsamples chroma values. In some embodiments, the color conversion
  • the color conversion application 112 can obtain downsampled chroma directly from Y'CbCr 4:4:4 color values, as shown in Figure 3.
  • the color conversion application 112 may downsample the linear RGB color values directly and then apply the OETF PQ and color transform to the down-sampled values.
  • the color conversion application 112 After obtaining the downsampled chroma, the color conversion application 112 upsamples the chroma by applying a selected upsampling filter. The color conversion application 112 then modifies one or more of the Y', Cb, and Cr values such that reconstructing these values to a linear RGB signal produces color values similar to the color values present in the original linear RGB signal. For reference, an example of an EOTF PQ that may be used to reconstruct linear RGB values from nonlinear R'G'B' values is shown below in Equation 2.
  • the color conversion application 112 modifies the Y', Cb, and Cr values by estimating each of the Y', Cb, and Cr values simultaneously.
  • the value of Y may be obtained while keeping the values of Cb and Cr constant. This latter approach may be beneficial because the filter used to upsample the chroma components is not defined by a standard and, thus, may vary across different hardware and software platforms.
  • modifying a chroma sample may also change the upsampled chroma values at neighboring locations.
  • the color conversion application 112 determines a value of the luma component Y'(x, y) by finding new Y, Cb, and Cr component values which correspond to RGB new (x,y) values that are substantially similar to the original RGB org (x, y) values.
  • x and y are horizontal and vertical positions of the sample, respectively.
  • the distance (D) between RGB new (x,y) and RGB org (x, y) could be measured as the Euclidean norm (e.g., the squared error for each
  • D (Rnew(X,y) - Rorg(X,y)) 2 + (G new (X,y) - G ORG (X,y)) 2 + (Snew(X,y) - Sorg(X,y)) 2 (3)
  • D WR (Rnew- ⁇ org) 2 + WG ( Gnew " G org ) 2 + WB (Snew - ⁇ org) 2 (5)
  • D WR (f (R'new) - f (R'org)) 2 + W G (f (G'new) " f (G'org)) 2 + W B (f (B'new) " f (B'org)) 2 (6)
  • the values R', G' and B' can be obtained from the original Y'CbCr values by applying an inverse color transform, which depends on the color space associated with the Y'CbCr values.
  • an inverse color transform for Y'CbCr values within the BT.709 and BT.2020 color spaces has the following form:
  • the EOTF may be approximated using a truncated Taylor series (e.g., the first, second, or third degree polynomials), enabling the cost function D to be minimized via a closed form solution.
  • a truncated Taylor series e.g., the first, second, or third degree polynomials
  • Equations 8-17 Two examples of closed form solutions for minimizing the cost function D shown in Equations 3-6 are described below in conjunctions with Equations 8-17.
  • the techniques described below implement RGB values, color spaces other than RGB can also be used.
  • values of the EOTF derivative squared f (X) 2 can be pre-computed and stored in a look-up table.
  • a look-up table with 1024 entries could be used when the fixed point processing of 10 bits is used. In case of a higher fixed point or floating point precision, more table entries could be used to improve accuracy.
  • R, G, and B color components are required to be within a certain interval range.
  • R, G, and B color components are typically required to be within the (0, 1 ) interval.
  • an appropriate integer range may be implemented such that the signal is quantized. For example, if 10-bit color components are implemented, then R, G, and B color components would be required to be integers within the 0 to 1023 range.
  • FIG. 4 illustrates a mapping of R' component values to linear light in the ST.2084 transfer function and non-constant luminance, according to various embodiments of the present invention. As shown, R' component values are clipped below zero and at levels above 10,000 nits. However, Equations 1 -13 described above assume that R', G', and B' values continue above 10,000 nits and below 0 nits.
  • the linear approximation indicated by the dashed line in Figure 4 assumes that R', G', and B' values continue above 10,000 nits. Because such approximations may fall outside of the acceptable range of color component values, a mismatch may exist between the calculated value of a color component and the real value of the color component, after clipping is applied. For example, if the calculated value(s) of one or more color components are outside of an acceptable range (e.g., 0 to 1 or 0 to 1023), then the color component value(s) may be clipped by a display device or decoder, creating visual artifacts in the resulting image.
  • an acceptable range e.g., 0 to 1 or 0 to 1023
  • Equations 6 and 9 do not depend on the value of Y', since the value of this component is equal to 1 or 0, which is the smallest or the highest value of the range. Therefore, the derivatives of these terms with respect to Y' are equal to zero, and the terms that correspond to those color components can be removed from the final equation for computing Y' new -
  • the R' new , G' new , or B' new values could be compared only to the upper end of the color component range (e.g., 1 ), since color component values below 0 tend to have a negligible effect on the resulting Y' value.
  • the upper end of the color component range e.g. 1
  • the value of Y' new could be calculated according to Equation 16 to take into account the color components that are being clipped at the upper end of the range of color component values.
  • the following computer code could be implemented to resolve clipping with respect to both a lower end (e.g., 0) and an upper end (e.g., 1 ) of a color component range.
  • the following computer code could be implemented to resolve clipping with respect to only an upper end (e.g., 1 ) of a color component range.
  • Equation 16 the EOTF could be approximated using a second degree polynomial, as shown in Equation 16:
  • X stands for R, G, and B
  • f ' (X) stands for the first derivative f ' (R'org), f ' (G'org), or f ' (S' org )
  • f " (X) stands for the second derivative f " (R' org ), f " (G'org), or f " (S'org)-
  • the derivatives are taken with respect to the corresponding nonlinear color component.
  • the cubic equation has either one or three real roots. In the case of three real roots, the minimum is achieved in either the root having the largest value or the smallest value, due to the fact that the cost function D is quadratic with a positive coefficient at the fourth degree term. The values of the cost function D are then calculated for both roots, and the root resulting in a smaller value is chosen as Y'new-
  • Equation 3 has only one real root, then the real part of the two complex roots may still be considered as a possible solution. Specifically, because an
  • Equation 19 having only one real root instead of three real roots.
  • the real part of the pair of complex roots represents a better approximation of the solution than the value of the remaining real root.
  • the distance (D) could be measured as a sum of weighted differences between individual R, G, and B components of RGB new (x,y) and RGB 0 rg(x, y).
  • the difference between the two values could be calculated according to Equation 20, where w R , w G , and w B are the weights corresponding to each color component and R, G, and B.
  • Equation 21 the cost function is defined by Equation 21 or Equation 22, where f(X) is the EOTF function:
  • the transfer function domain values R' , G', and B' can then be obtained from Y'CbCr by applying an inverse color transform, such as the BT.709 and BT.2020 transform:
  • the absolute value of the cost function D (i.e.,
  • the absolute value of the cost function shown in Equations 20-22
  • minimizing the absolute value of D is a non-trivial task, since the absolute value function is not differentiable at 0.
  • the function D 2 is instead minimized, since ⁇ D ⁇ and D 2 reach the minimum at the same point, and the function D 2 is differentiable on the entire range of real numbers R. Therefore, the optimization is instead performed for function D 1 .
  • the EOTF is approximated with a truncated Taylor series in order to find a closed form solution to minimizing the cost function D 1 .
  • a R in Equation 25 is substituted with (a ⁇ Y' new + eR) and similar substitutions are made for A G and ⁇ ⁇ .
  • the cost function D is then differentiated with respect to Y' to find a closed form solution for the local minimum.
  • eR, e G , and e B we first calculate eR, e G , and e B according to
  • Equation 10 The value of Y is equal to:
  • values of the EOTF derivative f ' (X) can be pre- computed and stored in a look-up table (e.g., a table having 1024 entries for video with a bit depth of 10 when the fixed point processing of 10 bits is used). In case of a higher fixed point or floating point precision, more table entries could be used to improve accuracy.
  • a look-up table e.g., a table having 1024 entries for video with a bit depth of 10 when the fixed point processing of 10 bits is used. In case of a higher fixed point or floating point precision, more table entries could be used to improve accuracy.
  • Equations 28 or 29 may be implemented to adjust the resulting Y' value.
  • R' new , G' new , and B'new are calculated according to Equation 23.
  • R' new , G' new , and B' new are then compared to the maximum and/or minimum of a color component range (e.g., 0 and 1 ), as shown in Equation 28. If R' new , G' new , or B' new are outside of the color
  • Equation 28 the value of Y' new is calculated according to Equation 28 to take into account the color components having clipped values.
  • the corresponding terms in Equations 22 and 25 do not depend on the value of Y', since the value of this component is equal to 1 or 0, which is the smallest or the highest value of the range. Therefore, the derivatives of these terms with respect to Y' are equal to zero, and the terms that correspond to those color components can be removed from the final equation for computing Y' new -
  • the R' new , G' new , or B' new values could be compared only to the upper end of the color component range (e.g., 1 ), since color components values below 0 tend to have a negligible effect on the resulting Y' value.
  • the upper end of the color component range e.g. 1
  • the value of Y' new could be calculated according to Equation 29 to take into account the color components that are being clipped at the upper end of the range of color component values.
  • the EOTF could be approximated using a second degree polynomial, as shown in Equation 16. Then, the solution, assuming
  • the cubic equation has either one or three real roots. If three real roots are determined, then the minimum is achieved in either the root having the largest value or the smallest value. The values of cost function D are then calculated for both real roots, and the real root resulting in a smaller value is chosen as Y'new- In case Equation 32 has only one real root, the real part of the remaining complex roots may be considered as a solution.
  • any of the techniques described above can be implemented with other types of transfer functions, including ST.2084 or BT.1886. Additionally, the techniques described above can be applied to other color spaces, such as BT.709 and BT.2020. Further, in some embodiments, the derivative of an EOTF can be obtained either by differentiating the EOTF or by numerically
  • a derivative e.g., dividing a change in the value of the EOTF by a change in the EOTF argument.
  • An average or a weighted average of two or more of EOTF could also be implemented in the closed form solutions - instead of a single EOTF - in order to optimize the approach for compatibility with several transfer functions.
  • the derivatives of the EOTFs could be replaced with an average or a weighted average (e.g., a weighted sum) of the derivatives of the EOTFs.
  • the techniques described herein could also use higher order polynomials to approximate an EOTF.
  • the cost function D could be minimized with respect to several values (e.g., Y', Cb, and Cr), such as several values that correspond to neighboring pixels.
  • partial derivatives could be taken with respect to each Y', Cb, and Cr component to find the optimal values of Y', Cb, and Cr.
  • the weights w R , w G , and w B could be chosen based on the desired precision or importance of each color component. For example, the weights could be set equal to 1 .
  • the weights w R , w G , and w B are chosen on a picture basis or a sequence basis.
  • the weights w R , w G and w B could also be set adaptively on a sample/pixel basis, for example, based on the original R, G, and B values for each sample (e.g., based on an inverse relationship with the intensity of R, G, and B for each sample) or based on some other algorithm.
  • the weights could be set based on the contribution of each color component to the luminance.
  • the weights could be set equal to the contributions of the R, G, and B, components, respectively, to the Y component of the XYZ color space. In the case of BT.709, weights of 0.212639, 0.715169, and
  • 0.072192 could be assigned to w R , w G , and w B , respectively.
  • weights of 0.262700, 0.677998, and 0.059302 could be assigned to w R , w G , and w B , respectively.
  • the algorithms described herein are able to efficiently improve the quality of HDR video when subsampling (e.g., chroma subsampling) is applied.
  • subsampling e.g., chroma subsampling
  • a decoder or other type of display equipment applies a different type of upsampling filter than the type of upsampling filter used to pre-process the video (e.g., as performed with respect to Equations 8-32)
  • visual artifacts can occur. It has been observed that a mismatch between the upsampling filter that was applied during pre-processing and the upsampling filter applied by a decoder or display may result in visual artifacts, often in the same areas which exhibit visual artifacts due to chroma subsampling.
  • small differences in color component values can occur when the upsampling filter applied during pre-processing implements different filter coefficients than the upsampling filter that is later used to reconstruct the color component values.
  • relatively small differences in R', G', and B' values of a reconstructed signal can result in significant differences in linear luminance values R, G, and B once the EOTF is applied. These differences are most noticeable when sample values are located in the range of approximately (0.5, 1 ), due to the highly non-linear shape of the EOTF PQ in this region, as shown in Figure 4.
  • an indication of the type of filter that was applied to a particular video bitstream during pre-processing may be signaled to a receiving device, such as a video decoder or a display device (e.g., a television, a laptop computer, a mobile phone, etc.).
  • the receiving device may then apply the same or substantially the same type of filter when upsampling the color component(s) for display, enabling the receiving device to produce a video bitstream having color component values (e.g., chroma and/or luma) that are sufficiently close to the original video bitstream.
  • the receiving device could use the same type of upsampling filter that is signaled to reconstruct the color components.
  • the receiving device could use the signaled upsampling filter to check that a set of upsampled color component values are not significantly different than the color components values that were upsampled using the type of upsampling filter that was signaled to the receiving device.
  • an indication of the type of filter that was used to upsample color component values at the pre-processing stage is signaled via an integer value, where the integer value is mapped to a particular type of upsampling filter or class of upsampling filters (e.g., a family of Lanczos filters).
  • entropy coding and/or fixed-length coding could be used to signal the upsampling filter type to the receiving device.
  • An indication of the type of upsampling filter could be signaled to the receiving device via an encoded video bitstream. For example, the type of upsampling filter could be signaled in a
  • supplemental enhancement information (SEI) message in a file format, in a file container, or by any other means.
  • SEI Supplemental Enhancement Information
  • an indication of coefficients of a digital filter that were used during pre-processing to upsample color component values could be signaled to the receiving device via any of the techniques described herein.
  • a chroma upsampling filter type is signaled via a variable-length coding (VLC).
  • VLC variable-length coding
  • upsampling filter types could be signaled via shorter codes (e.g., 1 -bit and/or 2-bit codes), and less frequently used filter types could be signaled via longer codes.
  • a chromaUpsamplingFilter element value of 0 could specify bilinear upsampling filter, and a value of 1 could specify a Lanczos 3 upsampling filter.
  • chromaUpsamplingFilterCoefficientsFlag is set equal to 1 to indicate that the chroma upsampling filter coefficients are being signaled in the video bitstream. Additionally, chromaUpsamplingFilterCoefficientsFlag element is set equal to 0 to indicate that chroma upsampling filter coefficients are not signaled in the video bitstream.
  • the numberOfCoeff element may specify the number of coefficients in the chroma upsampling filter, and the chromaUpsamplingFilterCoefficient[i] element may indicate the value of the i-th filter coefficient.
  • filter coefficients of a polyphase chroma upsampling filter are signaled to a receiving device.
  • chromaUpsamplingFilterCoefficientsFlag element may be set equal to 1 to indicate that chroma upsampling filter coefficients of a polyphase filter are being signaled in the video bitstream.
  • the chromaUpsamplingFilterCoefficientsFlag element may be set equal to 0 to indicate that chroma upsampling filter coefficients of a polyphase filter are not being signaled in the video bitstream.
  • the numberOfCoeff Phase05 element specifies the number of coefficients in the chroma upsampling filter with phase 0.5.
  • the chromaUpsamplingFilterCoefficientPhase05[i] element indicates the value of the i-th filter coefficient of phase 0.5.
  • the numberOfCoeff Phase025 element specifies the number of coefficients in the chroma upsampling filter with phase 0.25.
  • the chromaUpsamplingFilterCoefficientPhase025[i] element indicates the value of the i-th filter coefficient of phase 0.25.
  • Figure 7 illustrates a flow diagram of method steps for performing HDR color conversion correction on an image, according to various embodiments of the present invention. Although the method steps are described in conjunction with the systems of Figures 1 -3, persons skilled in the art will understand that any system configured to perform the method steps, in any order, falls within the scope of the present invention.
  • a method 700 begins at step 710, where the color conversion application 112 converts one or more linear RGB values to R'G'BO rg (x,y) values based on a nonlinear transfer function (e.g., an OETF PQ).
  • the color conversion application 112 converts the R'G'B' org (x,y) values to Y'CbCr org 4:4:4 values based on transform parameters associated with a target colored space.
  • the color conversion application 112 then subsamples the Y'CbCr org 4:4:4 values to generate Y'CbCr new 4:2:2 values or Y'CbCr new 4:2:0 values.
  • the color conversion application 112 upsamples the Y'CbCr 4:2:2 values or the Y'CbCr org 4:2:0 values to generate Y'CbCr new 4:4:4 values. Then, at step 750, the color conversion application 112 calculates a luma value Y' n ew(x,y) based on a closed form equation derived from an approximation of an EOTF. In various embodiments, the color conversion application 112 could calculate the luma value Y'new(x,y) based on any of Equations 11-13, 19, 26, 27, and/or 32.
  • the color conversion application 112 could calculate Y' n ew(x,y) based on the R'G'BOrg(x,y) values, the Y'CbCr org (x,y) values, the Y'CbCr new (x,y) values, one or more color space transform parameters, and/or an approximation of a nonlinear transfer function, in accordance with one or more of Equations 11-13, 19, 26, 27, and/or 32.
  • the color conversion application 112 computes R' new , G' new , and B'new based on a color space transform (e.g., via Equation 7).
  • the color conversion application 112 determines whether R' new , G' new , and/or B' new are outside of a range (e.g., 0 to 1 ) defined by the color space. If, at step 770, the color conversion application 112 determines that R' new , G' new , or B' new are outside of the range defined by the color space, then the method 700 proceeds to step 772, where the color conversion application 112 clips the color component value(s) to the color space range.
  • the color conversion application 112 then computes an updated Y' n ew(x,y) value and assigns the updated Y' n ew(x,y) value to the pixel (x,y).
  • the color conversion application 112 applies Equations 14, 15, 28, or 29 to compute an updated Y' new (x,y) value.
  • Equations 14, 15, 28, or 29 For example, as described above in conjunction with Equation 14, when the value of a color component is clipped, the corresponding terms in Equations 6 and 9 do not depend on the value of Y', since the value of this component is equal to 1 or 0, which is the smallest or the highest value of the range. Therefore, the derivatives of these terms with respect to Y' are equal to zero.
  • the color conversion application 112 can remove the terms that correspond to those color components from the final equation for computing Y' new - [0097] Additionally, in some embodiments, at steps 772 and 774, the color conversion application 112 updates Y' n ew(x,y) by selecting, from a lookup table, an entry that stores a Y' value that corresponds to the clipped color component value(s). For example, the color conversion application 112 could select an entry that is associated with the undipped value(s) of the one or more color component values determined at step 770.
  • the value stored in the entry of the lookup table may be a precomputed value that compensates for the clipping that will be applied by a receiving device (e.g., a television, laptop computer, mobile phone, etc.) to the one or more color component values determined at step 770.
  • a receiving device e.g., a television, laptop computer, mobile phone, etc.
  • step 770 the color conversion application 112 determines that R' new , G'new, or B' new are not outside of the range defined by the color space. If, at step 770, the color conversion application 112 determines that R' new , G'new, or B' new are not outside of the range defined by the color space, then the method 700 proceeds to step 780, where the color conversion application 112 assigns Y' n ew(x,y) to the corresponding pixel (x,y). At step 790, the color conversion application 112 determines whether a Y' new should be determined for one or more additional pixels.
  • step 750 the color conversion application 112 calculates one or more additional luma values Y' new , for example, based on the R'G'BO rg values, the Y'CbCr org values, the Y'CbCr new values, one or more color space transform parameters, and/or an approximation of a nonlinear transfer function.
  • the method 700 then terminates.
  • the color conversion application 112 may use any type of color values, color space parameters, transfer functions, and approximations to modify Y', Cb, and/or Cr values.
  • the complexity of the linear approximation approach is significantly lower than that of the conventional luma micro-grading approach.
  • the techniques described herein use a closed form solution to find the value of Y' in a single iteration.
  • the conventional micro-grading approach requires 10 iterations for a 10-bit video, including the last two boxes in the chain in Figure 2 - obtaining the R'G'B' values, applying the EOTF, and calculating the linear light luminance. Therefore, the proposed linear approximation approach has a good complexity versus quality trade-off that makes this approach well suited for real-time systems.
  • Figures 5A-5C and 6A-6C demonstrate the effect of the linear approximation algorithm, with weights w R , w G , and w B equal to the coefficients of the contribution of the R, G, and B components, respectively, to the Y component of the XYZ color space.
  • the weights could be 0.212639, 0.715169, and
  • the weights could be 0.262700, 0.677998, and 0.059302 for w R , w G , and w B , respectively.
  • the average PSNR is increased by more than 1 .75dB compared to conventional chroma downsampling. Additionally, a 3.65dB improvement is seen in tPSNR when implementing a linear approximation of EOTF.
  • the various embodiments disclosed herein yield a 0.07dB lower average PSNR when implementing a linear approximation of the EOTF and a 0.01 dB higher average PSNR for the second degree polynomial approximation of the EOTF.
  • the tPSNR metric is 0.17dB lower than the conventional luma micro-grading approach, on average.
  • the second degree approximation method results in a slightly better PSNR and tPSNR measures than the linear approximation.
  • each of the techniques disclosed herein significantly improve the subjective quality of the tested videos, removing perceptual artifacts. Another observation is that the techniques disclosed herein produce smoother luma than the conventional chroma downsampling, likely resulting in gains in subsequent
  • a color conversion application downsamples chroma values and then upsamples the chroma values by applying an upsampling filter.
  • the color conversion application modifies one or more of the downsampled Y'CbCr values such that the reconstruction of these values to a linear RGB signal produces values similar to those of the original linear RGB signal.
  • the color conversion application calculates an optimal value of Y', while keeping the values of Cb and Cr constant, by evaluating a closed form solution that is based on an approximation of a nonlinear transfer function.
  • the calculated value of Y' may further be converted into R', G', B' values in order to determine whether any of the R', G', B' are outside of a defined color space range. If one or more of the R', G', B' values is outside of the defined color space range, then the value(s) may be clipped, and Y' may be recomputed based on the clipped value(s).
  • At least one advantage of the disclosed techniques is that the resulting reconstructed HDR images are more accurate relative to the original HDR images than images generated via conventional chroma downsampling. Additionally, the complexity of the disclosed techniques is significantly lower than that of the conventional approaches, enabling an optimal component value (e.g., a luma value) to be determined in real-time via fewer iterations (e.g. , one or two iterations) than in conventional approaches. Further, because the type of filter applied during preprocessing of the HDR images can be signaled to a receiving device, the receiving device is able to account for the filter type and generate images that are more accurate relative to the original HDR images.
  • an optimal component value e.g., a luma value
  • aspects of the present embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "module” or "system.” Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
  • the computer readable medium may be a computer readable signal medium or a computer readable storage medium.
  • a computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.
  • a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
  • processors may be, without limitation, general purpose processors, special-purpose processors, application-specific processors, or field-programmable processors or gate arrays.
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s).
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Processing Of Color Television Signals (AREA)
  • Image Processing (AREA)

Abstract

La présente invention concerne, selon un mode de réalisation, une technique permettant de corriger des valeurs de couleur. La technique consiste à sous-échantillonner des premières valeurs d'espace de couleur pour générer des valeurs d'espace de couleur sous-échantillonnées, à suréchantillonner les valeurs d'espace de couleur sous-échantillonnées pour générer des secondes valeurs d'espace de couleur, et à déterminer une première nouvelle valeur pour au moins une valeur de composante incluse dans les valeurs d'espace de couleur sous-échantillonnées sur la base d'une première valeur de composante comprise dans les premières valeurs d'espace de couleur, une seconde valeur de composante incluse dans les secondes valeurs d'espace de couleur, et une approximation d'une fonction de transfert non linéaire. La technique consiste en outre à déterminer qu'une première valeur de composante de couleur associée à la première nouvelle valeur est située en dehors d'une plage d'espace de couleur, et à déterminer une seconde nouvelle valeur pour la valeur ou les valeurs de composante, la première composante de couleur associée à la seconde nouvelle valeur étant située dans la plage de l'espace de couleur.
PCT/US2018/054465 2017-10-04 2018-10-04 Correction de colorisation à plage dynamique élevée WO2019071045A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US15/725,266 US10715772B2 (en) 2015-11-09 2017-10-04 High dynamic range color conversion correction
US15/725,266 2017-10-04

Publications (1)

Publication Number Publication Date
WO2019071045A1 true WO2019071045A1 (fr) 2019-04-11

Family

ID=63998781

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2018/054465 WO2019071045A1 (fr) 2017-10-04 2018-10-04 Correction de colorisation à plage dynamique élevée

Country Status (1)

Country Link
WO (1) WO2019071045A1 (fr)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170134703A1 (en) * 2015-11-09 2017-05-11 Netflix, Inc. High dynamic range color conversion correction
US20170208310A1 (en) * 2015-02-13 2017-07-20 Telefonaktiebolaget Lm Ericsson (Publ) Pixel pre-processing and encoding

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170208310A1 (en) * 2015-02-13 2017-07-20 Telefonaktiebolaget Lm Ericsson (Publ) Pixel pre-processing and encoding
US20170134703A1 (en) * 2015-11-09 2017-05-11 Netflix, Inc. High dynamic range color conversion correction

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
ANDREY NORKIN (NETFLIX): "Closed form HDR 4:2:0 chroma subsampling (HDR CE1 and AHG5 related)", 23. JCT-VC MEETING; 19-2-2016 - 26-2-2016; SAN DIEGO; (JOINT COLLABORATIVE TEAM ON VIDEO CODING OF ISO/IEC JTC1/SC29/WG11 AND ITU-T SG.16 ); URL: HTTP://WFTP3.ITU.INT/AV-ARCH/JCTVC-SITE/,, no. JCTVC-W0107-v2, 18 February 2016 (2016-02-18), XP030117891 *
NORKIN ANDREY: "Fast Algorithm for HDR Color Conversion", 2016 DATA COMPRESSION CONFERENCE (DCC), IEEE, 30 March 2016 (2016-03-30), pages 486 - 495, XP033027730, DOI: 10.1109/DCC.2016.80 *
STRÖM (ERICSSON) J: "AhG-13 related: Multi-LUT Luma Adjustment Implementation", 25. JCT-VC MEETING; 14-10-2016 - 21-10-2016; CHENGDU; (JOINT COLLABORATIVE TEAM ON VIDEO CODING OF ISO/IEC JTC1/SC29/WG11 AND ITU-T SG.16 ); URL: HTTP://WFTP3.ITU.INT/AV-ARCH/JCTVC-SITE/,, no. JCTVC-Y0030, 6 October 2016 (2016-10-06), XP030118071 *
STRÖM J ET AL: "Modified Linearization of Luma Adjustment", 24. JCT-VC MEETING; 26-5-2016 - 1-6-2016; GENEVA; (JOINT COLLABORATIVE TEAM ON VIDEO CODING OF ISO/IEC JTC1/SC29/WG11 AND ITU-T SG.16 ); URL: HTTP://WFTP3.ITU.INT/AV-ARCH/JCTVC-SITE/,, no. JCTVC-X0036, 16 May 2016 (2016-05-16), XP030117963 *

Similar Documents

Publication Publication Date Title
US11265559B2 (en) High dynamic range image/video coding
US10750146B2 (en) High dynamic range color conversion correction
JP6312775B2 (ja) エンハンストダイナミックレンジ信号の階層符号化のための適応的再構成
JP2020171022A (ja) 高ダイナミックレンジおよび広色域シーケンスを符号化するシステム
US10432955B2 (en) High dynamic range adaptation operations at a video decoder
JP6609056B2 (ja) 高ダイナミックレンジおよび広色域シーケンスの再形成および符号化のためのシステム
KR101939012B1 (ko) 하이 다이내믹 레인지 이미지들을 위한 콘텐츠 적응적 지각 양자화기
CN107771392B (zh) 用于高动态范围图像的实时内容自适应感知量化器
US10742986B2 (en) High dynamic range color conversion correction
US10715772B2 (en) High dynamic range color conversion correction
WO2019071045A1 (fr) Correction de colorisation à plage dynamique élevée
EP3308541A1 (fr) Système de codage de séquences de données à plage dynamique élevée et gamme étendue de couleurs
WO2019071046A1 (fr) Plage dynamique élevée de correction de conversion de couleur

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18793336

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18793336

Country of ref document: EP

Kind code of ref document: A1