TWI684166B - Signal reshaping for high dynamic range signals - Google Patents

Abstract

In a method to improve backwards compatibility when decoding high-dynamic range images coded in a wide color gamut (WCG) space which may not be compatible with legacy color spaces, hue and/or saturation values of images in an image database are computed for both a legacy color space (say, YCbCr-gamma) and a preferred WCG color space (say, IPT-PQ). Based on a cost function, a reshaped color space is computed so that the distance between the hue values in the legacy color space and rotated hue values in the preferred color space is minimized. HDR images are coded in the reshaped color space. Legacy devices can still decode standard dynamic range images assuming they are coded in the legacy color space, while updated devices can use color reshaping information to decode HDR images in the preferred color space at full dynamic range.

Description

Signal reshaping for high dynamic range signals Cross reference case

This case is from U.S. Provisional Case No. 62/302,073 (application on March 1, 2016), No. 62/300,012 (application on February 25, 2016), and No. 62/278,362 (application on January 13, 2016 Japan), No. 62/202,980 (application on August 10, 2015) and No. 62/200,797 (application on August 4, 2015) request priority, each of which is referred to by its entirety Incorporated here.

The invention generally relates to images. More specifically, the embodiments of the present invention relate to signal reshaping of images with high dynamic range to improve backwards compatibility.

As used herein, the term "dynamic range" (DR; dynamic range) may be used to perceive on the image, for example, from the darkest dark color (black) to the brightest white (ie, high brightness) Intensity (for example, luminance, luma )) scope of the human visual system (HVS; human visual system) capabilities. In this sense, DR is about the intensity of "scene-referred". The DR can also be used to adequately or approximately represent a specific breadth of intensity range with respect to the display device. In this sense, DR refers to the intensity of "display-referred". Unless a specific meaning is clearly regulated to have a particular significance at any point in the description of the invention herein, it should be presumed that the term can be used in any meaning (eg, interchangeably).

As used herein, the term high dynamic range (HDR; high dynamic range) refers to the breadth of DR, which spans about 14 to 15 orders of magnitude of the human visual system (HVS) magnitude. In practice, humans can simultaneously detect a wide and wide DR in the intensity range, which can be shortened relative to HDR. As used herein, the terms enhanced dynamic range (EDR; enhanced dynamic range) or visual dynamic range (VDR; visual dynamic range) can be used individually or interchangeably to refer to the human visual system (HVS; human) within a scene or image Visual system) is a perceptible DR that includes eye movements, allowing some light adaptation changes across real scenes or images. As used herein, EDR can be about 5 to 6 orders of DR for Zhang Tuo. Therefore, while the HDR mentioned about the real scene is somewhat narrow, EDR stands for wide DR breadth and can also be called HDR.

8的影像(例如，彩色24位元JPEG影像)被認為是標準動態範圍之影像，同時在n>8的影像可被認為是增強的動態範圍之影像。EDR及HDR影像亦可使用高精度(例如，16位元)浮點格式來儲存及發行，像是由光影魔幻公司(Industrial Light and Magic)所發展的OpenEXR檔案格式。 In practice, the image includes one or more color components (for example, luminance Y and chrominance Cb and Cr), where each color component is represented by an accuracy of n bits per pixel (for example, n = 8). Use linear luminance translation, in n

An image of 8 (for example, a color 24-bit JPEG image) is considered as a standard dynamic range image, and an image with n > 8 can be considered as an enhanced dynamic range image. EDR and HDR images can also be stored and distributed using high-precision (for example, 16-bit) floating-point formats, such as the OpenEXR file format developed by Industrial Light and Magic.

Given a video stream, information about its coding parameters is typically embedded in the bitstream as metadata. As used herein, the term "metadata" refers to any auxiliary information that is transmitted as a partial stream of coded bits and assists the decoder in rendering the decoded image. Such meta data may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, such as those described herein.

Most consumer desktop computer (consumer desktop) displays currently support 200 to 300cd/m ² or nit (nit) brightness. Most consumer HDTVs range from 300 to 500 nits, with new models reaching 1,000 nits (cd/m ² ). Therefore, compared to HDR or EDR, this type of conventional display represents a lower dynamic range (LDR; lower dynamic range), also known as standard dynamic range (SDR). When the availability of HDR content grows due to developments in both capture devices (e.g. cameras) and HDR displays (e.g. PRM-4200 professional reference monitors from Dolby Laboratories), HDR content can be supported Color grade and display on HDR displays with higher dynamic range (eg, from 1000 nits to 5000 nits or more). In general, without limitation, the method of the present disclosure relates to any higher dynamic range than SDR. Here, as understood by the inventors, improvement techniques for coding of high dynamic range images are desirable.

The methods described in this section are methods that can be carried out, but are not necessarily methods that have been previously thought or carried out. Therefore, unless otherwise indicated, it should not be assumed that any methods described in this section qualify as prior art simply because they are included in this section. As such, problems identified against one or more methods should not be assumed to be recognized in any prior art based on this section unless otherwise indicated.

100‧‧‧Video delivery pipeline

105‧‧‧Image generation block

110‧‧‧ production stage

115‧‧‧ Post-production editor

120‧‧‧Overweight block

125‧‧‧Reference display

130‧‧‧decoding unit

135‧‧‧Display management box

140‧‧‧Target display

305‧‧‧Front color reshape square

310‧‧‧Encoder

315‧‧‧Encoder

320‧‧‧Backward color remodeling

500A‧‧‧Encoder

500B‧‧‧Encoder

515‧‧‧Reverse color rotation and scale box

520‧‧‧Color format conversion box

Embodiments of the present invention are illustrated in the accompanying drawings by way of example, not by way of limitation, and in which similar reference numbers refer to similar elements, and in which: FIG. 1 depicts Example program for a video delivery pipeline; Figure 2 depicts an example program for color conversion to IPT-PQ color space; Figure 3 depicts an example program for signal reshaping and encoding; Figure 4 depicts an implementation in accordance with the present invention An example tone mapping curve for brightness reshaping between ST 2084 IPT and BT 1866 IPT; FIG. 5 depicts an example system for retrospective compatible coding and encoding using color space reshaping according to an embodiment of the present invention; FIG. 6 depicts an example program flow for generating color rotation and scale matrices according to an embodiment of the present invention; FIGS. 7A and 7B depict hue and saturation reshaping functions according to an embodiment of the present invention; FIG. 8 depicts an embodiment of the present invention. Example of hue and saturation reshaping between IPT-PQ and YCbCr-Gamma color space; and FIG. 9 depicts an example of EETF function according to an embodiment of the present invention.

[Summary of the Invention and Implementation Modes]

Here, the signal reshaping and encoding of high dynamic range (HDR) images are described. In the following description, for the purpose of explanation, numerous specific details are presented in order to provide a thorough understanding of the present invention. However, it will be apparent that the invention may not be practiced with these specific details. In other instances, well-known structures and devices have not been described in full detail in order to avoid unnecessarily obscuring, obscuring or confusing the present invention.

Overview

The example embodiments described herein are about reshaping and encoding high dynamic range images. In the method for improving the backward compatible coding, in the encoder, the processor accesses the image database and calculates the first hue value of the image in the database in the first color space; The second hue of the image in the database in the second color space Value; calculate the hue rotation angle based on the minimized hue cost function, where the hue cost function is based on the difference measurement of the first hue value and the rotated second hue value; and generate a color rotation matrix based on the hue rotation angle.

In an embodiment, the first color space is a gamma-based YCbCr color space and the second color space is a PQ-based IPT color space.

In an embodiment, the color rotation matrix is used to generate a reshaped color space based on the preferred color space. The image is encoded using a reshaped color space, and information about the color rotation matrix is sent from the encoder to the decoder.

In an embodiment, in the decoder, in a method for reconstructing an input image coded in a reshaped color space, the decoder: receives the coded input image in a reshaped color space, wherein the reshaped The color space is obtained by rotating the chroma component of the preferred color space to about one or more parameters of the old color space; access to the meta data transmitted from the encoder to the decoder, where the post Let the data be associated with the encoded input image and include: a flag indicating the presence or absence of color rotation and a scaling matrix; and used when the flag indicates the presence of color rotation and a scaling matrix Complex coefficients of color rotation and scaling matrix; decode the encoded input image to be used in shaping the color space Generate decoded images; and generate decoded images in the preferred color space based on the decoded images in the reshaped color space and the color rotation and scaling matrix.

In another embodiment, in the encoder, the processor: receives the input image in a preferred color space; accesses the hue rotation function, where the hue value for pixels in the input image in the preferred color space , The hue rotation function produces a rotated hue output value that matches the hue value in the old color space according to the hue cost criterion; generates a reshaped image based on the input image and the hue rotation function; and encodes the reshaped image , Used to generate coded and reshaped images.

In another embodiment, in the decoder, the processor: accesses the input image encoded in the reshaped color space; accesses metadata associated with the input image, wherein the metadata includes and uses Transform the input image from the preferred color space to data associated with the hue rotation function of the reshaped color space, where the hue rotation function produces the hue value of the pixels in the input image in the preferred color space The rotated hue output value matches the hue value in the old color space according to the hue cost criterion; and generates an output image in the preferred color space based on the input image and the data associated with the hue rotation function.

Sample video delivery processing pipeline

FIG. 1 depicts an example sequence of traditional video delivery pipelines (100) showing various stages from video capture to video content display. A series of video frames (102) are captured or generated using an image generation block (105). The video frame (102) can be captured digitally (eg, by a digital camera) or generated by a computer (eg, using computer animation) to provide video data (107). Alternatively, the video frame (102) can be captured on the film by a film camera. The film is converted to a digital format to provide video data (107). In the production stage (110), the video data (107) is edited to provide a video production stream (112).

Next, the video data for creating the stream (112) is provided to the processor at block (115) for post-production editing. Post-production editing (115) may include adjusting or modifying color or brightness in specific areas of the image to enhance the image quality or achieve a specific appearance for the image based on the creative intent of the video creator. This is sometimes referred to as "color timing" or "color grading". Other editing (eg, scene selection and sorting, image cropping, adding computer-generated visual special effects, etc.) can be performed at block (115) to produce the final version (117) of the production for distribution. During post-production editing (115), the video image is viewed on the reference display (125).

After post-production (115), the video data of the final production (117) can be delivered to the encoding block (120) for downstream delivery to decoding and playback devices, such as televisions, set-top boxes ,Electricity Cinema, etc. In some embodiments, the encoding block (120) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other delivery formats, to generate encoded bit streams (122). In the receiver, the encoded bit stream (122) is decoded by the decoding unit (130) to generate a decoded signal (132) that represents the same or close approximation of the signal (117). The receiver may be attached to the target display (140), which may have completely different characteristics than the reference display (125). In this case, the display management block (135) can be used to map the dynamic range of the decoded signal (132) to the characteristics of the target display (140) by generating a display mapping signal (137).

IPT-PQ color space

In a preferred embodiment, without limitation, some processing pipelines, such as encoding (120), decoding (130), and display management (135), can be performed in what will be referred to as the IPT-PQ color space. An example use of the IPT-PQ color space for display management applications can be found in the WIPO publication WO 2014/130343 "Display Management for High Dynamic Range Video" by R. Atkins et al., which is cited by reference in its entirety Incorporate. The IPT color space described in the " Development and testing of a color space (ipt) with improved hue uniformity " of F. Ebner and MDFairchild at the 6th Color Imaging Conference: Color Science, Systems and Applications, IS&T, Sri Lanka Scottsdale, Arizona, November 1998, pp. 8-13 (to be referred to as the Ebner paper) (which is incorporated by reference in its entirety) is in human vision A model of the color difference between cones in the system. In this meaning, it is like YCbCr or CIE-Lab color space; however, it has been shown in some scientific studies to imitate human visual processing better than these color spaces. Like CIE-Lab, IPT is a normalized space for a reference brightness. In an embodiment, the normalization is based on the maximum brightness of the target display (eg, 5,000 nits).

The term "PQ" as used herein refers to perceptual quantization. The human visual system responds to increasing the light level in a very non-linear way. The ability of a human to see a stimulus is determined by the brightness of the stimulus, the size of the stimulus, the spatial frequency that makes up the stimulus, and the eye has adapted to the stimulus at a specific moment when we are watching the stimulus Affected. In a preferred embodiment, a perceptual quantizer function maps a linear input gray level to an output gray level that better matches the contrast sensitivity criticality in the human visual system. An example of a PQ mapping function is illustrated in US Patent No. 9,077,994 (referred to as the '994 patent) by JSMiller et al., which is incorporated by reference in its entirety, part of which has been published in 2014 The SMPTE ST 2084: 2014 manual titled "High Dynamic Range Electro-optical Transfer Function of Mastering Reference Displays" on August 16, 2014 was adopted by its entirety by reference, and for each The brightness level (ie, the stimulation level) is given a fixed stimulus size, The minimum visible contrast step at this brightness level is selected based on the most sensitive adaptation level and the most sensitive spatial frequency (based on the HVS model). Compared to a traditional gamma curve (which represents a physical cathode ray tube (CRT; cathode ray tube) device and coincidentally can have a very rough similarity to the response of the human visual system), as specified by the '994 patent The PQ curve uses a relatively simple functional/functional model to emulate the true visual response of the human visual system.

FIG. 2 depicts in more detail an example procedure (200) for color conversion to IPT-PQ color space according to an embodiment. As depicted in FIG. 2, given the input signal (202) on the first color space (eg, RGB), the color space transformation (color space transformation) in the perceptually corrected IPT color space (IPT-PQ) ) May include the following steps:

a) The optional step (210) may normalize the pixel values (eg, 0 to 4095) of the input signal (202) into pixel values having a dynamic range between 0 and 1.

b) If the input signal (202) is gamma coded or PQ coded (for example, per BT.1866 or SMPTE ST 2084), then the optional step (215) can use the signal's electro-optical transfer function (EOTF; electro-optical transfer function) (as provided by the signal meta data) is used to reverse or cancel the conversion from the code value to the source display of the brightness. For example, if the input signal is gamma coded, then an inverse gamma function is applied in this step. If the input signal is PQ coded according to SMPTE ST 2084, then follow this step to apply anti-PQ Function (inverse PQ function). In practice, the normalization step (210) and the inverse nonlinear coding (215) can be performed using a pre-calculated 1-D look-up table (LUT; Look-up table) to generate the linear signal 217.

c) In step (220), the linear signal 217 is converted from its original color space (eg, RGB, XYZ, etc.) to the LMS color space. For example, if the original signal is in RGB, then this step may include two steps: RGB to XYZ color conversion and XYZ to LMS color conversion. In an embodiment, without limitation, the XYZ to LMS transformation can be given as

In another embodiment, if the US Patent Provisional Application No. 62/056,093 titled "Encoding and decoding perceptually-quantized video content" was applied on September 26, 2014 (also applied for on September 24, 2015 as As described in PCT/US2015/051964) (which is incorporated by reference in its entirety), the overall coding efficiency in the IPT-PQ color space can be further increased, if one can incorporate a crosstalk matrix (cross -talk matrix)

As part of the XYZ to LMS conversion words. For example, for c = 0.02, multiply the crosstalk matrix by the 3 x 3 matrix in equation (1a) to obtain:

Similarly, in another embodiment, for c = 0.04, the crosstalk matrix is multiplied by the original XYZ to LMS matrix (eg, equation (1a)) to obtain:

d) According to Ebner's paper, traditional LMS to IPT color space conversion involves applying a first non-linear power function to LMS data and then applying a linear transformation matrix. While we can transform the data from LMS to IPT and then apply the PQ function in the IPT-PQ domain (IPT-PQ domain), in the preferred embodiment, in step (225), it is used for LMS to IPT The traditional power function of nonlinear coding is replaced by PQ nonlinear coding of each of the L, M, and S components.

e) Using LMS to IPT linear transformation (for example, as defined after the Ebner paper), step (230) completes the conversion of the signal 222 to the IPT-PQ color space. For example, in an embodiment, the L'M'S' to IPT-PQ transformation can be given by the following formula

In another embodiment, experiments have shown that it may be preferable to obtain the I ′ component independently of the S ′ component, so equation (2a) may become:

IPT-PQ vs YCbCr-Gamma

Most existing video compression standards, such as MPEG-1, MPEG-2, AVC, and HEVC, have been tested, evaluated, and optimized for gamma-encoded images in the YCbCr color space; however, experimental results have been Displaying the IPT-PQ color space can provide a better representation format for high dynamic range images with 10 or more bits per pixel per color component. Signal encoding in a color space that is better suited for HDR and wide color gamut signals (eg, IPT-PQ) can lead to better overall picture quality; however, older decoders (eg, set-top boxes (set-top boxes) top box) and the like) may not be able to perform proper decoding and color conversion. To improve traceability compatibility, as the inventors understand, new signal reshaping techniques are needed so that even devices that do not recognize the new color space can produce reasonable pictures.

Figure 3 depicts an example procedure for signal reshaping and encoding according to an embodiment. As depicted in FIG. 3, given the input (302), the color transformation and/or reshaping function is applied in the preferred color space (eg, IPT-PQ) as needed, forward color reshaping block (305) -r) generates a signal (307). Remodeling related meta data (309) can also be generated and communicated to the blocks of the subsequent encoding pipeline, such as encoder (310), decoder (315) and backward color reshaping (320).

After receiving the encoded signal (315), the decoder will apply decoding (315) (like HEVC decoding) to produce a decoded signal (317). Recognizing that the preferred HDR-WCG coded color space (for example, IPT-PQ-r) decoder will apply appropriate backward or inverse reshaping (320) to generate signals in the appropriate color space (for example, IPT-PQ) (322). The signal (322) can then be converted to YCbCr or RGB for additional post-processing, storage, or display.

Older encoders that do not recognize the preferred HDR-WCG coding space can view HDR-WCG as the old color space (eg, gamma-coded YCbCr); however, due to forward color reshaping (305) For the sake of reason, the output (317) can still have reasonable picture quality despite the fact that no reshaping or other color transformation is applied to the output (317) of the decoder.

Color remodeling

)值；b.5)計算YCbCr-伽瑪的信號之飽和值(例如，

)；c)以優選的色彩格式(例如，IPT-PQ)在資料庫中計算膚色調值。此步驟可包括下列子步驟：c.1)將XYZ轉換至LMS；c.2)藉由應用PQ(例如，per ST 2084)將LMS轉換至L’M’S’以及轉換至I’P’T’；c.3)計算色相值(例如，

)；c.4)計算飽和值(

)；d)計算用以旋轉IPT值的旋轉矩陣，使得在旋轉的或重塑的IPT-PQ(例如，IPT-PQ-r)中的膚色調與在YCbCr-伽瑪中的膚色調對準/一致。在實施例中，此步驟係藉由將關於在兩個色彩空間中的樣本之色相及飽和值的成本函數最佳化而計算。舉例來說，在實施例中，成本函數可代表在舊有色彩空間(例如，YCbCr)與經旋轉優選的HDR色彩空間(例如，IPT-PQ)之間的均方根誤差(MSE；mean square error)。舉例而言，使得

The IPT-PQ color space is considered without loss of generality. In an embodiment, a linear reshaping matrix (eg, a 3 x 3 matrix) is generated to perceptually match skin tone in an IPT-PQ signal with skin tone in YCbCr-Gamma signal space. This type of color conversion does not have the effect of expression in most image processing applications in the IPT color space, or the color reproduction is greatly improved by the old device. Instead of or in addition to skin tone, a similar transformation matrix can be generated to match other important colors, such as leaves, sky, etc. In an embodiment, the remodeling matrix can be calculated as follows: a) Load a database of skin tone tones, such as the reflection spectrum, and convert them to a device-independent color space, such as XYZ; b) Convert skin tone tones The library is converted from XYZ to the old color space format (for example, YCbCr, Rec.709). This step may include the following sub-steps, for example: b.1) Convert the database to RGB (Rec.709); b.2) Apply gamma to RGB values (eg, per BT.1886) to generate gamma Encoded R'G'B'signal; b.3) Convert R'G'B' signal to YCbCr-Gamma value (for example, per Rec.709); b.4) Calculate YCbCr-Gamma signal Hue (for example,

) Value; b.5) Calculate the saturation value of the YCbCr-Gamma signal (for example,

); c) calculate the skin tone value in the database in a preferred color format (for example, IPT-PQ). This step may include the following sub-steps: c.1) Convert XYZ to LMS; c.2) Convert LMS to L'M'S' and I'P'T' by applying PQ (for example, per ST 2084) ; C.3) Calculate the hue value (for example,

); c.4) Calculate the saturation value (

); d) Calculate the rotation matrix used to rotate the IPT value so that the skin tone in the rotated or reshaped IPT-PQ (for example, IPT-PQ-r) is aligned with the skin tone in YCbCr-Gamma / Consistent. In an embodiment, this step is calculated by optimizing the cost function regarding the hue and saturation value of the samples in the two color spaces. For example, in an embodiment, the cost function may represent the root mean square error (MSE; mean square) between the old color space (eg, YCbCr) and the rotated preferred HDR color space (eg, IPT-PQ) error). For example, making

Represents the cost function related to hue, where Hue _IPT-PQ-r represents the hue of the reshaped color (that is, IPT-PQ-r) and can be defined as

All inverse tan functions are calculated in (-π,π).

In an embodiment, one can apply any known optimization technique in the art to find the value of angle " a ", expressed as a' , to minimize the cost function according to given criteria. For example, we can apply the MATLAB function fminunc(fun, x0), with fun= Cost _H and x0=0.1. Given a' , the rotation matrix R can be defined as

As an example, based on the sample database, in the embodiment, for a' = 71.74 degrees

Given R and the original L'M'S' to I'P'T' matrix LMS2IPTmat (for example, see equation (2)), conversion to the reshaped IPT-PQ-r color space can use the new LMS2IPTmat-r matrix , Which is defined as LMS2IPTmat-r = R ^T * LMS2IPTmat = ( ( LMS2IPTmat ^T * R )) ^T , (7)

Where ^AT represents the transpose of matrix A.

其中，

以及c)使b’表示最佳化Cost _S 的b值。接著，吾人能應用標度向量(scaling vector)

到色度旋轉矩陣，用以形成單一色彩旋轉及標度3 x 3矩陣

In the embodiment, in addition to matching the hue used for skin tone adjustment, one can also saturate the same. This may comprise the steps of: a) R IPT-PQ applied to the original data, for generating a rotating color chroma (Chroma) value and P _{R R} T Information b) saturated cost function defined, for example in the target color space of the original MSE between saturation values:

among them,

And c) b 'b represents the best value of the Cost _S. Then, we can apply the scaling vector

To chroma rotation matrix to form a single color rotation and scale 3 x 3 matrix

等式(4)能被修改為

In some embodiments, the hue and saturation cost cost function (e.g., Equation (3) and (8)) may be combined into a single hue / saturation cost function and can simultaneously solve a 'and b' both. For example, from equation (11), in the embodiment, for

Equation (4) can be modified to

And I can solve the equation (3) to obtain the best a 'best and bi' (i = 1 to 4) scale factor.

For example, in one embodiment, for a' = 65 degrees and b 1 ' = 1.4, b 2 ' = 1.0, b 3 ' = 1.4, and b 4'= 1.0, equation (12) yields:

Reshaping

The proposed rotation matrix R can improve color reproduction; however, due to the non-linear EOTF encoding function (eg, ST 2084 vs. BT 1866), the decoded image (317) can still be perceived to have low contrast. In an embodiment, the contrast can be improved by applying a 1-D tone mapping curve to the lightness channel (eg, I'). This step may include the following sub-steps: a) Apply a tone mapping curve (eg, S-shaped) to map the original content from the original HDR maximum brightness (eg, 4,000 nits) to the SDR target brightness (eg, 100 nits) . Examples of such sigmoid functions can be found in US Patent No. 8,593,480 "Method and Apparatus for Image Data Transformation" by A. Ballestad and A. Kostin, which is incorporated herein by reference in its entirety. Examples of alternative reshaping functions are also disclosed in WIPO Publication No. WO 2014/160705 "Encoding perceptually-quantized video content in multi-layer VDR coding", which is incorporated herein by reference in its entirety. Let _{I 'T = f (I'} ) represents a tone-mapping function f () of the output, then b) adding I _'T linear (e.g., by reverse gamma function or PQ) to generate a linear I _T data; and c) EOTF old coding applications (e.g., BT.1866) I _T of the linearized signal coding, to generate the gamma coded brightness signals to be compressed and transmitted encoder.

An example of such mapping between ST 2084 (PQ) and BT 1866 is shown in FIG. 4. The curve has higher midtone contrast, lower black and brighter (with less contrast) high brightness. This closely aligns the tone scale with the standard SDR, so that when the input is decoded by the old device, the image is still viewable. In Figure 4, without a general loss, the input and output values are normalized to (0, 1).

The remodeled data can be sent from the encoder to the rest of the pipeline as metadata. The reshaping parameters can be determined at various time instances, such as on a per-frame basis, on a per-scene basis, or on a per-sequence basis, to be used for output The best possible performance for a given video sequence.

Although the description of the present invention focuses on the IPT-PQ color space, these techniques can be equally applied to other color spaces and color formats. For example, similar techniques can be applied to improve traceability across different YCbCr versions Compatibility, such as Rec.709 YCbCr and Rec.2020 YCbCr. Therefore, in an embodiment, when decoding using the old Rec. 709 decoder, the signal reshaping technique used in this description can be adjusted to adjust the Rec. 2020 bit stream to provide better hue and saturation output.

FIG. 6 depicts an example program flow for generating color rotation and scale matrices according to an embodiment. Given the image database (605), step (610) calculates the chroma and saturation values for the image in the database in the first (old) color space (eg, YCbCr-Gamma). Step (615) calculates the hue of the image in the database in the second (preferred) color space (eg, IPT-PQ).

Given hue correlation cost function (e.g., Equation (3)), the step (620) based on a minimum cost criteria (such as the root mean square error (the MSE)) solving for the optimal rotation angle a ', which minimizes the old The distance between the calculated hue in some color spaces and the calculated hue in the rotated preferred color space. In step (625) using a 'color values to produce the rotation matrix.

An optional saturation scale can also be calculated. Saturated given cost function (e.g., Equation 8), the step (630), optionally based on minimizing a cost criterion to solve the optimal dial b ', such as the saturation of the signal is rotated in a first color space in color The MSE between the saturation of the scaled signal in the preferred color space (640,645).

Finally, in step (635), the rotation angle and dial are combined to generate a color rotation and scale matrix (e.g., equation (11)).

In the encoder, the encoder will rotate the color and scale The matrix is applied to the input data in the preferred color space to generate the data in the reshaped color space. The data will be encoded and sent to the decoder along with information about color rotation and scale matrix.

In the decoder, the old decoder will decode the data assuming it was coded in the old color space. Despite the wrong color space information, the image will still be viewable with sufficient quality, even in the lower dynamic range. Newer, fully-enable decoders can use the received metadata information about color rotation and scaling matrix to decode image data in the preferred color space, thus providing the full high dynamic range of the data to the viewer.

SEI message syntax for reshaping information

As discussed earlier, in one embodiment, the rotation ( R ) matrix and the scale factor ( S ) can be absorbed by the L'M'S' to I'P'T' conversion matrix in (230). The hue reshaping curve may be partial forward color reshaping (305). In both cases, the adapted remodeling information (ie, matrix and tone mapping curve) can be used in US Provisional Application No. 62/193,390, filed on July 16, 2015, and also on April 19, 2016. The syntax proposed in the PCT application filed in Japanese PCT/US2016/02861 (which is incorporated by reference in its entirety) is transmitted from the encoder to the decoder.

In another embodiment, as depicted in FIG. 5, a new color rotation and scaling block (510) may be added to the encoder (500A). This block can be converted in color (200) (for example, RGB to IPT-PQ) The latter but preferably before the forward remodeling (305) is added. In the decoder (500B), the corresponding inverse color rotation and scaling blocks (515) can be added after the block (320) is reshaped backwards. As depicted in FIG. 5, optional color format conversion boxes (eg, 4:4:4 to 4:2:0 (505) or 4:2:0 to 4:4:4 (520)) may be Add to the encoding and/or decoding pipeline when needed.

In terms of syntax, since the lightness channel (eg, Y or I) is typically not changed, one can specify a 3 x 3 rotation matrix or only a 2 x 2 matrix. Table 1 provides examples of SEI communication to convey color rotation and scale matrix; however, the transmission is not limited to SEI messages, which can be inserted in any high-level syntax, such as SPS, PPS, etc.

Colour_rotation_scale_matrix_present_flag equal to 1 means that for c and i in the range of 0 to 1, including the upper and lower limits, the syntax element colour_rotation_scale_coeffs[c][i] exists. colour_rotation_scale_matrix_present_flag equal to 0 means that for c and i in the range of 0 to 1, the included syntax element colour_rotation_scale_coeffs[c][i] does not exist.

colour_rotation_scale_coeffs[c][i] specifies the values of 2 x 2 color rotation and scale matrix coefficients. colour_rotation_scale_coeffs The value of [c][i] should be in the range of -2^15 to 2^15-1, including the upper and lower limits. When colour_rotation_scale_coeffs[c][i] does not exist, the matrix of default/preset color rotation and scale matrix is used.

In an embodiment, both the encoder and the decoder recognize the color rotation and scaling matrix (eg, through a new color space mutual definition), so they may not need to encode the color rotation matrix from The transmitter sends a message to the decoder. In another embodiment, the color rotation and scaling matrix can be referenced with VUI (Video Availability Information) together with IPT-PQ.

Multiple hue and saturation remodeling

In some embodiments, it may be beneficial to apply remodeling on multiple hues. This will increase the correctness of the reshaped color space to match the old colors, but there will be additional computational cost at the decoder. For example, consider the problem of reshaping optimization for N hues (eg, skin tone, sky color, green color, etc.). In an embodiment, one can repeat the procedure discussed earlier to identify a set of optimal angles and saturation as a function of hue. For example, using database images for various hues, one can produce a set of optimal (rotation angle, saturation scale) values, such as {( a ₁ , b ₁ ), ( a ₂ , b ₂ ),...,( a _N , b _N )}. Or more generally, let the phase prime be pa ( p ) = f _H ( h ( p )), b ( p ) = f _S ( h ( p )), (14)

Represents the best chroma (hue) rotation and saturation scale values, where h(p) represents the hue measurement for pixel p . For example, for the IPT-PQ color space, the f _H and f _S functions can be calculated on the hue h(p) and saturation s ( p ) functions:

The functions f _H ( h ( p )) and f _S ( h ( p )) can be represented and stored in various ways known in the art, for example, like look-up tables or piece-wise linear or nonlinear It is polynomial and can be sent from the encoder to the decoder as back-end data.

Given f _H ( h ( p )) and f _S ( h ( p )), the encoder applies the following reshaping function to each pixel: h' ( p )= h ( p )+ a ( p ), s' ( p )= s ( p )＊ b ( p ), (16)

Used to generate appropriate reshaping signals. For example, color space for IPT-PQ, P for remodeling pixel p 'and T' color components using the following equation can be derived _{P 'r (p) = s} ' (p) cos (h' (p)) _{, T 'r (p) =} s' (p) sin (h' (p)). (17)

In the decoder, the program is reversed. For example, from equations (14) and (16), given f _H ( h ( p )) and f _S ( h ( p )), the decoder produces h ( p )= h' ( p )- a ( p ), s ( p ) = s' ( p )/ b ( p ). (18)

Note that in order to avoid division in the decoder, in some embodiments, the decoder may send the inverse of f _S ( h ( p )) (eg, 1/ b ( p ) value) to the decoder . In order to input data in the IPT-PQ space, the original data is generated as P ( p )= s ( p )cos( h ( p )), T ( p )= s ( p )sin( h ( p )). ( 19)

From equation (17), applying inverse reshaping to restore data in the preferred color space requires trigonometric operation. In some embodiments, trigonometric function operations may be performed using look-up tables. As an example, from equation (18), equation (19) can be rewritten as P ( p )= s ( p )cos( h' ( p )- a ( p )), T ( p )= s ( p )sin( h' ( p )- a ( p )). (20)

These operations can be further simplified using suitable look-up tables for cosine and sine functions.

Figure 7A depicts an example used to reverse the hue from the reshaped IPT-PQ-r (which appears to the old device as YCbCr) to the IPT-PQ when the old color space is YCbCr-Gamma. Plastic function. Figure 7B depicts the corresponding backward reshaping function to adjust saturation. Figure 8 depicts how the preferred color space IPT-PQ (820) can be adjusted to match the characteristics of the old YCbCr color space (810). Ray (830) depicts rotation and scale.

In another embodiment, instead of calculating the P and T values on the cosine or sine function of the hue, we can generate based on some other hue function (eg, f ( tan ^-1 ( h ( p )))) Lookup table to construct a simpler decoder. For example, the pixel component values given remodeling P _'r (p), and T' _r (p), in an embodiment, the decoder can respond to the original pixel value as in the following equation:

Where v () and w () represent the hue correlation function that has been generated, so that the image in the remodeled color space matches a set of hue and saturation in the old color space. As before, the v () and w () functions can already use meta data to communicate from the encoder to the decoder or they can establish a coding protocol for the part or a target known by both the encoder and the decoder.

IC _T C _P color space

IC _T C _P (also known as ICtCp (or IPT)) is a proposed new color space, which is specially designed to process high dynamic range and wide color gamut (WCG) signals. Like ITP-PQ, I (intensity) represents the brightness of the PQ-encoded signal, C _T (Tritan axis) corresponds to blue-yellow perception and C _P (Protan axis) corresponds to red-green color perception. In addition to the features discussed in IPT-PQ, in IC _T C _P :

‧As explained earlier, rotate the chroma to align the skin tone closer to YCbCr

‧Optimize XYZ to LMS matrix for better uniformity and linearity for WCG images

‧Optimize the L’M’S’ to ICtCp matrix to improve the isoluminance and stability of relative HDR and WCG images

As used herein, the term "brightness, etc." refers to the brightness (such as the I or ICtCp Y'Cb'Cr 'of Y') corresponding to a plurality of measurement light luminance Y. Indirectly, it measures how much the color space separates brightness and chroma. Experiments conducted by the inventor indicate that I of ICtCp is closer to brightness than Y of Y'Cb'Cr'.

From a construction point of view, using the IC _T C _P color space requires the same hardware and signal flow as YCbCr coded using traditional gamma. For example, consider using gamma-corrected YCbCr (Y'Cb'Cr') in the camera pipeline. Starting from XYZ, the program requires the following steps: a) using a 3 x 3 matrix to convert from XYZ to RGB BT.2020 b) applying anti-EOTF (or OETF) to the output of step a); and c) applying a 3 x 3 matrix Output to step b)

As described in Figure 2, the use of IC _T C _P color requires the following steps:

a) In step (220), in a preferred embodiment, the following 3 x 3 matrix is used to convert from XYZ to LMS:

It corresponds to combining the XYZ to LMS 3 x 3 matrix of equation (1a) with a crosstalk matrix with c = 0.04 (see also equation (1c)).

b) In step (225), as explained earlier, the signal (222) is converted to L’M’S’ by applying PQ nonlinearity

c) In step (230), a 3 x 3 matrix is used to convert from L'M'S' to IC _T C _P , which in a preferred embodiment can be defined as:

Equation (23) corresponds to multiplying the rotation matrix of equation (12b) by the L'M'S' to I'P'T' matrix of the original equation (2b).

其中

以及I=0.5L'+0.5M',C _T =(6,610L'-13,613M'+7,003S')/4096,C _P =(17,933L'-17,390M'-543S')/4096, (24) In another embodiment, steps a) to c) can also be expressed as follows:

among them

And I = 0.5 L '+0.5 M' , C T = (6,610 L '-13,613 M' +7,003 S ') / 4096, C P = (17,933 L' -17,390 M '-543 S') / 4096, ( twenty four)

表示依據SMPTE ST 2084反轉EOTF。在一些實施例中，

函數可由另一個非線性量化函數代替，像是混合對數-伽瑪(HLG；Hybrid Log-Gamma)函數。對於完整的參考，適當的等式亦整理在表2中，其中下標D指的是顯示光。 Among them, RGB _BT.2020 represents the triplet of RGB values in BT.2020,

Indicates that the EOTF is reversed according to SMPTE ST 2084. In some embodiments,

The function can be replaced by another nonlinear quantization function, such as the hybrid log-gamma (HLG; Hybrid Log-Gamma) function. For a complete reference, the appropriate equations are also summarized in Table 2, where the subscript D refers to the display light.

The conversion from IC _T C _P back to the original color space follows a similar approach, and in an embodiment, it may include the following steps: a) Conversion from IC _T C _P using the inversion of equation (23) or the following formula To L'M'S'

b) use the signal's EOTF function (for example, as defined in ST 2084) to convert the L'M'S' signal to LMS; and c) use the inversion of equation (22) to convert from LMS to XYZ, for example:

In an embodiment, the corresponding L'M'S' to RGB and IC _T C _P to L'M'S' matrices are given by the following equations:

Reference display management

High dynamic range content can be viewed on a display with a lower dynamic range than the reference display used to support the content. In order to view HDR on a monitor with a lower dynamic range, display mapping should be performed. This can take the form of EETF (Electro-Electrical Transfer Function) in the display, which is typically applied before EOTF is applied to the display. This function provides toes and shoulders to elegantly reproduce high brightness and shadows, which provides a balance between preserving artistic intention and maintaining detail. Figure 9 shows an example EETF mapping from a full dynamic range of 0 to 10,000 nits to a target that can be 0.1 to 1,000 nits. EETF can import PQ signals; this drawing shows the effect of mapping, that is, they explain how the expected light changes into the actual displayed light.

The following are the mathematical steps to build this tone map for various black and white light levels. EETF may be applied in the linear domain individually to any one of RGB is applied to the luminance channel or channel IC _T C _P or Y'C _'B C' _R.

Calculate EETF:

The central area of the tone mapping curve is defined as a one-to-one mapping from source to target. Additional toe and shoulder copies are calculated using Hermite splines to reduce the dynamic range to the target display's ability.

The turning points for the spline (toe start (TS; Toe Start) and shoulder start (SS; Shoulder Start)) are first defined. There is a point at the beginning of copying. Let minLum and maxLum denote the minimum and maximum brightness values of the target display, then: SS =1.5* maxLum -0.5

TS =1.5＊ minLum

Given E ₁ (the source input signal in the normalized PQ codeword), the output E ₂ is calculated as follows.

E ₁

TS For 0

E ₁

TS

X ₁ =0

Y ₁ = minLum

X ₂ , Y ₂ = TS

E ₂ = P ( E ₁ , X ₁ , Y ₁ , X ₂ , Y ₂ )

For TS < E ₁ < SS

E ₂ = E ₁

E ₁

1 For SS

E ₁

1

X ₁ , Y ₁ = SS

X ₂ =1

Y ₂ = maxLum

E ₂ = P ( E ₁ , X ₁ , Y ₁ , X ₂ , Y ₂ )

Hemet's spline equation:

B=(A, X ₁ , X ₂ )

P ( B , Y ₁ , Y ₂ )=(2 T ( B ) ³ -3 T ( B ) ² +1) Y ₁ +( T ( B ) ³ -2 T ( B ) ² + T ( B )) ( X ₂ - X ₁ )+(-2 T ( B ) ³ +3 T ( B ) ² ) Y ₂

In another embodiment:

step 1:

E ₂ = E ₁ for E ₁ < SS

E ₁

1 E ₂ = P ( E ₁ ) for SS

E ₁

1

Step 2:

E ₂

1 E ₃ = E ₂ + TS ＊(1- E ₂ ) ⁴ for 0

E ₂

1

The Hermitian spline equation P ( B )=(2 T ( B ) ³ -3 T ( B ) ² +1) SS +( T ( B ) ³ -2 T ( B ) ² + T ( B )) (1- SS )+(-2 T ( B ) ³ +3 T ( B ) ² ) maxLum , where

EETF resulting curve can be applied to IC _T C _P I-channel or the intensity Y'C _'B C' _R. The Y luminance channel. Here are some notable options:

1) I IC _T C _P - The intensity of the transmitted EETF processing IC _T C _P of the (I) channel

I ₂ = EETF ( I ₁ )

‧ Adjust grayscale more correctly

‧No color shift

‧The change in saturation will be required and should be applied to the C _T and C _P channels using the following equation:

2) Y'C _'B C' _R of Y '- processing y'c through EETF' Channel _B C 'of _R luminance Y'

_{Y '2 = EETF (Y'} 1)

‧Adjust gray scale more correctly

‧Limited color shift

‧ will require changes in the saturation and should be applied using the following equations to C _'B and C' _R:

Additional embodiments of the present invention are included in the appendix of this application.

Examples of computer system construction

Embodiments of the present invention may be a computer system, a system configured in electronic circuits and components, an integrated circuit (IC) like a microcontroller, a field programmable gate array (FPGA) or Another configurable or programmable logic device (PLD; programmable logic device), discrete time or digital signal processor (DSP; digital signal processor), application specific IC (ASIC; application specific IC) and/or including one or more Equipment of such systems, devices or components. The computer and/or IC can perform, control, or execute instructions regarding signal reshaping and encoding of images with enhanced dynamic range, such as those described herein. The computer and/or IC may calculate any of a number of parameters or values regarding the signal reshaping and encoding procedures described herein. The image and video embodiments can be implemented in hardware, software, firmware, and various combinations thereof.

Some implementations of the invention include a computer processor that executes software instructions that cause the processor to perform the method of the invention. For example, a display, encoder, set-top box, transcoder, or the like can be built by executing software instructions in program memory accessible to the processor as discussed above regarding HDR images Signal reshaping and code addition. The invention can also be provided in the form of a program product. The program product may include a non-transitory medium carrying a set of computer-readable signals containing instructions, which when executed by the data processor, causes the data processor to perform the method of the present invention. The program product according to the present invention can be in a wide variety of forms. For example, the program product may include physical media, such as magnetic data storage including floppy diskettes, hard disk drives, optical data storage media including CD ROM and DVD, and electronic data including ROM and flash RAM Storage media or similar. The computer-readable signal on the program product may be optionally compressed or encrypted.

When a component (eg, software module, processor, assembly, device, circuit, etc.) is referenced above, unless otherwise indicated, reference to that component (including reference to "means") should be interpreted to include equality of that component, Any group that describes the function of the component (for example, equal functions) The components, including unstructured, are equivalent to the structures disclosed to perform the functions in the exemplary embodiments described in the present invention.

Equalization, extension, selection and others

Therefore, an exemplary embodiment of signal reconstruction and encoding for efficient HDR images is described. In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that can be changed with construction. Therefore, what is the invention and what the inventor intends to be the only and exclusive indicator of the invention is the group of patent applications issued from this case, which are in the specific form used in the patent application , Including any subsequent amendments. Any definitions specifically directed to the terms included in the scope of the patent application mentioned herein shall govern the meaning of such terms as used in the scope of the patent application. Therefore, there are no restrictions, elements, properties, features, benefits or attributes that are not specifically documented in the request item and should limit the scope of the request item in any way. Accordingly, the description and drawings are to be regarded as illustrative rather than restrictive.

100‧‧‧Video delivery pipeline

102‧‧‧Video frame

105‧‧‧Image generation block

107‧‧‧ Video data

110‧‧‧ production stage

112‧‧‧ Production Stream

115‧‧‧ Post-production editor

117‧‧‧ final production

120‧‧‧Overweight block

122‧‧‧ bit stream

125‧‧‧Reference display

130‧‧‧decoding unit

132‧‧‧decoded signal

135‧‧‧Display management box

137‧‧‧Display mapping signal

140‧‧‧Target display

Claims (18)

A method for improving backward compatible decoding. The method includes: accessing an image database with a processor; calculating a first hue value in the image's first color space in the image database; calculating in the database The second hue value in the second color space of the image; the hue rotation angle is calculated by minimizing the hue cost function, where the hue cost function is based on the difference between the first hue value and the rotated second hue value Measurement; generate a color rotation matrix for the color rotation input image based on the hue rotation angle before encoding; calculate the first saturation value of the image in the first color space in the database; transform the image in the database into The second color space is used to generate transformed images; the color rotation matrix is applied to the transformed images to generate color rotated images; the second saturation value of the color rotated images is calculated; based on minimizing the saturation cost Function to calculate the saturation plate, wherein the saturation cost function is based on the difference between the first saturation value and the scaled second hue value; and a scale vector is generated based on the saturation plate. For example, the method of claim 1 of the patent application further includes combining the color rotation matrix and the scale vector to generate a reshaped color matrix. A method as claimed in claim 1, wherein the first color space includes a gamma coded YCbCr color space and the second color space includes a PQ coded IPT color space. A method as claimed in item 1 of the patent scope, wherein the first color space includes the Rec.709 YCbCr color space, and the first color space includes the Rec. 2020 YCbCr color space. As in the method of claim 1, the color rotation matrix contains

Where a 'represents the rotation angle of the hue. For example, the method of claim 1 of the patent scope, where the scale vector contains

Where b'represents the saturation plate. As in the method of claim 1, the first color space is the Rec.709 YCbCr color space, the second color space is the ST 2084 IPT color space (IPT-PQ) and the color rotation matrix contains

For example, the method of claim 2 of the patent scope, wherein the reshaped color matrix includes:

Where a 'represents the rotation angle of the hue and b' denotes the saturation disc. For example, the method of claim 1 of the patent scope, the encoder further includes: receiving the input image in the second color space; the step of applying forward color reshaping to the input image includes applying the color rotation matrix to the input Image, used to generate a color-rotated image; and use the encoder to encode the color-rotated image to generate a coded image; send the coded image to the decoder; send the remodeling-related metadata to The decoder, wherein the method in the decoder further includes: receiving the encoded image; receiving the remodeling-related metadata; decoding the encoded image to produce a decoded image; and remodeling the retrospective color Apply to the decoded image to produce an output image in the second color space. The method as claimed in item 2 of the patent scope further includes: receiving the input image in the second color space; applying the reshaped color matrix to the input image to generate the reshaped Image; and use the encoder to encode the reshaped image to generate a coded image. For example, the method of claim 9 of the patent scope further includes: applying the tone mapping function to the lightness value of the color-rotated image before the encoding step. For example, the method of claim 10 of the patent scope further includes: based on the reshaped color matrix, the information is sent to the decoder as meta data. For example, the method of claim 12, wherein the meta data includes: a flag indicating the existence of the color rotation and the scale matrix; and when the flag indicates the existence of the color rotation and the scale matrix, based on Complex coefficients of the color rotation and scaling matrix. A method as claimed in item 2 of the patent scope, wherein the second color space includes an IPT color space, and in the color converter, the preset LMS to IPT conversion matrix is the preset LMS to IPT conversion matrix and the reshaped color The product of the matrix is replaced. As in the method of claim 1, the hue cost function represents the root mean square error (MSE) between the first hue value and the rotated second hue value. The method as claimed in item 1 of the patent scope includes: receiving an input image in the second color space; generating a first reshaped image based on the input image and the color rotation matrix An image; generating a second reshaped image based on the first reshaped image and the scale vector; and encoding the second reshaped image to generate an encoded reshaped image. A device that contains a processor and is configured to perform the method described in item 1 of the patent application. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon for executing the method according to item 1 of the patent application scope with one or more processors.

Images