JP2018524870A - Content-adaptive application of fixed transfer functions to high dynamic range (HDR) and / or wide color gamut (WCG) video data - Google Patents

Abstract

  The present disclosure relates to processing video data, including processing video data to conform to a high dynamic range (HDR) / wide color gamut (WCG) color container. The technique applies color value pre-processing before application of the static transfer function and / or post-processing on the output from application of the static transfer function at the encoder side. By applying preprocessing, the example may generate color values that linearize the output codeword when reduced to a different dynamic range by applying a static transfer function. By applying post-processing, the example can increase the signal-to-quantization noise ratio. An example may apply the inverse of the operation at the encoding side at the decoding side to reconstruct the color values.

Description

  This application is based on U.S. Provisional Application No. 62 / 172,713 filed on June 8, 2015 and U.S. Provisional Application No. filed on June 24, 2015, the entire contents of each of which are incorporated herein by reference. It claims the benefit of 62 / 184,216.

  The present disclosure relates to video coding.

  Digital video capabilities include digital television, digital direct broadcast system, wireless broadcast system, personal digital assistant (PDA), laptop or desktop computer, tablet computer, ebook reader, digital camera, digital recording device, digital media player, video It can be incorporated into a wide range of devices including gaming devices, video game consoles, cellular or satellite radiotelephones, so-called “smartphones”, video conferencing devices, video streaming devices, and the like. Digital video devices are MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265, high efficiency Implement video coding techniques such as those described in standards defined by Video Coding (HEVC), and extensions of such standards. A video device may more efficiently transmit, receive, encode, decode, and / or store digital video information by performing such video coding techniques.

  Video coding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (e.g., a video frame, or a portion of a video frame) may be partitioned into video blocks, which may be tree blocks, coding units (CUs), and / or coding nodes. Sometimes called. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction on reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction for reference samples in neighboring blocks in the same picture, or temporal prediction for reference samples in other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame.

  Spatial or temporal prediction results in a predictive block for the block to be coded. The residual data represents the pixel difference between the original block to be coded and the prediction block. The inter-coded block is encoded according to a motion vector that points to the block of reference samples that form the prediction block, and residual data that indicates the difference between the coded block and the prediction block. Intra-coded blocks are encoded according to the intra-coding mode and residual data. For further compression, the residual data may be transformed from the pixel domain to the transform domain to yield residual transform coefficients, which may then be quantized. The quantized transform coefficients initially composed of a two-dimensional array may be scanned to generate a one-dimensional vector of transform coefficients, and entropy coding may be applied to achieve even further compression.

Bross et al., "High efficiency video coding (HEVC) text specification draft 10 (for FDIS & Last Call)", Video coding joint research group of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11 (JCT-VC) , 12th meeting, Geneva, Switzerland, January 14-23, 2013, JCTVC-L1003v34, http://phenix.int-evry.fr/jct/doc_end_user/documents/12_Geneva/wg11/JCTVC-L1003-v34 .zip Wang et al., `` High efficiency video coding (HEVC) Defect Report '', ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11 video coding joint research group (JCT-VC), 14th meeting, Vienna, Austria July 25-August 2, 2013, JCTVC-N1003v1, http://phenix.int-evry.fr/jct/doc_end_user/documents/14_Vienna/wg11/JCTVC-N1003-v1.zip ITU-T H.265, Series H: Audiovisual and Multimedia Systems, Infrastructure of audiovisual services-Coding of moving video, High efficiency video coding, Telecommunication Standardization Division, International Telecommunication Union (ITU), April 2013

  The present disclosure relates to processing video data, including processing video data to conform to a high dynamic range (HDR) / wide color gamut (WCG) color container. As described in more detail below, the techniques of this disclosure apply color value preprocessing and / or from application of a static transfer function at the encoder side prior to application of a static transfer function. Apply post-processing to the output. By applying preprocessing, the example may generate color values that linearize the output codeword when reduced to a different dynamic range by applying a static transfer function. By applying post-processing, the example can increase the signal-to-quantization noise ratio. An example may apply the inverse of the operation at the encoding side at the decoding side to reconstruct the color values.

  In one example, this disclosure describes a method of video processing, the method comprising receiving a first plurality of codewords representing a shortened color value of video data, wherein the shortened color value is a first dynamic range. A second plurality based on the first plurality of codewords using an inverse static transfer function that does not exhibit adaptability to the video data to represent the color at and to generate non-shortened color values The first plurality of codewords, wherein the non-shortened color values represent colors in a second different dynamic range, and the second plurality of codewords are reverse post-processed One of the codewords or one of the codewords from the first plurality of codewords and reverse post-processing the first plurality of codewords to generate a second plurality of codewords Or non-shortened color At least one of the inverse pre-processing the values, and a outputting a non-reducing color value or vice pretreated non shortened color values.

  In one example, this disclosure describes a method of video processing, the method receiving a plurality of color values of video data representing colors in a first dynamic range and a plurality of codewords representing shortened color values. To reduce color values using a static transfer function that is not adaptable to the shortened video data to generate, where the shortened color values represent colors in a second different dynamic range. At least one of representing and preprocessing the color value before shortening to produce a shortened color value, or post-processing the codeword resulting from shortening the color value Outputting a color value based on one of the shortened color value or the post-processed shortened color value.

  In one example, this disclosure describes a device for video processing, the device comprising a video data memory configured to store video data and at least one of fixed function or programmable circuitry. And a post processor. The video post processor receives a first plurality of codewords representing a shortened color value of the video data from the video data memory, wherein the shortened color value represents a color in a first dynamic range; Unshorten the second plurality of codewords based on the first plurality of codewords using an inverse static transfer function that is not adaptable to the video data to generate unshortened color values The non-shortened color value represents a color in a second different dynamic range, and the second plurality of codewords is the first plurality of codewords or the first plurality being reverse post-processed One of the codewords from the codeword and reverse post-processing the first plurality of codewords to generate a second plurality of codewords, or an unshortened color value Bract least one of the inverse pre-processing, configured to perform and outputting a non-reducing color value or vice pretreated non shortened color values.

  In one example, this disclosure describes a device for video processing, the device comprising a video data memory configured to store video data and at least one of fixed function or programmable circuitry. And a preprocessor. The video preprocessor is shortened to receive a plurality of color values of video data representing colors in a first dynamic range from the video data memory and to generate a plurality of codewords representing shortened color values. Shortening color values using a static transfer function that is not adaptable to video data, where the shortened color value represents a color in a second different dynamic range and the shortened color value At least one of pre-processing color values before shortening to generate or post-processing codewords resulting from shortening color values, and shortened color values or post-processing And outputting a color value based on one of the shortened color values.

  In one example, the disclosure is, when executed, receiving a first plurality of codewords representing a shortened color value of video data to one or more processors of a device for video processing, The shortened color value represents a color in the first dynamic range and uses an inverse static transfer function that is not adaptable to the video data to produce a non-shortened color value. Non-shortening the second plurality of codewords based on the second codeword, wherein the non-shortening color value represents a color in a second different dynamic range, and the second plurality of codewords, The first plurality of codewords being reverse post-processed or one of the codewords from the first plurality of codewords, and the first to generate a second plurality of codewords Multiple cordova Stores instructions that cause at least one of reverse post-processing of the data or reverse pre-processing of non-shortened color values and outputting non-shortened color values or reverse pre-processed non-shortened color values A computer-readable storage medium will be described.

  In one example, this disclosure describes a device for video processing, wherein the device is a means for receiving a first plurality of codewords representing a shortened color value of video data, wherein the shortened color value is Based on the first multiple codewords, using an inverse static transfer function that does not show adaptability to the video data to produce a non-shortened color value and means to represent the color in a dynamic range of 1 Means for non-shortening the second plurality of codewords, wherein the non-shortening color value represents a color in a second different dynamic range, and the second plurality of codewords are reverse post-processed The first plurality of codewords or one of the codewords from the first plurality of codewords, the means and the first plurality of codewords to generate the second plurality of codewords After reverse codeword Means for physical or at least one of the means for the non-reduced color values to inverse preprocessing, and means for outputting the non-reduced color values or reverse pre-processed non-reduced color values.

  The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

FIG. 3 is a block diagram illustrating an example video encoding and decoding system configured to implement the techniques of this disclosure. It is a figure which shows the concept of high dynamic range (HDR) data. It is a conceptual diagram comparing the color gamuts of video signals of high definition television (HDTV) (BT.709) and ultra high definition television (UHDTV) (BT.2020). It is a conceptual diagram which shows HDR / WCG expression conversion. It is a conceptual diagram which shows HDR / WCG reverse conversion. It is a conceptual diagram which shows an example transfer function. It is a conceptual diagram which shows the visualization of the output versus input with respect to PQ TF (ST2084 EOTF). It is a conceptual diagram which shows the content adaptive type HDR processing pipeline (encoder side) using an adaptive shape transfer function (TF). It is a conceptual diagram which shows the content adaptive type HDR processing pipeline (encoder side) using fixed TF. It is a conceptual diagram which shows the content adaptive type HDR processing pipeline (decoder side) using fixed TF. It is a conceptual diagram which shows the content adaptive HDR pipeline which uses static TF, and the encoder side. It is a conceptual diagram which shows the content adaptive type HDR pipeline and decoder side using static TF. It is a figure which shows an example of the histogram of the linear optical signal (red color component) of the HDR signal on which PQTF (ST2084) was set | placed for visualization. It is a figure which shows the example of the histogram of the nonlinear signal obtained from applying PQTF (ST2084) to a linear optical signal (red color component). FIG. 6 is a diagram illustrating a histogram output of a non-linear signal generated by processing according to the PQ TF and the techniques described in this disclosure. It is a figure which shows the influence of PQ TF with respect to the signal statistics after a pre-processing. It is a figure which shows the histogram of the output nonlinear signal produced | generated by PQTF. FIG. 4 is a diagram showing a histogram of a normalized nonlinear signal S. It is a figure which shows the histogram of the normalized nonlinear signal S2 after post-processing. It is a figure which shows the nonlinear PQTF for HDR as prescribed | regulated to ST2084. FIG. 6 is a diagram illustrating a linear transfer function y = x using the techniques described in this disclosure, modeled as Scale2 = 1 and Offset2 = 0. FIG. 6 illustrates a linear transfer function using the techniques described in this disclosure, modeled as Scale2 = 1.5 and Offset = −0.25. FIG. 6 is a conceptual diagram illustrating an example in which color conversion is performed after a transfer function is applied and before the post-processing techniques described in this disclosure. It is a conceptual diagram which shows the example in which reverse color conversion is performed after the reverse post-processing technique demonstrated by this indication, and before an inverse transfer function is applied. It is a figure which shows the post-processing accompanying the clipping to the range regarding the histogram of S2. It is another figure which shows the post-processing with the clipping to the range regarding the histogram of S2. It is another figure which shows the post-processing with the clipping to the range regarding the histogram of S2. FIG. 6 shows a histogram of codewords after post-processing techniques described in this disclosure with tail processing. FIG. 6 shows a histogram of codewords after post-processing techniques described in this disclosure with tail processing. It is a conceptual diagram which shows another example by the content adaptive HDR pipeline which uses static TF, and the encoder side. It is a conceptual diagram which shows another example by the content adaptive HDR pipeline which uses static TF, and the decoder side. FIG. 6 shows a histogram using two reserved encodings for processing color values. FIG. 4 illustrates a parameter adaptive function implemented by the techniques described in this disclosure. FIG. 4 illustrates a parameter adaptive function implemented by the techniques described in this disclosure. FIG. 4 is a diagram illustrating post-processing using a piecewise linear transfer function implemented with the techniques described in this disclosure applied to an input signal, and the effect of this post-processing on the histogram of the output signal. FIG. 4 is a diagram illustrating post-processing using a piecewise linear transfer function implemented with the techniques described in this disclosure applied to an input signal, and the effect of this post-processing on the histogram of the output signal. FIG. 4 is a diagram illustrating post-processing using a piecewise linear transfer function implemented with the techniques described in this disclosure applied to an input signal, and the effect of this post-processing on the histogram of the output signal. FIG. 4 is a diagram illustrating a hybrid log gamma transfer function and a potential target range as an example. It is a figure which shows the transfer function in which the gradient of the parabola in the knee point (knee point) vicinity is adjustable. 2 is a flowchart illustrating an exemplary method of video processing in a content adaptive high dynamic range (HDR) system. 6 is a flowchart illustrating another exemplary method of video processing in a content adaptive high dynamic range (HDR) system.

  The present disclosure relates to the field of coding video signals having high dynamic range (HDR) and wide color gamut (WCG) representations. More particularly, the techniques of this disclosure include signaling and operations applied to video data in several color spaces to allow more efficient compression of HDR and WCG video data. The proposed technique may improve the compression efficiency of hybrid-based video coding systems (eg, HEVC-based video coders) that are utilized to code HDR and WCG video data.

  Video coding standards, including hybrid-based video coding standards, include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, Includes ITU-T H.264 (also called ISO / IEC MPEG-4 AVC), including ISO / IEC MPEG-4 Visual, and its scalable video coding (SVC) and multiview video coding (MVC) extensions. The design of a new video coding standard, namely HEVC, was finalized by the Video Coding Joint Research Group (JCT-VC) of the ITU-T Video Coding Expert Group (VCEG) and the ISO / IEC Motion Picture Expert Group (MPEG). Bross et al., `` High efficiency video coding (HEVC) text specification draft 10 (for FDIS & Last Call) '', ITU-T SG16 WP3 and ISO / IEC JTC1 / Video Coordination Working Group with SC29 / WG11 (JCT-VC), 12th Meeting, Geneva, Switzerland, January 14-23, 2013, JCTVC-L1003v34, http://phenix.int-evry.fr Available from /jct/doc_end_user/documents/12_Geneva/wg11/JCTVC-L1003-v34.zip. The established HEVC standard is called HEVC version 1.

  Wang et al., `` High efficiency video coding (HEVC) Defect Report '', Video coding joint research group (JCT-VC) with ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11, 14th Meeting, Vienna, Austria, July 25-August 2, 2013, JCTVC-N1003v1, http://phenix.int-evry.fr/jct/doc_end_user/documents/14_Vienna/wg11/JCTVC-N1003-v1 Available from .zip. The confirmed HEVC standard document is ITU-T H.265, Series H: Audiovisual and Multimedia Systems, Infrastructure of audiovisual services-Coding of moving video, High efficiency video coding, Telecommunication Standardization Division of the International Telecommunication Union (ITU). It was announced as April 2013, and another version was announced in October 2014.

  FIG. 1 is a block diagram illustrating an example video encoding and decoding system 10 that may utilize the techniques of this disclosure. As shown in FIG. 1, the system 10 includes a source device 12 that provides encoded video data to be decoded later by a destination device 14. Specifically, source device 12 provides video data to destination device 14 via computer-readable medium 16. Source device 12 and destination device 14 are desktop computers, notebook (i.e. laptop) computers, tablet computers, set top boxes, telephone handsets such as so-called `` smart '' phones, so-called `` smart '' pads, televisions, cameras, Any of a wide range of devices may be provided, including display devices, digital media players, video game consoles, video streaming devices, and the like. In some cases, source device 12 and destination device 14 may be equipped for wireless communication.

  Destination device 14 may receive encoded video data to be decoded via computer readable medium 16. The computer readable medium 16 may comprise any type of medium or device capable of moving encoded video data from the source device 12 to the destination device 14. In one example, computer readable medium 16 may comprise a communication medium that enables source device 12 to transmit encoded video data directly to destination device 14 in real time. The encoded video data may be modulated according to a communication standard such as a wireless communication protocol and transmitted to the destination device 14. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. Communication media may include routers, switches, base stations, or any other equipment that may be useful for facilitating communication from source device 12 to destination device 14.

  In some examples, encoded data may be output from output interface 22 to a storage device. Similarly, encoded data may be accessed from a storage device by an input interface. The storage device can be a hard drive, Blu-ray® disk, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital for storing encoded video data It can include any of a variety of distributed or locally accessed data storage media, such as storage media. In a further example, the storage device may correspond to a file server or another intermediate storage device that may store the encoded video generated by the source device 12. Destination device 14 may access stored video data from the storage device via streaming or download. The file server can be any type of server that is capable of storing encoded video data and transmitting the encoded video data to the destination device 14. Exemplary file servers include a web server (eg, for a website), an FTP server, a network attached storage (NAS) device, or a local disk drive. Destination device 14 may access the encoded video data through any standard data connection, including an Internet connection. This includes wireless channels (e.g. Wi-Fi connections), wired connections (e.g. DSL, cable modems, etc.) or a combination of both suitable for accessing encoded video data stored on a file server. obtain. The transmission of the encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

  The techniques of this disclosure are not necessarily limited to wireless applications or settings. Techniques include over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission such as dynamic adaptive streaming over HTTP (DASH), digital video encoded on a data storage medium, data storage It may be applied to video coding in supporting any of a variety of multimedia applications, such as decoding digital video stored on a medium, or other applications. In some examples, the system 10 may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcast, and / or video telephony.

  In the example of FIG. 1, the source device 12 includes a video source 18, a video encoding unit 21 that includes a video preprocessor 19 and a video encoder 20, and an output interface 22. The destination device 14 includes an input interface 28, a video decoding unit 29 including a video decoder 30 and a video post processor 31, and a display device 32. In accordance with this disclosure, video preprocessor 19 and video postprocessor 31 may be configured to apply the exemplary techniques described in this disclosure. For example, the video preprocessor 19 and video postprocessor 31 may include a static transfer function unit configured to apply a static transfer function, but a pre-processing unit and a post-processing unit capable of adapting signal characteristics Have

  In other examples, the source device and destination device may include other components or configurations. For example, the source device 12 may receive video data from an external video source 18 such as an external camera. Similarly, destination device 14 may interface with an external display device rather than including an integrated display device.

  The illustrated system 10 of FIG. 1 is only an example. Techniques for processing video data may be performed by any digital video encoding and / or decoding device. In general, the techniques of this disclosure are performed by a video encoding device, but the techniques may also be performed by a video encoder / decoder, commonly referred to as a “codec”. For ease of explanation, this disclosure describes video preprocessor 19 and video postprocessor 31 performing the exemplary techniques described in this disclosure on each of source device 12 and destination device 14. Is done. Source device 12 and destination device 14 are only examples of coding devices in which source device 12 generates coded video data for transmission to destination device 14. In some examples, devices 12, 14 may operate substantially symmetrically such that each of devices 12, 14 includes a video encoding component and a decoding component. Thus, the system 10 may support one-way or two-way video transmission between the video devices 12 and 14, for example for video streaming, video playback, video broadcast, or video telephony.

  The video source 18 of the source device 12 may include a video capture device, such as a video camera, a video archive containing previously captured video, and / or a video feed interface for receiving video from a video content provider. As a further alternative, video source 18 may generate computer graphics-based data as a source video, or a combination of live video, archived video, and computer-generated video. In some cases, if video source 18 is a video camera, source device 12 and destination device 14 may form a so-called camera phone or video phone. However, as described above, the techniques described in this disclosure may be generally applicable to video coding and may be applied to wireless and / or wired applications. In each case, the captured video, pre-captured video, or computer-generated video may be encoded by video encoding unit 21. The encoded video information may then be output on the computer readable medium 16 by the output interface 22.

  The computer readable medium 16 may be a temporary medium such as a wireless broadcast or wired network transmission, or a storage medium such as a hard disk, flash drive, compact disk, digital video disk, Blu-ray® disk, or other computer readable medium ( That is, it may include a non-transitory storage medium. In some examples, a network server (not shown) may receive encoded video data from the source device 12 and provide the encoded video data to the destination device 14, for example, via a network transmission. Similarly, a computing device of a media manufacturing facility, such as a disk stamping facility, may receive encoded video data from source device 12 and manufacture a disc that includes the encoded video data. Accordingly, in various examples, computer readable medium 16 may be understood to include various forms of one or more computer readable media.

  The input interface 28 of the destination device 14 receives information from the computer readable medium 16. The information of the computer readable medium 16 may include syntax information defined by the video encoder 20 of the video encoding unit 21 and used by the video decoder 30 of the video decoding unit 29, where the syntax information includes blocks and Includes syntax elements that describe the characteristics and / or processing of other coding units, eg, picture groups (GOPs). The display device 32 displays the decoded video data to the user and various displays such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light emitting diode (OLED) display, or another type of display device. Any of the devices may be provided.

  As shown, video preprocessor 19 receives video data from video source 18. Video preprocessor 19 may be configured to process the video data and convert it to a form suitable for encoding using video encoder 20. For example, video preprocessor 19 may perform dynamic range reduction (eg, using a non-linear transfer function), color conversion to a more compact or robust color space, and / or conversion from a floating representation to an integer representation. Video encoder 20 may perform video encoding on the video data output by video preprocessor 19. Video decoder 30 may perform the inverse of video encoder 20 to decode the video data, and video postprocessor 31 may perform the inverse of video preprocessor 19 to convert the video data into a form suitable for display. For example, video post processor 31 may perform the inverse of integer to floating conversion, color conversion from a compact or robust color space, and / or dynamic range reduction to produce video data suitable for display. .

  Video encoding unit 21 and video decoding unit 29 are each one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, It can be implemented as any of a variety of fixed function and programmable circuit configurations, such as hardware, firmware, or any combination thereof. When the technique is partially implemented in software, the device may store instructions for the software in a suitable non-transitory computer readable medium and execute the instructions in hardware using one or more processors Thus, the techniques of this disclosure may be performed. Each of video encoding unit 21 and video decoding unit 29 may be included in one or more encoders or decoders, both of which are part of a combined encoder / decoder (codec) in the respective device. May be integrated.

  Video preprocessor 19 and video encoder 20 are illustrated as being separate units within video encoding unit 21, and video post-processor 31 and video decoder 30 are illustrated as being separate units within video decoding unit 29. However, the techniques described in this disclosure are not so limited. Video preprocessor 19 and video encoder 20 may be formed as a common device (eg, the same integrated circuit or housed in the same chip or chip package). Similarly, video post processor 31 and video decoder 30 may be formed as a common device (eg, the same integrated circuit or housed in the same chip or chip package).

  In some examples, video encoder 20 and video decoder 30 are ISO / IEC, including their scalable video coding (SVC) extension, multi-view video coding (MVC) extension, and MVC-based 3D video (3DV) extension. It operates according to video compression standards such as MPEG-4 Visual and ITU-T H.264 (also called ISO / IEC MPEG-4 AVC). In some cases, any bitstream that conforms to MVC-based 3DV always includes a sub-bitstream that conforms to an MVC profile, eg, a stereo high profile. Furthermore, work is underway to generate 3DV coding extensions to H.264 / AVC, ie AVC-based 3DV. Other examples of video coding standards are ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG -4 Visual, ITU-T H.264, ISO / IEC Visual included. In other examples, video encoder 20 and video decoder 30 may be configured to operate according to the ITU-T H.265, HEVC standard.

In HEVC and other video coding standards, a video sequence typically includes a series of pictures. A picture is sometimes called a “frame”. A picture may include three sample arrays denoted S _L , S _Cb , and S _Cr . S _L is a two-dimensional array (ie, block) of luma samples. S _Cb is a two-dimensional array of Cb chrominance samples. S _Cr is a two-dimensional array of Cr chrominance samples. A chrominance sample may also be referred to herein as a “chroma” sample. In other cases, the picture may be monochrome and may contain only an array of luma samples.

  Video encoder 20 may generate a set of coding tree units (CTUs). Each of the CTUs may comprise a syntax structure used to code a coding tree block of luma samples, two corresponding coding tree blocks of chroma samples, and a sample of coding tree blocks. For a monochrome picture, or a picture with three separate color planes, the CTU may comprise a single coding tree block and a syntax structure used to code samples of the coding tree block. The coding tree block may be an N × N block of samples. A CTU may also be referred to as a “tree block” or “maximum coding unit” (LCU). The HEVC CTU may be roughly similar to macroblocks of other video coding standards such as H.264 / AVC. However, a CTU is not necessarily limited to a particular size and may include one or more coding units (CUs). A slice may include an integer number of CTUs sequentially ordered in a raster scan.

  This disclosure refers to “video unit” or “video block” to refer to one or more blocks of samples and the syntax structure used to code samples of one or more blocks of samples. The term is sometimes used. Exemplary types of video units may include CTUs, CUs, PUs, transform units (TUs) in HEVC, macroblocks, macroblock partitions, etc. in other video coding standards.

  Video encoder 20 may partition the coding block of the CU into one or more prediction blocks. A prediction block may be a rectangular (ie, square or non-square) block of samples to which the same prediction is applied. The prediction unit (PU) of a CU may comprise a syntax structure used to predict a prediction block of luma samples, two corresponding prediction blocks of chroma samples, and a prediction block sample of a picture. For a monochrome picture, or a picture with three separate color planes, the PU may comprise a single predicted block and a syntax structure used to predict the predicted block samples. Video encoder 20 may generate a prediction luma block, a prediction Cb block, and a prediction Cr block for the luma prediction block, Cb prediction block, and Cr prediction block of each PU of the CU.

  Video encoder 20 may use intra prediction or inter prediction to generate a prediction block for the PU. If video encoder 20 uses intra prediction to generate a PU prediction block, video encoder 20 may generate a PU prediction block based on decoded samples of pictures associated with the PU.

  After video encoder 20 generates a predicted luma block, predicted Cb block, and predicted Cr block for one or more PUs of the CU, video encoder 20 may generate a luma residual block for the CU. Each sample in the CU's luma residual block indicates the difference between the luma sample in one of the CU's predicted luma blocks and the corresponding sample in the CU's original luma coding block. In addition, video encoder 20 may generate a Cb residual block for the CU. Each sample in the CU's Cb residual block may indicate the difference between the Cb sample in one of the CU's predicted Cb blocks and the corresponding sample in the CU's original Cb coding block . Video encoder 20 may also generate a Cr residual block for the CU. Each sample in the CU's Cr residual block may indicate the difference between the Cr sample in one of the CU's predicted Cr blocks and the corresponding sample in the CU's original Cr coding block .

  In addition, video encoder 20 uses quadtree partitioning to convert a CU's luma residual block, Cb residual block, and Cr residual block into one or more luma transform blocks, Cb transform blocks, and Cr Can be broken down into transform blocks. The transform block may be a sample rectangular block to which the same transform is applied. The transform unit (TU) of the CU may comprise a luma sample transform block, two corresponding transform blocks of chroma samples, and a syntax structure used to transform the transform block samples. For a monochrome picture, or a picture with three separate color planes, the TU may comprise a single transform block and a syntax structure that is used to transform transform block samples. Thus, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. The luma transform block associated with the TU may be a sub-block of the CU's luma residual block. The Cb transform block may be a sub-block of the CU's Cb residual block. The Cr transform block may be a sub-block of the CU's Cr residual block.

  Video encoder 20 may apply one or more transforms to the TU's luma transform block to generate a luma coefficient block for the TU. The coefficient block can be a two-dimensional array of transform coefficients. The conversion factor can be a scalar quantity. Video encoder 20 may apply one or more transforms to the TU's Cb transform block to generate a Cb coefficient block for the TU. Video encoder 20 may apply one or more transforms to the TU Cr transform block to generate a Cr coefficient block for the TU.

  After generating a coefficient block (eg, luma coefficient block, Cb coefficient block, or Cr coefficient block), video encoder 20 may quantize the coefficient block. Quantization generally refers to a process in which transform coefficients are quantized to reduce as much as possible the amount of data used to represent the transform coefficients, resulting in further compression. Further, video encoder 20 may dequantize the transform coefficients and apply the inverse transform to the transform coefficients to reconstruct the transform block of the CU TU of the picture. Video encoder 20 may use the reconstructed transform block of CU TU and the prediction block of PU of CU to reconstruct the coding block of CU. By reconstructing the coding block for each CU of the picture, video encoder 20 may reconstruct the picture. Video encoder 20 may store the reconstructed picture in a decoded picture buffer (DPB). Video encoder 20 may use the reconstructed picture in the DPB for inter prediction and intra prediction.

  After video encoder 20 quantizes the coefficient block, video encoder 20 may entropy encode syntax elements indicating quantized transform coefficients. For example, video encoder 20 may perform context adaptive binary arithmetic coding (CABAC) on syntax elements indicating quantized transform coefficients. Video encoder 20 may output entropy encoded syntax elements in the bitstream.

  Video encoder 20 may output a bitstream that includes a representation of a coded picture and a sequence of bits that form associated data. The bitstream may comprise a sequence of network abstraction layer (NAL) units. Each NAL unit includes a NAL unit header and encapsulates a raw byte sequence payload (RBSP). The NAL unit header may include a syntax element that indicates a NAL unit type code. The NAL unit type code specified by the NAL unit header of the NAL unit indicates the type of the NAL unit. An RBSP may be a syntax structure that includes an integer number of bytes encapsulated within a NAL unit. In some cases, the RBSP includes 0 bits.

  Different types of NAL units may encapsulate different types of RBSP. For example, a first type of NAL unit may encapsulate an RBSP for a picture parameter set (PPS), a second type of NAL unit may encapsulate an RBSP for a coded slice, and a third type of NAL unit Can encapsulate RBSP for supplemental enhancement information (SEI), and so on. PPS is a syntax structure that may include syntax elements that apply to zero or more coded pictures as a whole. A NAL unit that encapsulates an RBSP for video coding data (as opposed to an RBSP for parameter sets and SEI messages) may be referred to as a video coding layer (VCL) NAL unit. A NAL unit that encapsulates a coded slice may be referred to herein as a coded slice NAL unit. An RBSP for a coded slice may include a slice header and slice data.

  Video decoder 30 may receive the bitstream. In addition, video decoder 30 may parse the bitstream to decode syntax elements from the bitstream. Video decoder 30 may reconstruct a picture of the video data based at least in part on syntax elements decoded from the bitstream. The process for reconstructing video data may generally be the opposite of the process performed by video encoder 20. For example, video decoder 30 may use the motion vector of the PU to determine a prediction block for the PU of the current CU. Video decoder 30 may use one or more motion vectors of the PU to generate a prediction block for the PU.

  In addition, video decoder 30 may dequantize the coefficient block associated with the current CU's TU. Video decoder 30 may perform an inverse transform on the coefficient block to reconstruct the transform block associated with the current CU's TU. Video decoder 30 may reconstruct the current CU coding block by adding the sample of the predicted sample block for the current CU PU to the corresponding sample of the transform block of the current CU TU. By reconstructing the coding block for each CU of the picture, video decoder 30 may reconstruct the picture. Video decoder 30 may store the decoded picture in a decoded picture buffer for output and / or use in decoding other pictures.

  Next generation video applications are expected to work with video data representing captured scenes with HDR (High Dynamic Range) and WCG (Wide Color Gamut). The dynamic range and color gamut parameters utilized are two independent attributes of video content, and their specifications for digital television and multimedia services are defined by several international standards. For example, ITU-R Rec. 709 specifies parameters for HDTV (high definition television) such as standard dynamic range (SDR) and standard color gamut, while ITU-R Rec. 2020 specifies parameters such as HDR and WCG. Specifies UHDTV (ultra high definition television) parameters. There are other standards developing organization (SDO) documents that specify dynamic range and color gamut attributes in other systems, for example, the P3 color gamut is specified in SMPTE (American Film and Television Engineers Association) -231-2 Some parameters of HDR are defined in SMPTE-2084. A brief description of dynamic range and color gamut for video data is provided below.

  The dynamic range is usually defined as the ratio between the minimum and maximum luminance of the video signal. The dynamic range may also be measured in terms of “f-stop”, where 1 f-stop corresponds to doubling the signal dynamic range. According to the MPEG specification, HDR content is content characterized by luminance fluctuations exceeding 16 f stops. In some terms, the level between the 10f stop and the 16f stop is considered an intermediate dynamic range, but it may be considered HDR in other specifications. In some examples, the HDR video content is any video having a higher dynamic range than conventionally used video content having a standard dynamic range (e.g., video content specified by ITU-R Rec. BT.709). Can be content. At the same time, the human visual system (HVS) can perceive a much larger dynamic range. However, HVS includes an adaptive mechanism that narrows the so-called simultaneous range. The dynamic range visualization provided by HDTV SDR, expected UHDTV HDR, and HVS dynamic range is shown in FIG.

  Current video applications and services are regulated by Rec. 709 and provide SDR, usually in the range of luminance (i.e., luminance) near 0.1-100 candela (cd) / m2 (often called `` Nito '') Support and lead to less than 10f stop. Next-generation video services are expected to provide a dynamic range up to 16f stop. Detailed specifications are currently under development, but some initial parameters are specified in SMPTE-2084 and Rec.2020.

  Another aspect for a more realistic video experience besides HDR is color dimension, which is traditionally defined by a color gamut. Figure 3 shows the SDR gamut (triangles based on the red, green, and blue primaries of the BT.709 color) and the wider gamut for the UHDTV (red, green, and blue primaries of the BT.2020 color). FIG. FIG. 3 also shows a so-called spectral trajectory (defined by a tongue-shaped area) that represents an area of natural color. As illustrated by Figure 3, the move from the BT.709 primary to the BT.2020 primary aims to provide UHDTV services with about 70% increase in color. D65 specifies the white color for a given specification (eg, BT.709 specification and / or BT.2020 specification).

  Some examples of color gamut specifications are shown in Table 1.

  As can be seen in Table 1, the color gamut depends on the X and Y values of the white point, as well as the X values and the primary colors (e.g., red (R), green (G), and blue (B)). Can be defined by a Y value. The X and Y values represent the chromaticity (X) and luminance (Y) of the color, as defined by the CIE 1931 color space. The CIE 1931 color space defines a link between pure colors (eg, from a wavelength perspective) and how the human eye perceives such colors.

  HDR / WCG is typically collected and stored with extremely high accuracy (and even floating point) per component, using a 4: 4: 4 chroma format and a very wide color space (e.g. CIE1931 XYZ color space). . This representation targets high accuracy and is mathematically (almost) lossless. However, this format feature can include a lot of redundancy and is not optimal for compression purposes. Lower precision formats with HVS-based assumptions are typically used for advanced video applications.

  General video data format conversion for compression consists of three main processes as shown in FIG. The technique of FIG. 4 may be performed by the video preprocessor 19. The linear RGB data 110 may be HDR / WCG video data and may be stored in a floating point representation. The linear RGB data 110 can be shortened using a non-linear transfer function (TF) for dynamic range reduction. For example, video preprocessor 19 may include a transfer function (TF) unit 112 configured to use a non-linear transfer function for dynamic range reduction.

  The output of the TF unit 112 can be a set of codewords, where each codeword represents a range of color values (eg, illumination levels). The dynamic range shortening means that the dynamic range of the linear RGB data 110 can be the first dynamic range (for example, the human visual range as shown in FIG. 2). The dynamic range of the obtained codeword may be a second dynamic range (eg, HDR display range as shown in FIG. 2). Thus, the codeword captures a smaller dynamic range than the linear RGB data 110, and thus the TF unit 112 performs dynamic range shortening.

  The TF unit 112 performs a non-linear function in the sense that the mapping between codewords and input color values is not equally spaced (eg, the codeword is a non-linear codeword). Non-linear codeword means that the change in the input color value appears as a non-linear change in the codeword rather than as a linearly proportional change in the output codeword. For example, if the color value represents a low illuminance, a small change in the input color value will result in a small change in the codeword output by the TF unit 112. However, if the color value represents a high illuminance, a relatively large change in the input color value will be required for a small change in the codeword. The illuminance range represented by each codeword is not constant (e.g., the first codeword is the same for the first range of illuminance and the second codeword is for the second range of illuminance) And the first and second ranges are different). FIG. 7 described below shows the characteristics of the transfer function applied by the TF unit 112.

  As described in more detail, the technique may scale and offset the linear RGB data 110 received by the TF unit 112 and / or the code output by the TF unit 112 to better utilize the codeword space. Words can be scaled and offset. The TF unit 112 may be linear RGB data 110 (or scaled and offset) using any number of non-linear transfer functions (e.g., PQ (Perceptual Quantizer) TF as specified in SMPTE-2084). RGB data) can be shortened.

  In some examples, the color conversion unit 114 converts the shortened data into a more compact or robust color space that is more suitable for compression by the video encoder 20 (e.g., via the color conversion unit via the YUV color space or YCrCb Convert (in color space). As described in more detail, in some examples, the technique may scale and offset the codeword output by the TF application by the TF unit 112 before the color conversion unit 114 performs the color conversion. Color conversion unit 114 may receive these scaled and offset codewords. In some examples, some scaled and offset codewords may be larger or smaller than their respective thresholds, for which the technique may assign their respective set codewords. .

  This data is then encoded using a floating representation to integer representation (e.g., quantum representation) to generate video data (e.g., HDR data 118) that is to be encoded and transmitted to video encoder 20. Quantized (through the quantization unit 116). In this example, the HDR data 118 is an integer representation. HDR data 118 may currently be in a more appropriate format for compression by video encoder 20. It should be understood that the process sequence shown in FIG. 4 is given as an example and may be different in other applications. For example, color conversion may precede the TF process. In addition, video preprocessor 19 may apply more processing (eg, spatial subsampling) to the color components.

  Thus, in FIG. 4, the high dynamic range of input RGB data 110 in linear and floating point representations is shortened using a non-linear transfer function utilized by TF unit 112, for example, PQ TF as defined in SMPTE-2084 Followed by conversion to a target color space more suitable for compression, e.g., YCbCr (e.g., by color conversion unit 114), and then quantized to obtain an integer representation (e.g., quantization unit 116) . The order of these elements is given as an example and may be different in real world applications, for example, color conversion may precede a TF module (eg, TF unit 112). Additional processing such as spatial subsampling may be applied to the color components before the TF unit 112 applies the transfer function.

  The inverse transformation on the decoder side is shown in FIG. The technique of FIG. 5 may be performed by the video post processor 31. For example, the video post processor 31 receives video data (e.g., HDR data 120) from the video decoder 30, the inverse quantization unit 122 can inverse quantize the data, the inverse color conversion unit 124 performs the inverse color conversion, An inverse nonlinear transfer function unit 126 performs inverse nonlinear transfer to generate linear RGB data 128.

  The reverse color conversion process performed by the reverse color conversion unit 124 may be the reverse of the color conversion process performed by the color conversion unit 114. For example, the inverse color conversion unit 124 may convert the HDR data back from the YCrCb format to the RGB format. The inverse transfer function unit 126 may apply the inverse transfer function to the data to adjust the dynamic range shortened by the TF unit 112 to recreate the linear RGB data 128.

  In the exemplary techniques described in this disclosure, video postprocessor 31 may apply inverse post-processing before inverse transfer function unit 126 performs the inverse transfer function, and inverse transfer function unit 126 may apply the inverse transfer function. After execution, reverse preprocessing may be applied. For example, as described above, in some examples, video preprocessor 19 may apply preprocessing (eg, scaling and offsetting) before TF unit 112, and postprocessing after TF unit 112. (Eg, scaling and offsetting) may be applied. To compensate for pre-processing and post-processing, video post-processor 31 may apply reverse post-processing before inverse TF unit 126 performs the reverse transfer function, and after reverse TF unit 126 executes the reverse transfer function. Reverse pre-processing can be applied. It is optional to apply both pre-processing and post-processing and reverse post-processing and reverse pre-processing. In some examples, the video preprocessor 19 may apply one instead of both pre-processing and post-processing, and in such an example, the video post-processor 31 may be responsible for the processing applied by the video pre-processor 19. The reverse can be applied.

  The example video preprocessor 19 shown in FIG. 4 is described in further detail, with the understanding that the example video postprocessor 31 shown in FIG. A transfer function is applied to the data (eg, HDR / WCG RGB video data) to reduce its dynamic range and allow the data to be represented using a limited number of bits. These limited number of bits representing data are called codewords. This function typically reflects the inverse of the end-user display electro-optical transfer function (EOTF) as specified for SDR in Rec. 709, or SMPTE-2084 for HDR A one-dimensional (1D) nonlinear function that either approximates HVS perception to luminance changes, as for the PQ TF specified in. The reverse process of OETF (eg, as performed by video postprocessor 31) is EOTF (electro-optic transfer function), which maps code levels back to luminance. FIG. 6 shows some examples of nonlinear TFs. These transfer functions can also be applied to each R, G, and B component separately.

  In the context of this disclosure, the terms “signal value” or “color value” are used to describe the luminance level corresponding to the value of a particular color component (such as R, G, B, or Y) for an image element. Can be used. The signal value usually represents a linear light level (luminance value). The terms “code level”, “digital code value”, or “code word” may refer to a digital representation of an image signal value. Such digital representations typically represent non-linear signal values. EOTF represents the relationship between a non-linear signal value provided to a display device (eg, display device 32) and a linear color value generated by the display device.

  The ST2084 specification stipulated the application of EOTF as follows. TF is applied to the normalized linear R, G, B values, which results in a non-linear representation R'G'B '. ST2084 specifies normalization with NORM = 10000 related to peak luminance as 10000 nits (cd / m 2).

  Using the input values on the x-axis (linear color values) normalized to the range 0-1 and the normalized output values (non-linear color values) on the y-axis, PQ EOTF is visualized in FIG. As can be seen from the curve in FIG. 7, 1 percent (low illumination) of the dynamic range of the input signal is converted to 50% of the dynamic range of the output signal.

  There are a finite number of codewords that can be used to represent the input linear color value. Figure 7 shows that for PQ EOTF, nearly 50% of the available codewords are dedicated to low-light input signals, leaving fewer available codewords for higher-light input signals. Show. Thus, slight changes in the relatively bright input signal may not be captured because there are insufficient codewords to represent these slight changes. However, there can be an unnecessarily large number of available codewords for low illumination input signals, which can be represented by different codewords, even the slightest changes in relatively low illumination input signals Means that. Thus, there can be a large distribution of codewords available for input signals with low illuminance and a relatively small distribution of codewords available for input signals with high illuminance.

  Normally, EOTF is specified as a function with floating point precision. Therefore, when inverse TF, that is, so-called OETF is applied, an error is not introduced into a signal having this nonlinearity. The inverse TF (OETF) specified in ST2084 is defined as an inverse PQ function.

  With floating point precision, sequential application of EOTF and OETF results in complete reconstruction without error. However, this representation is not optimal for streaming services or broadcasting services. A more compact representation of non-linear R'G'B 'data with constant bit precision is described below. Note that EOTF and OETF are currently the subject of very active research, and the TF utilized in some HDR video coding systems may differ from ST2084.

  Other transfer functions are being considered for HDR. Examples include Philips TF or BBC hybrid “log gamma” TF. The BBC Hybrid “Loggamma” TF is based on the Rec709 TF for SDR backward compatibility. To extend the dynamic range, the BBC hybrid adds a third part to the Rec709 curve at higher input luminance. The new part of the curve is a logarithmic function.

  The OETF specified in Rec709 is

Where L is the luminance of the image 0 ≦ L ≦ 1 and V is the corresponding electrical signal. In Rec2020, the same formula is specified as follows:

Where E is a voltage that is normalized by the reference white level and proportional to the unconditional light intensity that will be detected using the reference camera color channels R, G, B. E ′ is the obtained nonlinear signal.
For a 10-bit system, α = 1.099 and β = 0.018.
For a 12-bit system, α = 1.0993 and β = 0.0181.

Although not explicitly stated, α and β are solutions to the following simultaneous equations.
^{4.5β = αβ 0.45 - (α-} 1)
4.5 = 0.45αβ ^-0.55

  The first expression is to treat the values of the linear function and the gamma function at E = β equally, and the second expression is also the slope of the two functions at E = β.

  Additional portions of the hybrid “log gamma” are shown below, and FIG. 27 shows the OETF. For example, FIG. 27 shows, as an example, a hybrid log gamma transfer function and potential target range.

  In FIG. 27, each curve is marked with A, B, C, D, and E to match each legend in the lower right box of the graph. Specifically, A supports 10-bit PQ-EOTF, B supports 8-bit BT.709 EOTF, C supports 10-bit BT.709 EOTF, and D supports 10-bit BBC hybrid log gamma EOTF E corresponds to 12-bit PQ-EOTF.

FIG. 28 shows a transfer function with adjustable parabola slope near the knee point. In FIG. 28, the gradient adjustment example is merely illustrated as an example of 5000 [cd / m ² ]. However, other examples of transfer functions with adjustable slope are possible.

  RGB data is usually generated by an image capturing sensor and is used as an input. However, this color space has great redundancy between its components and is not optimal for a compact representation. In order to achieve a more compact and more robust representation, the RGB components are usually converted into a less correlated color space, eg YCbCr, that is more appropriate for compression (ie color conversion is performed). This color space separates luminance in the form of luminance information and color information into different uncorrelated components.

  For modern video coding systems, a commonly used color space is YCbCr as specified in ITU-R BT.709 or ITU-R BT.709. The YCbCr color space in the BT.709 standard specifies the following conversion process from R'G'B 'to Y'CbCr (unsteady luminance representation).

The above can also be implemented using the following approximate transformation that avoids division on the Cb and Cr components.
・ Y '= 0.212600 * R' + 0.715200 * G '+ 0.072200 * B'
・ Cb = -0.114572 * R'-0.385428 * G '+ 0.500000 * B' (4)
・ Cr = 0.500000 * R'-0.454153 * G'-0.045847 * B '

  The ITU-R BT.2020 standard specifies the following conversion process from R'G'B 'to Y'CbCr (unsteady luminance representation).

The above can also be implemented using the following approximate transformation that avoids division on the Cb and Cr components.
・ Y '= 0.262700 * R' + 0.678000 * G '+ 0.059300 * B'
・ Cb = -0.139630 * R'-0.360370 * G '+ 0.500000 * B' (6)
・ Cr = 0.500000 * R'-0.459786 * G'-0.040214 * B '

  Note that both color spaces remain normalized. Thus, for input values normalized within the range 0-1 the resulting value is mapped to the range 0-1. In general, color conversion performed with floating point precision results in complete reconstruction, and thus the process is lossless.

  In the case of quantization and / or fixed-point transformations, processing the steps described above is usually performed with a floating-point precision representation, so they can be considered lossless. However, this type of accuracy can be considered redundant and expensive for most consumer electronics applications. For such services, input data in the target color space is converted to a target bit depth with fixed point precision. Several studies show that 10-12 bit accuracy combined with PQ TF is sufficient to provide 16f-stop HDR data with just less than a noticeable difference in distortion. Data represented using 10-bit precision can be further coded using most of the most advanced video coding solutions. This transformation process involves signal quantization, is a component of lossy coding, and is the source of inaccuracy introduced into the transformed data.

An example of such quantization applied to a codeword in the target color space, in this example YCbCr, is shown below. An input value YCbCr expressed in floating-point precision is converted into a signal having a fixed bit depth, that is, BitDepthY for a Y value and BitDepthC for a chroma value (Cb, Cr).
・ D _{Y '} = Clip1 _Y (Round ((1 << (BitDepth _Y -8)) * (219 * Y' + 16)))
・ D _Cb = Clip1 _C (Round ((1 << (BitDepth _C -8)) * (224 * Cb + 128))) (7)
・ D _Cr = Clip1 _C (Round ((1 << (BitDepth _C -8)) * (224 * Cr + 128)))
However,
Round (x) = Sign (x) * Floor (Abs (x) +0.5),
Sign (x) =-1 (if x <0), 0 (if x = 0), 1 (if x> 0),
Floor (x) is the largest integer less than or equal to x,
Abs (x) = x (if x> = 0), -x (if x <0),
Clip1 _Y (x) = Clip3 (0, (1 << BitDepth _Y ) -1, x),
Clip1 _C (x) = Clip3 (0, (1 << BitDepth _C ) -1, x),
Clip3 (x, y, z) = x (if z <x), y (if z> y), x (otherwise)
It is.

  Some techniques may not be suitable for HDR and WCG video data. Most of the currently published EOTFs for HDR video systems, such as PH, Phillips, and BBC, are independently applied to the R, G, and B components, and the spatiotemporal statistics or locality of the HDR video It is a static, content-independent 1D transfer function that does not take into account the brightness level. From the perspective of such EOTF usage in HDR video coding systems, such an approach will lead to bit allocation that is not optimal for the visual quality of the HDR video content provided.

  A second problem with currently utilized HDR processing pipelines is the static conversion of non-linear codewords in the target color space from floating point precision representations to representations. Usually, the codeword space is fixed, and HDR content that changes temporally and spatially will not get the optimal representation. This that follows provides further details on these two issues.

  Regarding the non-optimality of static 1D EOTF, the EOTF specified in ST2084 is probably shown as a PQ TF based on the perceptual sensitivity of the human visual system (HVS) at a specific level of brightness (numerical value as cd / m2) Specify a static, content-independent 1D transfer function. Despite the fact that most studies specify HVS perceptual sensitivity to luminance in cd / m2 units, PQ TF is as in Equation 1 (for example, to determine R ', G', and B ' ) Independently applied to each of the color values R, G, and B and does not utilize the luminance intensity of this RGB pixel. This leads to an estimated inaccuracy of the PQ TF in the approximation of the HVS sensitivity in the obtained R'G'B 'nonlinearity, for example, the combination of R'G'B' color values independently It can result in different levels of brightness that should be associated with different PQ TF values compared to those applied to each of the R, G, B components.

  Moreover, due to this design intent, the PQ TF combines the sensitivity of HVS in two modes: the so-called nighttime photopic vision and the scotopic vision (so-called nighttime) vision. The latter vision is effective when the scene brightness is less than 0.03 cd / m2, and features much higher sensitivity at the expense of reduced color perception. In order to allow high sensitivity, the PQ TF utilized gives a larger amount of codewords to lower illumination values, as reflected in FIG. Such codeword distribution may be optimal when the picture brightness is HVS and will operate in night vision mode. For example, such a codeword distribution may be optimal when the codeword is sensitive to minor changes in brightness. However, a typical HDR picture may be characterized by a bright scene and dark and noisy fragments, which darkness and noisy fragments will result in visual quality due to masking from the nearest bright sample. Although it will not affect, it will greatly contribute to the bit rate using the current static TF.

For inefficient use of codeword space, quantization of nonlinear codewords (Y ', Cb, Cr) expressed in floating-point precision and those using a fixed number of bits as shown in Equation (7) The representation (D _{Y ′} , D _Cb , D _Cr ) is the main compression tool of the HDR pipeline. Usually, the dynamic range of the input signal before quantization is less than 1.0 and belongs to the range within 0 to 1 for the Y component and the range -0.5 to 0.5 for the Cb and Cr components.

  However, the actual distribution of the HDR signal varies from frame to frame, so quantization as shown in equation (7) will not give the smallest quantization error and is consistent with the expected range equal to 1.0. This can be improved by adjusting the dynamic range of the HDR signal.

  To address the problems described above, the following solutions can be considered. As shown in FIG. 8, EOTF can be defined as a dynamic content-adaptive transfer function whose shape changes depending on content characteristics, for example, at the frame level. FIG. 8 shows a content adaptive HDR processing pipeline (encoder side) having an adaptive shape TF unit 112 ′. For example, FIG. 8 shows another example of a video preprocessor 19 that includes an adaptive shape TF function utilized by a TF unit.

  The components of FIG. 8 generally fit the components of FIG. In the example of FIG. 8, components having the same reference numbers as the reference numbers of the example in FIG. 4 are the same. However, instead of the TF unit 112 applying the static TF as shown in FIG. 4, the adaptive TF unit 112 ′ applies the adaptive TF in FIG. In this example, the manner in which adaptive TF unit 112 'performs data reduction is adaptive and can vary based on the video content. Although a corresponding video post-processor 31 is not shown, such an exemplary video post-processor 31 will perform an adaptive inverse transfer function rather than the static inverse transfer function of the inverse transfer function unit 126 of FIG. . The video preprocessor 19 will output information indicating parameters used by the TF unit 112 ′ to adapt the TF. Video postprocessor 31 will receive information indicative of parameters used by TF unit 112 'to adapt the adaptive inverse transfer function accordingly.

  However, this technique may require extensive signaling of transfer function adaptation parameters, as well as implementations that support this adaptability, eg, storing multiple lookup tables or implementation branches. Moreover, some aspects of PQ EOTF non-optimality can be solved through 3D transfer functions. For some implementations and for signaling costs, such an approach can be too expensive.

  In this disclosure, signaling refers to the output of syntax elements or other video data used to decode or otherwise reconstruct video data. The signaling parameters may be stored for later retrieval by the destination device 14 or may be sent directly to the destination device 14.

  This disclosure describes a content-adaptive HDR video system that employs a static fixed transfer function (TF). This disclosure describes keeping static TF in the pipeline but adapting signal characteristics to a fixed processing flow. This can be achieved by adaptive processing of either the signal to be processed by the TF or the signal obtained from the application of the TF. Some techniques may combine both of these adaptation mechanisms. At the decoder (eg, video postprocessor 31), an adaptation process opposite to that applied at the encoder side (eg, video preprocessor 19) will be applied.

  FIG. 9 is a conceptual diagram showing a content adaptive HDR processing pipeline (encoder side) using a fixed TF. As shown, the video preprocessor 19 includes a pre-processing unit 134, also called dynamic range adjustment (DRA1), a TF unit 112, a post-processing unit 138, also called DRA2, a color conversion unit 114, and a quantization unit 116.

  In the illustrated example, the video preprocessor 19 may be configured as a fixed function and programmable circuit configuration. For example, video preprocessor 19 includes transistors, capacitors, inductors, passive elements and active elements that together or separately form pre-processing unit 134, TF unit 112, post-processing unit 138, color conversion unit 114, and quantization unit 116. , An arithmetic logic unit (ALU), an elementary function unit (EFU), and the like. In some examples, video preprocessor 19 is programmable to execute instructions that cause pre-processing unit 134, TF unit 112, post-processing unit 138, color conversion unit 114, and quantization unit 116 to perform their respective functions. Includes core. In such an example, video data memory 132 or some other memory may store instructions executed by video preprocessor 19.

  In FIG. 9, a video data memory 132 is also shown for ease of understanding. For example, video data memory 132 may temporarily store video data before video preprocessor 19 receives the video data. As another example, any video data output by the video preprocessor 19 may be temporarily stored in the video data memory 132 (eg, stored in the video data memory before being output to the video encoder 20). ). Video data memory 132 may be part of video preprocessor 19 or may be external to video preprocessor 19.

  Video data stored in video data memory 132 may be obtained from video source 18, for example. The video data memory 132 may be a variety of dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM®), or other types of memory devices. It can be formed by any of the memory devices. In various examples, video data memory 132 may be on-chip with other components of video preprocessor 19 or may be off-chip with respect to those components.

  As described in more detail below, preprocessing unit 134 (eg, DRA1) preprocesses linear RGB data 110 before static transfer function unit 112 applies the static transfer function. Part of the preprocessing includes scaling and offsetting (eg, scaling the input value by a factor and adding the value to offset). In some examples, preprocessing unit 134 preprocesses video data based on video content (eg, scaling and offset factors are based on video content). The TF unit 112 then applies a transfer function to the scaled and offset input values to generate a plurality of codewords. In this example, the TF unit 112 is a static that reduces the dynamic range of input values so that the generated codewords represent colors with a smaller dynamic range than the input color values (e.g., linear RGB data 110). Apply the transfer function.

  Post-processing unit 136 (eg, DRA2) performs a post-processing function on the codeword generated by TF unit 112. For example, the output of TF unit 112 may be a non-linear codeword that represents a color value. Post processing unit 136 may also scale and offset the codeword output by TF unit 112. The color conversion unit 114 performs color conversion (for example, conversion from RGB to YCrCb) on the output of the post-processing unit 138, and the quantization unit 116 performs quantization on the output of the color conversion unit 114 To do.

  By pre-processing the linear RGB data 110, the pre-processing unit 134 can be configured to scale and offset the linear RGB data 110 so that the TF unit 112 applies a static transfer function while the TF unit The 112 inputs are adjusted so that there is greater linearity in the output codeword of the TF unit 112. Further, by preprocessing linear RGB data 110, preprocessing unit 134 may use video content to select scaling and offset parameters that take into account spatiotemporal statistics or local luminance levels. .

  There can be various ways in which the preprocessing unit 134 can determine the scaling and offset parameters. As an example, preprocessing unit 134 may determine a histogram for each of the colors in the picture (eg, determine a histogram for red, a histogram for green, and a histogram for blue). The histogram shows how many pixels of a particular color are at a particular illumination level. The pre-processing unit 134 can be configured to transfer and offset the color values to enhance the linearity of the codeword output by the TF unit 112 so that the TF unit 112 can apply the transfer function to be applied. It can be preprogrammed using mathematical expressions.

The preprocessing unit 134 may normalize the histogram of the input signal over a limited range [hist_min to hist_max] over the entire dynamic range [0 to 1] of the allowed codeword space of the input signal. Preprocessing unit 134 determines the scaling and offset parameters to be applied to the input signal by horizontally stretching each histogram of color values and determining the offset and scale based on the stretched histogram. obtain. For example, the equations for determining offset and scale may be as follows:
Offset1 = -hist_min
Scale1 = 1 / (hist_max-hist_min)

  Preprocessing unit 134 may determine offset and scaling parameters for each of the colors using equations for Offset1 and Scale1, and applies the offset1 and scale1 parameters to scale and color values as described in more detail. Can be offset. There may be various other ways in which the preprocessing unit 134 determines offset and scaling parameters for preprocessing, the above equation being provided as an example.

Except for post processing unit 138 applying offset and scaling parameters to the color codeword, post processing unit 138 may determine offset and scaling parameters in a similar manner. For example, post-processing unit 138 may determine a histogram for each of the codewords output by TF unit 112. The post-processing unit 138 may normalize the histogram of codewords over a limited range [hist_min to hist_max] over the entire dynamic range [0-1] of the allowed codeword space. Post-processing unit 138 stretches each histogram of the codeword for each of the color values, and determines the scaling and to be applied to the codeword by determining an offset and scale based on the stretched histogram. An offset parameter may be determined. For example, the equations for determining offset and scale may be as follows:
Offset1 = -hist_min
Scale1 = 1 / (hist_max-hist_min)

  Post-processing unit 138 may determine offset and scaling parameters for each of the colors using the equations for Offset2 and Scale2, and applies the offset2 and scale2 parameters to scale and codeword as described in more detail. Can be offset. There can be various other ways in which the pre-processing unit 138 determines offset and scaling parameters for post-processing, and the above equation is provided as an example.

  Although the color conversion unit 114 is illustrated as following the post-processing unit 138, in some examples, the color conversion unit 114 may first convert colors from RGB to YCrCb. Post processing unit 138 may perform operations on the YCrCb codeword. For the luma (Y) component, post-processing unit 138 may determine scaling and offset values using techniques similar to those described above. The following describes a technique for determining the scale and offset for the chroma component.

  Post processing unit 138 may determine scaling and offset parameters for the Cb and Cr color components from the colorimetry of the input video signal and the target colorimetry of the output video signal. For example, the target (T) color container specified by the primary color coordinates (xXt, yXt), where X is defined for the R, G, and B color components,

And the native (N) gamut specified by the primary color coordinates (xXn, yXn), where X is defined for the R, G, and B color components,

Is considered.

The white point coordinates for both gamuts are equal to whiteP = (xW, yW). A DRA parameter estimation unit (eg, post-processing unit 138) may derive scale Cb and scale Cr for the Cb and Cr color components as a function of the distance from the primary color coordinates to the white point. An example of such an estimation is given as follows.
rdT = sqrt ((primeT (1,1) -whiteP (1,1)) ^ 2+ (primeN (1,2) -whiteP (1,2)) ^ 2)
gdT = sqrt ((primeT (2,1) -whiteP (1,1)) ^ 2+ (primeN (2,2) -whiteP (1,2)) ^ 2)
bdT = sqrt ((primeT (3,1) -whiteP (1,1)) ^ 2+ (primeN (3,2) -whiteP (1,2)) ^ 2)
rdN = sqrt ((primeN (1,1) -whiteP (1,1)) ^ 2+ (primeN (1,2) -whiteP (1,2)) ^ 2)
gdN = sqrt ((primeN (2,1) -whiteP (1,1)) ^ 2+ (primeN (2,2) -whiteP (1,2)) ^ 2)
bdN = sqrt ((primeN (3,1) -whiteP (1,1)) ^ 2+ (primeN (3,2) -whiteP (1,2)) ^ 2)
scale Cb = bdT / bdN
scale Cr = sqrt ((rdT / rdN) ^ 2 + (gdT / gdN) ^ 2)
The Cb and Cr offset parameters for such an embodiment may be set equal to 0, ie offset Cb = offset Cr = 0.

  In some examples, the color conversion unit 114 may convert RGB to YCrCb before the preprocessing unit 134 applies preprocessing. In such an example, the pre-processing unit 134 may perform operations similar to those described above for the post-processing unit 138 on the YCrCb value, before the TF unit 112 applies the transfer function. Can be expected for input color values.

  When the preprocessing unit 134 is applied to the input color value, the output of the TF unit 112 is a linear codeword when the TF unit 112 applies the transfer function (e.g., the range of color values represented by the codeword). Can be determined based on the histogram to determine a scaling and offset factor that produces an output that is relatively the same across color and codeword spaces. In some examples, post-processing unit 138 may apply scaling and offsetting to the codeword that TF unit 112 outputs. Post processing unit 138 may modify the codewords to better utilize the range of available codewords. For example, the output of the TF unit 112 may be a codeword that does not use the entire codeword space. By spreading the codeword over the codeword space, the signal to quantization noise ratio by the quantization unit 116 may be improved.

  The signal-to-quantization noise ratio usually reflects the relationship between maximum nominal signal strength and quantization error (also called quantization noise), i.e., SNR = E (x ^ 2) / E (n ^ 2 ) Where E (x ^ 2) is the power of the signal, E (n ^ 2) is the power of the quantization noise, where ^ represents the exponential operation, E is the energy, n is noise.

  Multiplying the signal x by the scaling parameter> 1.0 will result in an increase in the power of the signal and thus an improved signal-to-quantization noise ratio, ie SNR2 = E ((scale * x) ^ 2) / E (n ^ 2)> SNR.

  Video encoder 20 receives the output of video preprocessor 19, encodes the video data output by video preprocessor 19, and outputs the encoded video data for later decoding by video decoder 30 and processing by video postprocessor 31. . In some examples, video encoder 20 may encode and signal information indicating a scaling and offset factor for one or both of pre-processing unit 134 and post-processing unit 138. In some examples, rather than encoding and signaling information indicative of scaling and offset factors, video preprocessor 19 may output information such as a histogram from which video postprocessor 31 determines scaling and offset factors. . Information output by the video preprocessor 19 that indicates the manner in which the video data has been preprocessed by the preprocessing unit 134 or postprocessed by the postprocessing unit 138 may be referred to as adaptive transfer function (ATF) parameters.

  It should be understood that in various examples, the transfer function applied by the TF unit 112 is static (eg, not content adaptive). However, data output to the TF unit 112 may be adapted (eg, by the pre-processing unit 134) and / or data output by the TF unit 112 may be adapted (eg, by the post-processing unit 138). In this way, video preprocessor 19 outputs a codeword that represents a color with a reduced dynamic range compared to the dynamic range of linear RGB data 110, where the output codeword is based on the video content. Thus, by adapting the input to TF unit 112 and the output of TF unit 112, the combination of pre-processing unit 134, TF unit 112, and post-processing unit 138 functions to apply an adaptive transfer function. To do.

  FIG. 10 is a conceptual diagram showing a content adaptive HDR processing pipeline (decoder side) using a fixed TF. As shown, the video post-processor 31 includes an inverse quantization unit 122, an inverse color conversion unit 124, an inverse post-processing unit 144, an inverse TF unit 126, and an inverse pre-processing unit 142.

  In the illustrated example, the video post processor 31 may be configured as a fixed function and programmable circuit configuration. For example, the video post-processor 31 may form an inverse quantization unit 122, an inverse color conversion unit 124, an inverse post-processing unit 144, an inverse TF unit 126, and an inverse pre-processing unit 142 together or separately, including transistors, capacitors, inductors , Passive and active elements, arithmetic logic units (ALU), elementary function units (EFU), and the like. In some examples, video post-processor 31 performs their respective functions on inverse quantization unit 122, inverse color transform unit 124, inverse post-processing unit 144, inverse TF unit 126, and inverse pre-processing unit 142. A programmable core that executes instructions to be executed. In such an example, video data memory 140 or some other memory may store instructions executed by video post processor 31.

  In FIG. 10, a video data memory 140 is also shown for ease of understanding. For example, video data memory 140 may temporarily store video data output by video post processor 31. As another example, any video data that video decoder 30 outputs to video post processor 31 may be temporarily stored in video data memory 140 (e.g., video data before being received by video post processor 31). Store in memory). Video data memory 140 may be part of video post processor 31 or may be external to video post processor 31.

  Video data stored in video data memory 140 may be output to display device 32, for example. Video data memory 140 may be a variety of dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM®), or other types of memory devices. It can be formed by any of the memory devices. In various examples, video data memory 140 may be on-chip with other components of video post processor 31 or may be off-chip with respect to those components.

  Video post processor 31 may be configured to perform the reverse process of video preprocessor 19. For example, the video decoder 30 outputs decoded video data to the video post processor 31, which may be substantially similar to the video data output by the video preprocessor 19 to the video encoder 20. In addition, video decoder 30 may output adaptive transfer function (ATF) parameters to video postprocessor 31 to perform the inverse of the pre-processing and post-processing performed by video preprocessor 19.

  The inverse quantization unit 122 receives video data from the video decoder 30 and performs the reverse operation of the quantization unit 116. The output of the inverse quantization unit 122 is video data that has not been quantized. The reverse color conversion unit performs an operation opposite to the operation of the color conversion unit 114. For example, when the color conversion unit 114 converts RGB color to YCrCb color, the reverse color conversion unit 124 converts the YCrCb color to RGB color.

  The reverse post-processing unit 144 may perform the reverse operation of the post-processing unit 138. The reverse post-processing unit 144 receives a codeword representing a color in the first dynamic range, which in this case is the same dynamic range as the output of the post-processing unit 138.

  Rather than multiplying by a scaling factor as was done with post-processing unit 138, inverse post-processing unit 144 divides by a scaling factor substantially similar to that post-processing unit 138 multiplied with. obtain. If post-processing unit 138 subtracts the offset from the scaled codeword, inverse post-processing unit 144 may add the offset to the descaled codeword. The output of the reverse post-processing unit 144 may be a codeword that is not spread over the entire codeword space. In some examples, inverse post-processing unit 144 may receive scaling and offset parameters (eg, scale and offset factor) from video preprocessor 19. In some examples, inverse post-processing unit 144 may receive information from which inverse post-processing unit 144 determines scaling and offset factors.

  The reverse TF unit 126 performs the reverse operation of the TF unit 112. For example, the TF unit 112 shortened the color value in a codeword that represents a smaller dynamic range compared to the color value dynamic range. The inverse TF unit 126 extends the color value from the smaller dynamic range back to the larger dynamic range. The output of the inverse TF unit 126 can be linear RGB data, but there can be some preprocessing that needs to be removed to yield the original RGB data.

  The reverse preprocessing unit 142 receives the output of the reverse TF unit 126 and performs the reverse of the preprocessing applied by the preprocessing unit 134. The inverse post-processing unit 144 receives a color value representing a color in the second dynamic range, which in this case is the same dynamic range as the output of the pre-processing unit 134.

  Rather than multiplying by a scaling factor as was done with preprocessing unit 134, inverse preprocessing unit 142 may divide by a substantially similar scaling factor that preprocessing unit 134 has multiplied with. If preprocessing unit 134 subtracts the offset from the scaled color value, inverse preprocessing unit 142 may add the offset to the descaled color value. In some examples, inverse preprocessing unit 142 may receive scaling and offset parameters (eg, scaling and offset factor) from video preprocessor 19. In some examples, inverse preprocessing unit 142 may receive information (eg, histogram information) from which inverse preprocessing unit 142 determines scaling and offset factors.

  The output of the inverse preprocessing unit 142 may be linear RGB data 128. The linear RGB data 128 and the linear RGB data 110 should be substantially similar. Display device 32 may display linear RGB data 128.

  Similar to TF unit 112, the inverse transfer function applied by inverse TF unit 126 is a static inverse transfer function (eg, does not exhibit adaptability to video content). However, data output to inverse TF unit 126 may be adapted (eg, by inverse post-processing unit 144) and / or data output by inverse TF unit 126 is adapted (eg, by inverse pre-processing unit 142). May be. In this way, the video post processor 31 outputs a color value representing a color that is not in the reduced dynamic range as compared to the reduced dynamic range of the output of the post-processing unit 138. Therefore, by adapting the input to the inverse TF unit 126 and the output of the inverse TF unit 126, the combination of the inverse post-processing unit 144, the inverse TF unit 126, and the inverse pre-processing unit 142 is an adaptive inverse transfer function. Function to apply.

  Although both pre-processing unit 134 and post-processing unit 138 are shown in FIG. 9, the exemplary techniques described in this disclosure are not so limited. In some examples, the video preprocessor 19 may include a preprocessing unit 134, but may not include a postprocessing unit 138. In some examples, video preprocessor 19 may include a post-processing unit 138, but may not include a pre-processing unit 134.

  Accordingly, FIG. 9 shows an example of a device (eg, source device 12) for video processing in a content adaptive HDR system, where the device includes a video data memory 132 and a video preprocessor 19. The video preprocessor 19 has at least one of fixed function or programmable circuitry, or a combination of both, and a plurality of colors of video data representing a color in a first dynamic range (e.g., linear RGB data 110) Configured to receive a value. Video preprocessor 19 uses a static transfer function that is not adaptable to the shortened video data to generate a plurality of codewords that represent the shortened color value in the second dynamic range. It includes a TF unit 112 configured to be shortened. The second dynamic range is more compact than the first dynamic range.

  Video preprocessor 19 may include at least one of pre-processing unit 134 or post-processing unit 138. Preprocessing unit 134 is configured to preprocess the color values prior to shortening to produce shortened color values. The post-processing unit 138 is configured to post-process the codeword obtained from the color value shortening. The video pre-processor 19 may select from among the shortened color values (e.g., in the example without the post-processing unit 138) or post-processed shortened color values (e.g., in the example where the post-processing unit 138 is part of the video pre-processor 19). One may be configured to output a color value based on one (eg, the output of quantization unit 116).

  Also, although both reverse post-processing unit 144 and reverse pre-processing unit 142 are shown in FIG. 10, the exemplary techniques described in this disclosure are not so limited. In some examples, video post-processor 31 may include reverse post-processing unit 144, but may not include reverse pre-processing unit 142 (e.g., in an example where video preprocessor 19 does not include pre-processing unit 134). However, the technique is not so limited). In some examples, video post-processor 31 may include reverse pre-processing unit 142, but may not include reverse post-processing unit 144 (e.g., in an example where video pre-processor 19 does not include post-processing unit 138, However, the technique is not so limited).

  Accordingly, FIG. 10 shows an example of a device for video processing (eg, destination device 14) in a content adaptive HDR system, where the device includes a video data memory 140 and a video post processor 31. Video post-processor 31 comprises at least one of fixed function or programmable circuitry, or a combination of both, and is configured to receive a first plurality of codewords representing a shortened color value of video data . The shortened color value represents the color in the first dynamic range (eg, a dynamic range substantially similar to the shortened dynamic range of the codeword output by the video preprocessor 19). The video post processor 31 uses the inverse static transfer function that is not adaptable to the video data to generate a non-shortened color value and uses the second plurality of codes based on the first plurality of code words. It includes an inverse TF unit 126 configured to unshorten the word. The non-shortened color value represents a color in a second dynamic range (eg, a dynamic range substantially similar to linear RGB data 110). The second plurality of codewords may be the first plurality of codewords being reverse post-processed (e.g., in an example including reverse post-processing unit 144) or the first plurality of codewords (e.g., reverse post-processing) One of the codewords from (in the example where unit 144 is not included).

  Video post processor 31 may include at least one of reverse post-processing unit 144 or reverse pre-processing unit 142. The reverse post-processing unit 144 is configured to reverse post-process the first plurality of codewords to generate a non-shortened second plurality of codewords. The reverse preprocessing unit 142 is configured to reverse preprocess the non-shortened color values obtained from the non-shortening of the second plurality of codewords. The video preprocessor 19 may select a non-shortened color value (e.g., the output of the reverse TF unit 126 in an example where the reverse preprocessing unit 142 is not included) or a reverse preprocessed non-shortened color value (e.g., the output of the reverse preprocessing unit 142 ) May be configured to output for display.

  9 and 10 illustrate exemplary techniques for content adaptive dynamic range adjustment (DRA) to reduce artifacts resulting from fixed (eg, static) transfer functions. However, by pre-processing and / or post-processing the data received or output by the TF unit 112 and by reverse post-processing and / or reverse pre-processing the data received or output by the reverse TF unit 126 Techniques may still use static transfer functions, but still achieve content adaptive DRA.

  This disclosure describes a content-adaptive HDR video system that employs a static fixed transfer function (TF). The following describes some examples that can be used together or maintained separately. For simplicity, they will be described separately, with the understanding that various permutations and combinations are possible.

  As explained above, the techniques described in this disclosure utilize static TF in the pipeline (eg, to generate encoded video content), but adapt the signal characteristics to a fixed processing flow. . This means that the signal to be processed by the TF unit 112 (e.g. using the pre-processing unit 134) or the signal obtained from the application of the TF by the TF unit 112 (e.g. using the post-processing unit 138) Either can be achieved by adaptive processing.

  In one example, the preprocessing unit 134 may allow adaptability by linear (scaling and offset) preprocessing of the input linear color values before the TF unit 112 applies the transfer function. In addition to or instead of the preprocessing unit 134, the postprocessing unit 138 is a linear (scaling and offset) of the nonlinear codeword obtained from the TF unit 112 applying TF to the linear color values. Post-processing may allow for adaptability. Reverse post-processing unit 144, reverse TF unit 126, and reverse pre-processing unit 142 may apply the reverse of post-processing unit 138, TF unit 112, and pre-processing unit 134, respectively.

  Preprocessing unit 134 applies linear preprocessing to the input signal (e.g., linear RGB 110) by applying offset Offset1 and scale Scale1 to provide a preferred codeword distribution of TF output (e.g., each codeword is a luminance etc. To represent an approximately equal range of input color values). Post processing unit 138 may apply linear post processing to the output of TF unit 112. The linear post-processing is defined by the parameters Scale2 and Offset2, allowing efficient use of the available codeword space (dynamic range) after the TF unit 112 applies the transfer function.

  In the example described above, the preprocessing unit 134 applies preprocessing to the linear RGB data 110. However, the techniques described in this disclosure are not so limited. RGB is one color space of HDR in which the preprocessing unit 134 can perform preprocessing there. In general, the preprocessing unit 134 is in any color space of the HDR processing flow (e.g., in the input linear RGB in the illustrated example, but in the YCbCr color space, etc.) that precedes the TF unit 112 applying the transfer function. Other color spaces are possible) and pre-processing can be performed.

  The reverse preprocessing unit 142 may also be configured to perform reverse preprocessing in any color space. In such an example, inverse preprocessing unit 142 may receive nonlinear color values (eg, nonlinear RGB data, but other color spaces including YCbCr are possible) from inverse TF unit 126.

  The above example illustrates that post-processing unit 138 performs post-processing in the RGB color space (eg, non-linear RGB due to the transfer function applied by TF unit 112). However, the techniques described in this disclosure are not so limited. RGB is one color space in HDR where the post-processing unit 138 can perform post-processing there. In general, the post-processing unit 138 is in any color space of the HDR processing flow (e.g., in the non-linear RGB in the illustrated example, but other in the YCbCr color space, etc.) following the TF unit 112 applying the transfer function. Post-processing may be performed.

  Reverse post-processing unit 144 may also be configured to perform reverse post-processing in any color space. In such an example, the inverse post-processing unit 144 obtains color values (e.g., non-linear shortened RGB data, but other color spaces including YCbCr are possible) from the video decoder 30, and possibly in inverse quantization. Can be received later.

  As explained above, the pre-processing unit 134 is on the encoder side, and in order to cause better parameter derivation, HDR signal characteristics such as histogram distribution input or target color space (e.g. linear RGB data 110, or color Parameters such as Scale1 and Offset1 factors may be derived and applied from YCrCb) when conversion unit 114 receives linear RGB data 110 and converts colors to YCrCb values) or in auxiliary color space. The post-processing unit 138 may also derive and apply parameters such as Scale2 and Offset2 factors from the output of the TF unit 112.

  The pre-processing unit 134 and / or the post-processing unit 138 are scale 1 and offset 1 factors, and / or the reverse pre-processing unit 142 and the reverse post-processing unit 144, respectively, reverse the operation of the pre-processing unit 134 and the post-processing unit 138, respectively. Scale2 and Offset2 factors used to execute can be output. In some examples, instead of outputting the Scale1, Offset1, Scale2, and / or Offset2 factors, the pre-processing unit 134 and the post-processing unit 138 have the reverse pre-processing unit 142 and the reverse post-processing unit 144 perform the same process. The information used to determine Scale1, Offset1, Scale2, and / or Offset2 may be output so that it can be used to determine Scale1, Offset1, Scale2, and / or Offset2 for the reverse process. In some cases, pre-processing unit 134 and post-processing unit 138 may output one or more Scale1, Offset1, Scale2, and / or Offset2, and derive other Scale1, Offset1, Scale2, and / or Offset2. Information that can be used for the purpose.

  In some examples, video preprocessor 19 may selectively enable the use of pre-processing unit 134 and / or post-processing unit 138 (eg, via a controller circuit). Accordingly, video preprocessor 19 may output information indicating whether pre-processing unit 134 and / or post-processing unit 138 are enabled. In response, video post-processor 31 may selectively enable the use of reverse post-processing unit 144 and / or reverse pre-processing unit 142 (eg, via a controller circuit).

  There may be various ways in which video encoder 20 may encode and output various parameters. For example, parameters through bitstreams with SEI (supplemental enhancement information) / VUI (video usability information), or parameters provided to the decoder side (e.g. video postprocessor 31) as side information, or input and output colors Video encoder 20 may signal and video decoder 30 may receive parameters derived by the decoder side (eg, by video post processor 31) from other identifications of space, transfer functions utilized, and the like.

  The following describes exemplary techniques that may be applied in accordance with the exemplary techniques described in this disclosure. Each of these techniques may be applied separately or in any combination. Each of these techniques is also described in further detail below.

  In the above example, video preprocessor 19 may signal parameter information in various ways. By way of example only, video encoder 20 includes an encoding unit (eg, an entropy encoding unit), which may encode parameter information that is signaled. Similarly, video decoder 30 includes a decoding unit (eg, an entropy decoding unit), which may decode the signaled parameter information and video postprocessor 31 may receive the parameter information from video decoder 30. There can be various ways in which parameter information can be signaled and received, and the techniques described in this disclosure are not limited to any specific approach. Further, as described above, the video preprocessor 19 does not always need to signal parameter information in all cases. In some examples, video postprocessor 31 may derive parameter information without necessarily receiving parameter information from video preprocessor 19.

  The following are exemplary implementations used for illustration only and should not be considered limiting. For example, the present disclosure may be applied by linear (scale and offset) preprocessing of input linear color values and / or for linear color values prior to the applied TF (eg, static, non-content adaptive) It will be explained that the linearity (scale and offset) post-processing of the non-linear codeword obtained from the TF is made possible. The following are some non-limiting examples of technique implementations.

  The proposed forward ATF processing flow is shown in FIG. In this example, a linear signal component s in an input color space, eg, RGB, has been preprocessed by preprocessing unit 134 to generate output signal s1. The TF unit 112 applies a transfer function to the value as s1, which results in the output codeword S1. In the following steps, the code word as S1 has been post-processed by the post-processing unit 138 to generate the output value S2.

  FIG. 12 is the reverse of the ATF process flow shown in FIG. For example, the reverse post-processing unit 144 performs reverse post-processing on the code word S2 ′. Codeword S2 ′ may be substantially similar to codeword S2. The output of the reverse post-processing by the reverse post-processing unit 144 is the code word S1 ′. Codeword S1 ′ may be similar to codeword S1. The inverse TF unit 126 receives the codeword S1 ′ and the output of the inverse TF unit 126 is s1 ′, where s1 ′ represents a color value and is similar to the color value s1. The inverse preprocessing unit 142 receives s1 ′ as input and outputs a color value s ′. The color value s ′ is substantially similar to the color value s.

  For ease of explanation, the various processes are not shown in FIGS. For example, color conversion and quantization and their respective inverse processes are not shown in FIG. 11 and FIG.

  The following describes exemplary operations and algorithms that preprocessing unit 134 and reverse preprocessing unit 142 may perform. The preprocessing unit 134 is defined by parameters Scale1 and Offset1. Preprocessing unit 134 targets to adjust the input signal characteristics to some characteristics of the transfer function applied by TF unit 112. An example of such processing is shown in FIGS. 13A and 13B. Note that FIG. 13A shows a histogram of the red color component of the HDR signal with the PQT transfer function on top, and the signal is shown on a different scale for visualization. A histogram of the non-linear signal obtained from PQTF is shown in FIG. 13B (eg, when TF unit 112 applies PQTF without using preprocessing unit 134).

  As shown, the PQTF curve (same as in Figure 7) gives PQTF more codewords for samples with lower brightness, for example, 0-1% of the input dynamic range uses 50% of the output dynamic range. Allow to be represented. Such a distribution may not be optimal for some classes of signals and / or applications.

For linear preprocessing, preprocessing unit 134 applies linear preprocessing to input signal s by applying offset Offset1 and scale Scale1 to achieve a preferred codeword distribution of the TF output from TF unit 112. obtain.
s1 = Scale1 * (s-Offset1) (8)

  The signal value s1 generated by Equation 8 is utilized in Equation 1, and the HDR processing pipeline specified in Equations (3-7) is applied (eg, by the color conversion unit 114 and the quantization unit 116).

  On the decoder side (for example, the video post processor 31), the reverse preprocessing unit 142 applies the reverse operation to the preprocessing as follows.

  Where the term s1 ′ indicates the linear color value generated by the inverse TF as specified in Equation 2. In other words, s1 ′ is the output of the inverse transfer function applied by the inverse TF unit 126.

  The effect of such preprocessing using the parameters (Scale1 = 0.1 and Offset1 = 0.0001) is shown in FIG. FIG. 14 is generated from the same inputs illustrated as FIG. 13A. Since the codeword of the S1 signal occupies the available dynamic range more efficiently compared to the example shown in Figure 13B, and the high luminance value occupies a much larger dynamic range (compared to the histogram peak in Figure 13B). 14), it can be seen that their representation is more accurate.

  In some examples, the input to TF unit 112 may be a bipolar signal. Bipolar signals mean that some of the values output by the preprocessing unit 138 are negative and the other is positive. The TF unit 112 adjusts the output of the preprocessing unit 134 to allow the application of a static TF by the TF unit 112 (and vice versa with the reverse TF unit 126) even for bipolar signals. And / or adjust the output after applying the transfer function. For example, if the output of the preprocessing unit 134 is a negative value, the TF unit 126 applies a transfer function to the absolute value of the output of the preprocessing unit 134 and uses the result as the output of the sign (s) function. May be multiplied, where s is the output of preprocessing unit 134, sign (s) =-1 (if s <0), 0 (if s = 0), 1 (if s> 0) It is. Therefore, in order to enable the application of static TF, the TF unit 112 may use a function defined by a positive value for a bipolar signal, which means that the number of signs of the bipolar signal Implying that special treatment.

  The following describes a transfer function using bipolar signal processing. In some examples, the technique may modify equation (1) so that bipolar input signals can be processed.

  This technique allows the TF unit 112 of FIG. 9 to allocate the desired steep slope of TF to the area where dynamic range is needed, for example, when a more accurate representation is needed for a moderately bright level of the signal. Can be tolerated. Visualization of this concept with parameters scale = 0.1 and offset = 0.1 is shown in FIGS. 15A and 15B.

  The inverse TF for the bipolar signal will be defined accordingly, as modified Equation 2.

  For example, the inverse TF unit 126 of FIG. 10 may apply an inverse transfer function for the case where the input codeword is positive. For input codewords that are negative, inverse TF unit 126 may determine the absolute value of the input codeword and may apply an inverse transfer function to the absolute value of the input codeword. The TF unit 126 may multiply the result with the output of the sign () function, where the input to the sign () function is the codeword received by the inverse TF unit 126.

  The following describes exemplary operations and algorithms that post-processing unit 138 and reverse post-processing unit 144 may perform. The post-processing unit 138 is defined by the parameters Scale2 and Offset2, and allows efficient use of the available codeword space (dynamic range) after the TF is applied by the TF unit 112. An example of such processing is described below.

  The following is an example of an HDR signal after TF has been applied by TF unit 112. FIG. 16A shows a histogram of the red color component of the HDR signal after TF application by the TF unit 112. As shown, the signal histogram does not occupy the entire codeword space, rather in this example only 60% of the codewords are actually utilized. Such a representation is not a problem until the signal is represented in floating point precision. However, an unused codeword budget can be utilized more efficiently to reduce the quantization error introduced by Equation 7.

Post-processing unit 138 may apply offset and scaling linear operations to codeword S generated after applying TF by TF unit 112.
S2 = scale2 * (S-offset2) (12)

In some frameworks, the post-processing unit 138 (Figure 9) may clip the resulting S value to the specified dynamic range as follows:
S2 = min (maxValue, max (S2, minValue)) (13)

  Following this post-processing, the signal value S2 is used in equation (3), followed by the HDR processing flow specified in equations (3-7).

  On the decoder side (eg, video post processor 31), inverse post-processing unit 144 may apply the reverse operation of post-processing, performed as follows.

In some frameworks, the inverse post-processing unit 144 may clip the resulting S value to the specified dynamic range as follows.
S '= min (maxValue, max (S', minValue)) (15)
However, the term S2 ′ indicates the decoded nonlinear color value.

  The effect of such post-processing using parameters (scale = 0.8 and offset = 0) is shown in FIG. 16B. Because the codeword of the S2 signal occupies the available codeword space (dynamic range) more efficiently, and the high luminance value occupies a much larger dynamic range (Figure 16B compared to the histogram peak in Figure 16A) It can be seen that their representation is more accurate.

  In the above example, instead of applying the operation of Equation (12), the post-processing unit 138 applies a second transfer function that is a linear transfer function instead of the nonlinear transfer function applied by the TF unit 112. Can be considered. Applying this second transfer function may be equivalent to multiplying the input codeword by Scale2 and adding Offset2, and the inverse post-processing unit 144 applies the inverse of this process. Thus, in some examples, the techniques of post-processing unit 138 and inverse post-processing unit 144 (FIG. 10) can be expanded through an additional linear transfer function TF2. The parameters of this additional linear transfer function TF2 can be specified through Scale2 and Offset2, for example, as shown in FIGS. 17A-17C.

  For example, FIG. 17A shows the same transfer function behavior as FIG. 7 (eg, a static transfer function applied by the TF unit 112). FIG. 17B shows the transfer function with Scale2 = 1 and Offset2 = 0 that the post-processing unit 138 applies to the output of the TF unit 112. FIG. 17C shows the transfer function with Scale2 = 1.5 and Offset2 = −0.25 that the post-processing unit 138 applies to the output of the TF unit 112.

  The above example describes the case where there is a pre-processing unit 134 (DRA1) and / or a post-processing unit 138 (DRA2), and a corresponding reverse pre-processing unit 142 and reverse post-processing unit 144. In some examples, post processing may be performed after color conversion (eg, in a target color space) rather than before color conversion. For example, as shown in FIG. 18, the TF unit 112 outputs to the color conversion unit 114, and the post-processing unit 150 applies post-processing after color conversion. Similarly, as shown in FIG. 19, the inverse post-processing unit 152 applies inverse post-processing to the output of the inverse quantization unit 122 before the inverse color conversion unit 124 applies color conversion. 18 and 19, the preprocessing unit 134 and the reverse preprocessing unit 142 are not shown for simplicity, and may be included in other examples.

  Post-processing unit 150 and reverse post-processing unit 152 can function in the same manner as post-processing unit 138 and reverse post-processing unit 144, but in different color spaces (eg, YCbCr color space rather than RGB color space). For example, post-processing unit 150 and reverse post-processing unit 152 may utilize the Scale2 and Offset2 parameters for scaling and offset. The following describes post-processing in the target color space.

  For some HDR systems that use some target color space (eg BT709 or BT2020 YCbCr), use post-processing that uses the same parameters for the three components as R, G, and B And post-processing can be applied in the output color space rather than in the input RGB color space.

  For example, for YCbCr color conversion as defined in Equation 3 and Equation 5,

Where variables a, b, c, d1, and d2 are color space parameters, and R ′, G ′, and B ′ are non-linear color values in the input color space after applying EOTF.

The post-processing defined in Equation 12 and Equation 13 is applied to the input color space, that is, R ′, G ′, B ′, resulting in R2 ′, G2 ′, B2 ′.
R2 '= scale2 * (R'-offset2)
G2 '= scale2 * (G'-offset2) (17)
B2 '= scale2 * (B'-offset2)

  In Equation 17, the post-processing parameters in the R, G, and B regions are assumed to be the same.

However, due to the process linearity defined in Equations 16 and 17, post-processing can be performed in the target color space.
Y2 '= scaleY * Y'-offsetY
Cb2 = scaleB * Cb-offsetB (18)
Cr2 = scaleR * Cr-offsetR
Here, the post-processing unit 150 calculates post-processing parameters as follows.

  On the decoder side (eg, video post processor 31), the reverse operation of post-processing by the reverse post-processing unit 152 may be performed as follows.

  However, the variables Y2 ′, Cb, and Cr are decoded and dequantized components, and the parameters of the inverse post-processing are calculated as shown in Equation 19.

  In the case of YCbCr color conversion defined in BT2020, parameters for post-processing performed in the target color space are derived as follows from parameters for post-processing and color conversion in the input color space.

  In some examples, Equation 20 implemented in the target color space using parameters derived from post-processing parameters in the input color space may be implemented as Equation 20A:

  In some examples, equation 20A above may be implemented through multiplication rather than division.

  For non-linear processing, in addition to the pre-processing and post-processing techniques described above, post-processing unit 138 or 150 and reverse post-processing unit 144 or 152 may include several methods to further improve codeword utilization. Non-linear techniques can be utilized. As an example of histogram tail processing, post-processing unit 138 or 150 and inverse post-processing unit 144 or 152 use a parameter piecewise specified transfer function applied to samples outside the specified range, The linear codeword post-processing defined above can be augmented.

The HDR signal processing flow specified above normally operates within a predetermined range.
Range = {minValue, MaxValue}

In the case of the normalized signal and the above equation, the value is
Range = {0.0-1.0},
minValue = 0.0 (22),
maxValue = 1.0
It is.

  The post-processing proposed in Equation 12 (eg, as performed by post-processing unit 138) overflows the specified range boundary with the resulting S2 value through applying Offset2 and Scale2 to codeword S1. Can lead to situations. To ensure that the data stays within the specified range, a clipping operation may be applied to the range {minValue-maxValue} in Equation 13, for example, as shown in FIGS. 20A-20C.

  FIG. 20A is a diagram illustrating post-processing in which clipping is applied to a range related to the histogram of S2. FIG. 20B is another diagram illustrating post-processing in which clipping is applied to the range related to the histogram of S2 after post-processing in Equation 10. FIG. 20C is another diagram illustrating post-processing with clipping applied to the range for the histogram of S2 after clipping in Equation 11.

  As shown in FIG. 20C, applying clipping after post-processing will result in an irrecoverable loss of information outside the specified range. In order to efficiently represent the HDR signal in a limited codeword space by applying Scale2 / Offset2 and maintain the TF assumption for HVS perception, the post-processing unit 138 or 150 performs content adaptive histogram tail processing. To allow, special processing of color samples that yields codewords outside of the specified Range can be used to augment the codeword post-processing defined above.

  The post-processing unit 138 or 150 may apply a transfer function specified in a parameter piecewise manner to all S2 obtained from Equation 12 that belong outside the range. These sample representation parameters are encoded and signaled by the video encoder 20 separately from the S2 codewords belonging to the specified range.

By way of illustration, as specified in Equation 22, an operation may be performed to classify the codeword S2 resulting from Equation 12 into a supported data range according to their relationship.
S2 _in = ∀S2∈Range (23)
S2 _low = ∀S2 ≦ minValue (24)
S2 _high = ∀S2 ≧ maxValue (25)
However, S2 _in samples belong to Range = [minValue ~ maxValue], S2 _low is a set of samples outside the range and below minValue, and S2 _high is a sample outside the range and above maxValue .

Technique, by S2 _low and S2 _high each individually addressable / determine / signaling codeword values for the relates to a color value as S2 _low and S2 _high in the range outside of the operation of clipping the formula 13 Or may be specified / determined / signaled with a group of codewords that best represents information outside of the Range.

The following is a non-limiting example of an exemplary decision process that can replace clipping for samples S2 _low and S2 _high .
a. Determine an average value for each of S2 _low and S2 _high .
Smin = mean (S2 _low ) (26)
Smax = mean (S2 _high )
b. Determine the median for each of S2 _low and S2 _high .
Smin = median (S2 _low ) (27)
Smax = median (S2 _high )
c. For each of S2 _low and S2 _high , a value that best represents each of those sample sets under some criteria, for example, the minimum sum of absolute differences (SAD), the mean square error (MSE) Or determine rate-distortion optimization (RDO) costs.
Smin = fun (S2 _low ) (28)
Smax = fun (S2 _high )

In such an example, each sample value S2 _low or S2 _high replaces the Smin and Smax values derived in equations (26, 27), respectively, rather than the clipped values as shown in equation (13). Is done.

The proposed nonlinear processing parameters are codewords representing each of S2 _low and S2 _high . These codewords are signaled to the decoder side (eg, video postprocessor 31) and utilized by inverse post-processing unit 144 or 152 in the reconstructed process.

  For example, in the above example of parameter piecewise processing, the post-processing unit 138 or 150 may use codewords resulting from color value reduction (e.g., dynamic range reduction resulting from application of a transfer function) by the TF unit 112. Scaling and offset (eg, applying Scale2 and Offset2) may be performed. Post processing unit 138 or 150 may determine that the set of scaled and offset codewords has a value that is less than the minimum threshold or greater than the maximum threshold. Post processing unit 138 or 150 may assign a first codeword (e.g., Smin) to a set of scaled and offset codewords having a value less than the minimum threshold, and is greater than the maximum threshold A second codeword (eg, Smax) may be assigned to a set of scaled or offset codewords having values. For codewords that are scaled and offset between the minimum and maximum values, post-processing unit 138 or 150 may apply scaling and offset as described in the example above.

  Inverse post-processing unit 144 or 152 (FIG. 10 or FIG. 19) may perform the inverse of the parameter piecewise processing. For example, the inverse post-processing unit 144 or 152 is configured such that the first set of codewords from the plurality of codewords received from the video decoder 30 is less than the minimum threshold or greater than the maximum threshold. Can be determined to represent a shortened color value (eg, where the dynamic range has been shortened based on the application of the transfer function by the TF unit 112). Inverse post-processing unit 144 or 152 may determine that the second set of codewords from the plurality of codewords represents a shortened color value having a value greater than or equal to the minimum threshold and less than or equal to the maximum threshold. In such an example, the reverse post-processing unit 144 or 152 may assign a first codeword (e.g., Smin) to a first set of codewords of a codeword that is less than the minimum threshold, and the maximum A second codeword (eg, Smax) may be assigned to the first set of codewords of the codeword that is greater than the threshold. The values of Smin and Smax may be signaled from video preprocessor 19 via video encoder 20. The inverse post-processing unit 144 or 152 reverses the second set of codewords (e.g., greater than the minimum value and less than the maximum value) using exemplary techniques such as those described above. Can be scaled and inverse offset.

  In the above example, applying the scaling and offset parameters piecewise is described. In some examples, such piecewise application of scaling and offset parameters may be extended for both pre-processing unit 134 and post-processing unit 138 or 152 and reverse post-processing unit 144 or 152 and reverse pre-processing unit 142. .

  For example, codewords received by inverse post-processing unit 144 or 152 may include a first set of codewords and a second set of codewords. The first set of codewords represents a shortened color value having a value belonging to the first section of the dynamic range, and the second set of codewords has a shortened color having a value belonging to the second section of the dynamic range Represents a value. In such an example, inverse post-processing unit 144 or 152 scales the first set of codewords using a first set of scaling and offset parameters (e.g., first Scale2 and first Offset2). And can be offset. Inverse post-processing unit 144 or 152 may scale and offset the second set of codewords using a second set of scaling and offset parameters (eg, second Scale2 and second Offset2).

  As another example, the color values received by the inverse preprocessing unit 142 may include a first set of non-shortened color values and a second set of non-shortened color values. The first set of non-shortened color values represents a non-shortened color value having a value belonging to the first section of the dynamic range, and the second set of non-shortened color values belongs to the second section of the dynamic range Represents an unshortened color value with a value. In such an example, the inverse preprocessing unit 142 uses the first set of scaling and offset parameters (e.g., first Scale1 and first Offset1) to scale and offset the first set of color values. Can do. Inverse pre-processing unit 142 may scale and offset the second set of color values using a second set of scaling and offset parameters (eg, second Scale1 and second Offset1).

  Similarly, the pre-processing unit 134 may determine that the first set of color values has a value belonging to the first partition of the dynamic range, and the second set of color values is the second partition of the dynamic range. Can be determined to have a value belonging to. Pre-processing unit 134 may scale and offset the first set of color values using a first set of scaling and offset parameters (eg, first Scale1 and first Offset1). Pre-processing unit 134 may scale and offset the second set of color values using a second set of scaling and offset parameters (eg, second Scale1 and second Offset1).

  The post-processing unit 138 or 152 may determine that the first set of codewords from the plurality of codewords has a value belonging to the first partition of the dynamic range, and the first of the codewords from the plurality of codewords. It can be determined that the set of 2 has a value belonging to the second segment of the dynamic range. Post-processing unit 138 or 152 may scale and offset the first set of codewords using a first set of scaling and offset parameters (e.g., first Scale2 and first Offset2). A second set of offset parameters (eg, second Scale2 and second Offset2) may be used to scale and offset the second set of codewords.

  FIG. 21A and FIG. 21B are diagrams showing histograms of codewords after post-processing using tail processing. FIG. 22 is a conceptual diagram showing another example of the content adaptive HDR pipeline using the static TF and the encoder side. FIG. 23 is a conceptual diagram showing another example of the content adaptive HDR pipeline using the static TF and the decoder side. Figures 22 and 23 show that in some examples, Smin and Smax codewords are output when the scaled and offset values are outside the range, and in some examples, the value is out of range. Figures 22 and 23 are the same as Figures 11 and 12, except that Smin 'and Smax' (both similar to Smin and Smax) indicate that they are received codewords when outside. Substantially the same.

  In the above example, a single codeword is reserved for codewords below the minimum value and a single codeword is reserved for codewords greater than the maximum value. In some examples, codewords that belong to the end of the histogram outside a specified Range may be represented using two or more reserved codewords.

  Post processing unit 138 may determine and signal a number of reserved codewords for a specified region of the dynamic range of codeword S. On the decoder side (eg, video post processor 31), inverse post-processing unit 144 may determine a reserved codeword for the specified region. The technique may determine, signal, and apply a signal value S ′ associated with each of the reserved codewords at the decoder.

FIG. 24 is a diagram showing a histogram using two reserved codewords for processing color values. For example, a visualization of this example using two reserved codewords that process color values at S2 _low is shown in FIG. In this example, the S2 _low codeword is divided into two subranges, where each subrange is coded using codewords Smin1 and Smin2, respectively. The value associated with each of these codewords is determined at the encoder side (eg, by video preprocessor 19) in a process similar to Equations 26-28, signaled to decoder 30, and at the decoder side (eg, video Applied by Equation 29) (by post processor 31).

  In some examples, the technique may be implemented in the form of a transfer function. The nonlinear part of the transfer function, i.e. the histogram tail processing in the given example, has two reserved codewords associated with the values determined at the decoder side, e.g. signaling and applied at the decoder side It can be modeled using an adaptive transfer function.

25A and 25B are diagrams showing parameter adaptive functions. Figure 25A shows a linear transfer function (TF2) modeled as Scale2 = 1.5 and Offset = -0.25, and Figure 25B was modeled using a parameterized nonlinear segment as Scale2 = 1.5 and Offset = -0.25. The linear transfer function (TF2) is shown. The non-linear segment is parameterized with two codewords that are associated with the two determined color values and signaled to the decoder, Smin1 and Smin2, and the S2 _high codeword is clipped to the maxValue of the Range .

The following describes post-processing using a piecewise linear transfer function (eg, using post-processing unit 138 or 150). Aftertreatment unit 138 or 150
Range = {r _i }, where i = 0 to N (30)
The codeword space of S can be divided into a finite number of segments r such that

For each of these segments r _i , the post-processing unit 138 or 150 determines independent post-processing parameters Scale2 _i and Offset2 _i , signaling the Scale2i and Offset2i parameters to the codeword S on the encoder side and the decoder side. Apply to generate output S2.
Scales = {Scale2i}, Offsets = {Offset2i}, i = 0 to N

  In the transfer function terms, an exemplary algorithm can be modeled as follows. 26A to 26C are diagrams illustrating post-processing using a piecewise linear transfer function. FIG. 26A shows a histogram of the S signal generated by the PQTF, FIG. 26B shows a post-processing parameter piecewise linear TF, and FIG. 26C shows a histogram of the S2 signal generated by the post-processing TF.

  In the histogram of the S signal in FIG. 26A, the segment into which the dynamic range is introduced is shown with a vertical grid. As can be seen, the signal occupies about 80% of the available codewords, and 20% of the available codewords are quantized errors as proposed in Equations 12 and 13. Can be used to reduce. At the same time, the histogram shows that a significant portion of the codeword is in the fourth segment extending from 0.6 to 0.8 as the codeword range. The available codeword budget may be used to improve the quantization error for this particular fourth segment, leaving the representation accuracy for the other segments unchanged.

Post-processing the parameters for these r segments (N = 5) can be written as:
Scales2 = {1,1,1,2,1}, Offsets2 = {-0.1, -0.1, -0.1, -0.1,0.1}
This will result in the piecewise linear transfer function shown in FIG. 26B. A histogram of the S2 signal resulting from such TF application is shown in FIG. 26C. As can be seen, the histogram is shifted to the left to occupy more codewords, and the peak that was in the fourth segment is stretched for a large dynamic range, thus resulting in higher accuracy of the representation and quantization error Reduce.

In general, the parameters of this scheme may include:
a. Number of segments for dynamic range
b. Range of each segment
c. Scale and offset for each of these segments

Apply
In some examples, techniques may include pre-processing techniques, post-processing techniques, as well as non-linear processing techniques. In some examples, the technique may augment the linear codeword post-processing defined above with a parameter piecewise specified transfer function that is applied to samples within a specified range. In one example, the technique may reserve a finite number of codewords within a specified range for identification of samples that invoke this parameter transfer function.
b. In some examples, pre-processing and post-processing techniques are defined as the scale and offset applied to the input signal, ie, Scale1 and Offset1 and / or Scale2 and Offset2. In some examples, the technique is implemented in the form of a transfer function.
c. In some examples, the non-linear processing is a single parameter for defining color values that fall outside the specified Range: one parameter for samples below minValue and another for samples above maxValue. Parameters. In some examples, a separately defined transfer function may be applied.
d. In some examples, dynamic range adjustment (DRA) is derived independently for each color component (independent parameter set), eg, independently for R, G, B, or Y. In some examples, a single inter-component parameter set is applied for all color components.
e. In some examples, the parameters may be determined for the entire video sequence. In some examples, the parameters can be adapted in time or are spatio-temporal adaptive.
f. In some examples, the proposed process may be utilized in an HDR system with SDR compatibility. In such a system, with some transfer functions in use, the proposed technique would be applied to codewords representing HDR signals, for example, greater than 100 nits and / or less than 0.01 nits. Leave the remainder as unprocessed codewords, thus providing SDR compatibility.

Parameter derivation
a. In some examples, pre-processing and post-processing parameters are derived independently. In some examples, the pre-processing and post-processing parameters are derived together.
b. In some examples, the parameters are derived from a process that minimizes the quantization error or by minimizing the cost function, where the cost function is the bit rate obtained from the transformation and coding, As well as the weighted sum of the strains introduced by these two lossy processes.
c. In some examples, the parameters can be determined separately for each component using information about that component and / or derived by using inter-component information.
d. In some examples, parameters such as the scope of application may be determined from the type and characteristics of the transfer function TF, for example, processing may be applied to codewords representing only HDR signals, for example, greater than 100 nits. , The codeword representing the SDR signal <= 100 remains unchanged.
e. In some examples, range parameters may be provided to the decoder as side information and may be applied depending on the TF utilized in the process.
f. In some examples, parameters such as the scope of application can be determined from the type and characteristics of the transfer function TF, for example, in the case of a TF characterized by a transition, to determine the dynamic range to which the processing is applied. The position of the transition (knee) can be used.
g. Parameters may include the identification of some reserved codewords and their actual values.
h. The parameter may include an identification of the process associated with the reserved codeword.
i. The parameters may include several sub-ranges in the codeword space for applying the determined processing.
j. The parameter may include a codeword value subrange identification for applying the determined processing.

Parameter signaling
In some examples, the parameters are estimated at the encoder side (e.g., video preprocessor 19) and signaled to the decoder (e.g., video postprocessor 31) through a bitstream (metadata, SEI message, VUI, etc.) . The decoder receives parameters from the bitstream.
b. In some examples, the parameters are derived from the input signal or from other available parameters associated with the input signal and processing flow at the encoder and decoder sides through the specified process.
c. In some examples, the parameters are explicitly signaled and are sufficient to perform DRA at the decoder side. In some examples, the parameters are derived from other input signal parameters, such as input gamut and target color container (primary color) parameters.
d. In some examples, the parameters of the proposed system may be signaled as parameters of the transfer function (TF) utilized in the system or provided to the decoder as side information for a specific transfer function .
e. The parameters of the proposed scheme can be signaled through the bit stream by SEI / VUI or provided to the decoder as side information, or other identification such as input and output color space, transfer function utilized Can be derived by the decoder.

  FIG. 29 is a flowchart illustrating an exemplary method of video processing in a content adaptive high dynamic range (HDR) system. The example of FIG. 29 is described with respect to a video post processor 31 for converting received video data into video data for display. In some examples, the video post processor 31 is exemplary for each color (once for the red component, once for the green component, and once for the blue component) so that the processing is independent. Techniques can be implemented.

  Video post processor 31 receives a first plurality of codewords representing shortened color values of video data, where the shortened color values represent colors in a first dynamic range (200). For example, to reduce the bit rate, the video preprocessor 19 may shorten the dynamic range of color values. The result of the shortening is the first plurality of codewords. Video post processor 31 may receive the first plurality of codewords directly from video decoder 30 or via video data memory 140. The shortened color value may be in a specific color space such as RGB or YCrCb. The shortened color value may be dequantized by the dequantization unit 122.

  Reverse post-processing unit 144 or 152 may apply a reverse post-processing function that reduces the range of codewords to generate a second plurality of codewords (202). In some examples, inverse post-processing unit 144 may apply reverse post-processing operations to codewords that have been color converted (eg, color converted via inverse color conversion unit 124). In some examples, reverse post-processing unit 152 may apply reverse post-processing prior to color conversion, and reverse color conversion unit 124 may convert the output of reverse post-processing unit 152.

  Inverse post-processing unit 144 or 152 may perform inverse scaling and inverse offset on the first plurality of codewords. Inverse post-processing unit 144 or 152 may receive scaling and offset parameters (eg, Scale2 and Offset2). In some examples, inverse post-processing unit 144 or 152 may receive information from which inverse post-processing unit 144 or 152 determines scaling and offset parameters. In general, inverse post-processing unit 144 or 152 may derive scaling and offset parameters for codeword inverse scaling and inverse offset from the received bitstream or separately signaled side information. When receiving the scaling and offset parameters in the bitstream, the inverse post-processing unit 144 or 152 receives the scaling and offset parameters along with the picture data to be processed. When receiving the scaling and offset parameters as separately signaled side information, the inverse post-processing unit 144 or 152 receives the scaling and offset parameters without any picture data to be processed (e.g., any picture Receive the scaling and offset parameters before the data).

  In some examples, instead of applying scaling and offset to all values of the codeword to the first plurality of codewords, the inverse post-processing unit 144 or 152 applies piecewise scaling and offset. obtain. For example, the reverse post-processing unit 144 or 152 may use a shortened color in which the first set of codewords from the first plurality of codewords has a value that is less than the minimum threshold or greater than the maximum threshold. Representing a value, and a second set of codewords from the first plurality of codewords representing a shortened color value having a value less than a maximum threshold and greater than a minimum threshold. Can be determined.

  The reverse post-processing unit 144 or 152 may assign a first codeword (e.g., Smin ') to a first set of codewords of codewords that is less than the minimum threshold and is below the maximum threshold. A second codeword (eg, Smax ′) may be assigned to a codeword of the first set of codewords that is larger. Inverse post-processing unit 144 or 152 may inverse scale and inverse offset the second set of codewords (eg, based on Scale2 and Offset2). Although one minimum threshold and one maximum threshold are described, in some examples there are multiple such thresholds with reserved codewords for each of the thresholds. Also good. The reverse post-processing unit 144 or 152 may receive Smin ′, Smax ′, or reserved codewords, may receive information about how to determine these codewords, or these codewords It may be pre-stored in the video data memory 140.

  The inverse TF unit 126 receives the output from the inverse post-processing unit 144 or from the inverse color conversion unit 124 in the example using the inverse post-processing unit 152. However, in some examples, the reverse post-processing unit 144 or 152 may not be enabled or available. In such an example, inverse TF unit 126 receives the first plurality of codewords.

  The inverse TF unit 126 uses the inverse static transfer function that is not adaptable to the video data to generate a non-shortened color value, and the second plurality of codes based on the first plurality of codewords. The word may be shortened (204). The non-shortened color value represents the color in the second dynamic range. The second plurality of codewords is a first plurality of codewords that are being reverse post-processed (e.g., output from reverse post-processing unit 144 or from reverse post-processing unit 152 after reverse color conversion) or One of the codes from the first plurality of codewords (eg, when reverse post-processing unit 144 or 152 is disabled or not available).

  In some examples, the inverse TF unit 126 may output non-shortened color values for output for display or further processing (208). However, in some examples, the inverse TF unit 126 may output the unreduced color value to the inverse preprocessing unit 142, such as when preprocessing is applied before the video preprocessor 19 shortens. In this case, the reverse preprocessing unit 142 may output the unprocessed color values that have been displayed or reverse preprocessed for further processing, as described below with respect to optional operation 206. The dynamic range of non-shortened color values may be greater than the dynamic range of codewords received by the inverse TF unit 126. In other words, the inverse TF unit 126 increases the dynamic range of the codeword in order to generate non-shortened color values having a larger dynamic range.

  In some examples, reverse preprocessing unit 142 may apply reverse preprocessing to the non-shortened color values (206). Inverse preprocessing unit 142 may receive scaling and offset parameters (eg, Scale1 and Offset1). In some examples, inverse preprocessing unit 142 may receive information (eg, a histogram of colors in a picture) from which inverse preprocessing unit 142 determines scaling and offset parameters. The inverse preprocessing unit 142 outputs the obtained inverse preprocessed non-shortened color value for display or further processing (208).

  In general, inverse preprocessing unit 142 may derive scaling and offset parameters for codeword inverse scaling and inverse offset from the received bitstream or separately signaled side information. When receiving the scaling and offset parameters in the bitstream, the inverse preprocessing unit 142 receives the scaling and offset parameters along with the picture data to be processed. When receiving the scaling and offset parameters as separately signaled side information, the inverse preprocessing unit 142 receives the scaling and offset parameters without any picture data to be processed (e.g., for any picture data). Receive scaling and offset parameters before).

  As described above, the video post-processor 31 performs reverse post-processing on the first plurality of code words to generate a second plurality of code words, or reverse pre-processing on non-shortened color values. At least one of doing may be performed. However, in some examples, video post-processor 31 reverses the first plurality of codewords (eg, via reverse postprocessing unit 144 or 152) to generate the second plurality of codewords. Both processing and reverse preprocessing the non-shortened color values (eg, via reverse preprocessing unit 142) to produce reverse preprocessed non-shortened color values may be performed.

  FIG. 30 is a flowchart illustrating another exemplary method of video processing in a content adaptive high dynamic range (HDR) system. The example of FIG. 30 is described with respect to a video preprocessor 19 for converting received video data into video data for transmission. In some examples, the video preprocessor 19 is illustrative for each color (once for the red component, once for the green component, and once for the blue component) so that the processing is independent. The technique can be performed.

  Video preprocessor 19 receives a plurality of color values of video data representing color in the first dynamic range (300). For example, video preprocessor 19 may receive video data stored in video data memory 132, where such data is received from video source 18. The color value may be in the RGB color space, but the color conversion unit 114 can convert the color to the YCrCb color space before processing.

  The preprocessing unit 134 may perform preprocessing on the color values as when the preprocessed color values are processed by the TF unit 112, and each resulting codeword approximates a range of color values. It is not at least severely weighted (302) to represent or represent a low illumination color by a much larger range of codewords and a high illumination color by a relatively small range of codewords. For example, the preprocessing unit 134 may scale and offset (eg, via Scale1 and Offset1). Preprocessing unit 134 may determine scaling and offset parameters based on a histogram of color values in the picture (eg, may adaptively determine scaling and offset factors based on input linear color values). Pre-processing unit 134 may output values for Scale1 and Offset1, or may output information that may be used to determine values for Scale1 and Offset1. For example, video preprocessor 19 may cause video encoder 20 to signal scaling and offset parameters for scaling and offsetting input linear color values in the bitstream or as side information.

  The TF unit 112 may receive the output of the preprocessing unit 134 to shorten the color value. However, as shown, in some examples, the preprocessing unit 134 may not be enabled or available. In such an example, the TF unit 112 may receive multiple color values without preprocessing.

  The TF unit 112 may shorten the color value using a static transfer function that is not adaptive to the shortened video data to generate a plurality of codewords that represent the shortened color value, where The shortened color value represents the color in the second dynamic range (304). The TF unit 112 may reduce the dynamic range of color values and may facilitate the transmission of such color values.

  In some examples, the TF unit 112 may output a color value based on the shortened color value represented by the codeword (308). However, in some examples, post-processing unit 138 or 150 may use color values to generate codewords that better use codeword space (e.g., increase use of available codeword space). The codeword resulting from the shortening may be post-processed (306). The post-processing unit 150 can receive the output of the color conversion unit 114 for post-processing, and the color conversion unit 114 receives the output of the TF unit 112. Post-processing unit 138 or 150 may scale using Scale2 and offset using Offset2. For example, video preprocessor 19 may cause video encoder 20 to signal scaling and offset parameters for codeword scaling and offset in the bitstream or as side information.

  In some examples, instead of applying scaling and offset to multiple codewords output by the TF unit 112 for all values of the codeword, the post-processing unit 138 or 150 performs piecewise scaling and offset. Applicable. For example, post-processing unit 138 or 150 may determine that the set of scaled and offset codewords has a value that is less than the minimum threshold or greater than the maximum threshold. The post-processing unit 138 or 150 may assign a first codeword (Smin) to a set of scaled and offset codewords having a value less than the minimum threshold, with a value greater than the maximum threshold. A second codeword (Smax) may be assigned to the set of scaled and offset codewords having. For other codewords, post-processing unit 138 or 150 may scale and offset other codewords (eg, based on Scale2 and Offset2). Although one minimum threshold and one maximum threshold are described, in some examples there are multiple such thresholds with reserved codewords for each of the thresholds. Also good. Post-processing unit 138 or 150 may output information about Smin, Smax, or reserved codewords, or may output information about how to determine these codewords.

  Video preprocessor 19 may output a color value based on one of the shortened color value or the post-processed shortened color value (308). For example, if the post-processing unit 138 or 150 is enabled, the video preprocessor 19 may output a color value based on the post-processed shortened color value. If post-processing unit 138 or 150 is not enabled or available, video preprocessor 19 may output a color value without post-processing based on the shortened color value as shortened by TF unit 112. However, the color conversion unit 114 and the quantization unit 116 may execute respective processes before outputting.

  Depending on the example, some acts or events of any of the techniques described herein may be performed in different sequences, added, integrated, or completely excluded. It should be appreciated that (for example, not all described acts or events are necessary for the implementation of the technique). Moreover, in some examples, actions or events may be performed in parallel rather than sequentially, for example, through multithreaded processing, interrupt processing, or multiple processors.

  In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. May be. A computer-readable medium is a computer-readable storage medium that corresponds to a tangible medium such as a data storage medium, or communication that includes any medium that facilitates transfer of a computer program from one place to another, eg, according to a communication protocol Media may be included. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementation of the techniques described in this disclosure. Medium. The computer program product may include a computer readable medium.

  By way of example, and not limitation, such computer readable storage media may be RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage device, flash memory, or instruction or data structure Any other medium that can be used to store the form of the desired program code and that can be accessed by the computer can be provided. Any connection is also properly termed a computer-readable medium. For example, instructions are sent from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, wireless, and microwave. In this case, coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that computer readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but instead are directed to non-transitory tangible storage media. The disc and disc used in this specification are a compact disc (CD), a laser disc (registered trademark) (disc), an optical disc (disc), a digital versatile disc (DVD) ), Floppy disk, and Blu-ray® disc, the disk normally reproduces data magnetically, and the disc uses lasers for data. Is reproduced optically. Combinations of the above should also be included within the scope of computer-readable media.

  Instructions can be one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic configurations Can be executed by one or more processors. Thus, as used herein, the term “processor” may refer to either the above structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in dedicated hardware and / or software modules configured for encoding and decoding, or in a composite codec May be incorporated. Also, the techniques may be fully implemented with one or more circuits or logic elements.

  The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (eg, a chip set). Various components, modules, or units are described in this disclosure to highlight functional aspects of a device configured to perform the disclosed techniques, but need not necessarily be implemented by different hardware units. Not always. Rather, as described above, the various units may be combined in a codec hardware unit, or interoperate including one or more processors as described above, with appropriate software and / or firmware. It may be provided by a collection of possible hardware units.

  Various examples have been described. These and other examples are within the scope of the following claims.

10 Video encoding and decoding system
12 Source device
14 Destination device
16 Computer-readable media
18 Video source
19 Video preprocessor
20 Video encoder
21 Video encoding unit
22 Output interface
28 Input interface
29 Video decoding unit
30 video decoder
31 Video post processor
32 display devices
110 Linear RGB data
112 Transfer function unit
112 'Applicable shape TF unit
114 color conversion unit
116 Quantization unit
118 HDR data
120 HDR data
122 Inverse quantization unit
124 Reverse color conversion unit
126 Inverse transfer function unit
128 linear RGB data
132 Video data memory
134 Pretreatment unit
138 Aftertreatment unit
140 Video data memory
142 Reverse pretreatment unit
144 Reverse post-processing unit
150 Aftertreatment unit
152 Reverse post-processing unit

Claims (38)

A method of video processing,
Receiving a first plurality of codewords representing shortened color values of video data, wherein the shortened color values represent colors in a first dynamic range; and
Non-shortening a second plurality of codewords based on the first plurality of codewords using an inverse static transfer function that is not adaptable to the video data to generate unshortened color values The first plurality of codewords, wherein the non-shortened color value represents a color in a second different dynamic range, and the second plurality of codewords are reverse post-processed Or one of codewords from the first plurality of codewords, and
At least one of reverse post-processing the first plurality of code words to generate the second plurality of code words, or reverse pre-processing the non-shortened color values;
Outputting the non-shortened color value or the reverse preprocessed non-shortened color value. At least one of reverse post-processing the first plurality of code words to generate the second plurality of code words, or reverse pre-processing the non-shortened color values;
Reverse post-processing the first plurality of codewords to generate the second plurality of codewords;
Reverse pre-processing the non-shortened color value to generate the reverse pre-processed non-shortened color value.
The method of claim 1.   The step of inverse post-processing comprises the step of inverse scaling and inverse offsetting the first plurality of codewords, wherein the step of inverse pre-processing is a non-shortened obtained from the step of non-shortening The method of claim 1, comprising inverse scaling and inverse offsetting the second plurality of codewords. Descale and reverse offset the first plurality of codewords, and descale and reverse offset the non-shortened second plurality of codewords obtained from the step of unshortening The method of claim 3, further comprising: deriving scaling and offset parameters for one or both of the steps from the received bitstream or separately signaled side information. Determining that a first set of codewords from the first plurality of codewords represents a shortened color value having a value less than a minimum threshold or greater than a maximum threshold;
Determining that a second set of codewords from the first plurality of codewords represents a shortened color value having a value greater than or equal to the minimum threshold and less than or equal to the maximum threshold. ,
Reverse post-processing the first plurality of codewords;
Assigning a first codeword to a codeword that is less than the minimum threshold of the first set of codewords;
Assigning a second codeword to a codeword greater than the maximum threshold of the first set of codewords;
Inverse scaling and inverse offsetting the second set of codewords;
The method of claim 1. Based on information received from side information signaled from a bitstream or separately signaled indicating the first codeword, the second codeword, the maximum threshold, or the minimum threshold, Determining one or more of a first codeword, the second codeword, the maximum threshold, and the minimum threshold, or the first codeword, the second code 6. The method of claim 5, further comprising: applying a process for determining a word, the maximum threshold, or the minimum threshold. Determining that a first set of codewords from the first plurality of codewords represents a shortened color value having a value belonging to a first partition of the first dynamic range;
Determining that a second set of codewords from the first plurality of codewords represents a shortened color value having a value belonging to a second segment of the first dynamic range; and
Reverse post-processing the first plurality of codewords;
Scaling and offsetting said first set of codewords using a first set of scaling and offset parameters;
Scaling and offsetting the second set of codewords using a second set of scaling and offset parameters;
The method of claim 1. Determining that the first set of non-shortened color values represents a non-shortened color value having a value belonging to a first segment of the second dynamic range;
Determining that the second set of non-shortened color values represents a non-shortened color value having a value belonging to a second segment of the second dynamic range; and
Reverse pre-processing the non-shortened color values comprises:
Scaling and offsetting the first set of non-shortened color values using a first set of scaling and offset parameters;
Scaling and offsetting the second set of non-shortened color values using a second set of scaling and offset parameters;
The method of claim 1. A method of video processing,
Receiving a plurality of color values of video data representing color in a first dynamic range;
Shortening the color value using a static transfer function that is not adaptable to the shortened video data to generate a plurality of codewords representing the shortened color value, the shortened color value comprising: A step whose value represents a color in a second different dynamic range;
At least one of pre-processing the color value before shortening to generate the color value to be shortened, or post-processing a codeword obtained from the step of shortening the color value; ,
Outputting a color value based on one of the shortened color value or the post-processed shortened color value. At least one of pre-processing the color value prior to shortening to generate the color value to be shortened, or post-processing a codeword obtained from the step of shortening the color value; ,
Pre-processing the color value before shortening to produce the color value to be shortened;
Post-processing the codeword obtained from the step of shortening the color value;
The method of claim 9.   The step of preprocessing comprises scaling and offsetting input linear color values, and the step of postprocessing comprises scaling and offsetting the codeword obtained from the step of shortening. The method described in 1. Scaling and offset parameters for one or both of the steps of scaling and offsetting input linear color values and scaling and offsetting the codeword obtained from the step of reducing in the bitstream The method according to claim 11, further comprising: signaling as side information.   Scaling and offsetting the input linear color value comprises adaptively determining a scaling factor and an offset factor based on the input linear color value, and the step of scaling and offsetting the codeword comprises an available code 12. The method of claim 11, comprising adaptively determining a scaling factor and an offset factor based on the codeword to enhance word space usage. Post-processing the codeword comprises:
Scaling and offsetting the codeword obtained from the step of shortening the color value;
Determining that the scaled and offset set of codewords has a value less than a minimum threshold or greater than a maximum threshold;
Assigning a first codeword to the set of scaled and offset codewords having a value less than the minimum threshold;
Assigning a second codeword to the set of scaled and offset codewords having a value greater than the maximum threshold.
The method of claim 9. Signaling information indicating one or more of the first codeword, the second codeword, the maximum threshold, or the minimum threshold in a bitstream or as side information 15. The method of claim 14, further comprising: Determining that the first set of color values has a value belonging to a first segment of the first dynamic range;
Determining that the second set of color values has a value belonging to a second segment of the first dynamic range; and
Pre-processing the color value comprises:
Scaling and offsetting the first set of color values using a first set of scaling and offset parameters;
Scaling and offsetting the second set of color values using a second set of scaling and offset parameters;
The method of claim 9. Determining that a first set of codewords from the plurality of codewords has a value belonging to a first partition of the second dynamic range;
Determining that a second set of codewords from the plurality of codewords has a value belonging to a second segment of the second dynamic range; and
Post-processing the plurality of codewords;
Scaling and offsetting said first set of codewords using a first set of scaling and offset parameters;
Scaling and offsetting the second set of codewords using a second set of scaling and offset parameters;
The method of claim 9. A device for video processing,
A video data memory configured to store video data;
A video post processor comprising at least one of a fixed function or a programmable circuit configuration, the video post processor comprising:
Receiving a first plurality of codewords representing a shortened color value of video data from the video data memory, wherein the shortened color value represents a color in a first dynamic range;
Non-shortening a second plurality of codewords based on the first plurality of codewords using an inverse static transfer function that is not adaptable to the video data to generate unshortened color values The first plurality of codewords, wherein the non-shortened color value represents a color in a second different dynamic range, and the second plurality of codewords are reverse post-processed Or one of the codewords from the first plurality of codewords;
At least one of reverse post-processing the first plurality of code words to generate the second plurality of code words, or reverse pre-processing the non-shortened color values;
Outputting the non-shortened color value or the reverse preprocessed non-shortened color value;
device. The video post processor is
Reverse post-processing the first plurality of codewords to generate the second plurality of codewords;
Configured to reverse pre-process the non-shortened color values to produce the reverse-preprocessed non-shortened color values;
The device of claim 18.   For reverse post-processing, the video post processor is configured to reverse scale and reverse offset the first plurality of codewords, and for reverse pre-processing, the video post processor 19. The device of claim 18, wherein the device is configured to inverse scale and inverse offset the non-shortened second plurality of codewords obtained from doing so. The video post processor is
De-scaling and inverse-offset the non-shortened second plurality of code words resulting from the de-scaling and reverse- offsetting the first plurality of code words and the non-shortening Configured to derive scaling and offset parameters for one or both of the received bitstreams or separately signaled side information;
21. A device according to claim 20. The video post processor is
Determining that a first set of codewords from the first plurality of codewords represents a shortened color value having a value less than a minimum threshold or greater than a maximum threshold;
Configured to determine that a second set of codewords from the first plurality of codewords represents a shortened color value having a value greater than or equal to the minimum threshold and less than or equal to the maximum threshold;
In order to reverse post-process the first plurality of codewords, the video post processor includes:
Assigning a first codeword to a codeword smaller than the minimum threshold of the first set of codewords;
Assigning a second codeword to a codeword greater than the maximum threshold of the first set of codewords;
Configured to inversely scale and inverse offset the second set of codewords;
The device of claim 18. The video post processor is
Based on information received from side information signaled from a bitstream or separately signaled indicating the first codeword, the second codeword, the maximum threshold, or the minimum threshold, Determining one or more of a first codeword, the second codeword, the maximum threshold, and the minimum threshold, or the first codeword, the second code Configured to apply a process for determining a word, the maximum threshold, or the minimum threshold;
23. A device according to claim 22. The video post processor is
Determining that a first set of codewords from the first plurality of codewords represents a shortened color value having a value belonging to a first partition of the first dynamic range;
Configured to determine that a second set of codewords from the first plurality of codewords represents a shortened color value having a value belonging to a second segment of the first dynamic range;
In order to reverse post-process the first plurality of codewords, the video post processor includes:
Using the first set of scaling and offset parameters to scale and offset the first set of codewords;
Configured to scale and offset the second set of codewords using a second set of scaling and offset parameters;
The device of claim 18. The video post processor is
Determining that the first set of non-shortened color values represents a non-shortened color value having a value belonging to a first segment of the second dynamic range;
Configured to determine that the second set of non-shortened color values represents a non-shortened color value having a value belonging to a second segment of the second dynamic range;
To reverse preprocess the non-shortened color values, the video post processor
Scaling and offset the first set of non-shortened color values using a first set of scaling and offset parameters;
Configured to scale and offset the second set of non-shortened color values using a second set of scaling and offset parameters;
The device of claim 18. A device for video processing,
A video data memory configured to store video data;
A video preprocessor comprising at least one of a fixed function or a programmable circuit configuration, the video preprocessor comprising:
Receiving a plurality of color values of video data representing colors in a first dynamic range from the video data memory;
Shortening the color value using a static transfer function that is not adaptable to the shortened video data to generate a plurality of codewords representing the shortened color value, The value represents a color in a second different dynamic range;
At least one of pre-processing the color value before shortening to generate the color value to be shortened, or post-processing a codeword resulting from the shortening of the color value; ,
Outputting a color value based on one of the shortened color value or the post-processed shortened color value,
device. The video preprocessor is
Pre-processing the color values before shortening to produce the color values to be shortened;
Configured to post-process codewords resulting from the shortening of the color values;
27. A device according to claim 26.   For preprocessing, the video preprocessor is configured to scale and offset the input linear color value, and for postprocessing, the video preprocessor scales and encodes the codeword resulting from the shortening. 27. The device of claim 26, configured to be offset.   A video encoder, wherein the video preprocessor is configured to either scale and offset the input linear color value to the video encoder and scale and offset the codeword resulting from the shortening. 30. The device of claim 28, configured to cause scaling and offset parameters for or both to be signaled in the bitstream or as side information.   To scale and offset the input linear color value, the video preprocessor is configured to adaptively determine a scaling factor and an offset factor based on the input linear color value, and to scale and offset the codeword 29. The device of claim 28, wherein the video preprocessor is configured to adaptively determine a scaling factor and an offset factor based on the codeword to enhance use of available codeword space. To post-process the codeword, the video preprocessor
Scaling and offsetting the codeword resulting from the shortening of the color value;
Determining that the set of scaled and offset codewords has a value less than a minimum threshold or greater than a maximum threshold;
Assigning a first codeword to the set of scaled and offset codewords having a value less than the minimum threshold;
Configured to assign a second codeword to the set of scaled and offset codewords having a value greater than the maximum threshold;
27. A device according to claim 26.   The video preprocessor further includes one or more of the first codeword, the second codeword, the maximum threshold, or the minimum threshold to the video encoder. 32. The device of claim 31, configured to cause the indicating information to be signaled in a bitstream or as side information. The video preprocessor is
Determining that the first set of color values has a value belonging to a first partition of the first dynamic range;
Configured to determine that the second set of color values has a value belonging to a second segment of the first dynamic range;
In order to preprocess the color values, the video preprocessor
Scaling and offsetting the first set of color values using a first set of scaling and offset parameters;
Configured to scale and offset the second set of color values using a second set of scaling and offset parameters;
27. A device according to claim 26. The video preprocessor is
Determining that a first set of codewords from the plurality of codewords has a value belonging to a first partition of the second dynamic range;
Configured to determine that a second set of codewords from the plurality of codewords has a value belonging to a second segment of the second dynamic range;
In order to post-process the plurality of codewords, the video preprocessor
Using the first set of scaling and offset parameters to scale and offset the first set of codewords;
Configured to scale and offset the second set of codewords using a second set of scaling and offset parameters;
27. A device according to claim 26. When executed, to one or more processors of the device for video processing,
Receiving a first plurality of codewords representing shortened color values of video data, wherein the shortened color values represent colors in a first dynamic range;
Non-shortening a second plurality of codewords based on the first plurality of codewords using an inverse static transfer function that is not adaptable to the video data to generate unshortened color values The first plurality of codewords, wherein the non-shortened color value represents a color in a second different dynamic range, and the second plurality of codewords are reverse post-processed Or one of the codewords from the first plurality of codewords;
At least one of reverse post-processing the first plurality of code words to generate the second plurality of code words, or reverse pre-processing the non-shortened color values;
A computer readable storage medium storing instructions for outputting the non-shortened color value or the reverse preprocessed non-shortened color value. Reverse post-processing the first plurality of codewords to generate the second plurality of codewords to the one or more processors, or reverse preprocessing the non-shortened color values. The instructions causing at least one of the instructions to the one or more processors,
Reverse post-processing the first plurality of codewords to generate the second plurality of codewords;
Instructions for reverse pre-processing the non-shortened color value to produce the non-shortened color value that has been reverse pre-processed;
36. A computer readable storage medium according to claim 35. A device for video processing,
Means for receiving a first plurality of codewords representing shortened color values of video data, wherein the shortened color values represent colors in a first dynamic range;
Non-shortening a second plurality of codewords based on the first plurality of codewords using an inverse static transfer function that is not adaptable to the video data to generate unshortened color values The non-shortened color value represents a color in a second different dynamic range, and the second plurality of codewords are reverse post-processed. Means, which is one of a codeword or a codeword from said first plurality of codewords;
At least one of means for reverse post-processing the first plurality of codewords to generate the second plurality of codewords, or means for reverse preprocessing the non-shortened color values; ,
Means for outputting the non-shortened color value or the reverse preprocessed non-shortened color value. The means for reverse post-processing the first plurality of codewords to generate the second plurality of codewords;
38. The device of claim 37, further comprising: said means for reverse preprocessing the non-shortened color values to produce the non-shortened color values that have been reverse preprocessed.