WO2018136432A1 - Remodelage à base de segments pour le codage de vidéo à plage dynamique élevée - Google Patents
Remodelage à base de segments pour le codage de vidéo à plage dynamique élevée Download PDFInfo
- Publication number
- WO2018136432A1 WO2018136432A1 PCT/US2018/013904 US2018013904W WO2018136432A1 WO 2018136432 A1 WO2018136432 A1 WO 2018136432A1 US 2018013904 W US2018013904 W US 2018013904W WO 2018136432 A1 WO2018136432 A1 WO 2018136432A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- frame
- segment
- frames
- dynamic range
- mapping
- Prior art date
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
- H04N19/98—Adaptive-dynamic-range coding [ADRC]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/136—Incoming video signal characteristics or properties
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/142—Detection of scene cut or scene change
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/179—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a scene or a shot
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
- H04N19/87—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving scene cut or scene change detection in combination with video compression
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
- H04N19/436—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation using parallelised computational arrangements
Definitions
- the present invention relates generally to images. More particularly, an embodiment of the present invention relates to the coding of video signals with high dynamic range using segment-based reshaping. BACKGROUND
- the term 'dynamic range' may relate to a capability of the human visual system (HVS) to perceive a range of intensity (e.g., luminance, luma) in an image, e.g., from darkest grays (blacks) to brightest whites (highlights).
- HVS human visual system
- DR relates to a 'scene-referred' intensity.
- DR may also relate to the ability of a display device to adequately or approximately render an intensity range of a particular breadth. In this sense, DR relates to a 'display-referred' intensity.
- the term may be used in either sense, e.g. interchangeably.
- the term high dynamic range relates to a DR breadth that spans the some 14-15 orders of magnitude of the human visual system (HVS).
- HVS human visual system
- the terms enhanced dynamic range (EDR) or visual dynamic range (VDR) may individually or interchangeably relate to the DR that is perceivable within a scene or image by a human visual system (HVS) that includes eye movements, allowing for some light adaptation changes across the scene or image.
- EDR may relate to a DR that spans 5 to 6 orders of magnitude.
- HDR high dynamic range
- n ⁇ 8 e.g., color 24-bit JPEG images
- images where n > 8 may be considered images of enhanced dynamic range.
- EDR and HDR images may also be stored and distributed using high-precision (e.g., 16-bit) floating-point formats, such as the
- a reference electro-optical transfer function (EOTF) for a given display characterizes the relationship between color values (e.g., luminance) of an input video signal to output screen color values (e.g., screen luminance) produced by the display.
- ITU Rec. BT. 1886 defines the reference EOTF for flat panel displays based on measured characteristics of the Cathode Ray Tube (CRT).
- CRT Cathode Ray Tube
- metadata relates to any auxiliary information that is transmitted as part of the coded bitstream and assists a decoder to render a decoded image.
- metadata may include, but are not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as those described herein.
- LDR lower dynamic range
- SDR standard dynamic range
- HDR content may be color graded and displayed on HDR displays that support higher dynamic ranges (e.g., from 1,000 nits to 5,000 nits or more).
- Such displays may be defined using alternative EOTFs that support high luminance capability (e.g., 0 to 10,000 nits).
- An example of such an EOTF is defined in SMPTE ST 2084:2014 (Ref. [2]).
- the methods of the present disclosure relate to any dynamic range higher than SDR.
- the term "reshaping" refers to a pre-processing operation on an HDR image, such as scaling, quantization, and the like, to map it from its original bit depth to an image of the same or lower bit depth, to allow for more efficient coding using existing coding standards and devices.
- 'Backward or Inverse reshaping' parameters used by an encoder may be communicated to a receiver as part of the coded bitstream using metadata so that a compliant decoder may apply an 'inverse' or 'backward reshaping' operation to reconstruct the original signal at its full dynamic range. Reshaping may be applied to any one or all of the color components of an HDR signal. In some embodiments, reshaping may also be constrained by the requirement to preserve on the decoded image the artistic intent of the original, for example, in terms of the accuracy of colors or "look,” as specified by a colorist under the supervision of the director.
- Existing reshaping techniques are typically scene-based.
- the term "scene” for a video sequence may relate to a series of consecutive frames in the video sequence sharing similar luminance, color and dynamic range characteristics.
- Scene-based methods work well in video-workflow pipelines which have access to the full scene; however, it is not unusual for content providers to use cloud- based multiprocessing, where, after dividing a video stream into segments, each segment is processed independently by a single computing node in the cloud.
- segment denotes a series of consecutive frames in a video sequence.
- a segment may be part of a scene or it may include one or more scenes. Thus, processing of a scene may be split across multiple processors.
- improved techniques for segment-based reshaping of HDR video are developed.
- FIG. 1 depicts an example process for data coding and decoding using reshaping techniques according to an embodiment of this invention
- FIG. 2 depicts an example of a backward luma reshaping function according to an embodiment of this invention
- FIG. 3 depicts an example of a two-stage distributed reshaping process according an embodiment of this invention
- FIG. 4 depicts an example of a single-stage distributed reshaping process according to an embodiment of this invention
- FIG. 5 depicts an example of an extended segment structure for segment-based reshaping according to an embodiment of this invention
- FIG. 6A depicts an example process for determining composer-metadata clusters according to an embodiment of this invention.
- FIG. 6B depicts an example of determining composer-metadata clusters according to the process depicted in FIG. 6A. DESCRIPTION OF EXAMPLE EMB ODIMENTS
- Example embodiments described herein relate to generating composer metadata which are transmitted from an encoder to a decoder for constructing in the decoder HDR images based on received SDR images.
- a processor accesses a database of representative input video clips in a first dynamic range (e.g., SDR) and converts them to output clips in a second dynamic range, higher than the first dynamic range (e.g., EDR or HDR).
- the processor computes a first histogram of luminance pixel values in the first dynamic range based on the frames of the input video clips, and computes a second histogram of luminance pixel values in the second dynamic range based on the frames of the output video clips.
- the processor computes a first luma cumulative distribution function (CDF) for luma pixel values in the first dynamic range based on the first histogram and a second luma CDF for luminance pixel values in the second dynamic range based on the second histogram.
- the processor computes a luma mapping based on the first and second CDFs for mapping luminance values of pixels in the first dynamic range into the second dynamic range.
- the processor Based on statistical data gathered from the input and output video data and a chroma prediction model, the processor computes a chroma mapping for mapping chroma values of pixel values in the first dynamic range into chroma values in the second dynamic range. Then, the processor generates static composer metadata for all frames to be encoded in the first dynamic range based on the luma mapping and the chroma mapping.
- a processor receives an input video signal in a first dynamic range and it divides it into segments. Each segment is distributed to a separate node processor.
- a node processor converts each frame of the input video signal in its segment into a corresponding frame of a second video signal in a second dynamic range, wherein the second dynamic range is higher than the first dynamic range.
- the node processor generates statistical data based on the input video frames and the second video frames in its segment.
- the node processor collects scene cut information for the frames of the input video frames in its segment.
- the post-processor identifies a scene boundary based the scene cut information from one or more segments.
- the post-processor generates updated statistics based on the generated statistics of all frames within the scene boundary, and determines a) a luma mapping for mapping luminance values of the frames within the scene boundary from the first dynamic range into the second dynamic range based on the updated statistics, and b) a chroma mapping for mapping chroma values of the frames within the scene boundary from the first dynamic range into the second dynamic range based on the updated statistics. Then, the post-processor generates composer metadata for the frames within the scene boundary based on the luma mapping and the chroma mapping.
- a processor receives an input video signal in a first dynamic range and divides it into segments, wherein each segment comprises primary frames (510) and secondary (padded) frames, wherein for two consecutive segments, padded frames for one of the segments overlap with the primary frames of the other segment.
- Each of the segments is processed by a node processor.
- the node processor converts each of the primary and padded frames of the input video signal in its segment into corresponding primary and padded frames of a second video signal in a second dynamic range (417), wherein the second dynamic range is higher than the first dynamic range.
- the node processor gathers statistical data (420) based on the input video frames and the second video frames in its segment and collects scene cut information (425) for the frames of the input video frames in its segment.
- the node processor generates a first sub-segment of the input video comprising the primary frames in its segment and a first part of the padded frames in its segment, then, for each frame in the first sub-segment of the input video (430): a) it computes a first support frame set (530) of smoothed statistical data based on a first sliding window centered on the frame of the first sub-segment and adjusted based on scene cuts in the segment, and b) determines a first luma mapping for mapping luminance values of the frame in the first sub- segment from the first dynamic range into the second dynamic range based on the statistical data and the first support frame set.
- the node processor for each primary frame in the segment of the input video (435): a) computes a second support frame set (545) of smoothed first luma mappings based on a second sliding window centered on the primary frame and adjusted based on scene cuts in the segment, b) determines a second luma mapping for mapping luminance values of the primary frame from the first dynamic range into the second dynamic range based on the first mapping and the second support frame set, and c) determines a chroma mapping for mapping chroma values of the primary frame from the first dynamic range into chroma values in the second dynamic range based on the statistical data and the second support frame set. Then, the node processor generates composer metadata for the frame in the segment based on the second luma mapping and the chroma mapping.
- SDI Serial Digital Interface
- H.264 or AVC
- H.265 or HEVC
- FIG. 1 depicts an example process (100) for data coding (100-E) and decoding (100-D) using luma and chroma reshaping according to an embodiment of this invention.
- a video sequence may be available in both high dynamic range (EDR, 102) and standard dynamic range (SDR, 104) formats.
- EDR, 102 high dynamic range
- SDR, 104 standard dynamic range
- the EDR sequence (referred herein also as iDM EDR) may be generated based on the SDR sequence using an inverse mapping process (iDM) (140). Examples for such a mapping process are given in Ref. [4] and Ref. [5].
- Input (102) may be coded according to a certain EOTF (e.g., gamma, ST 2084, and the like).
- luma reshaping (105-A) and chroma reshaping (105-B) processes may be applied to generate parameters (108) for backward or inverse reshaping functions (120) which when applied to the SDR input in a decoder will generate the EDR input (102).
- the original input SDR video signal (104) is passed to encoding block (110) for delivering downstream to decoding and playback devices such as television sets, set-top boxes, movie theaters, and the like.
- coding block (110) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other delivery formats, to generate coded bit stream (112).
- Coded data (112) and the backward reshaping parameters (108) are multiplexed into a coded bitstream to be stored or delivered to decoding devices.
- the coded data (112) is decoded by decoding unit (115) to generate a decoded signal (117) representing an identical or close approximation of SDR signal (104).
- decoded signal (117) may be displayed directly to SDR display (130).
- decoded signal (117) may be processed by a backward or inverse reshaping function (120) generated using the received backward reshaping parameters (108).
- Inverse reshaping converts the received SDR signal to an approximation of the (higher) dynamic range signal (102), to be displayed on an HDR display (125).
- Inverse reshaping may include separate inverse luma reshaping (120-A) and chroma reshaping (120-B).
- inverse reshaping (120) may also include additional (inverse) processes, such as inverse tone-mapping, color transformations, and the like.
- the backward or inverse reshaping function (120) may be integrated with a de-quantizer in decoder (115), e.g., as part of the de-quantizer in an AVC or HEVC video decoder.
- information about the backward reshaping parameters (108) may be communicated to downstream devices (such as decoders) using metadata, SEI messaging, and the like.
- CM Composer metadata
- Such metadata include all the backward reshaping parameters (108) needed for inverse or backward reshaping (120) in the decoder.
- Let/iX/) denote a forward luma reshaping function and let L -1 (7) denote its inverse, which when applied to the luma component of the decoder SDR data (117) generates the luma component (122_L) of the estimated EDR output (122).
- the / L -1 (7) function may be communicated as a piece-wise function.
- Such a function can be fully represented by a) a set of pivot points dividing the range of input codewords in the SDR input into segments, b) the order of the polynomial in each segment, and c) the polynomial coefficients in each segment.
- chroma reshaping may be performed using a multivariate, multiple regression (MMR) predictor.
- MMR multivariate, multiple regression
- Examples of such MMR predictors may be found in U.S. Patent 8,811,490 (Ref. [3]).
- MMR predictors may be found in U.S. Patent 8,811,490 (Ref. [3]).
- Such a predictor is characterized by a backward reshaping matrix M, which, when applied to the decoded SDR data (117) generates the chroma components (122_C) of the estimated EDR output.
- the composer metadata (108) may comprise a parametric representation of and a backward reshaping matrix M.
- Composer metadata may be updated at different levels of time intervals, such as: once for each sequence, for every scene, for each frame, or for clusters or groups of frames.
- Generating composer metadata for each frame may increase overall video quality, but requires higher computational resources on the encoder and increased overhead in the overall bitrate.
- this invention three different techniques for generating composer metadata are presented, each one addressing different trade-offs. The methods are
- This method provides a low overhead for composer metadata and is best suitable for a parallel-based computation environment with a very powerful postprocessing node.
- the composer metadata are always the same (fixed). Given a database of representative SDR reference clips, one may generate a corresponding set of EDR pictures using any of available inverse mapping techniques (140). Then, based on this pair of SDR/EDR data, one can generate a fixed set of composer metadata.
- CDF cumulative distribution functions
- s j y t , , and s ⁇ j denote the color components of the z ' -th pixel value at the j ' -th frame in the SDR data, where y denotes luminance (e.g., Y of YCbCr) and cO, cl, denote chrominance (e.g., Cb and Cr in YCbCr).
- v j y t , vj' and denote the color components of the corresponding z ' -th pixel value at the j-th frame in the inverse DM (iDM) EDR data. Denote as P the number of pixels in each frame.
- iDM inverse DM
- CDF-based luma reshaping maps SDR values to EDR values so that their corresponding cumulative density functions match. To accomplish this, typically, the following computations are needed:
- T b (k - 1) + ⁇ k-l
- Annex A provides in pseudocode an example implementation of the steps described above.
- FIG. 2 depicts an example SDR-to-EDR reshaping function for 8-bit SDR input values according to an embodiment using the steps described above and in Annex A.
- the vector sj j may be expressed using 15 values as
- equation (2) in some embodiments, some terms may be removed to reduce the computational load. For example, one may use in the model only one of the chroma components or one may eliminate completely certain high-order cross components. Without limitation, alternative linear or non- linear predictors may also be employed. [00038] Let the chroma components of the iDM EDR signal be denoted as c0
- chroma reshaping the goal is to generate an MMR backward reshaping matrix, M , such that the predicted EDR value, v , , is closest to v , .
- the overhead for composer metadata is very small since the same composer metadata are applicable to the whole video sequence. This low overhead may come at the expense of lower overall quality. For example, the reconstructed EDR images may look brighter than the reference iDM EDR data and some detail may be lost in the dark areas. Color accuracy may also be compromised.
- FIG. 3 depicts an example process for generating composer metadata using a two- stage, distributed, method. This method allows for creating new composer metadata for each scene in the video sequence, thus allowing for better luminance and color accuracy than the static reshaping method, at the cost of higher overhead for the composer metadata.
- the two-stage reshaping method may be deployed on a cloud- based, multi-processing computing system.
- a center dispatch node (305), may dispatch for processing to n parallel nodes (310) a fixed number of frames of the SDR input.
- This set of frames may be part of one or more scenes.
- some frames in node n may overlap with frames in neighbor nodes to improve scene-cut detection.
- each node (310- 0, 310-n-l) performs the following operations: a) inverse mapping (140, 315) to generate the corresponding iDM EDR frames (317) b) statistics collection (319) (e.g., generate luminance histograms ⁇ h j " b ⁇ , ⁇ h j v b ⁇ and chroma-based matrices ⁇ A ; .
- scene-cut detection (314) to determine whether each frame corresponds to a scene cut (e.g., if frame j starts a new scene, then SC j
- Scene-cut information and frame-statistics information will be sent for further processing to a central post-processing node (320).
- the post-processing node (320) identifies each scene boundary using the received 5C ; information, and for each scene generates the appropriate metadata for the luma and chroma backward reshaping functions using the same techniques as described earlier (e.g., equations (l)-(6)), but by taking into consideration only the statistical data generated within each scene. For example, let a scene (k) be determined in step (325) to be between frames fs and e. Then updated SDR and EDR histograms for scene k may be generated as:
- a luma reshaping function for the &-th scene may be generated in step (330) as described earlier.
- updated chroma-related matrices may be computed as
- FIG. 4 depicts an example process for generating composer metadata using a single-stage, distributed, method according to an embodiment.
- This architecture removes the need for a central post-processor at the expense of increased processing at each node, which now includes three additional processing steps: L2 smoothing (430), LI smoothing (435), and CM compression (440).
- L2 smoothing 430
- LI smoothing 435
- CM compression 440
- CM compression may also be performed by a central post-processing node (not shown).
- each node (except, perhaps, the node processing the last segment of the video) (410) receives a set of frames (F T ) defining a segment of a fixed-interval of the video.
- F T Typical segment sizes range in length between 10 to 30 seconds.
- a segment may have a total of 720 frames or pictures.
- a scene can cross multiple segments and a segment can include frames from multiple scenes.
- an encoder encodes each sub-scene individually, via local optimization.
- discontinuity in image characteristics such as a sudden luminance change or a sudden color change
- One approach to address this issue is by using extended or padded segments, where a padded segment may be partially processed by multiple computer nodes. For example, as depicted in FIG.
- fixed segment 510, encoded in Node t is padded by look-back overlap sub-segments 502 -b, 504-b, and 506-b, which will be encoded by Node t- 1 , and by look-ahead overlap segments 502-a, 504-a, and 506-a, which will be encoded in Node t+l.
- This approach offers better transition at the boundaries.
- there is no overlapped encoding that is, Node t will only compress frames in segment F T ⁇ however, reshaping functions and composer metadata are generated based on statistics from extended or padded segments.
- each of SDR frames will be converted to EDR frames (417).
- FIG.5 characterizes the following segments:
- step (425) scene cuts are detected for each frame j, within the range of frames in the L3 segment ( fs L ' 3 fe L ' 3 ), where:
- fe L ' 3 min ⁇ (t + 1) ⁇ F - 1 + Oll L2 + Oll Ll , F N - 1 ⁇ .
- the scene-cut sliding window (520) for each frame j has a range [ fl L ' 3 (j) fr L ' 3 ⁇ j) ], where
- r 3 -) minU + 0 ⁇ , ⁇ -1 ⁇ , (13) and is accessing all the frames in the full range of the Extended Primary t segment (540).
- step (420) frame-based statistics, such as ⁇ h j s h ⁇ , ⁇ h j v h ⁇ , ⁇ A . ⁇ , ⁇ B ; ⁇ , are also collected within the frame range of L3 segment (fs L ' 3 fe L ' 3 ).
- the next step is to smoothen out those statistics to take into consideration any scene cuts.
- the L2 sliding window (530) may be adjusted (bounded) according to any scene cuts detected within the extended L2 smoothing input range (525). Denote the nearest left scene cut for frame j as SL) and the nearest right scene cut for frame j as SR j ' . For each frame j, the L2 sliding window has a frame range:
- L2 smoothing refers to generating histograms and other statistics for frame j based on all the frames included in the L2 sliding window centered in frame j (see equation (15)).
- LI smoothing (to be defined later) may be performed for frames within the LI the range ( fs u ' fe u ' ) (510) using an LI sliding window (545), where
- LI smoothing refers to generating new statistical data by for frame j based on all the frames included in the LI sliding window centered in frame j (see equation (17)).
- Luma Reshaping is performed using both L2- and LI -level smoothing.
- the steps, described in more detail in pseudo-code in Tables 1 and 2, include: a) Compute statistics in each frame of the L3 segment (e.g., ⁇ h j s b ⁇ , ⁇ h j v h ⁇ and the like) b) Apply L2 smoothing to generated L2-smoothed statistical data to generate the CDFs c) Apply CDF matching (discussed earlier) to generate inverse luma reshaping LUTs T b ' d) Apply LI smoothing (e.g., averaging) to smooth the computed T b ' values to determine the final ⁇ LUT as
- the windowed LI and L2 smoothing techniques help to compensate for any false scene-cut detections and also assist later on in reducing ("compressing") the amount of composer metadata that need to be eventually transmitted by better detecting similar metadata.
- Table 1 Luma reshaping using L2 smoothing
- chroma reshaping is performed using only LI smoothing.
- An example embodiment is depicted in Table 3.
- fr L ' l ⁇ j min ⁇ + OL' L1 , fe L ' 2 , SR , F N - 1 ⁇
- a segment when a segment includes one or more complete scenes (e.g., Scene 1 in FIG.5), it may be possible to simplify the computations by generating composer metadata for the whole scene using the statistics generated within the L3 segment (525).
- LI and L2 smoothing may be performed only for frames outside the known scenes (e.g., in FIG. 5, for frames within Scene 0 and Scenes 2 and 3).
- Such approach relies on more accurate scene detection. Fade-ins and fade-outs may also affect such an approach, unless a fade-in-fade-out detector is added on top of the scene-cut detector.
- the sliding-window approach has better resilience in false scene-cut detection.
- CM Composer metadata
- the CM transmission syntax may include a composer metadata repeat flag, say, use_prev_cm_flag, which, when set to 1 indicates to the decoder that it can reuse a set of metadata which was previously transmitted.
- a composer metadata repeat flag say, use_prev_cm_flag
- use_prev_cm_flag when set to 1 indicates to the decoder that it can reuse a set of metadata which was previously transmitted.
- a repeat flag may be transmitted for every IDR (instantaneous decoding refresh) instance.
- a repeat flag may be transmitted at every IDR instance and at the beginning of each new scene.
- the composer metadata may change for each frame, thus a mechanism is desired to reduce the CM overhead. This CM reduction may be performed either inside each computing node (e.g., as part of the "CM compression" block (440)), or at a central, post-processing, node.
- CM may be computed for each frame
- new CM data are transmitted only when there is significant difference among their values.
- their similarity may be computed as
- each segment is subdivided into sub-partitions or clusters (say, denoted as ⁇ ⁇ ), where each sub-partition may be represented by a representative anchor frame (say, Ra). Then, when transmitting the composer metadata, only the CM data for the anchor frame of each sub-partition will be sent.
- the composer metadata may be reused, otherwise, a new cluster, with a new anchor frame is generated.
- the process may be repeated for all frames in the segment or for the entire sequence.
- Table 4 describes in pseudo code an example of this process according to an embodiment.
- R a 0 ; // set to store the anchor frame
- the above single-pass method is a greedy method. It simply picks as a new anchor the first frame which fails the similarity test. The processing can be easily done in a single scan; however, it might miss a better anchor frame later in the sequence. In other words, in terms of CM data, there is no need for the representative (or anchor) frame to be the first frame in a cluster.
- a more efficient two-pass method may be used.
- a binary map B is generated such that Then, new clusters and their anchor frames may be determined as follows: a) (Step 615): Let k define the start frame of a new cluster
- FIG.6B depicts an example of this process for a segment with 8 frames.
- the binary map B t (605).
- cluster 0 (607-0)
- frame no.2 is considered the anchor frame.
- cluster 1 (607-1)
- frame no.1 within this cluster (or frame no.5 in the segment) is considered the anchor frame for this cluster.
- Table 5 describes in pseudo-code an example of the two pass process described above.
- T b clip3(round(T b ),0, N v -1) ;
- Embodiments of the present invention may be implemented with a computer system, systems configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and/or apparatus that includes one or more of such systems, devices or components.
- IC integrated circuit
- FPGA field programmable gate array
- PLD configurable or programmable logic device
- DSP discrete time or digital signal processor
- ASIC application specific IC
- the computer and/or IC may perform, control, or execute instructions related to segment-based luma and chroma reshaping of images with enhanced dynamic range, such as those described herein.
- the computer and/or IC may compute any of a variety of parameters or values that relate to the reshaping processes described herein.
- the image and video embodiments may be implemented in hardware, software, firmware and
- Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention.
- processors in a display, an encoder, a set top box, a transcoder or the like may implement methods related to segment-based luma and/or chroma reshaping of HDR images as described above by executing software instructions in a program memory accessible to the processors.
- the invention may also be provided in the form of a program product.
- the program product may comprise any non-transitory medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention.
- Program products according to the invention may be in any of a wide variety of forms.
- the program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like.
- the computer-readable signals on the program product may optionally be compressed or encrypted.
- a component e.g. a software module, processor, assembly, device, circuit, etc.
- reference to that component should be interpreted as including as equivalents of that component any component which performs the function of the described component (e.g., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated example embodiments of the invention.
- EEE 1 A method for segment-based luma and/or chroma reshaping with a processor, the method comprising:
- each segment comprises primary frames (510) and padded frames, wherein for two consecutive segments padded frames for one of the segments overlap with the primary frames of the other segment;
- gathering statistical data comprises:
- EEE 3 The method of EEE 2, further comprising: computing where j denotes the index of the input video frame in the input signal, T denotes matrix transposition, V y comprises pixel values of the second video frame, and S J comprises values of the input video frame according to a prediction model used to generate predicted pixel values of the second video frames using pixel values of the input video frame.
- the prediction model is expressed as:
- V denotes the predicted pixel values of the second video frame and M denotes a matrix characterizing the prediction model.
- EEE 5 The method of EEE 3 or 4, wherein, for a primary frame in the segment, computing the chroma mapping comprises:
- a and B denote the updated Aj and Bj values for the primary frame
- M 7 denotes the parameter matrix of the chroma mapping
- fl L ' 1 (j) denote the start and end frames in the second support frame set for the primary frame in the segment.
- EEE 7 The method of EEE 6, wherein the composer metadata for the chroma mapping of the frame comprise the elements of the M 7 matrix.
- EEE 8 The method of any of EEEs 2-7, wherein, for a frame in the first sub-segment, computing the first luma mapping comprises:
- CDF cumulative distribution function
- EEE 9 The method of EEE 8, wherein, for a primary frame in the segment, computing the second luma mapping comprises:
- EEE 10 The method of EEE 9, wherein generating composer metadata for the frame based on the second luma mapping comprises:
- EEE 11 The method of any of EEEs 1-10, further comprising reducing the amount of metadata transmitted for the segment by applying a composer metadata compression step, wherein the composer metadata compression step comprises:
- EEE 12 The method of EEE 11, wherein dividing the primary frames into clusters comprises:
- EEE 13 The method of EEE 11 or 12, wherein dividing the primary frames into clusters comprises:
- step b) based on step b), find the longest run of similar frames and include them in the set of frames within the cluster a;
- d denote as the anchor frame the frame with the longest run.
- EEE 14 A method for segment-based luma and/or chroma reshaping with one or more processors, the method comprising:
- EEE 15 The method of EEE 14, wherein gathering statistical data comprises:
- B, S;V, , where j denotes the index of the input video frame in the input signal, T denotes matrix transposition, V Y comprises pixel values of the second video frame, and S J comprises values of the input video frame according to a prediction model used to generate predicted pixel values of the second video frames using pixel values of the input video frame.
- the prediction model is expressed as:
- V denotes the predicted pixel values of the second video frame and M denotes a matrix characterizing the prediction model.
- EEE 18 The method of EEE 16 or 17, wherein, for a frame within the scene boundary, computing the chroma mapping comprises:
- EEE 19 The method of any of EEEs 15-18, wherein, for a frame within the scene boundary, computing the luma mapping comprises:
- CDF cumulative distribution function
- EEE 20 An apparatus comprising a processor and configured to perform any one of the methods recited in EEEs 1-19.
- EEE 21 A non- transitory computer-readable storage medium having stored thereon computer-executable instruction for executing a method in accordance with any of the EEEs 1-19.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
La présente invention concerne des techniques de génération et de compression de métadonnées de composeur décrivant des fonctions de remodelage de luminance et de chrominance inverses en fonction d'une entrée vidéo de plage dynamique standard (SDR). En fonction de l'entrée SDR, les métadonnées de composeur permettent à un décodeur de générer une sortie correspondante dans une plage dynamique élevée. Trois techniques sont proposées : une architecture statique reposant sur des séquences, une solution distribuée reposant sur des scènes en deux étapes comportant un procédé de post-traitement centralisé, et une solution distribuée à une seule étape utilisant des segments chevauchés. L'invention concerne également des techniques destinées à réduire la quantité de métadonnées de composeur transmises.
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2019538633A JP6684971B2 (ja) | 2017-01-18 | 2018-01-16 | ハイダイナミックレンジ映像を符号化するためのセグメントベース再構成 |
CN201880006533.7A CN110178374B (zh) | 2017-01-18 | 2018-01-16 | 用于对高动态范围视频进行编码的基于段的整形 |
US16/462,496 US10902601B2 (en) | 2017-01-18 | 2018-01-16 | Segment-based reshaping for coding high dynamic range video |
EP18711430.1A EP3571843A1 (fr) | 2017-01-18 | 2018-01-16 | Remodelage à base de segments pour le codage de vidéo à plage dynamique élevée |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201762447517P | 2017-01-18 | 2017-01-18 | |
US62/447,517 | 2017-01-18 | ||
EP17152004 | 2017-01-18 | ||
EP17152004.2 | 2017-01-18 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2018136432A1 true WO2018136432A1 (fr) | 2018-07-26 |
Family
ID=57860700
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2018/013904 WO2018136432A1 (fr) | 2017-01-18 | 2018-01-16 | Remodelage à base de segments pour le codage de vidéo à plage dynamique élevée |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2018136432A1 (fr) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11310537B2 (en) | 2018-02-28 | 2022-04-19 | Dolby Laboratories Licensing Corporation | Linear encoder for image/video processing |
US12047686B2 (en) | 2019-08-12 | 2024-07-23 | Dolby Laboratories Licensing Corporation | Adaptive image data linearization for HDR image sensors |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2003077549A1 (fr) * | 2002-03-13 | 2003-09-18 | Imax Corporation | Systemes et procedes permettant de remasterisation ou de modification numerique de films cinematographiques ou autres donnees de sequences d'images |
US20140029675A1 (en) * | 2011-04-14 | 2014-01-30 | Dolby Laboratories Licensing Corporation | Multiple Color Channel Multiple Regression Predictor |
WO2015130797A1 (fr) * | 2014-02-25 | 2015-09-03 | Apple Inc. | Fonction de transfert adaptative pour un codage et un décodage vidéo |
WO2016040255A1 (fr) * | 2014-09-09 | 2016-03-17 | Dolby Laboratories Licensing Corporation | Procédé de prédiction auto-adaptatif pour codec multi-couche |
US20160253792A1 (en) | 2013-10-22 | 2016-09-01 | Dolby Laboratories Licensing Corporation | Guided color grading for extended dynamic range |
US20160358319A1 (en) | 2013-11-22 | 2016-12-08 | Dolby Laboratories Licensing Corporation | Methods and systems for inverse tone mapping |
-
2018
- 2018-01-16 WO PCT/US2018/013904 patent/WO2018136432A1/fr active Search and Examination
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2003077549A1 (fr) * | 2002-03-13 | 2003-09-18 | Imax Corporation | Systemes et procedes permettant de remasterisation ou de modification numerique de films cinematographiques ou autres donnees de sequences d'images |
US20140029675A1 (en) * | 2011-04-14 | 2014-01-30 | Dolby Laboratories Licensing Corporation | Multiple Color Channel Multiple Regression Predictor |
US8811490B2 (en) | 2011-04-14 | 2014-08-19 | Dolby Laboratories Licensing Corporation | Multiple color channel multiple regression predictor |
US20160253792A1 (en) | 2013-10-22 | 2016-09-01 | Dolby Laboratories Licensing Corporation | Guided color grading for extended dynamic range |
US20160358319A1 (en) | 2013-11-22 | 2016-12-08 | Dolby Laboratories Licensing Corporation | Methods and systems for inverse tone mapping |
WO2015130797A1 (fr) * | 2014-02-25 | 2015-09-03 | Apple Inc. | Fonction de transfert adaptative pour un codage et un décodage vidéo |
WO2016040255A1 (fr) * | 2014-09-09 | 2016-03-17 | Dolby Laboratories Licensing Corporation | Procédé de prédiction auto-adaptatif pour codec multi-couche |
Non-Patent Citations (4)
Title |
---|
"High Dynamic Range EOTF of Mastering Reference Displays", SMPTE ST 2084, 2014 |
"Reference electro-optical transfer function for flat panel displays used in HDTV studio production", ITU-R BT. 1886, March 2011 (2011-03-01) |
ROLLAND J P ET AL: "FAST ALGORITHMS FOR HISTOGRAM MATCHING: APPLICATION TO TEXTURE SYNTHESIS", JOURNAL OF ELECTRONIC IMAGING, SPIE / IS & T, 1000 20TH ST. BELLINGHAM WA 98225-6705 USA, vol. 9, no. 1, 1 January 2000 (2000-01-01), pages 39 - 45, XP000902783, ISSN: 1017-9909, DOI: 10.1117/1.482725 * |
ZHU DINGJU ED - XU ZHENG ET AL: "Overlapping boundary based multimedia slice transcoding method and its system for medical video and traffic video", MULTIMEDIA TOOLS AND APPLICATIONS, KLUWER ACADEMIC PUBLISHERS, BOSTON, US, vol. 75, no. 22, 12 January 2016 (2016-01-12), pages 14233 - 14246, XP036082930, ISSN: 1380-7501, [retrieved on 20160112], DOI: 10.1007/S11042-015-3235-8 * |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11310537B2 (en) | 2018-02-28 | 2022-04-19 | Dolby Laboratories Licensing Corporation | Linear encoder for image/video processing |
US12047686B2 (en) | 2019-08-12 | 2024-07-23 | Dolby Laboratories Licensing Corporation | Adaptive image data linearization for HDR image sensors |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10902601B2 (en) | Segment-based reshaping for coding high dynamic range video | |
US10397576B2 (en) | Reshaping curve optimization in HDR coding | |
US10575028B2 (en) | Coding of high dynamic range video using segment-based reshaping | |
EP3433833B1 (fr) | Codage et décodage de signaux vidéo en une couche à qualité de production réversible | |
CN112106357B (zh) | 用于对图像数据进行编码和解码的方法及装置 | |
US10015491B2 (en) | In-loop block-based image reshaping in high dynamic range video coding | |
US10419762B2 (en) | Content-adaptive perceptual quantizer for high dynamic range images | |
US10701359B2 (en) | Real-time content-adaptive perceptual quantizer for high dynamic range images | |
CN109792523B (zh) | 单层后向兼容编解码器的实时整形 | |
US10223774B2 (en) | Single-pass and multi-pass-based polynomial approximations for reshaping functions | |
WO2018049335A1 (fr) | Codage vidéo à plage dynamique élevée à l'aide d'un remodelage basé sur des segments | |
WO2018231968A1 (fr) | Codage efficace de gestion d'affichage inverse à couche unique de bout en bout | |
CN113170205A (zh) | 整形函数的插值 | |
EP3639238A1 (fr) | Codage efficace de gestion d'affichage inverse à couche unique de bout en bout | |
US11895416B2 (en) | Electro-optical transfer function conversion and signal legalization | |
WO2018136432A1 (fr) | Remodelage à base de segments pour le codage de vidéo à plage dynamique élevée | |
US20230291937A1 (en) | Workload allocation and processing in cloud-based coding of hdr video | |
US20230388555A1 (en) | Trim-pass correction for cloud-based coding of hdr video | |
EP4214926A1 (fr) | Segmentation récursive de segment à scène pour le codage en nuage de vidéo hdr |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
DPE1 | Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101) | ||
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18711430 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2019538633 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2018711430 Country of ref document: EP Effective date: 20190819 |