WO2016111799A1 - A method of encoding video with film grain - Google Patents

A method of encoding video with film grain Download PDF

Info

Publication number
WO2016111799A1
WO2016111799A1 PCT/US2015/064703 US2015064703W WO2016111799A1 WO 2016111799 A1 WO2016111799 A1 WO 2016111799A1 US 2015064703 W US2015064703 W US 2015064703W WO 2016111799 A1 WO2016111799 A1 WO 2016111799A1
Authority
WO
WIPO (PCT)
Prior art keywords
frames
video
film grain
score
encoder
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2015/064703
Other languages
French (fr)
Inventor
Wei Zhou
Jane Perera
Wayne D. MICHELSEN
Swaroop CHANDRA
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Arris Enterprises LLC
Original Assignee
Arris Enterprises LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Arris Enterprises LLC filed Critical Arris Enterprises LLC
Priority to MX2017008847A priority Critical patent/MX368521B/en
Priority to CA2972989A priority patent/CA2972989C/en
Priority to BR112017014421-2A priority patent/BR112017014421B1/en
Publication of WO2016111799A1 publication Critical patent/WO2016111799A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/184—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/115—Selection of the code volume for a coding unit prior to coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/117—Filters, e.g. for pre-processing or post-processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/119—Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/124—Quantisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H04N19/137—Motion inside a coding unit, e.g. average field, frame or block difference
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/179—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a scene or a shot
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/593—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial prediction techniques
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • H04N19/86—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving reduction of coding artifacts, e.g. of blockiness
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/144—Movement detection

Definitions

  • the present invention relates to a process for improving video quality when encoding video with film grain. More particularly, the present invention relates to a solution to improving video quality when film grain is present on a level similar to the "Red Lady" noise problem.
  • Film grain is hard to compress in an encoder. It requires more bits to encode than many other kinds of content for any level of video quality. Film grain may be thought of as a particular kind of spatial temporal noise. As such, film grain has low temporal predictability from one frame to any other frame of video. Thus, the encoding process is limited in its ability to leverage inter-frame estimation to achieve significant compression efficiency.
  • FIG. 1 A frame of the "Red Lady” video is illustrated with Fig. 1.
  • the "red lady” video shows that a lady walking alongside a soccer field with a grassy background.
  • the scene itself is simple, but the entire video is filled with film grain.
  • Film grain is like random noise. It requires a lot of bits to encode and is not temporally predictable, which makes high film grain video, in particular the "Red Lady” video, very difficult to encode.
  • a common practice to encode video with film grain is to encode a good quality I frame as a reference frame as a prediction frame for subsequent predictive frames (P or B frames).
  • P or B frames predictive frames
  • the I frame and P and B frames all contain film grain this does not predict well, and many bits are needed to encode unpredicted high frequency components. If too many bits are allocated to the I frame, later P and B frames may be allocated fewer bits than they need, and their quality suffers.
  • the good quality I frame thus, may not help with the subsequent P and B frames.
  • the difference between two consecutive frames is mostly noise. Encoding a good quality I frame for these costs too many bits and leaves fewer bits for predictive frames. Moreover, the high quality I frame, even with additional bits allocated, is not a good reference frame because the noisy temporal differences cannot be motion predicted well. With the I frame as a reference, the film grains in the predictive frames would be poorly encoded, and create a quality disparity between I and predictive frames, as illustrated in Fig. 3. Thus allocating more bits to the I frame creates a "Dirty Window" for future film grain elimination in P and B frames.
  • Embodiments of the present invention provide a system that enables improved video quality and compression efficiency during encoding by detecting video segments having film grain approaching the "Red Lady” problem and then optimizing the bit allocation between intra- and inter-predicted pictures using bit allocation variation between I, P and B type frames.
  • embodiments of the present invention encode smaller I frames and allocate more bits on P and B frames. Since allocating more bits to the I frame when the "Red Lady” film grain problem occurs does not enable better prediction for encoding in the P and B frames, additional bits to the I frame are not necessary. Thus, allocating extra bits allocated to the P and B frames and not using the extra bits in the I frame enables reduction of frame grain when the "Red Lady” like film grain problem occurs, and the "Dirty Window" I frame issue will no longer be a consideration.
  • a temporal analysis of motion-prediction data available is provided.
  • measurements of plotted frame-by-frame temporal differences (ME scores) of the received videos are determined.
  • ME scores plotted frame-by-frame temporal differences
  • two key indicators are identified: (1) The average temporal difference in frames with an intermediate motion level (i.e., greater than ME score of 20), is higher than frames of non-noisy video with intermediate motion; and (2) The fluctuation of the temporal differences between frames in a group is very small, unlike the non-noise video with natural motions which have higher motion differences without the noise.
  • These two indicators are set to identify when a special bit allocation ratio between the I, P and B frames should be applied so that there will tend to be less difference between frame types for film-grain content.
  • the system uses a preprocessing filter that analyzes video frames prior to the encoder.
  • the preprocessing filter computes the temporal difference score and stores it in a queue of data provided with the frames to the encoder.
  • the encoder analyzes the temporal difference scores. If it is detected that the average of all temporal differences is higher than a threshold and the variance of them are smaller than a threshold, it means the video contains significant film grain or noise. Based on the level of film grain or noise detected, the encoder allocates bits to I, P and B frames dynamically.
  • Fig. 1 shows a frame from the "Red Lady” video
  • Fig. 2 illustrates that for the "Red Lady” video frames, the difference between two consecutive frames is mostly noise
  • Fig. 3 illustrates that with an I frame as a reference, the film grains in the predictive frames would be poorly encoded, and create a quality disparity between I and predictive frames;
  • Figs. 4A-4F show the measured and plotted frame-by-frame temporal differences (ME scores) of various video clips
  • FIG. 5 is a diagram of components for implementing embodiments of the present invention in an encoding system.
  • Fig. 6 is a flow chart illustrating steps for implementing encoding with film grain according to embodiments of the present invention.
  • a clip can be identified as a "Red Lady” like clip, good quality can be achieved by encoding smaller I frames and allocating more bits on P and B frames.
  • Figs. 4A-4F show the measured and plotted frame- by-frame temporal differences (ME scores) of various video clips.
  • the X axis is a frame index and the Y axis shows a scaled ME score ranging from 0 to 100.
  • Figs. 4A-4C illustrate the ME score levels for video with noise and limited or no motion.
  • the Red Lady video has a constant noise level ME score of just over 20.
  • the "Sweep" video noise levels of Fig. 4B are very high with an ME noise level of near 100.
  • the "Zone Plate” noise with no motion and a set noise level has an ME score of just over 20 and can be used as a reference for ME levels.
  • the "Sprinkler Lady” of Fig. 4E meets both the two key factors of (1) an ME score over 20 and (2) the fluctuation of the temporal differences is small even taking into account motion.
  • the final video of "Basketball” in Fig. 4F has a relatively high ME score, but it is less than 20, and the motion in the video accounts for ME levels that on average may push the total ME score over 20.
  • the video of Fig. 4F does not require embodiments of the present invention to be used during encoding.
  • Fig. 5 is a diagram of one embodiment of components for implementing embodiments of the present invention in an encoding system.
  • the preprocessing filter 500 computes the temporal difference score and stores it in the queue 502.
  • the encoding in encoder 506 will be delayed by the frame buffer 504 until temporal difference scores of N frames are collected in the queue 502.
  • the encoder 506 analyzes the temporal difference scores of N frames. If it is detected that the average of all temporal differences is higher than a threshold and the variance of them are smaller than a threshold, it means the video contains significant film grain or noise. Based on the level of film grain or noise detected, the encoder 506 allocates bits to I, P and B dynamically according to embodiments of the present invention described herein. Generally, if the level of film grain or noise is high, the encoder allocates more bits to P and B frames than for other content.
  • Fig. 6 is a flow chart illustrating steps for implementing encoding when high film grain is detected according to embodiments of the present invention.
  • the video clips are received, such as the "Red Lady” clip illustrated in Fig. 4C.
  • the temporal difference score, or ME score is determined for the video frames and the result for each frame stored in a queue to provide to the encoder.
  • a determination is made to decide if film grain noise is high enough to constitute "Red Lady” type film grain that requires application of embodiments of the present invention.
  • step 602 if the average of a group of temporal differences is higher than a threshold and the variance is smaller than a threshold, the film grain noise is indicated to be significant for the frames of the video clip. [0029] Once the determination is made in 602, next in step 603 the determination is reviewed. If film grain noise for the clip is determined to be significant, then the program proceeds to step 604. If the film grain noise is determined to be insignificant, the program proceeds to step 605. In step 604 when high film grain noise is detected encoding is performed by allocating enough bits so that the I frame at the beginning has little additional bits and the P and B frames have additional bits for encoding. In step 605 when film grain noise is not detected as high, a normal bit allocation is performed by the encoder.
  • Appendix A shows an example of coding in "C" to implement the algorithm illustrated by Fig. 6.
  • each component can include a processor and memory to enable operation.
  • the memory of each device stores code that is executable by the processor to enable the processor to perform the processes described herein. Further the memory can be used to provide data storage with the data accessible by the processor to store or retrieve when performing operations.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Color Television Systems (AREA)

Abstract

A system for providing improved video quality and compression efficiency during encoding by detecting video segments having film grain approaching the "Red Lady" problem. The system detects when film grain approaches the level of the "Red Lady" problem by measuring frame-by-frame temporal differences (ME scores). From the ME scores, two key indicators are identified: (1) The average temporal difference in frames with an intermediate motion level higher than frames of non-noisy video; and (2) The fluctuation of the temporal differences between frames in a group is very small. When these indicators identify a high film video, a signal is provided to an encoder which allocates less bits to I frames and more bits to P and B frames than for other frames of video without comparable film grain.

Description

A METHOD OF ENCODING VIDEO WITH FILM GRAIN
CROSS REFERENCE TO RELATED APPLICATION
[0001] This Application claims priority under 35 U.S.C. §119(e) from earlier filed United States Provisional Application Serial No. 62/099,672 filed on January 5, 2015 and United States Application No. 14/962,814 filed on December 8, 2015 and incorporated herein by reference in its entirety.
BACKGROUND
TECHNICAL FIELD
[0002] The present invention relates to a process for improving video quality when encoding video with film grain. More particularly, the present invention relates to a solution to improving video quality when film grain is present on a level similar to the "Red Lady" noise problem.
RELATED ART
[0003] Film grain is hard to compress in an encoder. It requires more bits to encode than many other kinds of content for any level of video quality. Film grain may be thought of as a particular kind of spatial temporal noise. As such, film grain has low temporal predictability from one frame to any other frame of video. Thus, the encoding process is limited in its ability to leverage inter-frame estimation to achieve significant compression efficiency.
[0004] In some encoders, not enough bits are allocated to inter-predicted pictures. The result can be significant video quality artifacts such as I-frame beating and intermittent repetitive loss and recovery of spatial texture. Even encoders that can allocate significant bits might not be able to eliminate substantial noise, such as when noise is as high as in the "Red Lady" video frames.
A. The "Red Lady" Problem
[0005] A frame of the "Red Lady" video is illustrated with Fig. 1. The "red lady" video shows that a lady walking alongside a soccer field with a grassy background. The scene itself is simple, but the entire video is filled with film grain.
[0006] Film grain is like random noise. It requires a lot of bits to encode and is not temporally predictable, which makes high film grain video, in particular the "Red Lady" video, very difficult to encode.
[0007] A common practice to encode video with film grain is to encode a good quality I frame as a reference frame as a prediction frame for subsequent predictive frames (P or B frames). However, since the I frame and P and B frames all contain film grain this does not predict well, and many bits are needed to encode unpredicted high frequency components. If too many bits are allocated to the I frame, later P and B frames may be allocated fewer bits than they need, and their quality suffers. The good quality I frame, thus, may not help with the subsequent P and B frames.
[0008] In the Red Lady Video, the random noise level is very high. Thus, in the Red Lady Video, a beating effect will be seen, due to the quality of the pictures varying too much between frame types. Allocating more bits to the I frame does not help with reducing film grain in subsequent P and B frames in a typical encoder. B. The "Dirty Window" Problem
[0009] As shown in Fig. 2, the difference between two consecutive frames is mostly noise. Encoding a good quality I frame for these costs too many bits and leaves fewer bits for predictive frames. Moreover, the high quality I frame, even with additional bits allocated, is not a good reference frame because the noisy temporal differences cannot be motion predicted well. With the I frame as a reference, the film grains in the predictive frames would be poorly encoded, and create a quality disparity between I and predictive frames, as illustrated in Fig. 3. Thus allocating more bits to the I frame creates a "Dirty Window" for future film grain elimination in P and B frames.
[0010] Accordingly, it is desirable to provide better solutions for eliminating film grain comparable to the "Red Lady" video, and to avoid creating "Dirty Window" I frames.
SUMMARY
[0011] Embodiments of the present invention provide a system that enables improved video quality and compression efficiency during encoding by detecting video segments having film grain approaching the "Red Lady" problem and then optimizing the bit allocation between intra- and inter-predicted pictures using bit allocation variation between I, P and B type frames.
[0012] To optimize the bit allocation when a video clip is identified as a "Red Lady" like clip, embodiments of the present invention encode smaller I frames and allocate more bits on P and B frames. Since allocating more bits to the I frame when the "Red Lady" film grain problem occurs does not enable better prediction for encoding in the P and B frames, additional bits to the I frame are not necessary. Thus, allocating extra bits allocated to the P and B frames and not using the extra bits in the I frame enables reduction of frame grain when the "Red Lady" like film grain problem occurs, and the "Dirty Window" I frame issue will no longer be a consideration.
[0013] To identify the film grain level to determine when the optimization of bit allocation away from I frames to P and B frames should occur, a temporal analysis of motion-prediction data available is provided. For the temporal analysis, measurements of plotted frame-by-frame temporal differences (ME scores) of the received videos are determined. From the ME scores, two key indicators are identified: (1) The average temporal difference in frames with an intermediate motion level (i.e., greater than ME score of 20), is higher than frames of non-noisy video with intermediate motion; and (2) The fluctuation of the temporal differences between frames in a group is very small, unlike the non-noise video with natural motions which have higher motion differences without the noise. These two indicators are set to identify when a special bit allocation ratio between the I, P and B frames should be applied so that there will tend to be less difference between frame types for film-grain content.
[0014] The system according to embodiments of the present invention uses a preprocessing filter that analyzes video frames prior to the encoder. The preprocessing filter computes the temporal difference score and stores it in a queue of data provided with the frames to the encoder. The encoder analyzes the temporal difference scores. If it is detected that the average of all temporal differences is higher than a threshold and the variance of them are smaller than a threshold, it means the video contains significant film grain or noise. Based on the level of film grain or noise detected, the encoder allocates bits to I, P and B frames dynamically. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Further details of the present invention are explained with the help of the attached drawings in which:
[0016] Fig. 1 shows a frame from the "Red Lady" video;
[0017] Fig. 2 illustrates that for the "Red Lady" video frames, the difference between two consecutive frames is mostly noise;
[0018] Fig. 3 illustrates that with an I frame as a reference, the film grains in the predictive frames would be poorly encoded, and create a quality disparity between I and predictive frames;
[0019] Figs. 4A-4F show the measured and plotted frame-by-frame temporal differences (ME scores) of various video clips;
[0020] Fig. 5 is a diagram of components for implementing embodiments of the present invention in an encoding system; and
[0021] Fig. 6 is a flow chart illustrating steps for implementing encoding with film grain according to embodiments of the present invention.
DETAILED DESCRIPTION
[0022] For embodiments of the present invention, if a clip can be identified as a "Red Lady" like clip, good quality can be achieved by encoding smaller I frames and allocating more bits on P and B frames.
[0023] To help understand how a determination of when a clip is a "Red Lady" type clip, several different clips of data are analyzed. Figs. 4A-4F show the measured and plotted frame- by-frame temporal differences (ME scores) of various video clips. In Figs. 4A-4F, the X axis is a frame index and the Y axis shows a scaled ME score ranging from 0 to 100.
[0024] From the measurements in Figs. 4A-4F, two key indicators of noisy video are found: (1) The average temporal difference is at an intermediate level (greater than ME score of 20), and the ME score is higher than the one of non-noisy video with intermediate motion; and (2) The fluctuation of the temporal differences is very small, not like the non-noise video with natural motions. These two indicators are, thus, used to identify "Red Lady" like video frames from any streaming video.
[0025] Figs. 4A-4C illustrate the ME score levels for video with noise and limited or no motion. In Fig. 4C, the Red Lady video has a constant noise level ME score of just over 20. The "Sweep" video noise levels of Fig. 4B are very high with an ME noise level of near 100. The "Zone Plate" noise with no motion and a set noise level has an ME score of just over 20 and can be used as a reference for ME levels. Note that in the "Blacksmith" frame of Fig. 4D, a normal video that does not need special treatment during encoding using embodiments of the present invention that the temporal average noise difference is far below an ME level of 20 found with the "Red Lady" video of Fig. 4C.
[0026] Unlike the "Blacksmith" of Fig. 4D, the "Sprinkler Lady" of Fig. 4E meets both the two key factors of (1) an ME score over 20 and (2) the fluctuation of the temporal differences is small even taking into account motion. The final video of "Basketball" in Fig. 4F has a relatively high ME score, but it is less than 20, and the motion in the video accounts for ME levels that on average may push the total ME score over 20. However with the criteria of (1) an average temporal difference with a ME score greater than 20 and (2) the fluctuation of the temporal differences very small, the video of Fig. 4F does not require embodiments of the present invention to be used during encoding.
[0027] Fig. 5 is a diagram of one embodiment of components for implementing embodiments of the present invention in an encoding system. In Fig. 5, the preprocessing filter 500 computes the temporal difference score and stores it in the queue 502. The encoding in encoder 506 will be delayed by the frame buffer 504 until temporal difference scores of N frames are collected in the queue 502. The encoder 506 analyzes the temporal difference scores of N frames. If it is detected that the average of all temporal differences is higher than a threshold and the variance of them are smaller than a threshold, it means the video contains significant film grain or noise. Based on the level of film grain or noise detected, the encoder 506 allocates bits to I, P and B dynamically according to embodiments of the present invention described herein. Generally, if the level of film grain or noise is high, the encoder allocates more bits to P and B frames than for other content.
[0028] Fig. 6 is a flow chart illustrating steps for implementing encoding when high film grain is detected according to embodiments of the present invention. First, in step 600 the video clips are received, such as the "Red Lady" clip illustrated in Fig. 4C. Next, in step 601, the temporal difference score, or ME score, is determined for the video frames and the result for each frame stored in a queue to provide to the encoder. Next, in step 602 a determination is made to decide if film grain noise is high enough to constitute "Red Lady" type film grain that requires application of embodiments of the present invention. For the step 602 determination, if the average of a group of temporal differences is higher than a threshold and the variance is smaller than a threshold, the film grain noise is indicated to be significant for the frames of the video clip. [0029] Once the determination is made in 602, next in step 603 the determination is reviewed. If film grain noise for the clip is determined to be significant, then the program proceeds to step 604. If the film grain noise is determined to be insignificant, the program proceeds to step 605. In step 604 when high film grain noise is detected encoding is performed by allocating enough bits so that the I frame at the beginning has little additional bits and the P and B frames have additional bits for encoding. In step 605 when film grain noise is not detected as high, a normal bit allocation is performed by the encoder.
[0030] The results of applying the algorithm shown in Fig. 6 will reduce the "dirty window" effect. The algorithm also makes some high texture clips, such as "sprinkler" of Fig. 4E, look sharper. The algorithm does not change the quality of other non-noisy clips that do not rise to the detected level of the "Red Lady" video.
[0031] For reference, Appendix A below shows an example of coding in "C" to implement the algorithm illustrated by Fig. 6.
[0032] For components shown, like the pre-processing filter 500 and the encoder 506, each component according to embodiments of the present invention can include a processor and memory to enable operation. The memory of each device stores code that is executable by the processor to enable the processor to perform the processes described herein. Further the memory can be used to provide data storage with the data accessible by the processor to store or retrieve when performing operations.
[0033] Although the present invention has been described above with particularity, this was merely to teach one of ordinary skill in the art how to make and use the invention. Many additional modifications will fall within the scope of the invention as that scope is defined by the following claims.

Claims

CLAIMS What Is Claimed:
1. A method for encoding video comprising:
obtaining video frames provided to an encoder;
obtaining motion estimation (ME) data for the video frames;
calculating a ME score for the video frames;
comparing the ME score with a threshold;
calculating the value of a metric from a temporal variability of the ME score;
comparing the temporal variability metric with a predetermined value; and
providing a high film grain indication signal to the encoder indicating a particular segment of the video frames contains high film grain when the ME score exceeds a threshold and the ME score temporal variability metric is below a predetermined level.
2. The method of claim 1, wherein when the high film grain signal is provided, the encoder allocates less bits to I frames and more bits to P and B type frames than to other frames of the video.
3. The method of claim 1, wherein when the high film grain indication is not provided, the encoder allocates more bits to the I frame than when the high film grain indication is provided.
4. The method of claim 1, wherein the ME score threshold is greater than 20.
5. An apparatus to encode video frames, the apparatus comprising:
an encoder having a first input for receiving video frames to be processed and a second input for receiving parameter data to enable the encoder to allocate bits for frames for encoding;
a frame buffer having an input receiving the video frames and an output providing the first input to the encoder;
a preprocessor for receiving the video frames and having an output providing the second input to the controller;
a preprocessor memory connected to the processor for storing code that is executable by the preprocessor to determine the parameter data to enable the encoder to allocate bits, the code causing the preprocessor to perform the following steps;
obtaining motion estimation (ME) data for the video frames;
calculating a ME score for the video frames;
comparing the ME score with a threshold;
calculating the value of a metric from a temporal variability of the ME score; comparing the temporal variability metric with a predetermined value; and providing a high film grain indication signal to the encoder indicating a particular segment contains high film grain when the ME score exceeds a threshold and the ME score temporal variability metric is below a predetermined level.
6. The apparatus of claim 5, wherein when the high film grain signal is provided, the encoder allocates less bits to I frames and more bits to P and B type frames than to other frames of the video.
7. The apparatus of claim 5, wherein when the high film grain indication is not provided, the encoder allocates more bits to the I frame than when the high film grain indication is provided.
8. The apparatus of claim 5, wherein the ME score threshold is greater than 20.
PCT/US2015/064703 2015-01-05 2015-12-09 A method of encoding video with film grain Ceased WO2016111799A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
MX2017008847A MX368521B (en) 2015-01-05 2015-12-09 A method of encoding video with film grain.
CA2972989A CA2972989C (en) 2015-01-05 2015-12-09 A method of encoding video with film grain
BR112017014421-2A BR112017014421B1 (en) 2015-01-05 2015-12-09 A METHOD OF VIDEO ENCODING WITH FILM GRAIN

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201562099672P 2015-01-05 2015-01-05
US62/099,672 2015-01-05
US14/962,814 US11451798B2 (en) 2015-01-05 2015-12-08 Method of encoding video with film grain
US14/962,814 2015-12-08

Publications (1)

Publication Number Publication Date
WO2016111799A1 true WO2016111799A1 (en) 2016-07-14

Family

ID=55070691

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2015/064703 Ceased WO2016111799A1 (en) 2015-01-05 2015-12-09 A method of encoding video with film grain

Country Status (7)

Country Link
US (2) US11451798B2 (en)
EP (1) EP3041236B1 (en)
BR (1) BR112017014421B1 (en)
CA (1) CA2972989C (en)
GB (1) GB2533858B (en)
MX (1) MX368521B (en)
WO (1) WO2016111799A1 (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10454987B2 (en) 2016-10-28 2019-10-22 Google Llc Bitrate optimization for multi-representation encoding using playback statistics
FR3073999B1 (en) 2017-11-23 2024-02-09 Ateme INTELLIGENT COMPRESSION OF SEEDED VIDEO CONTENT
US11930222B1 (en) * 2021-03-30 2024-03-12 Amazon Technologies, Inc. Encoding video for film grain synthesis
CA3156314A1 (en) * 2021-04-19 2022-10-19 Comcast Cable Communications, Llc Methods, systems, and apparatuses for adaptive processing of video content with film grain
US20230059035A1 (en) * 2021-08-23 2023-02-23 Netflix, Inc. Efficient encoding of film grain noise
US11622123B1 (en) * 2021-09-02 2023-04-04 Amazon Technologies, Inc. Film grain preservation
CN115278265B (en) * 2022-06-23 2025-03-18 安谋科技（中国）有限公司 Parameter processing method, device, medium and electronic equipment

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2003077549A1 (en) * 2002-03-13 2003-09-18 Imax Corporation Systems and methods for digitally re-mastering or otherwise modifying motion pictures or other image sequences data
US20080152296A1 (en) * 2006-12-21 2008-06-26 Byung Tae Oh Methods and Systems for Processing Film Grain Noise

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7801215B2 (en) * 2001-07-24 2010-09-21 Sasken Communication Technologies Limited Motion estimation technique for digital video encoding applications
US7653129B2 (en) * 2004-12-28 2010-01-26 General Instrument Corporation Method and apparatus for providing intra coding frame bit budget
US7924922B2 (en) 2005-03-07 2011-04-12 Hewlett-Packard Development Company, L.P. Color registration in a digital video
US8243804B2 (en) 2005-12-01 2012-08-14 Lsi Corporation Hierarchical motion estimation for images with varying horizontal and/or vertical dimensions
EP2048888A1 (en) 2007-10-12 2009-04-15 Thomson Licensing Method and decoder for decoding an image frame of an image frame sequence
US20140112386A1 (en) * 2012-10-22 2014-04-24 General Instrument Corporation Algorithms for determining bitrate for a statistical multiplexing system to account for signal complexity including film mode and gop structural changes
US9813706B1 (en) * 2013-12-02 2017-11-07 Google Inc. Video content analysis and/or processing using encoding logs

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2003077549A1 (en) * 2002-03-13 2003-09-18 Imax Corporation Systems and methods for digitally re-mastering or otherwise modifying motion pictures or other image sequences data
US20080152296A1 (en) * 2006-12-21 2008-06-26 Byung Tae Oh Methods and Systems for Processing Film Grain Noise

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
PETER SCHALLAUER ET AL: "Rapid and Reliable Detection of Film Grain Noise", IMAGE PROCESSING, 2006 IEEE INTERNATIONAL CONFERENCE ON, 1 January 2006 (2006-01-01), Pi, pages 413 - 416, XP055252534, ISBN: 978-1-4244-0480-3, DOI: 10.1109/ICIP.2006.312481 *

Also Published As

Publication number Publication date
CA2972989C (en) 2019-11-12
GB2533858A (en) 2016-07-06
MX2017008847A (en) 2017-10-24
BR112017014421A8 (en) 2023-02-07
BR112017014421A2 (en) 2018-04-10
GB201522138D0 (en) 2016-01-27
US20160198165A1 (en) 2016-07-07
US20220394279A1 (en) 2022-12-08
CA2972989A1 (en) 2016-07-14
US11451798B2 (en) 2022-09-20
MX368521B (en) 2019-10-07
EP3041236B1 (en) 2021-02-17
EP3041236A1 (en) 2016-07-06
GB2533858B (en) 2017-08-16
BR112017014421B1 (en) 2023-05-09

Similar Documents

Publication Publication Date Title
CA2972989C (en) A method of encoding video with film grain
US9706203B2 (en) Low latency video encoder
US20140376624A1 (en) Scene change detection using sum of variance and estimated picture encoding cost
US9565440B2 (en) Quantization parameter adjustment based on sum of variance and estimated picture encoding cost
JP7145204B2 (en) Systems and methods for adaptive quality rendering of inductive encoders
US9066101B2 (en) Image processing apparatus and image processing method
CN106572352B (en) A kind of key frame sign control method and device
CN103731668A (en) Image encoding apparatus and image encoding method
US10129565B2 (en) Method for processing high dynamic range video in order to improve perceived visual quality of encoded content
EP2963928A1 (en) Apparatus and method for encoding and decoding based on region
US20150208072A1 (en) Adaptive video compression based on motion
CN117857815A (en) Hybrid Inter Coding Using Autoregressive Model
US20140226724A1 (en) System and Method for the Dynamic Reduction of the Entropy of a Signal Upstream to a Data Compression Device
CN112585968A (en) Benefit-based bit rate allocation for video coding
WO2016004721A1 (en) Bit rate control method and apparatus, and computer storage medium
GB2545447B (en) A method to encode video with close multiple scene changes
US11825088B2 (en) Adaptively encoding video frames based on complexity
US20160044340A1 (en) Method and System for Real-Time Video Encoding Using Pre-Analysis Based Preliminary Mode Decision
Merlos et al. Feature Compression for Machines with Range-Based Channel Truncation and Frame Packing
US20170061216A1 (en) Event sensing method and system using dct coefficient in surveillance camera
HK40024991A (en) Systems and methods for encoder-guided adaptive-quality rendering

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15828564

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: MX/A/2017/008847

Country of ref document: MX

ENP Entry into the national phase

Ref document number: 2972989

Country of ref document: CA

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15828564

Country of ref document: EP

Kind code of ref document: A1

REG Reference to national code

Ref country code: BR

Ref legal event code: B01A

Ref document number: 112017014421

Country of ref document: BR

ENP Entry into the national phase

Ref document number: 112017014421

Country of ref document: BR

Kind code of ref document: A2

Effective date: 20170703