Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/70—Denoising; Smoothing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/13—Edge detection
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/117—Filters, e.g. for pre-processing or post-processing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/136—Incoming video signal characteristics or properties
  • H04N19/14—Coding unit complexity, e.g. amount of activity or edge presence estimation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
  • H04N19/86—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving reduction of coding artifacts, e.g. of blockiness
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/14—Picture signal circuitry for video frequency region
  • H04N5/20—Circuitry for controlling amplitude response
  • H04N5/205—Circuitry for controlling amplitude response for correcting amplitude versus frequency characteristic
  • H04N5/208—Circuitry for controlling amplitude response for correcting amplitude versus frequency characteristic for compensating for attenuation of high frequency components, e.g. crispening, aperture distortion correction
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/14—Picture signal circuitry for video frequency region
  • H04N5/21—Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20004—Adaptive image processing
  • G06T2207/20012—Locally adaptive
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20172—Image enhancement details
  • G06T2207/20182—Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20172—Image enhancement details
  • G06T2207/20192—Edge enhancement; Edge preservation

Definitions

• the invention relates to image 3D noise reduction techniques primarily operable in real-time in an image or a sequence of images. More particularly, the invention relates to adaptive 3D techniques for artifact reduction in Discrete Cosine Transform (DCT) based decoded image applications.
• DCT Discrete Cosine Transform
• block-based DCT coding artifacts become perceptible. Such artifacts are known as mosquito noise or ringing noise occurring around edges within an image or near a smooth zone as well as the blocking effect. For still pictures or still parts of image, the blocking effect is dominant and visible in smooth regions. For dynamic video sequences and in high resolution large screen display, mosquito noise can become evident for the human vision system (HVS).
• HVS human vision system
• Low pass filtering is utilized also in U.S. Patent No. 5,850,294 to remindopoulos et al. for blocking artifact reduction purposes.
• the blocks that potentially exhibit block artifacts are detected in the DCT domain and low-pass filtering is applied only for the distorted blocks.
• B. Ramamurthi and A. Gersho "Nonlinear Space-variant post processing of block coded images", IEEE Transactions on Acoustics, Speech and Signal Processing, vol. ASSP-34, Oct 1986, pp. 1258-1268, the proposed adaptive filtering is based on the detection of edge orientation at each block boundary pixel. Many authors, as in, for instance, A.
• Nakajima For mosquito noise artifact reduction (MNR), in U.S. Patent No. 5,610,729, Nakajima teaches an estimation of block mean noise using the quantization step and the I, P, B coding mode when these data are available from the compressed bit stream. Nakajima teaches also the use of the well-known Minimum Mean Square Error (MMSE) filter proposed originally by J. S. Lee in "Digital image enhancement and noise filtering by use of local statistics", IEEE Transactions on PAMI-2, Mar 1980, pp. 165-168, for artifact reduction.
• MMSE Minimum Mean Square Error
• the quantization step or the coding mode is not necessary known or accessible.
• the Minimum Mean Square Error filter is efficient for edge reservation, it is not necessary for noise reduction near an edge.
• Mosquito Noise is a compression noise around edges.
• Tan et al. teach the use of separable low- pass filtering, when block boundaries are located, for a serial reduction of blocking effect and then, mosquito noise. For detected blocking effect, the pixels are firstly corrected by a proposed modified version of bilinear interpolation and secondly, by a mean value of homogenous neighboring pixels within the quantization step size.
• Yang et al. teach the use of iterative pixel clustering technique in a sliding window and the artifact correction mainly based on maximum likelihood estimation. There is no discussion about real-time processing.
• the present invention provides an apparatus and method for efficiently reducing noise or artifact in a block-based decoded image signal.
• an apparatus for reducing noise in a block-based decoded image signal including a luminance component.
• the apparatus comprises a noise power estimator responsive to said luminance component in a same frame of said image signal to classify the luminance pixel in a selected one of a plurality of predetermined image region classes associated with distinct image region spatial characteristics and to generate a corresponding selected region class indicative signal.
• the said noise power estimator further comprises a shape-adaptive luminance noise power estimator responsive to said luminance component and said selected region class indicative signal for estimating statistical characteristics of said luminance pixel by using local window segmentation data associated with the luminance pixel, to generate a corresponding luminance noise power statistical characteristics indicative signal.
• the distinct image region spatial characteristics include edge, near edge flat, flat and texture spatial characteristics.
• the noise power estimator comprises a shape-adaptive chrominance noise power estimator responsive to said chrominance components for estimating statistical characteristics of first and second chrominance pixels associated with said luminance pixel by using local window segmentation data associated with each said chrominance pixel to generate a corresponding chrominance noise power statistical characteristics indicative signal.
• the apparatus comprises further a minimum output variance temporal noise reducer and a spatial noise reducer for filtering said luminance and chrominance components according to said luminance and chrominance noise power statistical characteristics indicative signal.
• the said temporal noise reducer can be region-based varying or simply stationary for the whole image.
• the said temporal filter comprises further context-based soft motion detection for minimizing motion blur artifact.
• the said spatial noise reducer based on minimum mean squared error can be utilized shape adaptive windowing technique or region adaptive facet model parameters calculation.
• the apparatus further comprises a block localizer responsive to said luminance component for determining block position in a frame of said image.
• the said block localizer working on signal domain utilizes line matched filter and histogram analysis for block detection.
• the apparatus further comprises a blocking artifact reducer for said luminance and chrominance components.
• Blocking artifact reducer comprises edge-based filters for said luminance, horizontal and vertical filters for chrominance components.
• Blocking artifact reducer comprises also high frequency region detector for avoiding possible high frequency artifacts.
• the apparatus further comprises an optional detail enhancer for said luminance component. The said detail enhancer adaptively enhances the luminance signal differently in each of eight (8) principal directions.
• a method for reducing noise in a block-based decoded image signal including a luminance component comprises the steps of: i) noise power estimation according to a corresponding luminance pixel spatial context in a same frame of said image signal to classify the luminance pixel in a selected one of a plurality of predetermined image region classes associated with distinct image region spatial characteristics and to generate a corresponding selected region class indicative signal; ii) estimating, from said luminance component and said selected region class indicative signal, statistical characteristics of said luminance pixel by using shape- adaptive local window segmentation data associated with the luminance pixel, to generate a corresponding luminance noise power statistical characteristics indicative signal; and iii) spatio-temporal filtering said luminance component according to said luminance noise power statistical characteristics indicative signal.
• the distinct image region spatial characteristics include edge, near edge flat, flat and texture spatial characteristics.
• the block-based decoded image signal further includes first and second chrominance components and, method further comprises the steps of: iv) estimating, from said chrominance components statistical characteristics of first and second chrominance pixels associated with said luminance pixel by using shape-adaptive local window segmentation data associated with each said chrominance pixel to generate a corresponding chrominance noise power statistical characteristics indicative signal; and v) spatio-temporal filtering each said chrominance components according to said corresponding chrominance noise power statistical characteristics indicative signal.
• an apparatus and method for post-processing a decompressed image signal to reduce spatial mosquito noise and blocking artifact therein calls for an image multiple region segmentation, region noise power estimations for respectively luminance and chrominance signal components, and their associated adaptive noise corrections.
• the inventive apparatus and method employ edge/no-edge detectors and simple binary consolidation operators to classify and reinforce detected Edge (E), Near-Edge-Flat regions (NEF), Flat regions (F), and finally Texture (T) regions.
• E Edge
• NEF Near-Edge-Flat regions
• F Flat regions
• T Texture
• the preferred segmentation is based essentially on the following observations: First, almost strong mosquito noise is found in NEF regions; second, some important noise is also noticeable in picture edges; third, texture masks mosquito noise; and fourth, any excessive filtering in texture or flat regions will degrade eventually fine signal details.
• the apparatus and method In estimating local noise power of the luminance component of the image signal, the apparatus and method consider the diagonal high frequency component of the decoded image.
• the local noise power estimator comprises a local variance calculator that considers only local similar pixels to the current one, a look up table (LUT) for a conversion from observed diagonal high frequency component power to equivalent additive noise power.
• the noise power estimator also comprises a noise power weighting for each classified region and finally a low-pass filter for smoothing the variation of estimated local noise power between regions.
• the proposed method permits different smoothing degree for each segmented region and region transition to ensure resulting image quality.
• the. proposed apparatus and method are based on: i) minimization of the output noise variance for the temporal filter; ii) a shape adaptive local segmented window that considers only the similar intensity pixels to the current one for the local mean and local standard deviation estimations.
• a two-dimensional (2D) low pass filter is preferably required for the local adaptive windowing.
• the noise corrector further comprises a gain calculator in order to minimize the Mean Square Error (MMSE) for given local signal mean, local signal power and local additive noise power.
• MMSE Mean Square Error
• the combination of local shape adaptive windowing and Minimum Mean Square Error constitutes a noise corrector working on all of the above-cited classified regions.
• an adaptive apparatus and method for noise power estimation and noise correction for the chrominance components which are severely damaged at low bit rate in a decoded video signal.
• the proposed method is similar to luminance component processing.
• the region classification is not required. In other words, there is only a single region for the whole image.
• the above luminance-based shape adaptive windowing and the Minimum Mean Square Error technique are both utilized in a similar manner to the luminance case.
• the chrominance-sampling rate requires the use of suitable interpolation and decimation techniques for the chrominance signals.
• a method for reducing artifact in a DCT-based decoded image comprising associating each pixel of a plurality of pixels defining the image to corresponding image region having distinct spatial characteristics, estimating artifact statistical characteristics of each of the pixel using the associated corresponding image region and performing a tempo-spatial filtering of each of the pixels using the artifact estimated statistical characteristics of the pixel, whereby the filtered pixels produce the image having reduced noise or reduced artifact.
• a method for reducing artifact in a DCT-based decoded image comprising associating each pixel of a plurality of pixels defining the image to corresponding image region having distinct spatial characteristics, estimating artifact statistical characteristics of each of the pixel using the associated corresponding image region, performing a filtering of each of the pixels using the artifact estimated statistical characteristics of the pixel, whereby the filtered pixels produce the image having reduced noise or reduced artifact; and correcting the filtered pixels against artifact related to a compression technique used for encoding said image.
• an apparatus for reducing artifact in a DCT-based decoded image comprising a noise estimation unit for providing artifact statistical characteristics of each pixel of a plurality of pixels defining the image, wherein the artifact statistical characteristics of each pixel are estimated by associating to a given pixel a corresponding given image region having distinct spatial characteristics and a tempo- spatial filtering unit receiving the artifact statistical characteristics of each pixel and filtering the pixel accordingly to provide a temporally-spatially filtered signal.
• FIG. 1 is a block diagram of an preferred embodiment of a mosquito noise reducing apparatus
• FIG. 2 is also a block diagram of another embodiment of a mosquito noise reducing apparatus
• Figure 3 is a block diagram of an embodiment of a noise power estimation unit
• Figure 4a is a block diagram of an embodiment of a temporal filter for noise reduction with soft motion detection
• Figure 4b is a block diagram of one embodiment of a temporal filter coefficient calculation with embedded motion estimation
• Figure 5 is a block diagram of an embodiment of a shape adaptive window spatial noise reducer
• Figure 6 is a block diagram of one embodiment of a block detection and localization unit
• Figure 7a is a block diagram of one embodiment of a block artifact reducer
• Figure 7b is a block diagram of one embodiment of a line direction detector, part of blocking artifact reduction
• Figure 8 is a block diagram of an embodiment of an optional detail enhancer block correction
• Figure 9 illustrates sixteen curves stocked in a lookup table (LUT) for converting diagonal high frequency local standard deviation signal into local standard deviation of MPEG artifacts' equivalent additive noise;
• Figure 10 is a block diagram of an embodiment of a region adaptive facet model spatial noise reducer.
• FIG. 1 represents a block diagram of an embodiment of a MPEG noise reduction apparatus MNR2 10 in accordance with the invention.
• the MNR2 apparatus 10 receives two main system inputs.
• the first received input 101 is an image video signal comprising luminance Y and chrominance Cr/Cb components.
• Figure 1 illustrates only a video input signal.
• the second input corresponds to a user correction level which is applied at input 106.
• the user correction level at input 106 may represent, for instance, an intensity of noise correction or, if possible, coding transmission rate. In a preferred implementation, this user correction level input is controlled by an end-user in a heuristic manner.
• the Mosquito Noise Reducer apparatus 10 comprises a Noise Estimation
• NE Temporal Filter
• SNR Spatial Noise Reducer
• Block Localization Unit 102 a Block Artifact Reducer (BAR) 113 and an Optional
• Figure 3 receives the video input 101 and the user correction level 106 and generates noise power estimations at each pixel.
• the noise power estimations 108 and 107 are provided respectively to the Temporal Filter (TF) 103 and to the Spatial
• the Temporal Filter 103 determines its optimum filter coefficient 109 which is sent back in turn to the Noise Estimation unit 117 in order to establish a residual noise power 107 for the Spatial Noise Reducer 111.
• the Noise Estimation unit 117 receives also Offset x 104 and Offset y
• the Block Localization Unit 102 may not be required since for in this case the Offset x and the Offset y are both known equal to zero, i.e., no offset in accordance with image borders.
• Figures 4a and 4b receives the video input signal 101 and the region-based noise power estimation signal 108 and generates the optimum filter coefficient 109 and a temporally filtered image 110.
• the optimum filter coefficient 109 is applied to the Noise Estimation unit 117.
• the temporally filtered image 110 is sent in turn to the
• the Spatial Noise Reducer 111 receives the temporally filtered image signal 110 and the spatial noise power estimation signal 107 and performs a Minimum Mean, Squared Error filtering for spatial compression artifact reduction.
• the resulting image also referred to as spatio-temporally filtered image
• the Block Artifact Reducer 113 receives the temporally filtered image signal 110 and the spatial noise power estimation signal 107 and performs a Minimum Mean, Squared Error filtering for spatial compression artifact reduction.
• the resulting image also referred to as spatio-temporally filtered image
• Block Artifact Reducer 113 receives the temporally filtered image signal 110 and the spatial noise power estimation signal 107 and performs a Minimum Mean, Squared Error filtering for spatial compression artifact reduction.
• the resulting image also referred to as spatio-temporally filtered image 112 is sent to the Block Artifact Reducer 113.
• the Block Localization Unit 102 receives the luminance input image 101 and determines the horizontal
• Offset x and vertical (Offset y) offsets 104 and 105 respectively and sends these signals to the Noise Estimation unit 117 and to the Block Artifact Reducer 113. It is important to note that for economic and low latency purposes, the detected offsets in the current frame are applied for the next frame in supposing the same offsets between two consecutive frames.
• the Block Artifact Reducer 113 receives the spatio-temporally filtered image 112 and the offset value signals 104 and 105 and estimates edge directions, to determine block border pixels and to suitably apply filtering for blocking artifact reduction.
• the resulted image 114 is provided to the Optional Detail Enhancer 115.
• edge direction filtering is applied for the luminance video component. Meanwhile, only simple horizontal or vertical filters are used for Cr and Cb components.
• FIG. 2 there is illustrated in block diagram another embodiment of a mosquito noise reducing apparatus (also referred to as MNR2-S).
• MNR2-S mosquito noise reducing apparatus
• the mosquito noise reducing apparatus MNR2-S is similar to the previously described mosquito noise reducing apparatus 10 shown in Figure 1.
• a first difference is the temporal filter input 208 of noise variance signal which is now controlled by an end-user in contrast to the feedback signal 108 provided by the Noise Estimation unit 117.
• the temporal filter 103's functionality is now independent of the segmentation-based noise estimation.
• the temporal filter 103 becomes a temporal dynamic noise reducer for independent random noise and the spatial filter
• SNR is reserved therefore for coding artifact noise.
• a second difference is the Noise Estimation unit input 110 which is now provided by the temporal filter 103. Meanwhile, in Figure 1 , the Noise Estimation unit
• the Noise Estimation unit 117 receives directly the video input 101. In the former case, with temporally filtered image input 110, the Noise Estimation unit 117 does not require furthermore the temporal filter coefficient signal 109 in order to estimate residual noise power.
• the Noise Estimation unit 117 comprises an Image Segmentation unit
• the Image Segmentation unit 300 is necessary only for luminance component of the video signal. It will be appreciated that the Image segmentation for chrominance components Cr and Cb is not required.
• the Image Segmentation unit 300 receives a luminance component signal 301 of the video input signal 101 or the temporally filtered video signal 110 in accordance with the mosquito noise reducer 10 disclosed in Figure 1 or the mosquito noise reducer 12 disclosed in Figure 2.
• the luminance component signal 301 is sent to an Edge Detector 302 and to a Strong Texture Detector 304.
• the implemented Edge Detector 302 comprises a low pass filter, for some noise robustness, followed respectively by 4 parallel Sobel gradient compasses, a summing of absolute values, a threshold detector and some context-based binary filtering for removing isolated pixels or for reinforcing missed detected edges.
• the skilled addressee will understand the Edge
• the Edge Detector 302 provides at its output 303 a detected Edge (E) map signal.
• the detected Edge (E) map signal is provided to a Block Extension 305, to a negative input of gates 307, 313 and 315 and finally to a correction map 316.
• the Strong Texture Detector 304 is composed in series of Low pass filter, Sobel gradient compasses, Absolute values, Maximum detector, Threshold detector and some context-based binary filtering for removing isolated pixels or for reinforcing broken detection.
• the Strong Texture Detector 304 output signal 312 is applied now to the non negative input of the gate 313.
• the latter realizes detected texture (T) signal 314 as detected strong texture 312 but not edge 303. It might require a context-based binary filter, not shown, placed directly after the gate 313 if the gate output signal still contains isolated or broken detection.
• the detected texture signal (T) 314 is applied in turn as input to the NOR gate 315 and to the correction map 316.
• the previously described Edge (E) signal 303, together with the two block offset values 104 and 105 are applied to the Block Extension 305.
• the block extension output 306 is provided to the positive input of the gate 307 which defines Near Edge (NE) region signal 308 as Block Edge but not Edge.
• the Near Edge (NE) region signal 308 and the Flat (F) signal 315 are combined together with an AND gate 309 to provide a Near Edge and Flat (NEF) signal 310.
• the Near Edge and Flat (NEF) signal 310 is provided to the Correction Map (CM) 316.
• Block Extension is of dimension 8x8 for the progressive luminance signal or 4x8 (4 lines x 8 columns) for interlaced luminance signal.
• block extension sizes may be set wider than usual to reach dimensions such as 8x10 or 4x10.
• the Correction Map 316 receiving the 4 input signals (NEF), (E), (F) and (T) is used to solve any eventual ambiguity in the above segmentation.
• the Correction Map 316 performs the final segmentation defined with the following priorities for each pixel (E) > (NEF) > (F) > (T).
• the correction map output signal 317 is provided to the Noise Weighting sub module 340 in order to establish the noise level to be considered.
• the Noise Measurement unit 320 Prior to performing noise weighting, it is necessary to get first a Noise Measurement for each considered pixel.
• the Noise Measurement unit 320 is provided for that purpose.
• the Noise Measurement unit 320 receives the luminance component 301 of the video signal to provide estimated local compression noise power.
• the proposed Noise Measurement unit 320 uses a diamond high pass filter 321 applied on the luminance signal input Y 301 to extract only diagonal high pass component.
• the high pass filter impulse response is given as follows:
• the diamond filter output 322 is provided to a standard deviation estimator 323 which receives also the two inputs 329 and 328 respectively for ⁇ ,, and N defined in details below.
• Shape Adaptive Window 330 are defined as:
• (c,r) be the current pixel coordinates.
• (i,j) be the relative coordinates of a pixel in the rectangular window of size N c by N r around the current central pixel.
• Shape adaptive windowing yields a binary value for the pixel (i,j) using the following expression:
• Y(c,r) is the current luminance input
• Y, j (c,r) is the luminance at the relative coordinates (i,j) in the window centered around the current pixel (c,r).
• Y(c,r) is clearly Yoo(c.r).
• lp(Y,,(cr)) denotes the low pass filter output 332 at the relative coordinates (ij) in the current window centered at (c,r).
• the presence of the low pass filter 327 for robustness against noise is important for the followed local segmentation Shape Adaptive Window 330. Many low pass filters are possible.
• the low pass filter impulse response is given by:
• Standard deviation estimator output 324 is sent now to Lookup table 325 which also receives as input User Correction Level 331.
• the Lookup table 325 is pre- calculated look up table to convert the previously defined local standard deviation 324 s hf (c,r) of high frequency signal into the local standard deviation of MPEG artifacts' equivalent additive noise.
• the content of the Lookup table 325 is selected by the mode value and the selected Lookup table 325 has been obtained from extensive testing of various MPEG bit rates on many video sequences.
• the Lookup table 325 output 326 provides the local standard deviation of MPEG artifacts in terms of equivalent additive noise.
• 16 user-controlled levels 331 corresponding to 16 Lookup table curves illustrated in Figure 9 are provided.
• the Noise Weighting sub module 340 comprises Segmentation-based Weightings 341 to provide local noise variances for both temporal and spatial filtering in the case of the embodiment of the mosquito noise reducer shown in Figure 1 , or only local noise variance for spatial filter in the case of the embodiment of the mosquito noise reducer shown in Figure 2.
• the Segmentation-based Weightings 341 receives together the local standard deviation 326 and the correction map signal 317.
• the Segmentation-based Weightings 341 performs a weighting on the standard deviation 326 in function of the correction map signal 317:
• the Region-factor may depend on the mosquito noise reducer provided (i.e. the one disclosed in Fig. 1 or the one disclosed in Fig. 2).
• Region-factors are parameters allowed to the designer's discretion. In the proposed implementation, Region-factors are resumed in the following table:
• low pass filtering 343 and 347 on standard deviation images are recommended.
• the following low pass filter is used before the squaring processes 345 and 349:
• the squaring 345 output 108 corresponding to the estimated local noise variance to be processed is provided to the Temporal Filter 103 for the Mosquito Noise Reducer disclosed in Fig. 1.
• Low pass filter 343 and squaring 345 are not necessary.
• the squaring 349 output 350 is provided to the multiplier 351 to adjust residual noise variance 107 for the Spatial Filter 111.
• the other input 352 of the multiplier 351 is a coefficient ⁇ defined by selected configuration shown in Fig. 1 or Fig.2 and provided by the Temporal Filter 103.
• the multiplexer 353 and the multiplier 351 are not required.
• the noise estimation for chrominance sub module 360 in Figure 3 is much simpler with no required segmentation.
• the noise estimation for chrominance sub module 360 receives two multiplexed chrominance components Cr and Cb at the input 361.
• the multiplexed components Cr and Cb are sent to a diamond shape high pass filter 362.
• Ch _ hp(c, r) -1 0 -8 0 44 0 -8 0 - 1 / 64 0 0 - 2 0 -8 0 - 2 0 0 0 0 0 0 -1 0 0 0 0 0 0 0 0 0
• the filter output 363 is provided to Standard deviation estimator 364 which receives also two other inputs 373 and 375 respectively for ⁇ ijCh and N Ch - [0090]
• binary value signal coi jCh (c,r) 373 is determined by co-sited phase down sampler and hold 372.
• the down sampler and hold input is the corresponding binary signal ⁇ j j (c,r) 329 in the luminance case.
• N Ch (c,r) 375 is the number of ⁇ yc h Cc.r) in the considered window.
• Nc h is provided by counter 374.
• the effective window for each chrominance component is 5 lines by 5 columns.
• the co-sited phase down sampling and hold of the luminance window ⁇ y is equivalent to multiply element by element the window coy by the following sampling matrix
• the local standard deviation estimator 364 of the high frequency signal provides, in 4:2:2 sampling, multiplexed s hf c r and s hf c b signal 365 which are defined by
• the output 365 of Standard deviation Estimator 364 is applied to Lookup table 366 which combines also with User correction level signal 331 to provide estimated local standard deviations ⁇ c r (c,r) and ⁇ c b (c,r) signal 367.
• the Lookup table 366 is the same Lookup table 325 for luminance component.
• the standard deviations signal 367 is provided to low pass filter 368 to reduce eventual halo effect.
• the low pass impulse response is given by the followings:
• Low pass filter output 369 is provided to squaring operator 370 to provide variance signal 371 ⁇ 2 n scr and ⁇ 2 n scb for chrominance part of the Spatial Mosquito
• the Temporal Filter 103 for noise reduction receives the Video input 101 and a second input.
• the Video input 101 comprises luminance Y and chrominance Cr/Cb components.
• the second input may be 108 in the case of the mosquito noise reducer disclosed in Figure 1 or it may be 208 in the case of the noise reducer disclosed in Figure 2.
• the Temporal Filter 103 is based on well known temporal recursive first order filter of unitary gain. As illustrated by Figure 4a, it can realized sequentially with adder 401 , multiplier 404, adder 406 and frame buffer delay 408.
• MSD Embedded Motion Soft Detection
• the Minimum Output Noise Variance Calculation unit 430 receives as inputs the Video Input 101 and a Filtered Past Video 402. Each of these two video signals is decomposed by splitter, not shown, into three video components Y, Cr and Cb.
• the components are denoted as Y c (101 -Y), Cr c (101-Cr) and Cb c (101-Cb).
• the components are denoted as Y p (402-Y), Cr p (402-Cr) and Cb p (402-Cb).
• Component-wise Image Differences are firstly calculated with three respective subtractions 431 , 432 and 433.
• each chrominance component difference is furthermore horizontally up-sampled by 2 and hold by device 435 or 436. This operation is equivalent to column duplicating.
• Y, Cr and Cb 4:4:4 up-sampling and hold is not necessary.
• the three resulted image differences are now squared up respectively by 434, 437 and 438.
• the squaring operator outputs are combined together with summation 439.
• the summation result 440 is provided to Low Pass Filter 441 which approximates the local signal mean value.
• Low Pass Filter output 442 representing a variance signal s 2 ⁇ p is then provided to the Embedded Motion Soft Detection unit 460 and to the gain scaling
• the Past Image Noise Variance Estimator 445 is an estimator for residual noise variance Si 2 in the past filtered image.
• Past Image Noise Variance Estimator output 446 S 1 2 is applied then to an adder 447 and a divider 449.
• the former adder 447 receiving also temporal noise input variance ⁇ 2 n ⁇
• the filter coefficient b min 453 is provided to the Embedded Motion Soft
• the Embedded Motion Soft Detection 460 receives Low Pass Filter
• the Embedded Motion Soft Detection 460 comprises Comparator 461 , Weighted
• the Comparator 461 receives Low Pass Filter Output 442 and Sum (S 1 2 + ⁇ 2 n ⁇ ) 448 and provides a binary signal representing hard no-motion detection (hnm)
• Weighted Counter is used to avoid again an eventual division by 9 resulted from 3x3 window.
• no-motion decision nm is not a binary value but a fractional one varying from 0 to 1.
• the Filter coefficient b 0 109 is sent to the Temporal Filter 103 for noise filtering and also, in the case of full version MNR2, to the noise estimation unit 117 for residual noise power estimation.
• the filter coefficient b 0 is directly applied for luminance component or for 4:4:4 pattern sampling. However, for 4:2:2 case, b 0 shall be co-sited down sampling and hold, not shown, for chrominance component filtering.
• the shape adaptive windowing spatial noise reducer 111 has been described in PCT Application No PCT/CA02/00887, 2002 by the three present authors, the specification of which is hereby incorporated by reference. However banding effect, a resulted filtering artifact, is not discussed in the cited patent application. For completeness, proposed shape adaptive windowing spatial noise reducer 111 will be presented in some detail.
• the shape adaptive windowing spatial noise reducer 111 is a modified version of Lee's original Minimum Mean Squared Error (MMSE) reduction (see J. S.
• K(c,r) max [0, ( ⁇ 2 g (c,r) - ⁇ 2 n (c,r))/ ⁇ 2 g (c,r)]. Meanwhile, the error performance is written as:
• the modifications proposed comprise two major techniques Shape Adaptive Windowing for local mean and variance estimation and Banding Effect Reduction for small signal variance case.
• Figure 1 receives residual noisy three component (Y, Cr, Cb) image 110 provided from the temporal filtering unit 103 and theirs corresponding estimated noise powers
• the noise estimation unit 117 provided in turn by the noise estimation unit 117.
• the shape adaptive windowing spatial noise reducer 111 comprises a Luminance Spatial Filter 500 and a Chrominance
• the Luminance Spatial Filter 500 receives respectively luminance component signal Y ⁇ 110-Y and its residual noise power ⁇ 2 nS 107-Y.
• the luminance signal Y T 110- Y is applied as input to Low Pass Filter 501 , Local Mean Calculator
• Shape Adaptive Window 504 for providing local binary signal a>j j (c,r) 505 and N(c,r)
• ⁇ j j and N can be provided from the noise estimation unit 117 with only minor differences.
• Low Pass Filter 502 are already described in previous Sections. Briefly, in segmenting the window in two regions homogenous or not to the current pixel, the shape adaptive window technique allows a more precise estimation of local mean and local variance. Low Pass Filter acting as a pre-processor is required for robust window segmentation.
• the calculator 507 receives temporally filtered luminance Yj 110-Y and local shape adaptive window parameters 505 and 506 respectively for ⁇ i j (c,r) and
• N(c,r) provides at its output local mean signal 508 estimated by:
• m Y (c,r) (l/N (c, r)) ⁇ Y Tij (c, r) ffl jj (c, r) .
• the local mean value signal 508 is provided to negative input of adder 509 and to local variance calculator 511.
• the calculator 511 uses its inputs Y ⁇ 110-Y, coi j (c,r) and N(c,r) 505-506 and local mean 508, the calculator 511 estimates luminance variance signal 512 as follows:
• ⁇ 2 ⁇ (c,r) (l/N (c, r)) ⁇
• Adaptive local gain K calculator 513 which provides the following modification from Lee's original version:
• the term (be), which varies between 0 and 1 represents the banding effect in slowly varying regions of small variance.
• the banding effect (be) at a given pixel may be estimated as the ratio of detected "small variance" pixels on the total pixel number in a given sliding window.
• the window size is 3x3 with a weighted counter:
• the total number count is 64
• the weight 8 corresponds to the central and considered pixel position.
• the Chrominance Spatial Filter 550 shown in Figure 5 receives two chrominance components Cr ⁇ /Cb ⁇ denoted as 110-Cr/Cb, their respective residual noise powers ⁇ 2 n sc r / ⁇ 2 n scb 107-Cr/Cb, and luminance shape adaptive window coy 505.
• shape adaptive windowing coy 505 is horizontally down sampled by 2 and hold 551 to provide chrominance shape adaptive windowing coyc h 552 and by counter 553 the local pixel count N Ch 554. It will be appreciated that for 4:4:4, these operations are not necessary.
• m Cb (c,r) (l/N Ch (c,r))XCb TlJ (c, r) MlJ c h (c,r)
• Previously described temporal filter 103 and the shape adaptive windowing spatial noise reducer reduce partially blocking artifact.
• the proposed system includes also the Block Localization unit 102 and the Block Artifact Reducer 113 as illustrated by Figure 1.
• Block Localization is required only when picture is provided after some manipulation. In the present invention, the discussion is limited only for the case where blocks are shifted relatively from image boundaries.
• FIG. 6 there is illustrated an embodiment of a Block Localization unit 102.
• the Block Localization unit 102 receives only the luminance component 101 -Y of a noisy video input. The block detection will be based only in luminance component and used for three components Y, Cr and Cb in the proposed system. [0153] The Block Localization unit 102 is nearly row-column separable. The received signal 101 -Y is applied to two distinct vertical and horizontal branches, specifically to vertical and horizontal line masks 601 and 602. The used line mask impulse responses are given respectively in one embodiment by the two following expressions:
• Respective signal outputs 603 and 604 of the above masks are denoted furthermore as P 3v and P 3h .
• P 2V is pixel delay version of pixel input P 3v
• P 1v is line delay version of P 3v etc.
• P Ov corresponds furthermore to considered pixel position.
• the horizontal signal P 3h 604 is provided to horizontal network 606 which provides also eight (8) output signals 615-622.
• the relative positions of 6 cited pixels may be represented in
• P 2h is pixel delay version of pixel input
• P 1 h is line delay version of P 3h etc.
• Po h corresponds furthermore to considered pixel position.
• the signal set (607-614) is applied to the Vertical Block Border Test 623 for preliminary vertical block detection.
• the Vertical Test 623 in pseudo code is given as follows:
• Ch m1 corresponds to the first maximum histogram amplitude and its associative position lh m1
• Ch m2 and lh m2 correspond to the second maximum.
• lh m1 and lh m2 are possible 8 shift values from 0 to 7. It is interesting to note that vertical line detection implies horizontal position.
• Horizontal Histogram-based Detector 628 provides a set of Cvm1 , lv m1 , Cv m2 and lv m2 , denoted by (633-636).
• Horizontal Offset Detector 637 is described in pseudo code as follows:
• Offset_x lh m1 .
• Offset_y lv m1 .
• value 8 denotes no block border detected meanwhile 0 to 7 denote possible block border shifts.
• Offset_x and Offset_y, 639 and 640 are provided to Offset
• Offset_x [Offset_x + 1 ] modU
• Offset_y [Offset_y + 1 ] modU
• Offset_x 0
• Offset_y 0
• Offset_x Offset_x
• Offset_x Offset_x
• Offset_y Offset_y.
• the corrected Offset_x and Offset_y values 104 and 105 are provided to the Noise Estimation unit 117 and to the Block Artifact Reducer 1 13 as shown in
• Block Artifact Reducer unit 113 in accordance with the invention.
• the Block Artifact Reducer unit 113 receives the video signal 112 of three components Y mnr , Cr mnr and Cb mnr delivered by Spatial Mosquito Noise Reducer
• the Block Artifact Reducer (BAR) unit 113 receives also the two Offset value signals 104 and 105 provided in turn by the Block Localization unit 102.
• the Block Artifact Reducer (BAR) unit 113 receives also the two Offset value signals 104 and 105 provided in turn by the Block Localization unit 102.
• Artifact Reducer (BAR) unit 113 comprises a Luminance Block Artifact Reducer 700 and Chrominance Block Artifact Reducer 720.
• the Luminance Block Artifact Reducer 700 comprises a Border Mask
• BMG Bandwidth Generation
• HFRD High Frequency Region Detector
• LDD Direction Detector
• DLPF Directional Low Pass Filters (4)
• the Border Mask Generator 701 receives the two Offset value signals
• border size is one pixel width; block size is
• the Border Mask Generator 701 provides at its output 702 Bmask signal of three values: 0, 1 and 2 respectively for no border, vertical and horizontal border.
• the Bmask signal 702 is provided to a Luminance
• the High Frequency Region Detector 703 receives the luminance component Y mnr 112-Y provided by Spatial Mosquito Noise Reducer.
• the High Frequency Region Detector 703 receives the luminance component Y mnr 112-Y provided by Spatial Mosquito Noise Reducer.
• Frequency Region Detector 703 comprises in series, not shown, a high pass filter, an absolute value threshold detector and two context-based 3x3 binary filters for consolidation of decision results.
• isolmap, isolation map, is provided to the Luminance Decision Selector 712 to inhibit the correction for avoiding eventual high frequency alias contained in the incoming signal.
• the high pass filter impulse response is given by:
• the absolute value threshold detector comprises an absolute value operator followed by a comparator with a given threshold.
• the output of the comparator is equal to "1" if the input absolute value is greater than a given threshold, otherwise the output of the comparator is equal to "0".
• Line Direction Detector 705 receives the luminance component Y mnr 112-Y.
• the luminance component is applied to the inputs of four high pass filters HP 0 , HP 1 , HP 2 and HP 3 751-754, operating respectively in four principal spatial directions: 0°, 45°, 90° and 135°.
• Their impulse responses are:
• Each filter output is applied to an absolute value operator.
• the four absolute value operator outputs h 0 , h u h 2 and h 3 , 765-768, are provided in turn to Direction Decision unit 759.
• the Direction Decision unit 759 provides its output denoted as d and performs the following operations:
• J 1 ⁇ 0, 1 , 2, 3 ⁇ is the index associated to In 1 .
• the Direction Decision Output 760 is provided to the unit Decision Consolidation 0, 761.
• the latter output denoted as d 0 762 is provided from the following operations:
• dir max is the direction having the frequency freq max .
• dir max is the direction having the frequency freq max .
• the dir output 706 is provided finally to the Luminance Decision Selector 712 in referring back to Figure 7a.
• the received luminance signal Y mnr , 112- Y is also applied to a set 707 of four directional low pass filters LP 0 , LPi, LP 2 and LP 3 operating respectively in four directions: 0°, 45°, 90° and 135°.
• the Luminance Decision Selector 712 selects, in function of the three signals bmask 702, isolmap 704 and dir 706, one of the five luminance signals at its inputs as follows: If (isolmap ⁇ 0) then
• Chrominance Block Artifact Reducer 720 The part of Chrominance Block Artifact Reducer 720 is much simpler than the Luminance Block Artifact Reducer 700 in the embodiment disclosed.
• the received chrominance components Cr/Cb 112-Cr/Cb is provided to a set 721 of two low pass filters operating respectively in horizontal and vertical directions. Impulse responses of these filters are Ipo(c) and Ip 2 (r) previously described.
• the received chrominance components Cr/Cb 112-Cr/Cb and its filtered versions IpO, 722 and lp90, 723 are sent to the Chrominance Decision Selector 724.
• the latter receiving bmask signal 702 selects the followings for its output 114-Cr/Cb signals Cr mnr-bar and
• Enhancer provides the received signal Y mnr - bar 114- Y to an High Frequency Region
• Detector 802 to a set of eight directional masks, 810-817: Mask 0°, Mask 45°,..
• the High Frequency Region Detector 802 is similar to the High
• the High Frequency Region Detector 802 provides a control signal 803 to inhibit Detail Enhancer action in high frequency region.
• Each of the eight mask output signals is now applied to its corresponding multiplier 834-841 and its Amplitude Function 818- 825.
• Each Amplitude Function in the set 818-825 receiving its respective signal variation and its other parameters (max, min values) generates, from signal variation amplitude, a function output varying from zero to 1.
• the function output is equal to 1 if the signal variation amplitude is located between min and max value, elsewhere it is equal to zero.
• the min and max values are chosen such that eventual small or strong noise should be not enhanced.
• Each function signal output 826-833 is provided in turn to respective multiplier 834-841 for weighting the signal variation 850-857.
• Eight multiplier outputs 842-849 are added together via adders 860-866 to provide an enhancing signal 867.
• the latter is weighted furthermore by user controlled enhancement level 806 which is a varying value between 0 and 1.
• the user level weighting is performed by multiplier 807.
• Multiplier result 808 is combined with the luminance input 114-Y via adder 805 to form an enhanced luminance signal 809 which is applied jn turn to the input 0 of multiplexer 801.
• the Multiplexer 801 selects simply the luminance input 114-Y if considered pixel belongs to such region. If not, selected signal is the enhanced luminance signal 809.
• Multiplexer output 116-Y is denoted as Y mnr - enh -
• RAFB-SNR Region Adaptive Facet Based Spatial Noise Reducer
• RAFB-SNR region adaptive facet based spatial noise reducer
• the spatial noise reducer comprises two different innovations: a)- Minimum Mean Square Error denoising technique, b)- Facet models (piecewise linear or piecewise quadratic) adaptation depending on segmented regions.
• the region adaptive facet based spatial noise reducer (RAFB-SNR) 111 as illustrated in Figure 10 receives residual noisy three component (Y, Cr, Cb) image
• the received video 110 is provided to Adaptive Facet Parameters
• the Image Segmentation module 1001 provides a binary signal output
• the module illustrated in Figure 10 for comprehensive and completeness purpose, may be easily derived from the noise estimation unit 117 which is illustrated in Figure 3.
• the Flat/No-Flat regions signal 1002 is sent to
• the present invention can be carried out as a method, can be embodied in a system, a computer readable medium or an electrical or electro-magnetic signal.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (1)

Family

ID=35785883

Family Applications (1)

Country Status (5)

Cited By (12)

Families Citing this family (70)

Citations (4)

Family Cites Families (9)

• 2005
  • 2005-07-29 EP EP05772108A patent/EP1771818A4/en not_active Withdrawn
  • 2005-07-29 CA CA002616875A patent/CA2616875A1/en not_active Abandoned
  • 2005-07-29 US US11/191,912 patent/US7860167B2/en not_active Expired - Fee Related
  • 2005-07-29 JP JP2007522891A patent/JP2008508751A/ja active Pending
  • 2005-07-29 WO PCT/CA2005/001195 patent/WO2006010276A1/en not_active Ceased

Patent Citations (5)

Non-Patent Citations (4)

Also Published As

Similar Documents

Legal Events