KR102411033B1 - IPS bias-compensated noise reduction systems and methods - Google Patents

Abstract

Systems and methods are provided for effectively reducing noise while reducing bias introduced from various image signal processors (ISPs) or ISP components. ISP biases include biases from spatial noise filtering, high dynamic range (HDR) interpolation and demosaicing, among others. Specifically, the methods and systems of the present disclosure provide improved spatiotemporal noise by employing convex combining of multiple input frames or blocks of pixels to effectively reduce bias from a spatial noise filter, debaber unit or other ISP components. reduction solutions.

Description

IPS bias-compensated noise reduction systems and methods

BACKGROUND This disclosure relates generally to image signal processing. Specifically, the present disclosure relates to apparatus and methods for compensating for bias introduced by image signal processors (ISPs) or ISP components while effectively reducing noise. More specifically, high fidelity images and video by effectively reducing noise while reducing bias introduced from various ISPs or ISP components, including spatial noise reduction filters, HDR interpolation units and demosaicking units. Bias-compensated noise reduction systems and methods for generating

Noise reduction is becoming an important aspect of image and video capture systems as cameras and sensors continue to decrease along with the size of pixels while the availability of digital processing power improves. In general, video noise reduction can be roughly divided into spatial noise filters and temporal noise filters. Spatial filters are known to use neighboring pixels in each video frame to produce each output pixel. Temporal filters are known to use successive pixels between frames to produce each output pixel. Spatial and temporal noise reduction filters can be used simultaneously for good results.

Spatial noise reduction can be effective for still images, but most existing spatial noise reduction filters, in the final result, have artifacts such as ringing or blockiness, or This results in some form of bias, such as smoothing textures and microstructures. When applied to video, spatial noise reduction can generate visible residual temporal variations between frames that are not visible from a single frame.

Temporal noise reduction in one of the most common forms involves averaging or otherwise combining pixels in stationary portions of input frames. When the temporal filter converges slowly, i.e., when there are few frames available to combine pixels, the resulting images or videos create noise traces. In addition, ghost artifacts may occur when changing portions of input frames are incorrectly classified as static.

When a spatial noise filter is used in conjunction with a temporal noise filter, they are often referred to collectively as 3-D noise reduction filters or spatiotemporal noise reduction filters. However, existing forms of spatiotemporal filters present some similar problems from their component spatial noise filter or its component temporal noise filter. For example, when spatial filtering is applied first, this spatiotemporal filter introduces certain biases including smoothing of textures and details. However, if the temporal filter is applied first, motion detection or estimation is not effective due to noise. Convergence of the recursive temporal filter can also be slow. On the other hand, when switching of spatial and temporal filtering is implemented, motion detection is not effective due to noise and convergence of the recursive temporal filter remains slow. Moreover, in switching of these spatial and temporal filtering systems, the bias from temporal filtering persists in non-static regions if temporal filtering is not effective.

Thus, as is evident in existing systems, a typical ISP component, such as a spatial noise filter, may introduce bias in the resulting images and videos, thereby resulting in the overall camera or video communication system powered by the ISPs. weakens fidelity. Another example where an ISP introduces a bias into the resulting images and videos is a demosaicing unit or debayer unit, which may cause smoothing, zipping artifacts or incorrect colors in output frames. Additional examples include HDR (High Dynamic Range) interpolation for interleaved long and short exposure pixels that introduce their own bias.

Consequently, a need exists for improved methods and systems to reduce or compensate for bias introduced by an ISP in camera and video communication systems and to produce high fidelity images and videos.

Overview of various embodiments

Accordingly, it is an object of the present disclosure to provide methods and systems for generating high fidelity images and videos by effectively reducing noise while reducing the bias introduced by ISPs.

In particular, in accordance with the present disclosure, in one embodiment, a spatiotemporal noise reduction system for compensating for bias from image signal processing of raw video signals is provided. The system comprises: an image signal processor comprising a spatial noise reduction filter adapted to output each pixel based on neighboring pixels in each raw video frame; a signal change detector adapted to detect any signal changes in each pixel by receiving an output of the image signal processor and a previous output frame; and a signal combiner adapted to receive more than two input frames and generate a combined output frame therefrom. More than two input frames include a raw signal frame, an output from an image signal processor, and a previous output frame.

In another embodiment, the signal combiner is adapted to output a linear combination of more than two input frames. In another embodiment, the signal combiner is adapted to output a convex combination of more than two input frames.

In a further embodiment, the system further comprises a confidence updater adapted to determine the current confidence indicator by updating the previous confidence indicator for a previous output frame based on detection of any signal changes received from the signal change detector. A convex join weight is computed for each input of the convex join based on the current confidence indicator.

In another embodiment, the previous confidence indicator is assigned 0 for the first output frame from the raw video signals.

According to another embodiment, the system further comprises a motion compensator configured to reduce motion based on a previous output frame and an output from the image signal processor. The more than two input frames of the signal combiner include a raw signal frame, an output from the image signal processor, and an output from a motion compensator.

In a further embodiment, the raw video signals comprise an exposure mosaic with spatially interleaved long and short exposure time pixels. The image signal processor further includes a spatial HDR interpolation unit and a demosaicing unit. More than two input frames include a raw signal frame, an output from a spatial HDR interpolation unit, an output from a demosaicing unit, and a previous output frame.

In another embodiment, the raw video signals comprise a color mosaic having different color pixels spatially interleaved. The image signal processor further includes a demosaicing unit. More than two input frames include a raw signal frame, an output from a demosaicing unit, and a previous output frame.

According to another embodiment, the raw video signals include demosaiced frames.

In another embodiment, the raw video signals include frames that have not been processed by the spatial noise reduction filter and that have not been demosaiced, and the image signal processor further comprises a demosaicing unit. More than two input frames include a raw signal frame, an output from a demosaicing unit, and a previous output frame.

In a further embodiment, the signal combiner is adapted to output a linear combination of more than two input frames.

In another embodiment, the signal combiner is adapted to output a convex combination of more than two input frames.

In yet another embodiment, the system further comprises a confidence updater adapted to determine a current confidence indicator by updating a previous confidence indicator for a previous output frame based on detection of any signal changes received from the signal change detector. A convex join weight is computed for each input of the convex join based on the current confidence indicator.

In a further embodiment, the system further comprises a motion compensator adapted to reduce motion based on a previous output frame and an output from the image signal processor. The more than two input frames of the signal combiner include a raw signal frame, an output from an image signal processor, and an output from a motion compensator.

In accordance with the present disclosure, in one embodiment, a method is provided for compensating for bias from image signal processing of raw video signals. The method includes generating a linear combination of more than two input frames; and outputting a linearly combined frame. More than two input frames include a raw signal frame, an output from image signal processing, and a previous output frame.

In another embodiment, the linear combination is a convex combination of more than two input frames.

In another embodiment, a method includes generating a signal change detection classifier for each block of pixels based on an output of the image signal processor and a previous output frame; updating the confidence indicator for the current output frame based on the signal change detection classifier and the previous confidence indicator for the previous output frame; and calculating a weight for each input frame of the convex combination based on the updated confidence indicator for the current output frame.

In a further embodiment, calculating a weight for each input frame of the convex combination comprises: a weight for the unfiltered raw input frame and a weight for the output of image signal processing based on a confidence indicator of the current output frame and providing a decreasing function for the ratio between

According to another embodiment, the decreasing function is a monotonic decreasing function.

In yet another embodiment, calculating a weight for each input frame of the convex combination further comprises providing an incrementing function for the weight for a previous output frame based on a confidence indicator of the current output frame. do.

In a further embodiment, the confidence indicator is a numeric number having a range between 0 and 1.

According to another embodiment, the method further comprises generating a motion-compensated output by reducing motion based on a previous output frame and output from image signal processing. Convex combining is a convex combining of a raw signal frame, an output from image signal processing, and a motion-compensated output.

According to another embodiment, the raw video signals contain (i) spatially interleaved long and short exposure time pixels, (ii) color mosaic with spatially interleaved different color pixels, (iii) demosaiced frames. , and (iv) frames that are not spatially filtered and have not been demosaiced. The image signal processing is selected from the group consisting of (i) spatial HDR interpolation, demosaicing, and spatial noise reduction filtering, (ii) demosaicing and spatial noise reduction filtering, and (iii) spatial noise reduction filtering.

In a further embodiment, the more than two input frames include a raw signal frame, a previous output frame, and output from the group consisting of spatial HDR interpolation, demosaicing, and spatial noise reduction filtering.

In another embodiment, the method further comprises generating a motion-compensated output by reducing motion based on a previous output frame and output from image signal processing. Convex combining is a convex combining of a raw signal frame, a motion-compensated output, and an output from the group consisting of spatial HDR interpolation, demosaicing, and spatial noise reduction filtering.

According to another embodiment, a method includes generating a signal change detection classifier based on a previous output frame and an output of the image signal processing; determining a current confidence indicator for each pixel based on the signal change detection classifier and confidence indicators of neighboring pixels; and calculating a weight for each input frame for the convex combination based on the current confidence indicator.

1 illustrates a bias-compensated noise reduction system in accordance with an embodiment of the present disclosure;
2 illustrates a bias-compensated noise reduction system according to another embodiment.
3 illustrates a bias-compensated spatiotemporal noise reduction system according to another embodiment.
4 illustrates a bias-compensated space-time and debayer (demosaic) noise reduction system according to another embodiment.
5 illustrates a reliability updater of a bias-compensated noise reduction system according to an embodiment.
6 shows the convex coupling weight of each input of the convex coupling calculated by the weight calculator of the bias-compensated noise reduction system according to an embodiment.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

System and Methodology Overview

Methods and systems in accordance with various embodiments of the present disclosure employ a weighted combination of multiple input frames or blocks of pixels as part of a recursive temporal noise filter, such as an ISP, such as a spatial noise filter, demosaicing unit, It provides an improved noise reduction solution by reducing the bias from the HDR interpolation unit. The bias-compensation and noise reduction system in various embodiments of the present disclosure is designed to reduce biases introduced in the ISP stage of exposure mosaic interpolation, color demosaicing interpolation, or spatial noise reduction filtering.

In one embodiment, referring to FIG. 3 , for example, the spatial noise reduction filter is an ISP combined with a low complexity temporal noise filter. The resulting spatiotemporal filter reduces the bias from the spatial noise filter and at the same time reduces the disadvantages of temporal filtering. According to some embodiments, the spatial noise reduction filter is provided in hardware or firmware, and the temporal noise reduction filter is customized and implemented in software. The customized temporal filter of the present disclosure may be combined with an existing spatial filter in certain embodiments to provide an improved noise reduction solution by forming an extension to existing equipment.

In other embodiments where the raw input data includes a color mosaic with spatially interleaved long and short exposure time pixels or spatially interleaved different color pixels, the demosaicing unit or debayer unit is combined with a temporal noise filter It is an ISP that becomes The resulting bias-compensation filter reduces bias or artifacts from demosaicing and achieves higher fidelity in image or video output frames. In further embodiments, additional ISPs including spatial noise filters are combined in the system with a demosaicing unit and a recursive temporal filter. See FIG. 4 . The resulting bias-compensation filter reduces bias or artifacts from the spatial noise filter and Debayer unit and achieves improved fidelity in output frames.

Signal combiners and reliability indicators

1 , a signal combiner according to an embodiment is a central part of the recursive temporal filter of the present disclosure. The signal combiner provides a linear combination in one embodiment of at least two input frames, including an unfiltered input frame, an ISP-filtered input frame, and a previous output frame. In another embodiment, the signal combiner provides a convex combining of at least two input frames, including an unfiltered input frame, an ISP-filtered input frame, and a previous output frame. A convex combination according to various embodiments is made per block of pixels or per pixel. A confidence indicator is associated with each convex combination per pixel or per block of pixels. A convex combination based on the updated confidence indicator determines a new output frame.

According to one embodiment, for each input frame a confidence indicator is assigned for every input frame or blocks of pixels. In various embodiments the confidence indicator generally indicates the degree to which the previous output frame is a good representation of the current input frame. According to one embodiment, it is defined as a numerical number having a predetermined range, for example from 0 to 1. The confidence indicator is stored in the frame buffer along with the pixel values. It is calculated by the confidence updater based on the confidence update function.

Referring to FIG. 5 , the confidence update function is based on the signal change classifier (“true” or “false”) derived for each block of pixels and the confidence indicator of the previous output frame (C(n−1)) to compute the fidelity indicator (C(n)) of the current frame or current block of pixels.

In one embodiment, the updated confidence indicator is defined to be a positive increment of the previous confidence indicator for the previous frame or block of pixels plus the positive increment of the pixels or blocks of pixels classified as static. And, for a pixel or blocks of pixels classified as changing, the updated confidence indicator is defined to be the lowest possible value for the system, such as zero, in some embodiments. According to one embodiment, the increment according to certain embodiments depends on the previous confidence indicator, but the updated confidence indicator is constrained to be within a predetermined range, such as between 0 and 1. See FIG. 5 .

1 , the signal change classifier for which the reliability indicator is calculated is determined by a signal change detector coupled to an image signal processor (ISP) stage. An ISP stage according to a particular embodiment includes a spatial noise reduction filter, wherein the signal change detector detects position changes in blocks of related pixels based on an output of the spatial filter and data from a previous output frame.

Convex joint weights

Referring to FIG. 1 , a weight calculator takes input from the reliability updater in calculating a convex joint weight for each input frame or block of pixels. According to one embodiment, the raw unfiltered input is weighted more than the ISP-filtered input or the spatially filtered input, where ISP is the spatial noise reduction filter, as the confidence indicator increases. Thus, the bias of the spatial filter is reduced as more frames are averaged out in the process. On the other hand, when the confidence indicator for the previous output frame is low, the spatially filtered input is weighted more, thereby reducing noise fluctuations in the output.

Thus, in certain embodiments, the ratio between the weight for the ISP-filtered input frame and the weight for the unfiltered raw input frame is a monotonic decreasing function of the reliability indicator. Also, the weight for the previous output frame is an increasing function of the confidence indicator. The weight for this previous output frame is zero or near zero when the confidence indicator is zero and close to one when the confidence indicator is one, according to certain embodiments.

6 , the convex joint weights according to one embodiment are derived by the following formula based on the confidence indicator: gamma = c, beta = (1.0-c)*(1.0-c), alpha = ( 1.0-c)*c, where c refers to the confidence indicator, alpha refers to the weight for the raw unfiltered input, beta refers to the weight of the ISP-filtered input or the spatially filtered input, and is a spatial filter, and gamma refers to the weight for the previous output frame. In this embodiment, the sum of alpha, beta and gamma is 1, and the confidence indicator has a value between 0 and 1. Thus, according to the manner in which alpha and beta change as a function of the confidence indicator for the previous frame in this embodiment, beta (weight for ISP or spatially filtered frame) is high at the start of processing, and therefore While the convergence of H occurs faster, alpha (weight for the unfiltered raw input) increases gradually to reduce the bias from the ISP or spatial filter where ISP is a spatial filter.

In alternative embodiments, piecewise linear approximations may be employed to derive weights for each input frame or block of pixels in the combined output frame. The combined output frame is based on a linear or convex combination of more than two input frames in alternative embodiments.

signal change detector

As discussed above, the confidence updater updates the confidence indicator for each input frame or each block of pixels based on the change detection classifier generated by the signal change detector in certain embodiments. See, for example, FIGS. 1 , 2 and 5 . The signal change detector takes as inputs an ISP-filtered signal (or a spatially filtered signal where the ISP is a spatial filter) along with the previous output frame, and sets the status for each pixel or block of pixels that indicate "static" or "change". Outputs a binary change detection classifier (“true” or “false”). For example, in a basic embodiment, the difference between the two inputs is evaluated and a threshold is determined for the classification of “change”. The difference between the two inputs above the threshold leads to a "true" reading by the signal change classifier indicating "change", otherwise "false" reading by the signal change classifier as indicating "static" do.

According to other embodiments, the signal change detector takes as inputs the spatial neighbors of pixels in previous and current input frames to improve classification of signal changes for the system. In further embodiments, the signal change classifier employs a background and foreground estimation algorithm to classify pixels as static or changing.

Thus, a bias-compensated noise reduction system according to various embodiments can be configured such that, on the one hand, inputs to a signal change detector for classification of changes are decoupled from inputs to a signal combiner for convex combining, on the other hand, a new output Provides the flexibility to form a frame. The inputs for determining the signal detection classifier are selected in various embodiments to maximize classification performance, while the inputs to the convex combination are the spatial filters employed by the system and the visual degradation of ISPs such as Debayer units. are selected to effectively reduce s and biases.

motion compensator

2 , the bias-compensated noise reduction system in another embodiment further includes a motion compensator. The motion compensator employs motion estimation techniques to modify the previous output frame and compensate for motion between the previous output frame and the current input frame. As motion is estimated and compensated, the bias-compensated noise reduction system according to this embodiment enables excellent opportunities for combining frames in the time dimension. For convex combining, a motion compensated video frame according to this embodiment is taken as an input frame instead of a previous output frame.

3 provides another example of a bias-compensated spatiotemporal noise reduction system in which motion detection and compensation is implemented. In this embodiment, the convex combining is based on the output from the motion compensator, the spatial-filtered input frame, and the unfiltered input frame.

4 provides a further example of a bias-compensated space-time and DeBayer noise reduction system in which motion detection and compensation are implemented. In this embodiment, the convex combining is based on the output from the motion compensator, the spatial-filtered input frame, the debayered frame, and the raw Bayer input frame.

ISP stage and reduced corresponding bias

As discussed above, the systems and methods of this disclosure reduce biases introduced by ISPs or ISP components during ISP-filtering of raw input data, also referred to as the ISP stage. The raw input data to the system is of various types in various embodiments. The ISP stage in various embodiments employs and operates one or more ISPs of different utilities. In one embodiment, the ISP stage includes exposure mosaic interpolation. In another embodiment, the ISP stage includes color demosaicing interpolation. In a further embodiment, the ISP stage includes spatial noise reduction filtering.

For example, in certain embodiments, a block mosaic of long and short exposures, or interleaved lines with different exposures, are generated by existing sensors. In one embodiment HDR sensing is employed at the ISP stage. According to one embodiment, since spatial interleaving and spatial reconstruction are involved in HDR interpolation, which inherently introduces some bias, an enhanced reconstruction with good quality is applied to the previous output frame, the raw interleaved exposed pixels and the bias. It is provided through convex combining based on weighted input frames containing the received spatial reconstruction.

Thus, in this embodiment, the raw data is exposure mosaic, and the ISP filtering stages are spatial HDR interpolation, demosaicing, and spatial noise filtering. In another embodiment, the raw data is color mosaic, and the ISP filtering stage is demosaicing and spatial noise filtering. In a further embodiment, the raw data is demosaiced frames and the ISP filtering stage is spatial noise filtering. Convex combining in various embodiments is achieved by combining weighted input frames comprising temporal filtered data (previous output frame or motion-compensated output), raw data, and ISP filtered data.

The descriptions of the various embodiments provided in this disclosure, including various drawings and examples, are illustrative of the invention and its various embodiments and are not limiting.

Claims (30)

A noise reduction system for compensating for bias from image signal processing of raw video signals, comprising:
an image signal processor comprising a spatial noise reduction filter adapted to output each pixel based on neighboring pixels in each raw video frame;
a signal change detector adapted to detect any signal changes in each pixel by receiving an output of the image signal processor and a previous output frame;
a signal combiner adapted to compensate for bias from the image signal processor by combining more than two input frames and generating a combined output frame, wherein the more than two input frames are a raw signal frame, an output from the image signal processor and the signal combiner including a previous output frame. The method of claim 1,
and the signal combiner is adapted to output a linear combination of the more than two input frames. 3. The method of claim 2,
and the signal combiner is adapted to output a convex combination of the more than two input frames. 4. The method of claim 3,
and a confidence updater adapted to determine a current confidence indicator by updating a previous confidence indicator for a previous output frame based on detection of any signal changes received from the signal change detector, wherein a convex combining weight is the current reliability indicator. calculated for each input of the convex combination based on an indicator. 5. The method of claim 4,
and the previous confidence indicator is assigned zero for a first output frame from the raw video signals. 5. The method of claim 4,
and a motion compensator adapted to reduce motion based on the previous output frame and an output from the image signal processor, wherein the more than two input frames of the signal combiner include the raw signal frame, the image signal processor and an output from the motion compensator. The method of claim 1,
wherein the raw video signals comprise an exposure mosaic having spatially interleaved long and short exposure time pixels, the image signal processor further comprising a spatial HDR interpolation unit and a demosaicing unit, wherein the more than two input frames are a raw signal frame, an output from the spatial HDR interpolation unit, an output from the demosaicing unit, and a previous output frame. 8. The method of claim 7,
and the signal combiner is adapted to output a linear combination of the more than two input frames. 9. The method of claim 8,
and the signal combiner is adapted to output a convex combination of the more than two input frames. 10. The method of claim 9,
and a confidence updater adapted to determine a current confidence indicator by updating a previous confidence indicator for a previous output frame based on detection of any signal changes received from the signal change detector, wherein a convex combining weight is the current reliability indicator. calculated for each input of the convex combination based on an indicator. 11. The method of claim 10,
and a motion compensator adapted to reduce motion based on the previous output frame and an output from the image signal processor, wherein the more than two input frames of the signal combiner include the raw signal frame, the image signal processor and an output from the motion compensator. The method of claim 1,
wherein said raw video signals comprise a color mosaic having different color pixels spatially interleaved, said image signal processor further comprising a demosaicing unit, said more than two input frames comprising a raw signal frame, said demosaicing A noise reduction system comprising an output from the unit, and a previous output frame. 13. The method of claim 12,
and the signal combiner is adapted to output a linear combination of the more than two input frames. 14. The method of claim 13,
and the signal combiner is adapted to output a convex combination of the more than two input frames. 15. The method of claim 14,
and a confidence updater adapted to determine a current confidence indicator by updating a previous confidence indicator for a previous output frame based on detection of any signal changes received from the signal change detector, wherein a convex combining weight is the current reliability indicator. calculated for each input of the convex combination based on an indicator. 16. The method of claim 15,
and a motion compensator adapted to reduce motion based on the previous output frame and an output from the image signal processor, wherein the more than two input frames of the signal combiner are the raw signal frame, the image signal processor and an output from the motion compensator. delete delete A method for compensating for bias from image signal processing of raw video signals, comprising:
generating a convex combination of more than two input frames, wherein the more than two input frames include a raw signal frame, an output from the image signal processing, and a previous output frame. to do;
generating a signal change detection classifier for each block of pixels based on the previous output frame and the output of the image signal processing;
updating a confidence indicator for a current output frame based on a previous confidence indicator for the previous output frame and the signal change detection classifier;
calculating a weight for each input frame of the convex combination based on the updated confidence indicator for the current output frame; and
and outputting a frame of the convex combination. delete delete 20. The method of claim 19,
The step of calculating a weight for each input frame of the convex combination comprises: a weight value for an output of the image signal processing and a weight for an unfiltered raw input frame based on a confidence indicator of the current output frame. A method for compensating for bias from image signal processing of raw video signals, further comprising providing a reduction function for ratio. 23. The method of claim 22,
wherein the reduction function is a monotonic reduction function. 23. The method of claim 22,
wherein calculating the weight for each input frame of the convex combination further comprises providing an increasing function for the weight for the previous output frame based on a confidence indicator of the current output frame. A method for compensating for bias from image signal processing of video signals. 23. The method of claim 22,
wherein the confidence indicator is a numeric number having a range between 0 and 1. 23. The method of claim 22,
generating a motion-compensated output by reducing motion based on the previous output frame and output from the image signal processing, wherein the convex combining comprises the raw signal frame, the output from the image signal processing , and a convex combination of the motion-compensated output. delete delete delete delete

Images