Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/70—Denoising; Smoothing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/70—Determining position or orientation of objects or cameras
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2200/00—Indexing scheme for image data processing or generation, in general
  • G06T2200/21—Indexing scheme for image data processing or generation, in general involving computational photography
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10024—Color image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10048—Infrared image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20004—Adaptive image processing
  • G06T2207/20012—Locally adaptive
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20172—Image enhancement details
  • G06T2207/20192—Edge enhancement; Edge preservation

Definitions

• the present disclosure relates to imaging systems and methods that include a de-noising process.
• the disclosure relates to systems and methods that use a near infrared image to guide the de-noising process.
• the first option is to use a high ISO when capturing the image. Increasing sensor gain can effectively increase signal strength to get bright images within short exposure time. However, the image noise is inevitably increased as well, thus the signal-to-noise ratio (SNR) is not improved.
• SNR signal-to-noise ratio
• a second option is to capture the image using a large aperture. Allowing more light pass through camera lenses is a very straightforward way to improve image quality, however changing aperture size will also affect depth of field. Further, the effect is very limited when sensor and lens have been built to accommodate a small form factor, e.g. cell phone cameras.
• a third option is to use long exposure time to capture the image.
• Extended exposure time can increase SNR, but may increase undesired motion blur in the captured image.
• a fourth option is to employ flash or other strong artificial light to the scene in order to obtain a sharp, noise-free image.
• flash may ruin ambience atmosphere and introduce unwanted artifacts, such as red eye, undesired reflections, harsh shadows, and intense light highlights and reflections.
• Image de-noising is an intensively studied problem and numerous methods exist. However, even with the state-of-the-art image de-noising methods, it is still very difficult to obtain a high quality noise-free photo, especially when noise level is high.
• Conventional single-image de-noising solutions consist of several different methods. Image filtering based methods selectively smooth parts of a noisy image. Wavelet-based methods rely on the careful shrinkage of wavelet coefficients. Image prior based methods learn a dictionary from noise-free images and de-noise images by approximating them using a sparse linear combination of the elements in the dictionary.
• dual-image methods introduce another image to guide the de-noising process.
• Two images are captured of the same scene.
• Image filtering of the first image using the guidance of the second image is then applied to better preserve image structure, and image detail transfer may be applied to enhance fine image details.
• the guidance image may be captured under different lighting conditions than the first image, and therefore may contain a different level of detail of the image scene.
• the additional details in the guidance image may be used to enhance the quality of the first image.
• the first type of dual-image de-noising methods uses a visible flash image as the guidance image. However, this method can easily blur weak edges and introduce artifacts from the guidance image. Additionally, the visible flash is intrusive to use under low-light conditions, and is even prohibited in certain environments.
• Figure 1 is a schematic block diagram of a multispectral imaging system, according to one implementation
• Figure 2 is a flowchart of an embodiment of a method for capturing multispectral image data of a particular scene, according to one implementation
• Figure 3A illustrates an example RGB and NIR image
• Figure 3B illustrates various regions of the red, green, and blue channels of the RBG image of Figure 3A and the corresponding regions of the NIR image of Figure 3 A;
• Figure 4 is a flowchart of an embodiment of a method for generating a gradient scale map and using the gradient scale map for de-noising
• Figure 5 illustrates an example of images at various steps of a de- noising process.
• Implementations disclosed herein provide systems, methods and apparatus for image de-noising applications. For example, as explained herein, it can be desirable to process a high-quality image from a noisy visible light image using a corresponding dark flash image, such as a near infrared (NIR) image.
• NIR near infrared
• embodiments are described herein as employing NIR images, it will be appreciated that infrared and ultraviolet flash images may be used as well.
• the implementations disclosed herein can be used to capture a pair of visible light, for example RGB, and NIR images of the same image scene.
• the visible light and NIR images may be aligned on a pixel bases, and a gradient scale map may be introduced to relate gradient fields of the aligned visible light and NIR images.
• the gradient scale map may be generated based on the differences and similarities between the gradient vectors of the visible light image and the gradient vectors of the NIR image.
• the gradient vectors can be used to detect continuities and discontinuities in intensity values between regions of pixels in the image.
• the continuities and discontinuities may represent boundaries of objects in the image, variations in scene illumination, discontinuities in depth, and changes in material properties.
• the gradient vectors of an image can be determined at each pixel by convolving the image with horizontal and vertical derivative filters. For example, gradient vectors along the x- and y- directions may be calculated for a group of pixels, such as an overlapping region between the NIR and visible light images. In some embodiments, the gradient vectors may be calculated by convolving the image with horizontal and vertical derivative filters. For each group of pixels considered, the derivative filters can be applied to calculate the first or second derivative in the horizontal and vertical directions, representing the horizontal and vertical changes in pixel intensity values. This produces a gradient vector value which can be assigned to the center pixel in the group of pixels. Accordingly, the intensity values in a region around each pixel can be used to approximate the corresponding gradient vector value at the pixel.
• the gradient vector values of the pixels in the image can be combined to calculate gradient magnitude vectors and gradient direction vectors.
• the gradient direction vectors and gradient magnitude vectors provide information on the location of object boundaries and about the different intensity values of regions of pixels on either side of the boundaries. For example, object boundary pixels can be located at local maxima of gradient magnitudes..
• the visible light image and the NIR image may be aligned on a pixel-by-pixel basis and cropped to an overlapping region.
• a gradient scale map may be generated by computing the difference in, or ratio between, the values of the gradient vectors of the visible light image and the gradient vectors in the NIR image.
• the gradient scale map accordingly captures the nature of structure discrepancies and similarities between images, and may have clear statistical and numerical meanings for use in the denoising processes described herein.
• the gradient scale map may be assigned a positive value or a negative value for each pixel location.
• a positive value may indicate that an edge or structure is present with similar gradient direction vectors in both the visible light image and the NIR image.
• a negative value may indicate that an edge or structure is present in both the visible light image and the NIR image, but the direction of the local gradient vector in the NIR image is reversed relative to the local gradient vector in the visible light image.
• a value of zero may indicate that an edge or structure is present in the NIR image which is not present in the visible light image, for example due to highlights and shadows resulting from the NIR flash, or that an edge or structure present in the visible light image is not present in the NIR image, for example due to the different reflectance properties of an object in the scene to red and infrared light.
• an optimal ratio map for de-noising by gradient transfer may be generated considering adaptive smoothing, edge preservation and guidance strength manipulation.
• the optimal ratio map may represent the amount of guidance from the NIR image which will be applied to the visible light image at each pixel.
• the de-noising methods described herein are better able to leverage the NIR image as a guidance image to achieve high quality image de-noising, and may do so without introducing additional artifacts into the de-noised visible light image from the guidance image.
• an imaging apparatus may capture images using two different types of imaging sensors, for example a NIR sensor and a visible light sensor.
• This type of multispectral imaging can be useful de-noising the image captured by the visible light sensor.
• the light received at the visible light sensor can carry color information of a scene, while the light captured by the NIR sensor can be used to enhance the quality of the visible light image by performing image de-noising.
• the NIR image and visible light image may be decomposed into gradient vectors and used to create a gradient scale map.
• the gradient scale map may be used to guide de-noising of the visible light image by the NIR image to improve the quality of the captured visible light image.
• an image captured by a NIR sensor is used to enhance the quality of visible light images captured in low light conditions.
• a visible flash system is often used to illuminate the object to be imaged.
• such artificial light may ruin the ambience of the image and may introduce unwanted artifacts like red eye, undesired reflections, and shadows.
• a NIR flash lamp may be used instead of a visible flash to capture a NIR image, and a conventional visible imaging sensor (such as a RGB color CCD) captures a corresponding visible light image.
• the NIR image may not be contaminated with noise as in the visible light image, and the NIR image may be used in conjunction with a denoising technique (such as a weighted least squares smoothing technique) to remove noise from the visible (e.g., RGB) image caused by low lighting conditions.
• a denoising technique such as a weighted least squares smoothing technique
• the NIR guidance image may also be captured in some embodiments without a flash using an image sensor configured to detect the infrared or near-infrared light naturally present in an image scene.
• NIR near-infrared
• image sensors With non-intrusive NIR flash, a high quality noise-free NIR photo is captured to guide de-nosing and detail recovery in a corresponding RGB image.
• Some embodiments may capture a pair of RGB and NIR images simultaneously and subsequently align the RGB and NIR images pixel-wise. Such an image pair can be captured using dual camera system, a single camera which captures sequential RGB- NIR shots in static scenes, by a single RGB-IR camera, or by a stereo RGB-NIR camera system as described in U.S. Application No. 13/663,897, filed October 30, 2012, the entirety of which is herein incorporated by reference.
• a process is terminated when its operations are completed.
• a process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.
• its termination may correspond to a return of the function to the calling function or the main function, or a similar completion of a subroutine or like functionality.
• the multispectral imaging system 100 can include a multispectral imaging apparatus 110.
• the multispectral imaging apparatus 110 can be configured to include one or more multispectral imaging sensors that can sense a broad band of wavelengths, including at least visible light wavelengths and near infrared (MR) light wavelengths.
• the multispectral imaging apparatus 1 10 can be configured to detect light at wavelengths between about 390 nm and about 1400 nm.
• the imaging apparatus 1 10 can also be configured to detect a much broader range of wavelengths as well.
• a charge-coupled device (CCD) can be used as the multispectral imaging sensor(s).
• CMOS imaging sensor can be used as the multispectral imaging sensor(s).
• a flash module 1 18 and/or other accessories may be included in the multispectral imaging apparatus 110 to help illuminate the scene.
• the flash module 8 can include visible and/or R flash devices.
• the multispectral imaging apparatus 1 10 includes two separate sensors instead of a single multispectral imaging sensor.
• the multispectral imaging apparatus 110 includes a visible light sensor 115 and a separate NIR sensor 117.
• the multispectral imaging apparatus 1 10 has a first, visible sensor 115 such as a CCD/CMOS sensor capable of detecting visible light at least in the range between about 390 nm and about 800 nm.
• the multispectral imaging apparatus 1 10 further includes a second, NIR sensor 117, such as a CCD/CMOS sensor that is capable of detecting NIR light in the range between about 800 nm and about 1400 nm.
• the wavelength ranges for the visible and NIR sensors can overlap or can even be substantially the same.
• imaging filters such as a NIR pass filter, can be used on a suitable CCD/CMOS sensor to detect only the NIR data.
• both the visible light sensor 1 15 and the NIR sensor 117 can be implemented on a Samsung® S5K4E1GX QSXGA CMOS sensor.
• the NIR sensor 1 17 can be implemented by removing an IR-cutoff filter to enable the sensor to receive NIR light.
• the modified, NIR sensor 1 17 can be further covered by an IR filter to filter out lower wavelengths, e.g., wavelengths less than about 800 nm.
• a Kodak® Wratten IR filter (#87C) can be applied over the NIR sensor 1 17.
• various other sensors or combinations thereof can be used to capture visible and NIR image data.
• the multispectral imaging system 100 may capture a pair of RGB and NIR images simultaneously.
• the multispectral imaging system 100 can, in other embodiments, capture the NIR image within a predetermined time interval of capturing the RGB image, and the predetermined time interval may be short enough that the pair of images can be of substantially the same scene.
• the NIR image may be captured using NIR flash or by capturing an image using NIR ambient light present in the target image scene.
• the visible image may be captured using flash or ambient light.
• the multispectral imaging system 100 further includes a processor 140 and a memory 130 that are in data communication with each other and with the imaging apparatus 1 10.
• the processor 140 and memory 130 can be used to process and store the images captured by the imaging apparatus 1 10.
• the multispectral imaging system 100 can include a user interface (UI) 150 configured to provide input/output (I/O) operations to the user.
• the UI 150 can include a display that presents various icons to the user.
• the UI 150 may also include various input devices, such as a keyboard, touch-screen input, mouse, rollerball, data inputs (e.g., USB or wireless), and/or any other suitable type of input device.
• the UI 150 can be programmed to allow the user to manipulate image data and/or to select the imaging mode that the user desires to use.
• the UI 150 may also include controls for capturing the multispectral image data.
• the processor 140 and memory 130 can be configured to implement processes stored as software modules in an image processing module 120 and configured to process multispectral image data captured by the multispectral imaging apparatus 1 10.
• the image processing module 120 can be implemented in any suitable computer-readable storage medium, such as a non-transitory storage medium.
• the image processing module 120 can have any number of software modules.
• a communications module 121 can be implemented on the image processing module 120.
• the communications module 121 can comprise computer-implemented instructions that manage the flow of data between the components of the multispectral imaging system 100.
• the communications module 121 can include instructions that manage the receipt of image data from the multispectral imaging apparatus 1 10.
• the image processing module 120 also includes a storage module 129 configured to store various types of data, source code, and/or executable files.
• the storage module 129 can be programmed to store image data received by the imaging apparatus 110 and/or image data processed by the image processing module 120.
• the image processing module 120 can also include various modules that are programmed to implement various multispectral imaging applications.
• the image processing module 120 includes a dark flash imaging module 123 that is programmed capture both the NIR image and the visible light image.
• the dark flash imaging module 123 may cause the visible light sensor 115 and the NIR sensor 117 to capture images of a target scene at substantially the same time.
• the dark flash imaging module 123 may cause the visible light sensor 1 15 and the NIR sensor 1 17 to capture images of a target scene in succession.
• the image processing module 120 may include a hybrid imaging module 125.
• the hybrid imaging module 125 can be programmed to process both still images and video images captured by the imaging apparatus 1 10.
• the hybrid imaging module 25 can be programmed to process still image data from the visible light sensor 115 and video image data from the NIR sensor 1 17, or vice versa.
• the still image data and the video image data can be simultaneously captured by the respective sensors in some arrangements; in other arrangements, the still and video image data can be captured at separate times.
• the still image data can be captured at a higher resolution than the video image data in order to reduce motion blur in images of a scene.
• the image processing module 120 also includes a multispectral information processing module 127.
• the multispectral information processing module 127 can be programmed to process NIR image data captured from the NIR sensor 1 17 to enhance the quality of visible light image data captured from the visible light sensor 115 through a de-noising process.
• the multispectral information processing module 127 can include various sub-modules for carrying out a noise reduction process, as described below.
• the multispectral information processing module 127 may include an image alignment module 124.
• the NIR sensor 1 17 and visible light sensor 115 may be spaced apart by a distance and may therefore capture a target scene using different framing and/or angles. NIR and visible light images may be aligned before the images are processed for noise reduction.
• the image alignment module 124 can be programmed to perform any preliminary operations on the NIR and visible light images, such as, e.g., confirming that the visible light sensor 1 15 and the NIR sensor 117 are aligned vertically.
• the image alignment module 124 can further be programmed to match pixels in the NIR image with pixels in the visible light image to form a plurality of matched pixel pairs.
• the image alignment module 124 can thereby provide an initial, pixel-by -pixel alignment of the NIR and visible light images based on image descriptors for each image. The image descriptors may be based in part on image gradients measured in the respective images.
• the image alignment module 124 also can be programmed to generally align sparse portions of the NIR and visible light images. For example, the image alignment module 124 can be programmed to calculate pixel disparities for each matched pixel pair. In various implementations, the pixel disparities can be based on a pixel separation distance for each matched pixel pair. The image alignment module 124 can be programmed to assign weights to each matched pixel pair based at least in part on the calculated pixel disparities for each matched pixel pair. Once the weights are assigned, in various implementations, the matches with the higher weights are kept, while the matches with the lower weights are discarded. A piece-wise homographic constraint between the visible and NIR images can be estimated, and the matches that satisfy the homographic constraint can be kept for subsequent alignment processing.
• the image alignment module 124 can be programmed to align the NIR image with the visible light image based at least in part on the assigned weights. For example, the image alignment module 124 can propagate the sparse matching results to the dense matching results based on an intensity similarity value and/or a confidence map, e.g., such as the weights that were assigned. In some aspects, the NIR image can be warped to the perspective of the visible light image, or vice versa, to confirm that the alignment of the images is accurate.
• the gradient scale map generation module 126 can be programmed to generate a gradient scale map from the NIR and RGB images.
• the gradient scale map may be generated based on the differences and similarities between gradient vectors in the visible light image and the NIR image. Accordingly, the gradient scale map generation module 126 can be programmed to calculate the gradient vector values at each pixel in each of the visible and NIR images, for example by applying a derivative filter to pixel blocks and assigning a gradient vector value to a center pixel in each pixel block.
• the gradient scale map generation module 126 may be further programmed to generate a gradient scale map based on a comparison of gradient vectors of the NIR image and visible light image.
• the gradient scale map accordingly may capture the nature of structure discrepancies and similarities between the NIR and visible light images.
• the gradient scale map generation module 126 may assign a positive value or a negative value to each pixel location within the gradient scale map. For example, a positive value may indicate that an edge or object boundary is present with similar gradient direction vectors in both the visible light image and the NIR image. A negative value may indicate that an edge is present in both the visible light image and the NIR image, but the direction of the local gradient vector in the NIR image is reversed relative to the local gradient vector in the visible light image.
• a value of zero may indicate that an edge is present in the NIR image which is not present in the visible light image, for example due to highlights and shadows resulting from the NIR flash, or that an edge or structure present in the visible light image is not present in the NIR image, for example due to the different reflectance properties of an object in the scene to red and infrared light.
• the denoising module 128 may be programmed to receive the gradient scale map values from the gradient scale map generation module 126 and to utilize such values for denoising the visible light image.
• an optimal ratio map for de-noising by gradient transfer may be generated based on an analysis of the pixel-by-pixel values in the gradient scale map.
• the denoising module 128 may take into account image quality enhancement factors such as adaptive smoothing, edge preservation and guidance strength manipulation.
• the optimal ratio map may represent the amount of guidance from the NIR image which the denoising module 128 may apply to the visible light image at each pixel.
• the denoising module 128 may be further programmed to perform an iterative denoising process on the visible light image. For example, a denoised visible light image may be sent with the NIR image back to the gradient scale map generation module 126 in order to generate an updated gradient scale map representing the similarities and differences between the gradient vector fields in the denoised visible light image and the NIR image. The denoising module 128 may use this updated gradient scale map to further denoise the denoised visible image. This process may be repeated, in some embodiments, until the updated gradient scale map varies less than a threshold amount from a previous gradient scale map. In some embodiments, the process may be repeated until a resulting denoised visible light image varies less than a threshold amount from the visible light image used to construct the updated gradient scale map.
• FIG. 2 is a flowchart of a method 200 for capturing multispectral image data of a particular scene, according to one implementation.
• the illustrated method 200 begins in a block 205 wherein the NIR and visible light sensors are activated.
• the user may power up and/or boot- up the imaging system 100 to enable the sensors to capture the scene.
• an icon on a mobile device may be engaged to activate the NIR and visible light sensors 1 15 and 1 17.
• the method then moves to a block 210, wherein a visible light image is captured by the visible light sensor and a NIR image is captured by the NIR sensor.
• a flash e.g., visible and/or NIR flash
• the process 200 moves to a block 215, wherein the NIR image and the visible light image can be aligned with respect to the image alignment module 124 of Figure 1.
• the aligned images may be cropped to an overlapping region for subsequent processing.
• the process 200 then moves to block 230 in which a gradient scale map representing the differences and similarities between the gradient vector fields of the NIR and visible light images is generated.
• the gradient scale map may be generated by computing the difference between gradient vector direction and gradient vector magnitude for each pixel of the NIR and visible light images.
• the gradient scale map is used for image processing.
• the gradient scale map may be used to determine a level of guidance applied from the NIR image to the visible light image for denoising the visible light image.
• the gradient scale map may be used for dehazing or deblurring the visible light image, increasing sharpness or contrast in the visible light image, or for skin smoothening applications.
• the process 200 then moves to a decision block 230, to determine whether additional images are to be processed. If a decision is made in decision block 230 that additional images are to be processed, then the method 200 returns to block 210 to capture additional NIR image data and visible light image data. If a decision is made in decision block 230 that no additional images are to be processed, then the method 200 terminates.
• NIR and RGB Gradients Overview NIR and RGB Gradients Overview
• Figure 3A illustrates an example visible light image 300 and an example NIR image 310.
• the visible light image 300 is discussed as an RGB image, however the visible light image 300 may also be grayscale.
• Figure 3B illustrates zoomed in views of various regions of the red, green, and blue channels of the RGB image 300 of Figure 3A and the corresponding regions of the NIR image 310, noted by the rectangular borders 320, 330, and 340 over the images of Figure 3 A.
• Using the NIR image 310 to guide the denoising of the RGB image 300 may make a significant difference in detail distribution and intensity formation, compared to each channel of the RGB image (the red, green, and blue channels).
• Edge structure and gradient inconsistencies between the RGB 300 and NIR 310 images exists for nearly all pixels, as illustrated in Figures 3A and 3B and discussed in more detail below.
• One structure inconsistency which may exist is gradient magnitude variation.
• the letter 'D' on the book cover was captured with much lower contrast in the NIR image 310 than that in the RGB image 300. This effect may be due to the different reflectance characteristics to infrared and visible light.
• Gradient direction divergence is another structure inconsistency, in which some gradient direction vectors in the RGB image 300 may be reversed with respect the gradient direction vectors located along a corresponding edge in the NIR image 310.
• the red channel, green channel, and blue channel of the lower area 330 of the book cover are illustrated in Figure 3B in comparison with the NIR channel of the same area 330.
• the gradient vector directions of the two images are reversed; that is, the lower pixel region in the red, green, and blue channels is lighter than the upper pixel region, while in the NIR image the upper pixel region is lighter than the lower pixel region.
• the pixels in upper and lower image regions change their relative intensity levels greatly, but in opposite directions in the RGB and NIR images.
• Another structural inconsistency is gradient loss, in which some gradients presented in the RGB image 300 may be completely lost in the NIR image 310. As shown in the last row of Figure 3B, the text visible in the red, green, and blue channels in the region 340 of the Jewel Scarabs book cover is totally missing from that same region 340 of the NIR image. This gradient loss may also be due to the different reflectance of the object in the scene to infrared and visible light. Though not illustrated, other structural inconsistencies may exists between pixel regions in the RGB and NIR images, for example resulting from differences in the highlights or shadows present in the respective image due to their different illumination sources.
• the content of the NIR image light may yield better contrast than the corresponding visible light image and carry richer levels of detail, which generates great potential for using the NIR image as a guidance image to enhance the quality of the RGB image.
• the denoising methods described herein employ the gradient scale map to describe the relationship between the gradient fields of the two images.
• Figure 4 is a flowchart illustrating an embodiment of a method for generating a gradient scale map and using the gradient scale map for de-noising.
• the gradient scale map may be generated to better handle the inherent discrepancy of structures between the visible light and NIR images.
• the gradient scale maps may be defined as follows:
• the gradient scale map s may represent a scale or ratio map between the gradient fields of the NIR and ground-truth images. It may have a few unique properties corresponding to structural discrepancies between VG and VI*.
• the process 400 begins at block 405 when image data is received for an RGB image IQ, which may be a noisy image, as well as image data for the corresponding NIR flash image G, which may have neglectable noise.
• an RGB image IQ which may be a noisy image
• image data for the corresponding NIR flash image G which may have neglectable noise.
• the process 400 then moves to block 410 to determine x- and y- direction gradient vectors for each pixel in both the NIR and RGB images. In some embodiments this may be accomplished by applying horizontal and vertical linear convolution filters to pixel blocks of the NIR and RGB, as described above, to obtain gradient vector values for a center pixel in each block.
• the gradient vector values for each pixel may be defined as a coordinate pair, e.g., (x, y), and the value of the "x" variable of the coordinate pair may indicate whether the gradient increases in the right or left directions, and the value of the "y" variable of the coordinate pair may indicate whether the gradient increases in an up or down direction, relative to the orientation of the image.
• the process 400 moves to block 420 in which the gradient scape map values are computed from the gradient vectors.
• the gradient scale map may be a ratio between the values in the NIR and RGB gradient vectors.
• the gradient scale map may include a positive value or a negative value for each pixel location. For example, a positive value may indicate that an edge is present with similar gradient directions in both the visible light image and the NIR image, while a negative value may indicate that an edge is present in both the visible light image and the NIR image, but that the direction of the local gradient vector in the NIR image is reversed relative to the local gradient vector in the visible light image.
• a value of zero may indicate that an edge is present in the NIR image which is not present in the visible light image, for example due to highlights and shadows resulting from the NIR flash, or that an edge is not present in the NIR image which is present in the visible light image, for example due to the different reflectance properties of a material in the images to visible and near-infrared light.
• the process 400 moves to block 425 to use the gradient scale map values to guide image de-noising of the visible light image 7 0 .
• the problem of obtaining a de-noised image I from I 0 using the gradient scale map 5 can be formulated as maximizing the following conditional probability.
• E(s, I) E x (s, I) + AE 2 (I) + ⁇ , (s).
• Eq. (3) contains three terms: E 1 (s, I) is the data term for 5 and /, representing the cost of using 5 to relate the estimated image / and the guidance image G; E 2 ) is the date term for / representing how much the estimated image deviates from the input noisy image I 0 ; and E 2 (s) is a regularization term for s, enforcing penalty on the smoothness of s.
• ⁇ controls the confidence on the noisy image I 0 , ranging from 1-10.
• ⁇ corresponds to smoothness of s. Its value may typically be set to [0.2-0.8] empirically.
• E x (s, I) X (p(s, - p x>i V i )+ p ⁇ s t - p y .V y I i )), (6)
• the threshold m may be used to avoid division by zero and in some embodiments is set to 0.004 empirically.
• p is the same robust function defined in Eq. (7).
• the data term may require that the restoration result does not deviate above a threshold amount from the input noisy image I Q especially for important and salient edge areas.
• the robust function may beneficially be used to reject noise from / 0 .
• the regularization term may be defined specially with anisotropic gradient tensors. It is based on the fact that the values in the gradient scale map 5 are similar locally only in certain directions. For instance, 5 values should change smoothly or be constant along an edge more than the values across the edge. Uniformly smoothing 5 in all directions may either create discontinuity along a continuous edge or blur a sharp boundary.
• the anisotropic tensor scheme is able to preserve sharp edges according to gradie sor may be expressed as where VG ⁇ is a vector perpendicular to Gi. 1 is an identity matrix and scalar that controls the isotropic smoothness.
• VG ⁇ is a vector perpendicular to Gi. 1 is an identity matrix and scalar that controls the isotropic smoothness.
• the anisotropic tensor reduces to be isotropic.
• the anisotropic tensor can be decomposed to where the two orthogonal eigenvectors of D(VGj) are
• the objective function in Eq. (15) is non-convex due to the spars ity terms. Joint representation for 5 and / in optimization further complicates the problem. Naively solving it by simple gradient decent cannot guarantee optimality and may lead to very slow convergence even for a local minimum.
• the illustrated embodiment employs a weighted least square scheme, which enables the original problem- the objective function in Eq. (15)- to be solved using a few corresponding linear systems without losing the original properties.
• ⁇ may be a small number to avoid division by zero.
• This form enables splitting the robust function into two parts where ⁇ ( ⁇ ) can be regarded as a weight for x 2 .
• ⁇ ( ⁇ ) and x 2 are updated alternatively during optimization, because each of ⁇ ( ⁇ ) and x 2 can be utilized together with other terms to form simpler representation, profiting optimization.
• C X and C Y are discrete backward difference matrices that are used to compute image gradients in the x— and — directions.
• P x , P y , A X , A y and B are all diagonal matrices, whose i— th diagonal elements are defined as
• a x A y and B account for the re-weighting process and are typically computed using estimates from previous iteration; P ⁇ and P y are normalization terms from the guided input G.
• the process 400 moves to block 430, where the de-noised image is used to generate an additional gradient scale map, such that the process 400 advantageously employs an alternating minimization process to solve Eq. (18).
• This initialization also makes the starting V/ same as VG with many details. Then at iteration t + 1, the following two subproblems may be solved sequentially.
• (21) ⁇ c x T P x A x ' + ) + C y T P y A y ,+ ) )s + B ,+ ,) I 0 ,
• a +U and A ⁇ +U) are calculated using s ⁇ + ⁇ and / (t) .
• B (t+U) depends on / (t) .
• the two steps repeat until, at decision block 435, the process 400 determines that s and I do not change above a threshold amount. In some embodiments, three to six iterations may be enough to generate visually pleasing results.
• the iterative process 400 disclosed herein provides several advantages such as preservation of colors, contrast, and image structures in the final de-noised image and avoidance of artifacts along edges. Specifically, compared with the result of existing dark flash de-noising methods, the result of the denoising processes described herein has better overall contrast and is able to avoid staircase artifacts along noisy edges. Both methods enforce color similarity with the original RGB image and gradient similarity with the guided image.
• existing dark flash methods use a heuristic sparse gradient constraint, which inherently is not well adapted to address the differences between the original RGB image and the NIR flash image.
• the method 400 as described herein uses the gradient scale map to well model the discrepancies and similarities between the image pair. Accordingly, the de-noising method 400 is able to constrain the gradients of the noisy RGB image to be close to those of the ground-truth image of the target image scene with no noise. In addition, the anisotropic tensor scheme used in the process 400 helps to reduce staircase artifacts caused by sparse gradient constraint. De-noising Process Images Overview
• Figure 5 illustrates the various images and stages of the denoising process 400 described above.
• the original RGB image 500 has a high noise level, and some gradients of the NIR flash image 505 are reverted and weaker compared with the gradients of the noisy RGB image 500.
• Figure 5 also illustrates a gradient scale map 510, an iteration of an estimated de-noised image 515, and a final de-noised image 520.
• DSP digital signal processor
• ASIC application specific integrated circuit
• FPGA field programmable gate array
• a general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine.
• a processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
• a software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of non-transitory storage medium known in the art.
• An exemplary computer- readable storage medium is coupled to the processor such the processor can read information from, and write information to, the computer-readable storage medium.
• the storage medium may be integral to the processor.
• the processor and the storage medium may reside in an ASIC.
• the ASIC may reside in a user terminal, camera, or other device.
• the processor and the storage medium may reside as discrete components in a user terminal, camera, or other device.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Image Processing (AREA)
• Transforming Light Signals Into Electric Signals (AREA)
• Studio Devices (AREA)

Priority Applications (4)

Applications Claiming Priority (4)

Publications (1)

Family

ID=50640037

Family Applications (1)

Country Status (6)

Cited By (5)

Families Citing this family (43)

Citations (1)

Family Cites Families (11)

• 2014
  • 2014-01-09 US US14/151,589 patent/US9171355B2/en active Active
  • 2014-04-07 EP EP14721712.9A patent/EP2984621B1/en active Active
  • 2014-04-07 JP JP2016507589A patent/JP6416208B2/ja active Active
  • 2014-04-07 CN CN201480020052.3A patent/CN105103189B/zh not_active Expired - Fee Related
  • 2014-04-07 KR KR1020157032226A patent/KR102190717B1/ko active Active
  • 2014-04-07 WO PCT/US2014/033197 patent/WO2014168880A1/en not_active Ceased

Patent Citations (1)

Non-Patent Citations (7)

Also Published As

Similar Documents

Legal Events