WO2016183744A1

WO2016183744A1 - Color correction system and method

Info

Publication number: WO2016183744A1
Application number: PCT/CN2015/079094
Authority: WO
Inventors: Wei Chen; Zisheng Cao
Original assignee: SZ DJI Technology Co., Ltd.
Priority date: 2015-05-15
Filing date: 2015-05-15
Publication date: 2016-11-24
Also published as: US20200228681A1; US20170318194A1; EP3202131A4; US20180262650A1; CN106471567A; US20170041507A1; CN106471567B; EP3202131A1; US9742960B2; US9998632B2; CN109963133B; US10244146B2; CN109963133A; US20190215418A1; US10560607B2

Abstract

A system and method for calibrating a digital imaging device for color correction is disclosed. The method comprises obtaining an input color value and a reference color value for each of a plurality of color references, as well as a noise evaluation image having a color noise for evaluating noise reduction, the input color values and reference color values being in a non-linear color space. A plurality of color correction parameters are determined as optimized based on evaluating a fitness function in the non-linear color space. The non-linear color space can be a CIE L*a*b*color space. The fitness function can include a color correction error and a noise amplification metric so as to reduce noise amplification during color correction.

Description

无标题

A portion of the disclosure of this patent document contains material which is subject to (copyright or mask work) protection. The (copyright or mask work) owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all (copyright or mask work) rights whatsoever.

COLOR CORRECTION SYSTEM AND METHOD

FIELD

The disclosed embodiments relate generally to digital image processing and more particularly, but not exclusively, to systems and methods for color correction of digital images.

BACKGROUND

Because digital imaging devices acquire colors differently from the way that human eyes perceive color, images acquired by digital imaging devices typically benefit from color correction. However, the color correction process may be prone to introducing and/or amplifying different types of noise. This is the general area that embodiments of the invention are intended to address.

SUMMARY

Described herein are systems and methods that can calibrate a digital imaging device for color correction is disclosed. The method comprises obtaining an input color value and a reference color value for each of a plurality of color references, as well as a noise evaluation image having a color noise for evaluating noise reduction, the input color values and reference color values being in a non-linear color space. A plurality of color correction parameters are determined as optimized based on evaluating a fitness function in the non-linear color space. The non-linear color space can be a CIE L*a*b*color space. The fitness function can include a color correction error and a noise amplification metric so as to reduce noise amplification during color correction

In accordance with a first aspect herein, there is disclosed a method for calibrating a digital imaging device for color correction, the method comprising obtaining an input color value and a reference color value for each of a plurality of color references and a noise evaluation image having a color noise for evaluating noise reduction, the input color values and reference color values being in a non-linear color space； and determining a plurality of color correction parameters that are optimized based on evaluating a fitness function in the non-linear color space.

In some embodiments, the non-linear color space is a CIE L*a*b*color space.

In some embodiments, determining the plurality of color correction parameters comprises adjusting the color correction parameters based on the input color values, and reference color values, and the noise evaluation image.

In some embodiments, the fitness function comprises a color correction error and a noise amplification metric.

In some embodiments, adjusting the plurality of color correction parameters comprises determining the color correction error by color correcting the input color values and comparing the corrected input color values with the reference color values.

In some embodiments, adjusting the plurality of color correction parameters comprises determining the noise amplification metric by color correcting the noise evaluation image using the parameters and comparing the corrected noise evaluation image with the pre-correction noise evaluation image.

In some embodiments, determining the noise amplification metric comprises determining the noise amplification metric using a peak signal-to-noise ratio (PSNR) .

In some embodiments, determining the noise amplification metric using a PSNR comprises finding a PSNR difference that is a difference between a PSNR for the pre-correction noise evaluation image and a PSNR for the corrected noise evaluation image.

In some embodiments, determining the noise amplification metric using a PSNR comprises determining a downsampled PSNR difference.

In some embodiments, determining the downsampled PSNR difference comprises downsampling the pre-correction noise evaluation image to obtain a downsampled pre-correction noise evaluation image； downsampling the corrected noise evaluation image to obtain a downsampled corrected noise evaluation image； and finding the downsampled PSNR difference as a difference between a PSNR for the downsampled pre-correction noise evaluation image and a PSNR for the downsampled corrected noise evaluation image.

In some embodiments, determining the noise amplification metric comprises determining a weighted average of the PSNR difference and at least one downsampled PSNR difference.

In some embodiments, determining the noise amplification metric using a PSNR comprises determining a plurality of successively downsampled PSNR differences and determining the noise amplification metric as a weighted average of the PSNR difference and the plurality of successively downsampled PSNR differences.

In some embodiments, determining the noise amplification metric comprises weighing each of the PSNR difference and the plurality of successively downsampled PSNR difference with a respective weight, and at least one of the weights is not zero.

In some embodiments, determining the noise amplification metric comprises determining the noise amplification metric based on color variances.

In some embodiments, adjusting the plurality of color correction parameters comprises determining the noise amplification metric by converting the color corrected noise evaluation image into a YUV color space.

In some embodiments, the noise evaluation image comprises a plurality of color patches, and wherein the noise amplification metric is determined as a weighted sum of noise amplification levels of each of the color patches.

In some embodiments, each color patch is weighted according to an average sensitivity of human perception to the color patch.

In some embodiments, adjusting the color correction parameters comprises optimizing the fitness function by minimizing a weighted sum of the color correction error and the noise amplification metric.

In some embodiments, adjusting the color correction parameters comprises adjusting the parameters using a genetic process. Some embodiments comprise further adjusting the color correction parameters using a direct search method.

In some embodiments, obtaining the input color values of the color calibration images comprises imaging one or more color references comprising a plurality of color patches.

In some embodiments, the color correction parameters are in the form of a matrix.

In some embodiments, the color correction parameters are in the form of a look-up table.

In some embodiments, adjusting the color correction parameters comprises look-up operations and interpolation operations in the look-up table.

In some embodiments, the interpolation operations comprise a Shepard interpolation.

In some embodiments, the noise evaluation image is a virtual noise evaluation image.

In some embodiments, the virtual noise evaluation image comprises noise added to a virtual noise-free image.

In another aspect herein, there is disclosed a color correction apparatus configured for calibration for color correction based upon images of a plurality of color references each having a reference color, comprising a memory for storing a plurality of color correction parameters； and a processor for performing color correction of a digital image, wherein the processor is configured to obtain an input color value and a reference color value for each of the color references and a noise evaluation image having a color noise for evaluating noise reduction, the input color values and reference color values being in a non-linear color space, and determine a plurality of color correction parameters that are optimized based on evaluating a fitness function in the non-linear color space.

In some embodiments, the non-linear color space is a CIE L*a*b*color space.

In some embodiments, the processor is configured to determine the plurality of color correction parameters by adjusting the color correction parameters based on the input color values, and reference color values, and the noise evaluation image.

In some embodiments, the processor is configured to adjust the plurality of color correction parameters by color correcting the input color values and comparing the corrected input color values with the reference color values.

In some embodiments, the processor is configured to adjust the plurality of color correction parameters by color correcting the noise evaluation image using the parameters and comparing the corrected noise evaluation image with the pre-correction noise evaluation image.

In some embodiments, the processor is configured to determine the noise amplification metric using a peak signal-to-noise ratio (PSNR) .

In some embodiments, the processor is configured to determine the noise amplification metric using a PSNR by finding a PSNR difference that is a difference between a PSNR for the pre-correction noise evaluation image and a PSNR for the corrected noise evaluation image.

In some embodiments, the processor is configured to determine the noise amplification metric using a PSNR by determining a downsampled PSNR difference.

In some embodiments, the processor is configured to determine the downsampled PSNR difference by downsampling the pre-correction noise evaluation image to obtain a downsampled pre-correction noise evaluation image； downsampling the corrected noise evaluation image to obtain a downsampled corrected noise evaluation image； and finding the downsampled PSNR difference as a difference between a PSNR for the downsampled pre-correction noise evaluation image and a PSNR for the downsampled corrected noise evaluation image.

In some embodiments, the processor is configured to determine the noise amplification metric by determining a weighted average of the PSNR difference and at least one downsampled PSNR difference.

In some embodiments, the processor is configured to determine a plurality of successively downsampled PSNR differences and determine the noise amplification metric as a weighted average of the PSNR difference and the plurality of successively downsampled PSNR differences.

In some embodiments, the processor is configured to determine the noise amplification metric by weighing each of the PSNR difference and the plurality of successively downsampled PSNR difference with a respective weight, and at least one of the weights is not zero.

In some embodiments, the processor is configured to determine the noise amplification metric based on color variances.

In some embodiments, the processor is configured to determine the noise amplification metric by converting the color corrected noise evaluation image into a YUV color space.

In some embodiments, the processor is configured to optimizing the fitness function by minimizing a weighted sum of the color correction error and the noise amplification metric.

In some embodiments, the processor is configured to adjust the color correction parameters using a genetic process.

In some embodiments, the processor is configured to further adjust the color correction parameters using a direct search method.

In some embodiments, the processor is configured to obtain the input color values of the color calibration images by imaging one or more color references comprising a plurality of color patches.

In some embodiments, the processor is configured to adjust the color correction parameters by look-up operations and interpolation operations in the look-up table.

In some embodiments, the color correction apparatus is mounted aboard a mobile platform.

In some embodiments, the mobile platform is an unmanned aerial vehicle (UAV) .

In another aspect herein, there is disclosed a digital imaging device, comprising an image sensor for imaging a plurality of color references； and a processor for performing color correction of a digital image, wherein the processor is configured to obtain an input color value and a reference color value for each of the color references and a noise evaluation image having a color noise for evaluating noise reduction, the input color values and reference color values being in a non-linear color space, and determine a plurality of color correction parameters that are optimized based on evaluating a fitness function in the non-linear color space.

In some embodiments, the non-linear color space is a CIE L*a*b*color space.

In some embodiments, said interpolation operations comprise a Shepard interpolation.

In some embodiments, the mobile platform is an unmanned aerial vehicle (UAV) .

In another aspect herein, there is disclosed a non-transitory readable medium storing instructions for calibrating a digital imaging device for color correction, wherein the instructions comprise instructions for obtaining an input color value and a reference color value for each of a plurality of color references and a noise evaluation image having a color noise for evaluating noise reduction, the input color values and reference color values being in a non-linear color space； and determining a plurality of color correction parameters that are optimized based on evaluating a fitness function in the non-linear color space.

In some embodiments, the non-linear color space is a CIE L*a*b*color space.

In some embodiments, said determining the plurality of color correction parameters comprises adjusting the color correction parameters based on the input color values, and reference color values, and the noise evaluation image.

In some embodiments, said adjusting the plurality of color correction parameters comprises determining the color correction error by color correcting the input color values and comparing the corrected input color values with the reference color values.

In some embodiments, said adjusting the plurality of color correction parameters comprises determining the noise amplification metric by color correcting the noise evaluation image using the parameters and comparing the corrected noise evaluation image with the pre-correction noise evaluation image.

In some embodiments, said determining the noise amplification metric comprises determining the noise amplification metric using a peak signal-to-noise ratio (PSNR) .

In some embodiments, said determining the noise amplification metric using a PSNR comprises finding a PSNR difference that is a difference between a PSNR for the pre-correction noise evaluation image and a PSNR for the corrected noise evaluation image.

In some embodiments, said determining the noise amplification metric using a PSNR comprises determining a downsampled PSNR difference.

In some embodiments, said determining the downsampled PSNR difference comprises downsampling the pre-correction noise evaluation image to obtain a downsampled pre-correction noise evaluation image； downsampling the corrected noise evaluation image to obtain a downsampled corrected noise evaluation image； and finding the downsampled PSNR difference as a difference between a PSNR for the downsampled pre-correction noise evaluation image and a PSNR for the downsampled corrected noise evaluation image.

In some embodiments, said determining the noise amplification metric comprises determining a weighted average of the PSNR difference and at least one downsampled PSNR difference.

In some embodiments, said determining the noise amplification metric using a PSNR comprises determining a plurality of successively downsampled PSNR differences and determining the noise amplification metric as a weighted average of the PSNR difference and the plurality of successively downsampled PSNR differences.

In some embodiments, said determining the noise amplification metric comprises weighing each of the PSNR difference and the plurality of successively downsampled PSNR difference with a respective weight, and at least one of the weights is not zero.

In some embodiments, said determining the noise amplification metric comprises determining the noise amplification metric based on color variances.

In some embodiments, said adjusting the plurality of color correction parameters comprises determining the noise amplification metric by converting the color corrected noise evaluation image into a YUV color space.

In some embodiments, said noise evaluation image comprises a plurality of color patches, and wherein the noise amplification metric is determined as a weighted sum of noise amplification levels of each of the color patches.

In some embodiments, said adjusting the color correction parameters comprises optimizing the fitness function by minimizing a weighted sum of the color correction error and the noise amplification metric.

In some embodiments, said adjusting the color correction parameters comprises adjusting the parameters using a genetic process.

Some embodiments comprise further adjusting the color correction parameters using a direct search method.

In some embodiments, said obtaining the input color values of the color calibration images comprises imaging one or more color references comprising a plurality of color patches.

In some embodiments, said adjusting the color correction parameters comprises look-up operations and interpolation operations in the look-up table.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is an exemplary top-level block diagram illustrating an embodiment of a color correction apparatus for color-correcting a digital image.

Fig. 2 is an exemplary diagram illustrating an embodiment of an imaging system including the color correction apparatus of Fig. 1, wherein the imaging system is shown imaging a color reference using an image sensor.

Fig. 3 is an exemplary diagram illustrating an alternative embodiment of the imaging system of Fig. 2, wherein the color correction apparatus is shown as acquiring various color values for calibration of color correction parameters.

Fig. 4 is exemplary top-level flow chart illustrating an embodiment of a method for calibrating a digital imaging device.

Fig. 5 is exemplary flow chart illustrating an alternative embodiment of the method of Fig. 4, wherein color correction parameters are optimized for calibrating a digital imaging device.

Fig. 6 is exemplary flow chart illustrating an alternative embodiment of the method of Fig. 5, wherein the method includes a two-step optimization method.

Fig. 7 is exemplary flow chart illustrating another alternative embodiment of the method of Fig. 5, wherein the method includes a genetic process.

Fig. 8 is an exemplary diagram illustrating an embodiment of the method of Fig. 5, wherein the method includes sampling at different spatial frequencies to determine a noise amplification metric.

Fig. 9 is an exemplary diagram illustrating an embodiment of the method of Fig. 8 with spatial downsampling.

Fig. 10 is an exemplary flow chart illustrating an embodiment of the method of Fig. 5, wherein the method includes sampling at different spatial frequencies to determine a noise amplification metric.

Fig. 11 is an exemplary diagram illustrating an embodiment of an imaging system installed on an unmanned aerial vehicle (UAV) .

Fig. 12 is an exemplary diagram illustrating a chrominance diagram showing a color error of a first experiment testing the efficacy of optimizing color correction parameters with noise regulation.

Fig. 13 is an exemplary diagram illustrating noise values of the experiment of Fig. 9 testing the efficacy of optimizing color correction parameters with noise regulation.

Fig. 14 is an exemplary diagram illustrating a chrominance diagram showing a color error of a second experiment testing the efficacy of optimizing color correction parameters without noise regulation.

Fig. 15 is an exemplary diagram illustrating noise values of the experiment of Fig. 11 testing the efficacy of optimizing color correction parameters without noise regulation.

Fig. 16 is an exemplary diagram illustrating a two-step method for optimizing color correction parameters with noise regulation.

It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are generally represented by like reference numerals for illustrative purposes throughout the figures. It also should be noted that the figures are only intended to facilitate the description of the preferred embodiments. The figures do not illustrate every aspect of the described embodiments and do not limit the scope of the present disclosure.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The invention is illustrated, by way of example and not by way of limitation, in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to “an” or “one” or “some” embodiment (s) in this disclosure are not necessarily to the same embodiment, and such references mean at least one.

Color correction is a process that transforms signals acquired by photosensors of digital imaging devices into colors that look realistic. Color correction is a transformation defined by a set of color correction parameters. These parameters are typically calibrated for each individual digital imaging device to find customized parameter values that accurately reflect the color response characteristics of the individual device.

The calibration of color correction parameters entails an optimization process in which colors values acquired by the imaging device are compared with known reference color values. Typically, the goal during the optimization process is to minimize a difference between the acquired colors post-color-correction and the known reference values. A drawback of this approach, however, is that accounting for color correction accuracy alone can often result in parameters that excessively amplify noise. Image noise can include color and brightness variations in an image. These variations are not features of an original object imaged but, instead, are attributable to artifacts introduced by the acquisition and processing of the image. Sources of noise include, for example, quantum exposure noise, dark current noise, thermal noise, readout noise, and others. Since image noise is inversely proportional to the size of the imaging device, the problem of noise is especially acute for smaller imaging devices. When image acquisition is performed aboard mobile platforms such as unmanned aerial vehicles (UAVs) , the image noise problem is especially acute both because of the smaller cameras used on the mobile platforms and because of noise introduced by movement of the mobile platforms. In view of the foregoing, there is a need for improved color correction systems and methods that increase color correction accuracy while limiting noise amplification.

The present disclosure sets forth systems and methods for color correction of a digital image which overcome shortcomings of existing color correction techniques by increasing color correction accuracy while limiting noise amplification. Based on color reference images, color correction parameters are calibrated to increase color correction accuracy while limiting noise amplification. The calibration can be performed in the CIE L*a*b*color space to more closely reflect human perception of distances between colors. The calibration can be performed with reference to a virtual noisy image that can be sampled at different spatial frequencies. At each spatial frequency, a peak signal-to-noise ratio (PSNR) can be used to evaluate the amount of noise introduced by color correction. The color correction parameters can be optimized by using a genetic process. A two-step parameter optimization method can be used that avoid the optimization process being trapped in local optima. The present systems and methods advantageously are suitable for use, for example, by unmanned aerial vehicles (UAVs) and other mobile platforms.

Turning now to Fig. 1, an exemplary color correction apparatus 100 is shown as including a processor 110 and a memory 120. The processor 110 can comprise any type of processing system. Exemplary processors 110 can include, without limitation, one or more general purpose microprocessors (for example, single or multi-core processors) , application-specific integrated circuits, application-specific instruction-set processors, graphics processing units, physics processing units, digital signal processing units, coprocessors, network processing units, audio processing units, encryption processing units, and the like. In certain embodiments, the processor 110 can include an image processing engine or media processing unit, which can include specialized hardware for enhancing the speed and efficiency of certain operations for image capture, image filtering, and image processing. Such operations include, for example, Bayer transformations, demosaicing operations, white balancing operations, color correction operations, noise reduction operations, and/or image sharpening/softening operations. In certain embodiments, the processor 110 can include specialized hardware and/or software for performing various color correction parameter calibration functions and operations described herein. Specialized hardware can include, but are not limited to, specialized parallel processors, caches, high speed buses, and the like.

The memory 120 can comprise any type of memory and can be, for example, a random access memory (RAM) , a static RAM, a dynamic RAM, a read-only memory (ROM) , a programmable ROM, an erasable programmable ROM, an electrically erasable programmable ROM, a flash memory, a secure digital (SD) card, and the like. Preferably, the memory 120 has a storage capacity that accommodates the needs of the color correction parameter calibration functions and operations described herein. The memory 120 can have any commercially-available memory capacity suitable for use in image processing applications and preferably has a storage capacity of at least 512 Megabytes, 1 Gigabyte, 2 Gigabytes, 4 Gigabytes, 16 Gigabytes, 32 Gigabytes, 64 Gigabytes, or more. In some embodiments, the memory 120 can be a non-transitory storage medium that can store instructions for performing any of the processes described herein.

The color correction apparatus 100 can further include any hardware and/or software desired for performing the color correction parameter calibration functions and operations described herein. For example, the color correction apparatus 100 can include one or more input/output interfaces (not shown) . Exemplary interfaces include, but are not limited to, universal serial bus (USB) , digital visual interface (DVI) , display port, serial ATA (SATA) , IEEE 1394 interface (also known as FireWire) , serial, video graphics array (VGA) , super video graphics array (SVGA) , small computer system interface (SCSI) , high-definition multimedia interface (HDMI) , audio ports, and/or proprietary input/output interfaces. As another example, the color correction apparatus 100 can include one or more input/output devices (not shown) , for example, buttons, a keyboard, keypad, trackball, displays, and/or a monitor. As yet another example, the color correction apparatus 100 can include hardware for communication between components of the color correction apparatus 100 (for example, between the processor 110 and the memory 120) .

Turning now to Fig. 2, an exemplary embodiment of an imaging system 200 is shown as including a color correction apparatus 100, an image sensor 130, and a color filter 140. The color correction apparatus 100 can be provided in the manner discussed in more detail above with reference to Fig. 1. The memory 120 of the color correction apparatus 100 is shown as storing color correction parameters 125, noise generation parameters 126, pre-correction and post-correction image data 127, and intermediate values 128 produced during various color correction parameter calibration functions and operations described herein. The image sensor 130 can perform the function of sensing light and converting the sensed light into electrical signals that can be rendered as an image. Various image sensors 130 are suitable for use with the disclosed systems and methods, including, but not limited to, image sensors 130 used in commercially-available cameras and camcorders. Suitable image sensors 130 can include analog image sensors (for example, video camera tubes) and/or digital image sensors (for example, charge-coupled device (CCD) , complementary metal-oxide-semiconductor (CMOS) , N-type metal-oxide-semiconductor (NMOS) image sensors, and hybrids/variants thereof) . Digital image sensors can include, for example, a two-dimensional array of photosensor elements that can each capture one pixel of image information. The resolution of the image sensor 130 can be determined by the number of photosensor elements. The image sensor 130 can support any commercially-available image resolution and preferably has a resolution of at least 0.1 Megapixels, 0.5 Megapixels, 1 Megapixel, 2 Megapixels, 5 Megapixels, 10 Megapixels, or an even greater number of pixels. The image sensor 130 can have specialty functions for use in various applications such as thermography, creation of multi-spectral images, infrared detection, gamma detection, x-ray detection, and the like. The image sensor 130 can include, for example, an electro-optical sensor, a thermal/infrared sensor, a color or monochrome sensor, a multi-spectral imaging sensor, a spectrophotometer, a spectrometer, a thermometer, and/or an illuminometer.

The color filter 140 is shown in Fig. 2 as separating and/or filtering incoming light based on color and directing the light onto the appropriate photosensor elements of the image sensor 130. For example, the color filter 140 can include a color filter array that passes red, green, or blue light to selected pixel sensors to form a color mosaic (not shown) . The layout of different colors on the color mosaic can be arranged in any convenient manner, including a Bayer pattern. Once a color mosaic is formed, a color value of each pixel can be interpolated using any of various demosaicing methods that interpolate missing color values at each pixel using color values of adjacent pixels. As an alternative to filtering and demosaicing, the image sensor 130 can include an array of layered pixel photosensor elements that separates light of different wavelengths based on the properties of the photosensor elements. In either case, an image can be acquired by the image sensor 130 as intensity values in each of a plurality of color channels at each pixel.

The imaging system 200 is further shown in Fig. 2 as acquiring an image of a color reference 150 to perform calibration of color correction parameters 125. The color reference 150 preferably has a known reference color value C_ref that is known or that can be otherwise determined in advance, making the color reference 150 suitable for use as a color standard. Stated somewhat differently, the reference color value C_ref is a property of the color reference 150 that is independent of how the color reference 150 is imaged. The reference color value C_ref can be designated based on an average human perception of the color reference 150. The reference color value C_ref can thus serve as an objective measure how a color imaged by the image sensor 130 can be corrected so as to match the average human perception.

The color reference 150 is preferably, but not necessarily, homogeneous in color. Flatness of the color reference 150 is preferable, though not essential, to avoid variations attributable to differential light scattering. The optical properties of the color reference 150 need not be ideal for purposes of performing color correction, so long as the optical properties do not interfere with imaging the color reference 150. The color reference 150 can be made of one or more of a variety of materials such as plastic, paper, metal, wood, foam, composites thereof, and other materials. Furthermore, the color, reflectance, and/or other optical properties of the color reference 150 can advantageously be calibrated as desired using an appropriate paint or other coating. In some embodiments, the color reference 150 can advantageously include multiple color patches 151, each of which has a different reference color value C_ref. This embodiment enables multiple color references 150 to be imaged at the same time, reducing the number of image capture operations for color correction. This embodiment is particularly suitable when a large number of color references 150 are to be imaged in order to calibrate color correction parameters 125 with greater accuracy. Commercially available color references 150 include, for example, MacBeth ColorChecker, MacBeth ColorChecker SG, and the like.

Although images acquired by the image sensor 130 are described above in an RGB (red, green, and blue) color space for illustrative purposes only, the images can be acquired in other color spaces, as well. The color space in which images are acquired depends generally on the properties of the image sensor 130 and any color filters 140. Furthermore, the color space in which an image is acquired need not be three-dimensional but can have any number of dimensions as desired to capture the spectral composition of the image. The number of dimensions can depend on the number of color channels of the image sensor 130. The color space of an acquired image can be one-dimensional, two-dimensional, three-dimensional, four-dimensional, five-dimensional, or more.

Once acquired by the image sensor 130, an image can be converted between color spaces as desired for processing and/or calibration. For example, a conversion from a sRGB color space with coordinates (R_sRGB, G_sRGB, B_sRGB) to a CIE 1931 XYZ color space with coordinates (X, Y, Z) entails a linear conversion, which can be represented by the following three-dimensional matrix:

In some embodiments, it can be desirable to express the color of an image in a non-linear color space. One suitable non-linear color space for imaging applications is a CIE L*a*b*color space (for example, a CIE 1976 L*a*b*color space) as defined by the International Commission on Illumination. The color of an image in a CIE L*a*b*color space can be computed from the colors of the image in a CIE 1931 XYZ color space using the following non-linear transformation:

where

In the above equations (2) - (5) , X_n, Y_n, and Z_n are the CIE XYZ values of the color at a reference white point. The CIE L*a*b*color space is designed to mimic the color response of human perception. The non-linearity of the transformation from the CIE XYZ color space to the CIE L*a*b*color space reflects the nonlinearity of human perception. Representing a color in the CIE L*a*b*color space has the advantage that the CIE L*a*b*color space is perceptually uniform to human beings, meaning that a change of a given amount in a color value will produce a proportional change of visual significance. According, calibration of color correction parameters 125 can advantageously be performed after converting input and reference colors into a CIE L*a*b*color space representation.

In some embodiments, it can be desirable to express the color of an image in a YUV color space (for example, a Y’ UV color space) . The YUV color space is represented by one luminance component Y representing image brightness and two chrominance components U and V representing image color. A conversion from a RGB color space with coordinates (R, G, B) to a YUV color space with coordinates (Y, U, V) entails a linear conversion, which can be represented by the following three-dimensional matrix:

Although conversions between specific color spaces are shown and described for illustrative purposes only, an image can be converted from any first predetermined color space to any other second predetermined color space as desired.

Turning now to Fig. 3, an alternative embodiment of the imaging system 200 is shown. The imaging system 200 of Fig. 3 includes a color correction apparatus 100, which is shown as obtaining several inputs for calibration of color correction parameters 125. Without limitation, an image sensor 130 is shown as acquiring an image of a color reference 150. The image is then passed to the color correction apparatus 100, which can obtain an input color value C_in of the image. The input color value C_in represents a pre-color-corrected value that reflects the image acquisition properties of the image sensor 130, filtering properties of an image filter 140, as well as any other optical properties of the imaging system 200. In one embodiment, the input color value C_in can be transformed from the color space of the color reference image to a non-linear color space—for example, a CIE L*a*b*color space. The transformation can be performed, for example, by first using a linear transformation from a RGB color space to an intermediate CIE XYZ color using Equation (1) shown above. The color values in the intermediate CIE XYZ color space can be non-linearly transformed to a CIE L*a*b*color space as shown above in Equations (2) - (5) . Such transformations can be performed on a processor 110 (shown in Fig. 1) of the color correction apparatus 100.

Similarly, the color correction apparatus 100 can obtain a reference color value C_ref that corresponds to the input color value C_in for color reference 150. If desired, the reference color value C_ref can be transformed into a non-linear color space—for example, the CIE L*a*b*color space. In some embodiments, the reference color value C_ref advantageously can be directly inputted into the color correction apparatus 100 in the CIE L*a*b*color space, thereby making the transformation step unnecessary.

In Fig. 3, the color correction apparatus 100 is further shown as obtaining noise evaluation color values C_noise from a noise evaluation image 160. The noise evaluation image 160 can be any image containing noise. As color correction tends to amplify noise, the noise evaluation image 160 can be used to calibrate color correction parameters 125 in order to limit noise amplification. Stated somewhat differently, the noise evaluation image 160 can used to evaluate now noise is amplified with a given set of color correction parameters 125 (shown in Fig. 2) , and thereby select a set of color correction parameters 125 with reduced noise amplification. In one embodiment, the noise evaluation color values C_noise can be transformed into the YUV color space, as further described below with reference to Fig 7. The transformation can be performed, for example, using the linear transformation from the RGB color space to the YUV color space shown above in Equation (6) . This transformation can be performed using the processor 110 of the color correction apparatus 100.

In one embodiment, the noise evaluation image 160 can be an image acquired by the image sensor 130 with or without filtering through the color filter 140. In this embodiment, the noise evaluation image 160 is preferably an image of the color reference 150. Imaging the color reference 150 advantageous allows the simultaneous determination of the input color values C_in and the noise evaluation color values C_noise.

Alternatively and/or additionally, the noise evaluation image 160 can be a virtual noise evaluation image 160A. The virtual noise evaluation image 160A can be generated by the color correction apparatus 100 using a pre-determined set of noise generation parameters 126 (shown in Fig. 2) . The noise generation parameters 126 can, for example, reflect the distribution of the noise that is generated virtually (for example, Poisson or Gaussian noise) . The specific noise generation parameters 126 can reflect the types of noise that the imaging system 200 can be expected to encounter in usage. A virtual noise evaluation image 160A can be used because the evaluation of noise amplification does not require information about the color of an underlying object that is imaged. Instead, an arbitrary image containing noise can be evaluated for how the noise of that image would be amplified under a given set of color correction parameters 125. For example, the noise evaluation color values C_noise of the virtual noise evaluation image 160A can be represented as follows:

C_noise＝C_{noise_free}+n Equation (7)

where C_{noise_free} represents the color of the virtual noise evaluation image 160A before noise is added, and n represents the noise added.

Once the inputs for color correction parameter calibration (for example, input color values C_in, reference color values C_ref, and noise evaluation color values C_noise) are obtained by the color correction apparatus 100, these inputs can be stored for later use by the color correction apparatus 100 (for example, in a memory 120 as shown in Fig. 1) . For example, the inputs for color correction parameter calibration can be obtained as part of an initialization process for a new imaging device 200 prior to usage. The inputs for color correction parameter calibration can be stored in the memory 120 and called upon periodically to re-calibrate the color correction parameters 125 as desired (for example, as image response characteristics of the imaging device 200 change after wear and tear) . The inputs for color correction parameter calibration can be, but do not need to be, re-obtained for each new color correction parameter calibration.

Turning now to Fig. 4, an exemplary top-level method 400 of calibrating color correction parameters 125 is shown. The method 400 advantageously can be applied to calibrating the color correction parameters 125 for a digital imaging device 200 (shown in Figs. 2 and 3) . At 401, input color values C_in and reference color values C_ref are obtained for each of a plurality of color references 150 (shown in Figs. 2 and 3) . Preferably, the input color values C_in and reference color values C_ref are obtained or transformed into a non-linear color space—for example, a CIE L*a*b*color space—as described above with reference to Fig. 3. Additionally, a noise evaluation image 160 having a color noise for evaluating noise reduction is obtained.

At 402, a plurality of color correction parameters 125 are adjusted so as to optimize a fitness function J. In some embodiments, the fitness function J can comprise a color correction error e_color and/or a noise amplification metric D_noise based on the input color values C_in, the reference color values C_ref , and the noise evaluation image 160. An exemplary embodiment of the adjusting is described in more detail below with respect to Fig. 5.

Turning now to Fig. 5, an exemplary method 500 of calibrating color correction parameters 125 (shown in Fig. 2) for a digital imaging device 200 (shown in Figs. 2 and 3) is shown. At 501, input color (or pre-correction) values C_in for a color references 150 are color corrected using the current values of the color correction parameters 125 to obtain post-correction input color values

This operation can be represented as:

In the above equation (8) , CC represents a color correction operation. The specific implementation of the color correction operation CC depends on the underlying form of the color correction parameters 125. In one embodiment, the color correction parameters 125 can take the form of a matrix having dimensions n x m, where m is dimensionality of the pre-correction color value and n is the dimensionality of the post-correction color value. In this embodiment, the color correction operation CC will take the form of a matrix multiplication that transforms an m-dimensional color value vector into an n-dimensional color value vector. Preferably, the pre-correction color value and the post-correction color value have the same dimensionality, in which case CC will take the form of a square matrix. Preferably, the pre- correction color value and the post-correction color value are each three-dimensional (for example, for color values in the RGB, CIE XYZ, CIE L*a*b*, and LUV color spaces) , in which case CC will take the form of a 3x3 matrix. An advantage of using a matrix is that a matrix can describe a color correction operation CC using only n x m correction parameters 125, allowing decreased memory usage. However, linear color correction using a matrix may be unsuitable for some applications.

In another embodiment, the color correction parameters 125 can take the form of a look-up table (LUT) indexed in m dimensions that contains ordered m-tuples (a₁, a₂, …, a_m) each mapping to an n-dimensional vector, where m is dimensionality of the pre-correction color value and n is the dimensionality of the post-correction color value. Preferably, the look-up table is three-dimensional, that is, indexed in three dimensions. An advantage of using a look-up table to implement the color correction parameters 125 is that a look-up table can account for a non-linear relationship between a pre-correction color value and a post-correction color value. Furthermore, since the entries in the look-up table are discrete, interpolation operations can be performed when pre-correction color values fall in between discrete entries. Such interpolation operations can include finding look-up table entries that have the closest distance (for example, Euclidian distance) to the pre-correction color value, and interpolating a corrected color value using the closest look-up table entries. For example, linear interpolations can be performed for one-dimensional look-up tables, and multi-linear interpolations can be performed for look-up tables in higher dimensions. In this embodiment, the color correction operation CC will take the form of a look-up operation in the look-up table, followed by an interpolation operation, if desired. The color correction parameters 125 can be implemented in multiple ways simultaneously； for example, a combination of a matrix and a look-up table can be used.

In one embodiment, a Shepard interpolation can be used to perform color correction where the color correction parameters 125 take the form of a look-up table (LUT) . In one embodiment, a color-corrected value for a given color p can be found as follows:

In the above equation (9) , i is an index over the different input color values C_in and their corresponding reference color values C_ref , c_i represents the ith value of the input color values C_in,

represents the ith value of the reference color values C_ref , (‖p-c_i‖) represents a distance (for example, a Euclidian distance) between the given color p and c_i, and w_i represents a weight of the ith input color value C_in.

At 502, the post-correction input color values

are compared with the reference color values C_ref, and the color correction error e_color is computed based on the comparison. For example, where the post-correction input color values

and reference color values C_ref are represented in a CIE L*a*b*color space, the color correction error e_color can be expressed as:

In the above equation (10) , C_{in_j} and

represent the jth component of the reference color values C_ref and the post-correction input color values

respectively. Stated somewhat differently, the color correction error e_color is the Euclidian distance between the post-correction input color values

and the reference color values C_ref in the color space in which the color values are represented. Where the color correction error e_color is to be determined over multiple color references 150 (or, equivalently, over multiple color patches 151 of a given color reference 150) , the color correction error e_color can be taken as a weighted and/or unweighted average over the color patches 151.

At 503, noise evaluation color values C_noise are color corrected using the current values of the color correction parameters 125 to obtain post-correction noise evaluation color values

This operation can be represented as:

In the above equation (11) , CC represents a color correction operation as described above with reference to 501. The specific color correction operation CC depends on the implementation of the color correction parameters 125 and, as described above with reference to 501, can take the form of a matrix or a look-up table with each form having respective advantages.

At 504, the post-correction noise evaluation color values

are compared with pre-correction noise evaluation color values C_noise, and the noise amplification metric D_noise is found based on the comparison. The noise amplification metric D_noise can be any measure of the distance between post-correction noise evaluation color values

and the pre-correction noise evaluation color values C_noise. That is, the greater the value of the noise amplification metric D_noise, the more noise is amplified after applying a color correction.

Where the noise amplification metric D_noise is to be determined over multiple color references 150 (or, equivalently, over multiple color patches 151 of a given color reference 150) , the noise amplification metric D_noise can be taken as a weighted and/or unweighted average over the color patches 151. In one embodiment, the noise amplification metric D_noise can be taken as a weighted average over the color patches 151.

In the above equation (12) , i is an index over the color patches 151, N is the total number of color patches 151, and ω_i is a non-negative weight for color patch i. The weights ω_i can be set according to the sensitivity of the average human perception to the color of each color patch 151. For example, colors having greater sensitivity of human perception can be given greater weights ω_i.

At 505, a fitness function J can be determined. In some embodiments, the fitness function J can be found as a weighted and/or unweighted sum of the color correction error e_color and the noise amplification metric D_noise. For example, an unweighted fitness function J can be represented as the following sum:

J＝e_color+D_noise Equation (13)

In some embodiments, a weighted fitness function J can used to advantageously weight the color correction error e_color more than the noise amplification metric D_noise, or vice versa. The amount of weighting for the fitness function J can be determined, for example, by repeating a color correction parameter calibrations for different weights and taking the weight that gives the best (for example, the lowest) value of the fitness function J. Alternatively and/or additionally, the amount of weighting for the fitness function J can be determined based on prior color correction parameter calibrations (for example, using different imaging devices) .

Turning now to Fig. 6, an exemplary method 600 for calibrating color correction parameters 125 (shown in Fig. 2) is shown as including two steps. At 601, a first optimization process is applied to obtain initial values CC₀ for the color correction parameters 125. The first optimization preferably samples broadly the space of possible color correction parameter values so as to avoid becoming trapped in local optima. Any of various optimization processes can be used in the first optimization at 601, including a genetic process, a simulated annealing method, and other non-greedy methods that avoid local optima. At 602, a second optimization process is applied using the initial values CC₀ as a starting point to obtain further optimized values CC_opt for the color correction parameters 125. At the second optimization, at 602, a goal is to find the local optimum value. Accordingly, direct optimization methods are suitable for the second optimization at 602. Exemplary direct optimization methods include, but are not limited to, gradient descent methods.

Turning now to Fig. 7, an exemplary genetic process 700 is shown for calibrating color correction parameters 125 (shown in Fig. 2) . A genetic process is an optimization method loosely based on evolutionary principles in biology, where possible solutions to a problem are generated as members of a “population, ” and the members are selected based on a fitness function over a number of selection rounds. The genetic process 700 can be used to find an optimal solution to the problem of selecting a set of color correction parameters 125 to optimize (for example, minimize) the fitness function J that includes a color correction error e_color and a noise amplification metric D_noise. At 701, a predetermined number N of initial sets of candidate color correction parameters 125A are selected as the initial “population” of solutions. The predetermined number N can comprise any suitable number of initial sets and, for example, can be at least 10, 50, 100, 500, 1000, or more. The initial population of the N sets of candidate color correction parameters 125A can be selected, for example, by sampling the space of possible parameters at specified intervals. Alternatively and/or additionally, the sampling can be done at random.

At 702, the fitness function J is evaluated for the members of the “population, ” that is, for each of the N sets of candidate color correction parameters 125A. From among the N initial sets of candidate color correction parameters 125A, the initial set that has the best value of the fitness function J (for example, the minimal value, if the fitness function J to be minimized) is chosen. At 703, if the best value passes a predefined threshold, the genetic process stops at 704. Alternatively and/or additionally, at 705, if certain conditions are met (for example, the genetic process has been run for more than a certain number of rounds, or the genetic process has not produced more than a specific amount of improvement in the fitness function J from the prior round) , the genetic process stops at 704. After the genetic process stops, at 704, the candidate color correction parameters 125A giving the best value of the fitness function J is declared to be the “winner, ” and these candidate color correction parameters 125A can be outputted and/or used as a starting point for further optimization.

If the “best” candidate correction parameters 125 do not pass a predefined threshold and/or certain conditions for stopping the genetic process, at 703, are not met, the genetic process continues, at 706, by discarding and replacing candidate color correction parameters 125A having the lowest values of the fitness function J. In one embodiment, a given percentile of the candidate color correction parameters 125A having the lowest fitness function J can be discarded and replaced with new candidate color correction parameters 125A. The new candidate color correction parameters 125A can, for example, be generated in the same way as the initial candidate color correction parameters 125A. In some embodiments, at least 10％, 20％, 30％, 40％, 50％, or more the lowest scoring fitness functions J can be discarded.

At 707, “mutation” operations can be applied to the candidate color correction parameters 125A, simulating biological mutations of chromosomes between successive generations of individuals. Here, each set of candidate color correction parameters 125A can be conceptually treated as a “chromosome” that is also subject to mutation. Mutations to the candidate color correction parameters 125A include, for example, “point mutations” changing individual parameters at random and/or “crossover” mutations between two sets of candidate color correction parameters 125A. For example, where the candidate color correction parameters 125A take the form of a matrix, a crossover can be performed by swapping corresponding rows and/or columns or portions thereof between two candidate matrices. Where the candidate color correction parameters 125A take the form of a look-up table, a crossover can be performed by swapping one or more corresponding entries in the look-up table. Once the mutations are applied to the candidate color correction parameters 125A, the method 700 can return to 702 for evaluating the fitness function J for the new round of the genetic process.

The noise amplification metric D_noise can be determined using any suitable approach, including but not limited to using peak signal-to-noise ratios (PSNR) and/or color variances.

Turning now to Fig. 8, an exemplary diagram is shown for finding the noise amplification metric D_noise using a peak signal-to-noise ratios (PSNR) . Beginning with noise-free color values C_{noise_free}, noise-evaluation color values C_noise can be found by adding noise n, as described in Equation (7) . A color correction CC can be applied to the noise-free color values C_{noise_free} and noise-evaluation color values C_noise , respectively, to find corresponding post-correction values of the noise-free color values

and noise evaluation color values

For example, the color correction is shown in Equation (11) and in Equation (14) below:

Based on the parameters C_{noise_free}, C_noise,

and

apair of PSNR values PSNR and

can be found through determining a mean squared error (MSE) , as shown in Equations (15) through (18) :

In the above equations (15) - (16) , S, i.e. ∑_j S_j, is the number of pixels and MAX_I is the maximum value of C_noise and C_{noise_free}, and j is an index over virtual color patches.

In the above equations (17) - (18) , S, i.e. ∑_j S_j, is the number of pixels and MAX_I is the maximum value of

and

and j is an index over virtual color patches

In some embodiments, determining the noise amplification metric D_noise can include finding a PSNR difference that is a difference between a PSNR for the pre-correction noise evaluation image and a

for the corrected noise evaluation image.

In some embodiments, the noise amplification metric D_noise can be determined by downsampling. For example, the downsampling can be a spatial downsampling, as illustrated in Fig. 9. In particular, Fig. 9 illustrates an embodiment of downsampling in which an image (for example, images having pre-correction noise evaluation color values C_noise, or pre-correction noise-free color values C_{noise_free}) is sample at every other pixel in a first downsampling. In some embodiments, the downsampled image can be downsampling again, and the downsampling process can be repeated as often as desired up to M iterations. Although not shown in Fig. 9, a similar downsampling process can be performed for images that have been color-corrected (for example, images having post-correction noise evaluation color values

or post-correction noise-free color values

) . Since downsampling can be an iterative process, color values and PSNR values at particular iterations are denoted with a subscript from 0 to M corresponding to the iteration, as shown in Figs. 8-9.

After each round of downsampling, the downsampled images can be used to determine one or more downsampled PSNRs as well as a downsampled PSNR difference. Returning to Fig. 8, pre-correction noise-free color values

and pre-correction noise evaluation color values

that have undergone one round of downsampling can be used to find a corresponding downsampled PSNR₁. Likewise, post-correction noise-free color values

and post-correction noise evaluation color values

that have undergone one round of downsampling can be used to find a corresponding downsampled

After M downsampling rounds, a set of PSNR values PSNR_i. and

will be obtained, where i ranges from 0 to M. The set of PSNR values can be used to find corresponding PSNR differences for value of i ranging from 0 to M, where i＝0 corresponds to a PSNR difference that has not been downsampled, and i＝m corresponds to a PSNR difference that has been downsampled m times.

In some embodiments, the noise amplification metric D_noise can be obtained by taking a weighted average of a PSNR difference and at least one downsampled PSNR difference. In some embodiments, the noise amplification metric D_noise can be obtained by taking a weighted average of the PSNR difference and the plurality of successively downsampled PSNR differences. The weight applied to each PSNR difference and/or downsampled PSNR difference can represented as w_i, where i ranges from 0 to M. An exemplary method of finding the noise amplification metric D_noise is shown as follows in Equation (19) , which is reproduced in Fig. 8:

where M is the total number of downsampling iterations and w_i is the weight given to each downsampling iteration i.

In some embodiments, at least one of the weights w_i is non-zero. Stated somewhat differently, PSNR differences at one or more iterations i can be given a weight of zero to effectively ignore that PSNR difference, provided that not all of the weights are ignored.

Turning now to Fig. 10, an exemplary method 1000 is shown for finding the noise amplification metric D_noise that locates and compares peak signal-to-noise ratios (PSNR) at successively downsampled frequencies. At 1001a, an initial value of a pre-correction PSNR PSNR₀ can be found using the pre-correction noise evaluation color values

and pre-correction noise-free color values

as described above with reference to Figs. 8 and 9. At 1002a,

and

can each be downsampled to obtain

and

respectively. At 1003a, a downsampled PSNR₁ can be found from

and

Optionally, at 1004a, the process of downsampling and finding a corresponding downsampled PSNR can be repeated for M iterations, as desired.

Similarly, the iterative downsampling process can be repeated for color-corrected images. At 1001b, an initial value of a post-correction PSNR

can be found using the post-correction noise evaluation color values

and post-correction noise-free color values

as described above with reference to Figs. 8 and 9. At 1002b,

and

can each be downsampled to obtain

and

respectively. At 1003b, a downsampled

can be found from

and

Optionally, at 1004b, the process of downsampling and finding a corresponding downsampled

can be repeated for M iterations, as desired.

Finally, at 1005, the set of PSNR values and color-corrected PSNR values found at iterations 0 to M can be used to find the noise amplification metric D_noise —for example, as shown above in Equation (19) .

In another embodiment, the noise amplification metric D_noise can be obtained based on a variance of Y, U, and V components of the pre-correction noise evaluation color values

and post-correction noise evaluation color valuesIn an exemplary embodiment, the noise amplification metric D_noise can be obtained using the Equation (20) :

wherein_represents the variance of the Y components of the pre-correction noise evaluation color values , and represents the variance of the Y components of the post-correction noise evaluation color values^, and likewise for the U and V components, and is the weight given to each downsampling iteration i, where ≥0.

Turning now to Fig. 11, an exemplary embodiment of the imaging system 200 is shown wherein the imaging system 200 is shown as being installed aboard an unmanned aerial vehicle (UAV) 1100. A UAV 1100, colloquially referred to as a “drone, ” is an aircraft without an onboard human pilot and whose flight is controlled autonomously and/or by a remote pilot. The imaging system 200 is suitable for installation aboard any of various types of UAVs 1100, including, but not limited to, rotocraft, fixed-wing aircraft, and hybrids thereof. Suitable rotocraft include, for example, single rotor, dual rotor, trirotor, quadrotor (quadcopter) , hexarotor, and octorotor rotocraft. The imaging system 200 can be installed on various portions of the UAV 1100. For example, the imaging system 200 can be installed within a fuselage 1110 of the UAV 1100. Alternatively, the imaging system 200 can be mounted onto an exterior surface 1020 (for example, on the underside 1025) of the UAV 1100. Furthermore, the various components of the imaging system 200 can be installed on the same portion, and/or different portions, of the UAV 1100. For example, an image sensor 130 can be mounted on an exterior surface 1120 to facilitate image acquisition； while, a color correction apparatus 100 advantageously can be installed within the fuselage 1110 for protection against wear and tear. Likewise, the various components of the color correction apparatus 100 can be installed on the same portion, and/or different portions, of the UAV 1100. Although shown and described with respect to a UAV 1100 for purposes of illustration only, the imaging system 200 can include, or be mounted on, any type of mobile platform. Exemplary suitable mobile platforms include, but are not limited to, bicycles, automobiles, trucks, ships, boats, trains, helicopters, aircraft, various hybrids thereof, and the like.

Example 1

The following color correction parameter calibration experiment was performed to determine the efficacy of a method of calibration with noise regulation in comparison to the method without noise regulation. First, an input image was used to calibrate color correction parameters by using a fitness function that includes the noise amplification metric in the manner described above. Fig. 12 shows a chrominance diagram of resulting color errors in a CIE L*a*b*color space (showing a cross section in the a*and b*dimensions) , showing a mean color correction error of 16.8 with a maximum color correction error of 29.7. Fig. 13 shows a plot of the resulting noise levels from the same experiment, showing that the average Y(luminance) noise is 0.83％； while, the average chrominance noise in the R, G, and B components are 1.38％, 1.31％, and 1.63％, respectively.

In contrast, the same input image was used to calibrate color correction parameters by using a fitness function that does not include the noise amplification metric. Fig. 14 shows a chrominance diagram of resulting color errors in a CIE L*a*b*color space, showing a mean color correction error of 17.7 with a maximum color correction error of 35, both of which are significantly greater than the corresponding errors obtained with noise regulation. Fig. 15 shows a plot of the corresponding noise levels of the experiment, showing that the average Y(luminance) noise is 0.86％； while, the average chrominance noise in the R, G, and B components are 1.56％, 1.31％, and 1.66％, respectively, which are significantly greater than the noise obtained by calibration with noise regulation. Accordingly, it can be seen from this experiment that color correction parameter calibration with noise regulation is an improvement over color correction parameter calibration without noise regulation.

Example 2

The following color correction parameter calibration experiment was performed to determine the efficacy of a method of calibration with conversion to a CIE L*a*b*color space in comparison to the method that performs the calibration in a CIE XYZ color space. First, an input image was used to calibrate color correction parameters after the input and reference colors of the input image are converted to a CIE L*a*b*. Optimization yielded the following matrix of color correction parameters:

having an optimized e_c of 2.412169304.

Next, the same input image was used to calibrate color correction parameters after the input and reference colors of the input image are converted to a CIE XYZ. Optimization yielded the following matrix of color correction parameters:

having an optimized e_c of 3.0107447. This comparison shows that using a non-linear color space—here, a CIE L*a*b*color space—yields improved results over using a CIE XYZ color space.

Example 3

The following example shows the process of optimizing a set of color correction parameters using the two-step method of Fig. 6. In the first step of the two-step method, a genetic process is used to find a set of initial parameters so as to avoid becoming trapped in local optima. The fitness value of the parameters for the genetic process over six hundred generations is shown in Fig. 16 at the upper panel, showing that the fitness value reaches a best value of 335.134 after 600 generations. In the second step of the two-step method, a direct optimization process is used starting from the initial parameters produced at the end of step one. In the second step, after another 600 generations, the direct optimization method reduces the average distance between the corrected input colors and the corresponding reference colors, as shown in Fig. 15 at the lower panel. This example shows that it is advantageous to use a two-step optimization method.

The disclosed embodiments are susceptible to various modifications and alternative forms, and specific examples thereof have been shown by way of example in the drawings and are herein described in detail. It should be understood, however, that the disclosed embodiments are not to be limited to the particular forms or methods disclosed, but to the contrary, the disclosed embodiments are to cover all modifications, equivalents, and alternatives.

Claims

A method for calibrating a digital imaging device for color correction, the method comprising:

obtaining an input color value and a reference color value for each of a plurality of color references and a noise evaluation image having a color noise for evaluating noise reduction, the input color values and reference color values being in a non-linear color space； and

determining a plurality of color correction parameters that are optimized based on evaluating a fitness function in the non-linear color space.
The method of claim 1, wherein the non-linear color space is a CIE L*a*b* color space.
The method of claim 1 or claim 2, wherein said determining the plurality of color correction parameters comprises adjusting the color correction parameters based on the input color values, and reference color values, and the noise evaluation image.
The method of claim 3, wherein the fitness function comprises a color correction error and a noise amplification metric.
The method of claim 4, wherein said adjusting the plurality of color correction parameters comprises determining the color correction error by color correcting the input color values and comparing the corrected input color values with the reference color values.
The method of claim 4 or claim 5, wherein said adjusting the plurality of color correction parameters comprises determining the noise amplification metric by color correcting the noise evaluation image using the parameters and comparing the corrected noise evaluation image with the pre-correction noise evaluation image.
The method of claim 6, wherein said determining the noise amplification metric comprises determining the noise amplification metric using a peak signal-to-noise ratio (PSNR) .
The method of claim 7, wherein said determining the noise amplification metric using a PSNR comprises finding a PSNR difference that is a difference between a PSNR for the pre-correction noise evaluation image and a PSNR for the corrected noise evaluation image.
The method of claim 7 or claim 8, wherein said determining the noise amplification metric using a PSNR comprises determining a downsampled PSNR difference.
The method of claim 9, wherein said determining the downsampled PSNR difference comprises:

downsampling the pre-correction noise evaluation image to obtain a downsampled pre-correction noise evaluation image；

downsampling the corrected noise evaluation image to obtain a downsampled corrected noise evaluation image； and

finding the downsampled PSNR difference as a difference between a PSNR for the downsampled pre-correction noise evaluation image and a PSNR for the downsampled corrected noise evaluation image.
The method of claim 9 or claim 10, wherein said determining the noise amplification metric comprises determining a weighted average of the PSNR difference and at least one downsampled PSNR difference.
The method of claim 11, wherein said determining the noise amplification metric using a PSNR comprises determining a plurality of successively downsampled PSNR differences and determining the noise amplification metric as a weighted average of the PSNR difference and the plurality of successively downsampled PSNR differences.
The method of claim 12, wherein said determining the noise amplification metric comprises weighing each of the PSNR difference and the plurality of successively downsampled PSNR difference with a respective weight, and at least one of the weights is not zero.
The method of claim 6, wherein said determining the noise amplification metric comprises determining the noise amplification metric based on color variances.
The method of any one of claims 6-14, wherein said adjusting the plurality of color correction parameters comprises determining the noise amplification metric by converting the color corrected noise evaluation image into a YUV color space.
The method of any one of claims 4-15, wherein said noise evaluation image comprises a plurality of color patches, and wherein the noise amplification metric is determined as a weighted sum of noise amplification levels of each of the color patches.
The method of claim 16, wherein each color patch is weighted according to an average sensitivity of human perception to the color patch.
The method of any one claims 4-17, wherein said adjusting the color correction parameters comprises optimizing the fitness function by minimizing a weighted sum of the color correction error and the noise amplification metric.
The method of any one of the above claims, wherein said adjusting the color correction parameters comprises adjusting the parameters using a genetic process.
The method of claim 19, comprising further adjusting the color correction parameters using a direct search method.
The method of any one of the above claims, wherein said obtaining the input color values of the color calibration images comprises imaging one or more color references comprising a plurality of color patches.
The method of any one of the above claims, wherein the color correction parameters are in the form of a matrix.
The method of any one of the above claims, wherein the color correction parameters are in the form of a look-up table.
The method of claim 23, wherein said adjusting the color correction parameters comprises look-up operations and interpolation operations in the look-up table.
The method of claim 24, wherein said interpolation operations comprise a Shepard interpolation.
The method of any one of the above claims, wherein the noise evaluation image is a virtual noise evaluation image.
The method of claim 26, wherein the virtual noise evaluation image comprises noise added to a virtual noise-free image.
A color correction apparatus configured for calibration for color correction based upon images of a plurality of color references each having a reference color, comprising:

a memory for storing a plurality of color correction parameters； and

a processor for performing color correction of a digital image,

wherein the processor is configured to:

obtain an input color value and a reference color value for each of the color references and a noise evaluation image having a color noise for evaluating noise reduction, the input color values and reference color values being in a non-linear color space, and

determine a plurality of color correction parameters that are optimized based on evaluating a fitness function in the non-linear color space.
The color correction apparatus of claim 28, wherein the non-linear color space is a CIE L*a*b* color space.
The color correction apparatus of claim 28 or claim 29, wherein the processor is configured to determine the plurality of color correction parameters by adjusting the color correction parameters based on the input color values, and reference color values, and the noise evaluation image.
The color correction apparatus of claim 30, wherein the fitness function comprises a color correction error and a noise amplification metric.
The color correction apparatus of claim 31, wherein the processor is configured to adjust the plurality of color correction parameters by color correcting the input color values and comparing the corrected input color values with the reference color values.
The color correction apparatus of claim 31 or claim 32, wherein the processor is configured to adjust the plurality of color correction parameters by color correcting the noise evaluation image using the parameters and comparing the corrected noise evaluation image with the pre-correction noise evaluation image.
The color correction apparatus of claim 33, wherein the processor is configured to determine the noise amplification metric using a peak signal-to-noise ratio (PSNR) .
The color correction apparatus of claim 34, wherein the processor is configured to determine the noise amplification metric using a PSNR by finding a PSNR difference that is a difference between a PSNR for the pre-correction noise evaluation image and a PSNR for the corrected noise evaluation image.
The color correction apparatus of claim 34 or claim 35, wherein the processor is configured to determine the noise amplification metric using a PSNR by determining a downsampled PSNR difference.
The color correction apparatus of claim 36, wherein the processor is configured to determine the downsampled PSNR difference by:

downsampling the pre-correction noise evaluation image to obtain a downsampled pre-correction noise evaluation image；

downsampling the corrected noise evaluation image to obtain a downsampled corrected noise evaluation image； and

finding the downsampled PSNR difference as a difference between a PSNR for the downsampled pre-correction noise evaluation image and a PSNR for the downsampled corrected noise evaluation image.
The color correction apparatus of claim 36 or claim 37, wherein the processor is configured to determine the noise amplification metric by determining a weighted average of the PSNR difference and at least one downsampled PSNR difference.
The color correction apparatus of claim 38, wherein the processor is configured to determine a plurality of successively downsampled PSNR differences and determine the noise amplification metric as a weighted average of the PSNR difference and the plurality of successively downsampled PSNR differences.
The color correction apparatus of claim 39, wherein the processor is configured to determine the noise amplification metric by weighing each of the PSNR difference and the plurality of successively downsampled PSNR difference with a respective weight, and at least one of the weights is not zero.
The color correction apparatus of claim 33, wherein the processor is configured to determine the noise amplification metric based on color variances.
The color correction apparatus of any one of claims 33-41, wherein the processor is configured to determine the noise amplification metric by converting the color corrected noise evaluation image into a YUV color space.
The color correction apparatus of any one of claims 31-42, wherein the noise evaluation image comprises a plurality of color patches, and wherein the noise amplification metric is determined as a weighted sum of noise amplification levels of each of the color patches.
The color correction apparatus of claim 43, wherein each color patch is weighted according to an average sensitivity of human perception to the color patch.
The color correction apparatus of any one claims 31-44, wherein the processor is configured to optimizing the fitness function by minimizing a weighted sum of the color correction error and the noise amplification metric.
The color correction apparatus of any one of claims 28-45, wherein the processor is configured to adjust the color correction parameters using a genetic process.
The color correction apparatus of claim 46, wherein the processor is configured to further adjust the color correction parameters using a direct search method.
The color correction apparatus of any one of claims 28-47, wherein the processor is configured to obtain the input color values of the color calibration images by imaging one or more color references comprising a plurality of color patches.
The color correction apparatus of any one of claims 28-48, wherein the color correction parameters are in the form of a matrix.
The color correction apparatus of any one of claims 28-49, wherein the color correction parameters are in the form of a look-up table.
The color correction apparatus of claim 50, wherein the processor is configured to adjust the color correction parameters by look-up operations and interpolation operations in the look-up table.
The color correction apparatus of claim 51, wherein said interpolation operations comprise a Shepard interpolation.
The color correction apparatus of any one of claims 28-52, wherein the noise evaluation image is a virtual noise evaluation image.
The color correction apparatus of claim 53, wherein the virtual noise evaluation image comprises noise added to a virtual noise-free image.
The color correction apparatus of any one of claims 28-54, wherein the color correction apparatus is mounted aboard a mobile platform.
The color correction apparatus of claim 55, wherein the mobile platform is an unmanned aerial vehicle (UAV) .
A digital imaging device, comprising:

an image sensor for imaging a plurality of color references； and

a processor for performing color correction of a digital image,

wherein the processor is configured to:

obtain an input color value and a reference color value for each of the color references and a noise evaluation image having a color noise for evaluating noise reduction, the input color values and reference color values being in a non-linear color space, and

determine a plurality of color correction parameters that are optimized based on evaluating a fitness function in the non-linear color space.
The digital imaging device of claim 57, wherein the non-linear color space is a CIE L*a*b* color space.
The digital imaging device of claim 57 or claim 58, wherein the processor is configured to determine the plurality of color correction parameters by adjusting the color correction parameters based on the input color values, and reference color values, and the noise evaluation image.
The digital imaging device of claim 59, wherein the fitness function comprises a color correction error and a noise amplification metric.
The digital imaging device of claim 60, wherein the processor is configured to adjust the plurality of color correction parameters by color correcting the input color values and comparing the corrected input color values with the reference color values.
The digital imaging device of claim 60 or claim 61, wherein the processor is configured to adjust the plurality of color correction parameters by color correcting the noise evaluation image using the parameters and comparing the corrected noise evaluation image with the pre-correction noise evaluation image.
The digital imaging device of claim 62, wherein the processor is configured to determine the noise amplification metric using a peak signal-to-noise ratio (PSNR) .
The digital imaging device of claim 63, wherein the processor is configured to determine the noise amplification metric using a PSNR by finding a PSNR difference that is a difference between a PSNR for the pre-correction noise evaluation image and a PSNR for the corrected noise evaluation image.
The digital imaging device of claim 63 or claim 64, wherein the processor is configured to determine the noise amplification metric using a PSNR by determining a downsampled PSNR difference.
The digital imaging device of claim 65, wherein the processor is configured to determine the downsampled PSNR difference by:

downsampling the pre-correction noise evaluation image to obtain a downsampled pre-correction noise evaluation image；

downsampling the corrected noise evaluation image to obtain a downsampled corrected noise evaluation image； and

finding the downsampled PSNR difference as a difference between a PSNR for the downsampled pre-correction noise evaluation image and a PSNR for the downsampled corrected noise evaluation image.
The digital imaging device of claim 65 or claim 66, wherein the processor is configured to determine the noise amplification metric by determining a weighted average of the PSNR difference and at least one downsampled PSNR difference.
The digital imaging device of claim 67, wherein the processor is configured to determine a plurality of successively downsampled PSNR differences and determine the noise amplification metric as a weighted average of the PSNR difference and the plurality of successively downsampled PSNR differences.
The digital imaging device of claim 68, wherein the processor is configured to determine the noise amplification metric by weighing each of the PSNR difference and the plurality of successively downsampled PSNR difference with a respective weight, and at least one of the weights is not zero.
The digital imaging device of claim 62, wherein the processor is configured to determine the noise amplification metric based on color variances.
The digital imaging device of any one of claims 62-70, wherein the processor is configured to determine the noise amplification metric by converting the color corrected noise evaluation image into a YUV color space.
The digital imaging device of any one of claims 60-71, wherein the noise evaluation image comprises a plurality of color patches, and wherein the noise amplification metric is determined as a weighted sum of noise amplification levels of each of the color patches.
The digital imaging device of claim 72, wherein each color patch is weighted according to an average sensitivity of human perception to the color patch.
The digital imaging device of any one claims 60-73, wherein the processor is configured to optimizing the fitness function by minimizing a weighted sum of the color correction error and the noise amplification metric.
The digital imaging device of any one of claims 57-74, wherein the processor is configured to adjust the color correction parameters using a genetic process.
The digital imaging device of claim 75, wherein the processor is configured to further adjust the color correction parameters using a direct search method.
The digital imaging device of any one of claims 57-76, wherein the processor is configured to obtain the input color values of the color calibration images by imaging one or more color references comprising a plurality of color patches.
The digital imaging device of any one of claims 57-77, wherein the color correction parameters are in the form of a matrix.
The digital imaging device of any one of claims 57-78, wherein the color correction parameters are in the form of a look-up table.
The digital imaging device of claim 79, wherein the processor is configured to adjust the color correction parameters by look-up operations and interpolation operations in the look-up table.
The digital imaging device of claim 80, wherein said interpolation operations comprise a Shepard interpolation.
The digital imaging device of any one of claims 57-81, wherein the noise evaluation image is a virtual noise evaluation image.
The digital imaging device of claim 82, wherein the virtual noise evaluation image comprises noise added to a virtual noise-free image.
The digital imaging device of any one of claims 57-83, wherein the color correction apparatus is mounted aboard a mobile platform.
The digital imaging device of claim 84, wherein the mobile platform is an unmanned aerial vehicle (UAV) .
A non-transitory readable medium storing instructions for calibrating a digital imaging device for color correction, wherein the instructions comprise instructions for:

obtaining an input color value and a reference color value for each of a plurality of color references and a noise evaluation image having a color noise for evaluating noise reduction, the input color values and reference color values being in a non-linear color space； and

determining a plurality of color correction parameters that are optimized based on evaluating a fitness function in the non-linear color space.
The non-transitory readable medium of claim 86, wherein the non-linear color space is a CIE L*a*b* color space.
The non-transitory readable medium of claim 86 or claim 87, wherein said determining the plurality of color correction parameters comprises adjusting the color correction parameters based on the input color values, and reference color values, and the noise evaluation image.
The non-transitory readable medium of claim 88, wherein the fitness function comprises a color correction error and a noise amplification metric.
The non-transitory readable medium of claim 89, wherein said adjusting the plurality of color correction parameters comprises determining the color correction error by color correcting the input color values and comparing the corrected input color values with the reference color values.
The non-transitory readable medium of claim 89 or claim 90, wherein said adjusting the plurality of color correction parameters comprises determining the noise amplification metric by color correcting the noise evaluation image using the parameters and comparing the corrected noise evaluation image with the pre-correction noise evaluation image.
The non-transitory readable medium of claim 91, wherein said determining the noise amplification metric comprises determining the noise amplification metric using a peak signal-to-noise ratio (PSNR) .
The non-transitory readable medium of claim 92, wherein said determining the noise amplification metric using a PSNR comprises finding a PSNR difference that is a difference between a PSNR for the pre-correction noise evaluation image and a PSNR for the corrected noise evaluation image.
The non-transitory readable medium of claim 92 or claim 93, wherein said determining the noise amplification metric using a PSNR comprises determining a downsampled PSNR difference.
The non-transitory readable medium of claim 94, wherein said determining the downsampled PSNR difference comprises:

downsampling the pre-correction noise evaluation image to obtain a downsampled pre-correction noise evaluation image；

downsampling the corrected noise evaluation image to obtain a downsampled corrected noise evaluation image； and

finding the downsampled PSNR difference as a difference between a PSNR for the downsampled pre-correction noise evaluation image and a PSNR for the downsampled corrected noise evaluation image.
The non-transitory readable medium of claim 94 or claim 95, wherein said determining the noise amplification metric comprises determining a weighted average of the PSNR difference and at least one downsampled PSNR difference.
The non-transitory readable medium of claim 96, wherein said determining the noise amplification metric using a PSNR comprises determining a plurality of successively downsampled PSNR differences and determining the noise amplification metric as a weighted average of the PSNR difference and the plurality of successively downsampled PSNR differences.
The non-transitory readable medium of claim 97, wherein said determining the noise amplification metric comprises weighing each of the PSNR difference and the plurality of successively downsampled PSNR difference with a respective weight, and at least one of the weights is not zero.
The non-transitory readable medium of claim 98, wherein said determining the noise amplification metric comprises determining the noise amplification metric based on color variances.
The non-transitory readable medium of any one of claims 91-99, wherein said adjusting the plurality of color correction parameters comprises determining the noise amplification metric by converting the color corrected noise evaluation image into a YUV color space.
The non-transitory readable medium of any one of claims 89-100, wherein said noise evaluation image comprises a plurality of color patches, and wherein the noise amplification metric is determined as a weighted sum of noise amplification levels of each of the color patches.
The non-transitory readable medium of claim 101, wherein each color patch is weighted according to an average sensitivity of human perception to the color patch.
The non-transitory readable medium of any one claims 89-102, wherein said adjusting the color correction parameters comprises optimizing the fitness function by minimizing a weighted sum of the color correction error and the noise amplification metric.
The non-transitory readable medium of any one of claims 86-103, wherein said adjusting the color correction parameters comprises adjusting the parameters using a genetic process.
The non-transitory readable medium of claim 104, comprising further adjusting the color correction parameters using a direct search method.
The non-transitory readable medium of any one of claims 86-105, wherein said obtaining the input color values of the color calibration images comprises imaging one or more color references comprising a plurality of color patches.
The non-transitory readable medium of any one of claims 86-106, wherein the color correction parameters are in the form of a matrix.
The non-transitory readable medium of any one of claims 86-107, wherein the color correction parameters are in the form of a look-up table.
The non-transitory readable medium of claim 108, wherein said adjusting the color correction parameters comprises look-up operations and interpolation operations in the look-up table.
The non-transitory readable medium of claim 109, wherein said interpolation operations comprise a Shepard interpolation.
The non-transitory readable medium of any one of claims 86-110, wherein the noise evaluation image is a virtual noise evaluation image.
The non-transitory readable medium of claim 111, wherein the virtual noise evaluation image comprises noise added to a virtual noise-free image.