WO2021155925A1

WO2021155925A1 - Color filter array and pipeline for image processing

Info

Publication number: WO2021155925A1
Application number: PCT/EP2020/052917
Authority: WO
Inventors: Samu Koskinen; Petri Nenonen
Original assignee: Huawei Technologies Co., Ltd.
Priority date: 2020-02-06
Filing date: 2020-02-06
Publication date: 2021-08-12
Also published as: CN115039402B; CN115039402A

Abstract

A device (1201) for processing a set of color pixels of an image to form, for at least one pixel, an output on four channels, wherein the device is configured to: form outputs for three of the channels according to respective first (601), second (602) and third (603) wavelength- dependent weighting functions which each have a positive response peak in the visible wavelength spectrum within 20 nm of a positive response peak of a respective one of first (301, 401, 501), second (302, 402, 502) and third (303, 403, 503) color standard observer functions in a common color space; and form the output for the fourth channel according to a fourth wavelength-dependent weighting function (604) which has a positive response peak in the visible wavelength spectrum offset by more than 10 nm from the positive response peaks of each of the said standard observer functions (301, 302, 303, 401, 402, 403, 501, 502, 503). Also described are an image signal processor (900) and a method (1100) for transforming image sensor readings (901). This may allow the color transform performed in an image signal processing pipeline to be independent of the illumination color of a scene.

Description

COLOR FILTER ARRAY AND PIPELINE FOR IMAGE PROCESSING

FIELD OF THE INVENTION

This invention relates to color filter arrays (CFA) and related processing methods for digital cameras.

BACKGROUND

In the processing of image sensor data to an RGB image, color filters are needed because typical image sensors, such as active pixel sensors (CMOS), detect light intensity with little or no wavelength specificity and therefore do not themselves differentiate colors. The photosensitive elements of a color filter array (CFA) filter the light according to wavelength- dependent functions and so the separate filtered intensities include information regarding the color of light so that an RGB image can be formed. In order to differentiate three different color channels that can be further transformed into RGB data, a CFA must have at least three different spectral responses. A commonly used CFA is the Bayer filter, as shown in Figure 1, which gives information about the intensity of light in the red, green and blue wavelength regions. This pattern has a recurring 2x2 mosaic that is tiled across the image. At each pixel, either a red 101, green 102 or blue color 103 is acquired.

The CFA-recorded data is not recorded with color coordinates of a standard color space and therefore cannot be shown natively to end users, as it does not contain correct or pleasing colors. A color transform is therefore required to convert the values into a common color space.

One aspect of the robustness of the color transform is how much the transform varies between different scene illuminants. Another aspect is how much the color transform has noise amplification properties. A third aspect is how well all colors are captured, for example, are the metameric failures minimal. In this context, the metameric failure means that camera does not separate all colors that the human or standard observer does.

A straightforward method for making the color transform robust would be to use the color spectral responses of the standard RGB channels as the CFA filters. However, these responses have negative values, which cannot be implemented in reality. These filters would give stable and robust transform coefficients, but negative responses are not physically possible. One typical solution is to use RGB filters, i.e. some responses filtering the reddish, greenish and bluish wavelengths of the color spectrum to different pixels of the sensor. Then the sensor RGB data is, in the later phase of the image processing system, transformed into standard RGB format, or some other target image format.

The disadvantages of this solution are that the required color transforms are illumination and scene color dependent. For example, a yellow light illuminating the scene colors requires a different color transform for achieving the standard RGB colors than a bluish light illuminating the same scene colors. This has been solved with additional algorithms which aim to analyze the type and color of the illumination and then selecting some color transform optimized for that color. The problem in this case is that illumination estimation or detection is a complex task. The selection of the transform is not always robust and final colors may not be optimal.

Another class of solutions attempt to maximize the sensitivity of the camera. These solutions can be divided into two categories. One approach is to use color filters that bear less resemblance to the RGB colors, but have a wider transmission bandwidth. Then the sensor detects more light than for the narrower bandwidth RGB type filters. For example, combinations such as Cyan-Magenta-Yellow or Red-Yellow-Blue have been used. These solutions may help the sensor to detect some light signal in extremely low light conditions. However, the disadvantage is that the color transform processing may result in more noise being gained. This is at least partly because the camera color space will be very different to the target standard color space and therefore stronger processing is required. Furthermore, the risk of metameric failures increases.

Another approach is to add a fourth channel that records the light with minimized reduction of signal. For example, a neutral channel (i.e. no filter at all) may be used as the fourth channel for recoding the whole spectral range that the sensor is able to detect. The disadvantage is the complexity of the color transform and the possible increased noise amplification due to the transform.

From a color transform simplicity viewpoint, one possible approach is to use CFA filter responses that are transforms of the standard color space. If the standard color space is achieved with an illuminant independent transform from the well-known standard CIE XYZ color representation and the CFA filter responses are a linear transform of the CIE XYZ responses then the camera fulfills the well-known Luther-lves condition and it is proven in the literature that the camera can detect the colors as the standard observer with a linear transform and the target standard RGB colors can be achieved with an illuminant independent transform. There is no metameric failure. This kind of camera is commonly called a colorimetric camera. There are multiple options for how these kind of filter responses can be selected. A trivial solution would be to use the CIE XYZ standard observer spectral responses. However, one disadvantage of this solution is that the responses are quite narrow, therefore reducing the amount of light entering the imaging sensor. Another disadvantage is that the responses, for example the X channel, may have multiple passbands in the spectrum range and are therefore hard to implement in reality. In addition, the transform, for example from CIE XYZ to standard RGB, has a high noise amplification property.

The most difficult parts of the problem are to design and implement the CFA filters so that they do not degrade the light sensitivity of the imaging sensor but still produce a good color accuracy and color separation, and further design the CFA filters so that the recorded image data processing to standard RGB data is robust and produces the least unwanted artefacts (such as noise), and also to design the color transform method jointly with the CFA filter design.

It is desirable to develop a CFA and a pipeline for processing image data that overcomes these problems.

SUMMARY OF THE INVENTION

According to a first aspect there is provided a device for processing a set of color pixels of an image to form, for at least one pixel, an output on four channels, wherein the device is configured to: form outputs for three of the channels according to respective first, second and third wavelength-dependent weighting functions which each have a positive response peak in the visible wavelength spectrum within 20 nm of a positive response peak of a respective one of first, second and third color standard observer functions in a common color space; and form the output for the fourth channel according to a fourth wavelength-dependent weighting function which has a positive response peak in the visible wavelength spectrum offset by more than 10 nm from the positive response peaks of each of the said standard observer functions.

The wavelength-dependent weighting functions of the channels of a color filter array are therefore determined based on the theoretical perfect spectral responses in a color space. As a result, the color transform performed in an image signal processing pipeline may be independent of the illumination color of a scene. The filters of a CFA having these wavelength- dependent weighting functions have good sensitivity and are realizable in practice, as each of the filters has a positive wavelength response. This may also result in reduced noise amplification during color processing. According to some embodiments, a wavelength-dependent weighting function may indicate an entity’s sensitivity to a specific wavelength of light.

The three of the channels may be red, green and blue color channels respectively. This may allow CFA-recorded data to processed in an ISP pipeline and transformed into a standard RGB format used by many current display devices.

The fourth wavelength-dependent weighting function may have a positive response peak between 510 to 530 nm. An additional CFA channel function with a peak in this range may result in optimized color processing.

The first to fourth wavelength-dependent weighting functions may each have a single positive response peak in the visible wavelength spectrum. This allows the corresponding color filters to be easily implementable.

At least one positive response peak associated with each of the first to fourth wavelength- dependent weighting functions may be a local maximum value or a global maximum value. The local maximum value or global maximum value may be in the visible wavelength spectrum.

One of the said standard observer functions in the common color space may have a positive response peak in the red region of the visible wavelength spectrum and a negative response peak outside the red region of the visible wavelength spectrum, and wherein the fourth wavelength-dependent weighting function may have a positive response peak within 20 nm of that negative response peak. The one of the said standard observer functions in the common color space may be the standard observer function of the red color channel. The positive response peak of the fourth wavelength-dependent weighting function may have the same peak wavelength as the negative response peak. The fourth wavelength-dependent weighting function may therefore be formed by taking the negative wavelength response of the red standard observer function and adding it as a new color channel with a positive wavelength response.

The wavelength-dependent weighting function of the red color channel may be negligible when the fourth wavelength-dependent weighting function is greater than zero. The common color space may be one of the sRGB color space, the CIE RGB color space, the Adobe RGB color space, the Rec2020 color space or the DCI-P3 color space. Therefore, the approach described herein may be used for a variety of different color spaces.

Each of the first to fourth wavelength-dependent weighting functions may have a positive wavelength response. This allows the corresponding color filters to be realizable in practice.

Each of the first to fourth wavelength-dependent weighting functions may have a single passband. This allows the corresponding color filters to be easily implementable.

The fourth wavelength-dependent weighting function may be different to the first, second and third wavelength-dependent weighting functions. Each of the first to fourth wavelength- dependent weighting functions may have a single peak in the visible wavelength spectrum.

The device may comprise an image sensor having one or more two-dimensional arrays of photosensitive elements, wherein the one or more two-dimensional arrays are configured to detect the intensity of light on each of the four channels. Each of the elements may comprise a color filter configured to filter light according to one of the first to fourth wavelength- dependent weighting functions. The photosensitive elements of a color filter array filter the light according to wavelength-dependent functions and so the separate filtered intensities include information about the color of light. This allows an RGB image to be formed.

According to a second aspect there is provided an image signal processor for transforming image sensor readings, the image signal processor being configured to: receive the image sensor readings corresponding to a scene of a source image in a source color space on a plurality of channels; perform a first transformation to transform the image sensor readings to a target color space to form a set of transformed readings; estimate an illumination color for the source image; and perform a second transformation to transform the set of transformed readings based on the estimated illumination color to form an output image.

The first transformation may be performed by applying a predetermined transformation to the image sensor readings. The first transformation may be independent of the illumination color of the source image. Therefore, a single matrix may be used for the color transform for scenes having different illumination colors.

The image signal processor may be further configured to receive a set of spectral sensor readings. The first transformation may be determined based on the set of spectral sensor readings. The illumination color for the source image may be estimated based on the set of spectral sensor readings. One or more of the first and second transformations may be further determined based on the image sensor readings. Therefore, readings from an external spectral sensor may be used to assist the determination of the transformations in the ISP.

The first transformation may be performed by applying a matrix to the image sensor readings. The approach is therefore compatible with standard image processing techniques for applying the color transform.

The first transformation and the second transformation may be applied to the image sensor readings as a single matrix. This may improve the efficiency of the image processing operation.

The image sensor readings received by the image signal processor may be on three or four channels. The matrix may be a 3x3 matrix or a 3x4 matrix. The image signal processor described herein may be used to process traditional three-channel CFA data (after interpolation, or demosaicking) or four-channel CFA data.

The image sensor readings received by the image signal processor may be output by the device described above. The combination of the four-channel CFA and image signal processor described above may result in more efficient image processing because only a linear transform may be needed from an imaging device’s own RGB color space to a standard RGB color space.

The target color space may be one of the sRGB color space, the Adobe RGB color space, the Rec2020 color space or the DCI-P3 color space. The output of the image signal processor may be image data in an RGB color space or a YUV color space. Therefore, the approach described herein may be used for a variety of different color spaces.

The illumination color for the source image may be estimated subsequent to performing the first transformation. If the first transformation may be performed using an illuminant- independent transform, this may improve the efficiency of the image processing operation.

The output image may represent the scene of the source image under a canonical illuminant (white light). The image signal processor may be further configured to perform lens shading correction prior to performing the first transformation. The image signal processor may be further configured to enhance the color of the output image. This may improve the resulting image quality.

According to a third aspect there is provided a method for transforming image sensor readings, the method comprising: receiving image sensor readings corresponding to a scene of the source image in a source color space on a plurality of channels; performing a first transformation to transform the image sensor readings to a target color space to form a set of transformed readings; estimating an illumination color for the source image; and performing a second transformation to transform the set of transformed readings based on the estimated illumination color to form an output image.

BRIEF DESCRIPTION OF THE FIGURES

The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

Figure 1 schematically illustrates the standard Bayer pattern color filter array on the sensor. At each pixel, either a blue, green, or red color is acquired.

Figure 2 illustrates the CIE XYZ standard observer functions.

Figure 3 illustrates the standard observer functions in the sRGB color space.

Figure 4 schematically illustrates how the fourth color channel can be created from the standard observer functions in the sRGB color space.

Figure 5 illustrates the CIE RGB color matching functions.

Figure 6 schematically illustrates the CFA responses on the four channels. Version 1 (solid lines) is based on the sRGB curves and version 2 (dashed lines) is based on the CIE RGB curves. The fourth channel is marked as J (jade color).

Figure 7 shows an example of a spectral comparison after a color transform (CCM) has been applied to the image sensor readings collected using the CFA described herein, shown as thick dashed lines (e.g. 701), compared to a normal camera with a CCM optimized for D65 (e.g. 702), a normal camera with a CCM optimized for halogen (e.g. 703) and the standard observer functions in the sRGB color space, shown as solid lines (e.g. 704). Figure 8 schematically illustrates the color transform coefficients as a function of scene illumination color. The dashed lines show typical RGB sensor transform coefficients under different illuminants. The solid lines show the transform matrix coefficients of the four-channel CFA described herein.

Figure 9 schematically illustrates an image signal processing pipeline for color processing.

Figure 10 schematically illustrates a comparison between a traditional color pipeline and the pipeline design of Figure 9.

Figure 11 is a flowchart illustrating an example of a method for transforming image sensor readings.

Figure 12 schematically illustrates an example of an imaging device configured to implement the device, image signal processor and method described herein to process image data captured by an image sensor in the device.

DETAILED DESCRIPTION OF THE INVENTION

In the present invention, the CFA filters may be essentially designed jointly with the color transform to a target colorspace in the image signal processing (ISP) pipeline so that, for RGB color spaces, the solution can be very close to a colorimetric camera, but yet the filter response functions have good sensitivity (i.e. wide passbands) and the filters are realizable in practice.

The embodiments described below are for examples where the source, target and/or output color spaces are RGB color spaces. However, the approach is also applicable to other colors, such as CMY or RYB.

The wavelength-dependent weighting functions of the channels of the CFA are determined based on the theoretical perfect spectral responses in a color space. Here, the functions describe the physical relationship between wavelengths and their contribution to the respective output signals.

“Standard observer functions” are typically associated with the CIE XYZ standard observer. The CIE XYZ standard observer color matching functions, shown in Figure 2, are the numerical description of the CIE standard observer. They can be thought of as the spectral sensitivity curves of three light detectors yielding the CIE tristimulus values X, Y and Z. The x- axis represents wavelength (nm) and the y-axis represents the relative response of the channels. The red, green and blue functions are indicated at 201 , 202 and 203 respectively.

The phrase standard observer functions in a color space is used herein to describe the CIE XYZ standard observer color matching functions transformed to a color space, such as sRGB. The transformed functions are the color matching functions of that color space. For example, the standard observer functions in the sRGB color space are therefore the CIE XYZ standard observer functions transformed to the sRGB color space (which can also be called the color matching functions of the sRGB color space). The color space may be one of the sRGB color space, the CIE RGB color space, the Adobe RGB color space, the Rec2020 color space or the DCI-P3 color space. Alternatively, any other suitable color space may be used.

In one example, the standard observer functions in the sRGB color space are used to determine the wavelength-dependent CFA functions on the four channels. These functions are shown in Figure 3. The red, green and blue functions are indicated at 301, 302 and 303 respectively.

Figure 4 shows an example of how the wavelength-dependent responses of the four CFA filters can be formed using the curves of the standard observer functions in the sRGB color space. The red, green and blue functions are indicated at 401, 402 and 403 respectively.

The CFA described herein is a four-channel CFA where three of the channels output image sensor readings according to respective first, second and third wavelength-dependent weighting functions and the fourth channel outputs readings according to a fourth wavelength- dependent weighting function. The first, second and third wavelength-dependent weighting functions have positive response peaks at positions (in nm) corresponding to, or close to, the positions of the peaks of the positive wavelength response of the standard observer functions in a common (i.e. the same) color space. A positive response peak may be located at positions corresponding to a local maximum intensity value or a global maximum intensity value. A positive response peak may be at a position (in nm) where the first derivative of intensity with respect to wavelength is equal to zero and the second derivative is less than zero. The peak may be the major, or highest intensity, peak in the visible wavelength spectrum for a particular wavelength-dependent weighting function.

The first, second and third wavelength-dependent weighting functions each have a positive response peak in the visible wavelength spectrum within 20 nm, preferably within 15 nm, more preferably within 10 nm, of a positive response peak of a respective one of first, second and third color standard observer functions in a common color space. Most preferably, the first, second and third wavelength-dependent weighting functions each have a positive response peak that has the same peak wavelength as a positive response peak of a respective one of first, second and third color standard observer functions in the common color space.

In one example, three of the channels of the CFA have wavelength-dependent weighting functions that are based on the positive wavelength response of the three standard observer functions in the sRGB color space. These channels are red, green and blue color channels. The peak wavelengths of the functions of the red, green and blue color channels of the CFA are preferably within 20 nm of the peak wavelengths of the red, green and blue standard observer functions in the color space (sRGB in the example above). More preferably, they may be within 15 nm or 10 nm of the peak wavelengths of the red, green and blue standard observer functions in the color space. Most preferably, they have the same peak wavelengths as the red, green and blue standard observer functions in the color space.

A fourth channel is added to the CFA which has a fourth wavelength-dependent weighting function which has a positive response peak in the visible wavelength spectrum offset by more than 10 nm, preferably more than 15 nm, more preferably more than 20 nm or 25 nm, from the positive response peaks of each of the said standard observer functions in the common color space.

In the present example, the fourth channel has a wavelength-dependent weighting function that is determined based on at least one of the red, green and blue standard observer functions in the sRGB color space.

In one example, the negative wavelength response (i.e. the part of the curve that is below the x-axis) of the red curve (shown at area 404) is flipped to the positive side and added as a new color channel, such that the function of the fourth channel has a positive wavelength response. This may be achieved in a preferably simple technical implementation.

The negative (and positive) minor peaks for red and green at around the 440 nm range (shown at area 405) can also be approximated from the using the blue channel, i.e. “blue channel (B) * -x = negative green part of the spectrum (negG)”. This can also be performed using a matrix- type operation ,i.e. negG = -x*B +y*G + z*B. y and z can be positive, negative or zero.

However, not all of the peaks/bumps of the functions may be implemented as a new channel. The most dominant and/or most “non-colorimetric” part can be selected, as explained below. As can be seen in Figure 3, the red standard observer function 301 in the sRGB color space has a positive response peak in the red region of the visible wavelength spectrum at approximately 610 nm (plus a further minor positive response peak at approximately 440 nm) and a negative response peak outside the red region of the visible wavelength spectrum, at approximately 520 nm. The fourth wavelength-dependent weighting function has a positive response peak within 20 nm, preferably within 15 nm, or more preferably within 10 nm, of that negative response peak (of the red standard observer function). Most preferably, the positive response peak of the fourth wavelength-dependent weighting function has the same peak wavelength (i.e. peak position in nm) as the negative response peak of the red standard observer function 301.

The negative part of the red standard observer function response can therefore be cut and used to create the fourth color channel. The red wavelength-dependent weighting function minus the fourth wavelength-dependent weighting function is close to the red standard observer function. A simple matrix can handle and reverse negative and positive values to form the fourth function. The blue channel can be used to compensate the negative values for green in the blue region and positive values for red in the blue region.

The channel responses can be normalized as magnitude differences to the standard observer functions, as shown in Figure 6. This can be also handled with a linear transform, i.e. the Luther-lves condition is not violated. The functions of the red, green and blue channels are indicated at 601 , 602 and 603 respectively and the function of the new ‘jade’ color channel is shown at 604.

In another implementation, color matching functions can be used to form the functions of the channels that are a linear transform of the target color space for the color transform (such as sRGB). For example, the functions of the four channels of the CFA can be based on the CIE RGB color matching functions. These functions are shown in Figure 5. The red, green and blue functions are indicated at 501, 502 and 503 respectively. These functions have the benefit that there are fewer parts of the functions that have a negative wavelength response (only in the red channel). The CFA functions determined from the CIE RGB color matching functions are shown as v2 (solid lines) in Figure 6.

In this example, the peak wavelengths of the functions of the red, green and blue color channels of the CFA are within 20 nm, preferably within 15 nm, more preferably within 10 nm, of the peak wavelengths of the red, green and blue standard observer functions in the CIE RGB color space. Most preferably, they have the same peak wavelengths as the red, green and blue standard observer functions in the CIE RGB color space.

In this example, the fourth wavelength-dependent weighting function has a highest positive response peak within 20 nm, preferably with 15 nm, more preferably within 10 nm, of the negative response peak of the red standard observer function. Most preferably, the positive response peak of the fourth wavelength-dependent weighting function has the same peak wavelength as the negative response peak of the red standard observer function 501.

For an RGB CFA, the wavelength-dependent weighting function of the fourth channel of the CFA preferably has a highest positive response peak between approximately 510 to 530 nm.

Each of the four functions of the CFA channels has a positive wavelength response and a single passband.

Where the four functions of the CFA channels are determined based on the CIE RGB standard observer functions (or color matching functions) the transformation from the CIE RGB color space to the sRGB color space is a fixed matrix and the combination of the transform is simply Matrix_CIERGB2SRGB * Matrix_4ChSensor2CIERGB (inverting the originally negative parts of the red channel and scaling the sensitivities and compensating the inaccuracies (difference to the ideal function)).

As can be seen in Figure 6, the function of the red color channel 601 is preferably negligible when the function of the fourth channel 604 is greater than zero. Ideally, the response of the red channel is zero across this range, however in a real implementation (for example, for manufacturability reasons) the response may be very small. For example, the fourth function may have a relative intensity that is less than 10%, preferably less than 5%, more preferably less than 3% or even most preferably less than 1 % of the intensity of the red color channel for a given wavelength. The fourth function 604 is preferably negligible when the function of the red channel 601 is greater than zero. For example, the red function may have a relative intensity that is less than 10%, preferably less than 5%, more preferably less than 3% or even most preferably less than 1% of the intensity of the fourth function for a given wavelength.

The CFA described herein is therefore designed based on the theoretical perfect spectral response (such as that for sRGB or CIE RGB) and taking the parts of the functions which have a negative wavelength response and flipping them as new positive color channels. The four channels of the CFA therefore have wavelength-dependent weighting functions that are determined in dependence on the standard observer functions in the target color space of the color transform to be performed by the image processing pipeline. As a result, only a linear transform is needed from a camera’s own RGB color space to the standard RGB color space.

Figure 7 shows a spectral comparison after a color transform (such as a color correction matrix, CCM) has been applied to the image sensor readings collected using the CFA described above, shown as thick dashed lines (e.g. 701), compared to a normal camera with a CCM optimized for D65 (e.g. 702), a normal camera with a CCM optimized for halogen (e.g.

703) and the standard observer functions in the sRGB color space, shown as solid lines (e.g.

704). The results from the four-channel CFA described herein are a close match to the standard observer functions in the sRGB color space and are therefore very close to satisfying the Luther-lves condition.

Figure 8 shows the color transform coefficient values for each of the four channels of the CFA as a function of scene illumination color. The dashed lines show typical RGB sensor transform coefficients under different illuminants. The solid lines show the transform matrix coefficients of the four-channel CFA described herein under different illuminants. The solid lines are much more robust and stable over different illuminants.

The CFA filters are therefore essentially designed jointly with the color transform that is performed in the image signal processing pipeline so that the solution is very close to a colorimetric camera, but yet the functions have good sensitivity, i.e. wide passbands, and the filters are realizable in practice (i.e. they have positive wavelength responses). A primary advantage is that the color transform to the sRGB color space is substantially illuminant independent and the camera is still sensitive to light. The noise amplification of the end-to-end processing from sensor data to the standard RGB colors is also low, as shown in Figure 8. The noise amplification can be approximated from the coefficient values.

The CFA described herein preferably has similar or lower coefficients (depending on the color channel) than a typical Bayer sensor. All filters have a positive wavelength response and fairly smooth transitions and are therefore realizable. The filters also have only one passband and are therefore easily implementable. The four-channel CFA spectral response design may, in some implementations, enable close to a colorimetric color reproduction.

The CFA can be implemented as an image sensor having a two-dimensional array (i.e. an ordered series or arrangement) of photosensitive elements configured to detect the intensity of light on each of the four channels. Each of the elements comprises a color filter configured to filter light according to one of the functions. The raw image data captured by the image sensor can then be converted to a full color image in an image signal processing pipeline, as will be described in more detail below.

Schematically illustrated in Figure 9 is an example of an image signal processing pipeline for color processing which can conveniently process the output of an image sensor which implements the four-channel CFA described above. In other embodiments, the pipeline may include additional blocks for altering different features of the image.

Existing color processing pipelines largely assume three-channel input data from a CFA (after Bayer interpolation, or demosaicking) that is then transformed to the target standard color space and pleasing colors, where the output is also on three channels. The color pipeline described herein is able to process four-channel input data, but can also be used to process image sensor readings on three channels or more than four channels.

The color processing pipeline receives as input image sensor readings 901. The image sensor readings may be on three, four, or greater than four channels. The image sensor readings received by the processor may be on three channels, where a conventional CFA is used (after demosaicking), or on four channels where image sensor readings using the CFA described previously is used as an input. In the processing of conventional CFA data, demosaicking forms the full resolution image from the Bayer sampled RAW image. Between the sensor and the demosaicking module, four-channel data (RGGB) can be transferred.

In this example, a lens shading correction module (LSC) 902 corrects the color non-idealities in the image sensor readings (i.e. performs color shading) caused by the incident angle of the light and spatial location of the sensor. The LSC module may operate on data input on two different green channels separately. The resulting image 903 has these aspects corrected. Alternatively, the image sensor readings may proceed to the transform module 904 directly.

The transform module 904 applies a first transformation (the color transform) to the image sensor readings. The first transformation transforms the image sensor readings from the source color space (the color space of the image sensor of the camera that captured the image) to a target color space. The target color space may be one of the sRGB color space, the Adobe RGB color space, the Rec2020 color space, the DCI-P3 color space, or any other suitable color space. The first transformation may be performed by applying a matrix to the image sensor readings. Alternatively, the first transformation may be applied by another method, such as using a lookup table. The color transform can be calculated in many ways or may be replaced with a different transform method. Where a matrix is used, the matrix may be a 3x3 matrix or a 3x4 matrix, depending on the number of input channels of the image sensor readings (i.e. when a three-channel or four-channel CFA is used).

Where the four-channel CFA described above is used, the first transformation may be a predetermined transformation, because in this case the color transform may be independent of the illumination color of the source image.

Once the image sensor readings have been transformed to the target color space, resulting in image 905, the illumination color of the scene of the source image is estimated and adaptation is performed at the adaptation module 906. At the adaptation module 906, a second transformation is performed to transform the set of transformed readings based on the estimated illumination color to form an output image 907. The output image 907 may represent the scene of the source image under a canonical illuminant (white light). Incomplete adaptation models may also be used. In this case, the image is corrected so that when looked under the final color space white point colored illumination (for example, standard color space compatible display white), the image sensation is the same as when the human observer adaptation happened under the original illumination color. The illumination color for the source image is therefore estimated and corrected for subsequent to performing the first transformation.

The illumination color of the scene may be estimated based on the image sensor readings. This may be performed using an AWB algorithm, an illuminant estimation algorithm, or other traditional methods. Alternatively, the scene illumination color may be estimated based on readings from an external spectral sensor. The readings from the spectral sensor are illustrated at 910 in Figure 9. It may be assumed that the scene illuminant is the same as that recorded by the spectral sensor. In one example, the spectral sensor can measure the illuminant color. Then the optimal transform can be calculated for all possible (or significant) surface reflectance colors. In another example, the spectral sensor can record the reflected spectrum (or spatial spectra) and the transform can be optimized for the specific scene colors only. A combination of these approaches is also possible. Therefore, the illumination color of the scene, and hence the second transformation, may be determined from the spectral sensor readings. The scene illumination color may also be a mixture of illuminant and scene content, so the second transformation may be determined in dependence on a combination of the image sensors readings and the spectral sensor readings.

In an additional beneficial hardware implementation, with a wide field of view (FOV), the spectral sensor information can be used to estimate human observer white balance. As this is performed in human space, any error in this process may be more natural. With FOV close to human observer, the error is naturally similar to what was observed. With a narrow FOV, the error can be made more natural by, for example, reducing the degree of adaptation, then the image will naturally be less adapted (e.g. white balanced).

The output of the image signal processor is image data in, for example, an RGB color space or a YUV color space. Other color spaces may be used.

As shown in Figure 9, the image signal processor is optionally configured to enhance the color of the output image at an enhancement module 908, using for example a 3D lookup table (LUT), to form an enhanced image 908.

If a traditional three-channel CFA input to the ISP is used, the color transform (first transformation) is not illuminant independent. In this case, the external spectral sensor can be used to determine the first and second transformations. The readings from the spectral sensor are illustrated at 910 in Figure 9. It may be assumed that the scene illuminant is the same as that recorded by the spectral sensor and the illumination color may be determined from the spectral sensor readings as described above. Therefore, the illumination color of the scene, and hence the second transformation, can also be determined from the spectral sensor readings. The scene illumination color may also be a mixture of illuminant and scene content, so the second transformation may further be determined in dependence on the image sensors readings. Therefore, in this case, the first and second transformations are based on the spectral sensor readings.

The image signal processor is therefore configured to receive a set of image sensor readings corresponding to a scene of the source image in a source color space on a plurality of channels and perform a first transformation (color transform) in which the set of image sensor readings corresponding to a scene of the source image in a source color space are transformed to a target color space to form a set of transformed readings. After the illumination color of the scene has been estimated, the image signal processor is configured to perform a second transformation (adaptation/auto white balance (AWB)) in which the set of transformed readings are transformed based on the estimated illumination color to form an output image. Therefore, the first transformation is performed to transform the image sensor readings to some predetermined color space, and then the result can be transformed to some other color space. In one embodiment, the first transformation and the second transformation are combined and are applied to the image sensor readings as a single matrix.

The ISP pipeline described herein works more similarly to a human visual system than traditional ISPs. The adaptation (or AWB) is performed later in the pipeline after the color transform. Adaptation is the most challenging and critical part of the color processing. When the color transform is performed before the AWB, any error in AWB will not be enhanced as much as if the AWB was performed before the color transform. Traditionally, the color transform enhances the errors. Any AWB/adaptation error in the standard observer space (i.e. after the transform) tends to be more natural and pleasing to the end user.

The image signal processor described above is particularly advantageous when it receives as input the output of an image sensor which has four channels, wherein three of the channels have outputs formed according to respective first, second and third wavelength-dependent weighting functions which each have a highest positive response peak in the visible wavelength spectrum within 20 nm of a highest positive response peak of a respective one of first, second and third color standard observer functions in a common colorspace, and a fourth channel with an output formed according to a fourth wavelength-dependent weighting function which has a highest positive response peak in the visible wavelength spectrum offset by more than 10 nm from the positive response peaks of the said standard observer functions. It is particularly advantageous when the first, second and third standard observer functions are red, green and blue standard observer functions in a common color space, such as sRGB or CIE RGB. In this case, the fourth wavelength-dependent weighting function preferably has a peak between 510 to 530 nm. The wavelength-dependent weighting functions of each of the channels may be formed as described above.

In some implementations, the same device can have multiple image sensors. In this case the image sensor readings from all of the image sensors can be transferred to the same common color space first. Then one optimization for the AWB algorithm can be used for all sensors, whether the algorithm is Al (artificial intelligence) or learning-free. This is difficult when using the traditional approach, as it requires extra calculations to reduce the differences between the sensors.

The color transform can be optimized by obtaining a set of color spectral responses and calculating how the CFA sees them and how the standard observer sees them. Then a matrix can be optimized that minimizes the transform from the camera colors (source color space) to the standard color space (i.e. colors seen by the standard observer). This can be done, for example, with a simple pseudoinverse function or a gradient descent algorithm. The basic equation is shown below:

Tstd ^— M S_cam (1) where T_std is the target standard colors, M is the transform matrix (3x3 using the output of a traditional CFA, after demosaicking, but with the four-channel CFA it is 3x4), S_cam is the colors seen natively by the camera.

Figure 10 compares the pipeline of the present invention with a traditional image signal processing pipeline (LSC = lens shading correction, AWB = auto white balance, CCM = color correction matrix, LUT = lookup table). In the traditional pipeline, the image sensor readings 1001 are received and LSC is performed at module 1002. AWB is then performed at module 1003 where the scene illumination color is estimated and the image is corrected accordingly. After the AWB, the color transform is performed at module 1004, and the color enhancements can be performed at module 1005 to give the final image 1006.

In such traditional pipelines, it is not possible to transform to the target color space directly because the transform is illuminant dependent and the illuminant is not known at this stage. Therefore, firstly AWB has traditionally been applied that estimates the illuminant and corrects the white point. After this, the color transform could be performed. The transform is traditionally optimized based on the illuminant estimated by the AWB.

In the approach described herein, the color transform is performed before the illuminant estimation and adaptation/AWB. This is possible because, for the image sensor readings output from the four-channel CFA, the transform is illuminant independent. Where a three- channel CFA is used, the color transform can be determined using an external spectral sensor. The extra spectral sensor information enables calculation of the transform on the fly.

The four-channel CFA in combination with the pipeline function in a similar manner to a human visual system (HVS) but are optimized for noise and robust color processing.

The CFA may, in some implementations, enable easier and more accurate color processing. The CFA supports the color processing approach where the camera RGB to standard observer transform is performed first before the adaptation/AWB. No significant changes are needed to the current ISP. The illuminant independent transformation is applied at the beginning of the pipe, after which it works in a typical (but much closer to a standard observer) RGB pipe.

Although the order of the operations is changed in the approach described herein, the old hardware blocks can be utilized. The AWB + CCM or 3x3-4 transform + adaptation transformations can be calculated into a single matrix. The pipeline is therefore different to the traditional color processing pipeline but may utilize the CCM function of traditional color processing.

Furthermore, part of the gaining in the combined CCM can be extracted to be implemented with the AWB gaining block of the ISP. This can solved mathematically as:

CCM1 = CCM2 * WB (2)

Here, CCM1 is a combined color correction matrix which performs both transformations (the color transform and the adaptation). In Equation (2), this matrix is split into two phases, as in a traditional ISP: diagonal white balance gains (WB) and a CCM (CCM2). ImageOut = CCM1 * Imageln is modified as ImageOut = CCM2 * WB * Imageln. In order to keep the total processing effect the same, it is required to solve Equation (2).

WB can be set as a proper diagonal white balancing-type matrix, or any diagonal matrix that can be fed to the ISP AWB gaining block. Then, one possible way to solve Equation (2) is:

CCM2 * WB * WB ¹ = CCM 1 * WB ¹ (3)

Therefore:

CCM2 = CCM1 * WB ¹ (4)

The WB values for the split can be set by solving an optimization problem, with a target of optimizing all desired properties of the processing. For example, minimizing the total gaining of the processing (lower noise benefit), or limiting the CCM gains, if, for example, the ISP CCM block has some maximum coefficient limit. There may be also some other optimization criteria. For example, there may be some other processing algorithms, such as highlight recovery, that may benefit from a certain range of gaining in the ISP white balance block. The optimization of the white balance part of the split may be performed in different ways. For example, using an analytical mathematical method or an iterative method. The reverse processing order (color transform before AWB) in the ISP can also be utilized for transforming the data into some camera independent color space. This can, for example, minimize the Al training effort (same training for multiple cameras).

Figure 11 shows a flowchart illustrating an example of a method for transforming image sensor readings. At step 1101 , the method comprises receiving image sensor readings corresponding to a scene of the source image in a source color space on a plurality of channels. At step 1102, the method comprises performing a first transformation to transform the image sensor readings to a target color space to form a set of transformed readings. At step 1103, the method comprises estimating an illumination color for the source image. At step 1104, the method comprises performing a second transformation to transform the set of transformed readings based on the estimated illumination color to form an output image.

Figure 12 shows an example of an imaging device, such as a camera, configured to implement a device for processing a set of color pixels of an image using the CFA and the image signal processor to process images taken by an image sensor 1202 in the device 1201. Such a device 1201 typically includes some onboard processing capability. This could be provided by the processor 1204. The processor 1204 could also be used for the essential functions of the device. The device typically also comprises memory a 1203.

The transceiver 1205 is capable of communicating over a network with other entities 1210, 1211. Those entities may be physically remote from the device 1201. The network may be a publicly accessible network such as the internet. The entities 1210, 1211 may be based in the cloud. In one example, entity 1210 is a computing entity and entity 1211 is a command and control entity. These entities are logical entities. In practice they may each be provided by one or more physical devices such as servers and datastores, and the functions of two or more of the entities may be provided by a single physical device. Each physical device implementing an entity comprises a processor and a memory. The devices may also comprise a transceiver for transmitting and receiving data to and from the transceiver 1205 of device 1201. The memory stores in a non-transient way code that is executable by the processor to implement the respective entity in the manner described herein.

The command and control entity 1211 may store and/or train algorithms or models used in the pipeline. This is typically a computationally intensive task, even though the resulting model may be efficiently described, so it may be efficient for the development of the algorithm or model to be performed in the cloud, where it can be anticipated that significant energy and computing resource is available. It can be anticipated that this is more efficient than forming such a model at a typical imaging device.

In one implementation, once algorithms have been developed in the cloud, the command and control entity can automatically form a corresponding model and cause it to be transmitted to the relevant device. In this example, the system is implemented at the device 1201 by processor 1204.

In another possible implementation, an image may be captured by the sensor 1202 and the image data may be sent by the transceiver 1205 to the cloud for processing in the system. The resulting target image could then be sent back to the device1201, as shown at 1212 in Figure 12.

Therefore, the method may be deployed in multiple ways, for example in the cloud, on the device, or alternatively in dedicated hardware. As indicated above, the cloud facility could perform training to develop new algorithms or refine existing ones. Depending on the compute capability near to the data corpus, the training could either be undertaken close to the source data, or could be undertaken in the cloud, e.g. using an inference engine. The system may also be implemented at the device, in a dedicated piece of hardware, or in the cloud.

The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

1. A device (1201) for processing a set of color pixels of an image to form, for at least one pixel, an output on four channels, wherein the device is configured to: form outputs for three of the channels according to respective first (601), second (602) and third (603) wavelength-dependent weighting functions which each have a positive response peak in the visible wavelength spectrum within 20 nm of a positive response peak of a respective one of first (301 , 401 , 501), second (302, 402, 502) and third (303, 403, 503) color standard observer functions in a common color space; and form the output for the fourth channel according to a fourth wavelength-dependent weighting function (604) which has a positive response peak in the visible wavelength spectrum offset by more than 10 nm from the positive response peaks of each of the said standard observer functions (301 , 302, 303, 401 , 402, 403, 501, 502, 503).

2. The device (1201) as claimed in claim 1 , wherein the three of the channels are red, green and blue color channels respectively.

3. The device (1201) as claimed in claim 1 or 2, wherein the fourth wavelength-dependent weighting function (604) has a positive response peak between 510 to 530 nm.

4. The device (1201) as claimed in any preceding claim, wherein the first to fourth wavelength- dependent weighting functions (601, 602, 603, 604) each have a single positive response peak in the visible wavelength spectrum.

5. The device (1201) as claimed in any preceding claim, wherein at least one positive response peak associated with each of the first to fourth wavelength-dependent weighting functions (601 , 602, 603, 604) is a local maximum value or a global maximum value.

6. The device (1201) as claimed in any preceding claim, wherein one of the said standard observer functions (301 , 401 , 501) in the common color space has a positive response peak in the red region of the visible wavelength spectrum and a negative response peak outside the red region of the visible wavelength spectrum, and wherein the fourth wavelength- dependent weighting function (604) has a positive response peak within 20 nm of that negative response peak.

7. The device (1201) as claimed in claim 6 as dependent on claim 2, wherein the one of the said standard observer functions (301 , 401 , 501) in the common color space is the standard observer function of the red color channel.

8. The device (1201) as claimed in claim 6 or claim 7, wherein the positive response peak of the fourth wavelength-dependent weighting function (604) has the same peak wavelength as the negative response peak.

9. The device (1201) as claimed in claim 2 or any of claims 3 to 8 as dependent on claim 2, wherein the wavelength-dependent weighting function of the red color channel (601) is negligible when the fourth wavelength-dependent weighting function (604) is greater than zero.

10. The device (1201) as claimed in any preceding claim, wherein the common color space is one of the sRGB color space, the CIE RGB color space, the Adobe RGB color space, the Rec2020 color space or the DCI-P3 color space.

11. The device (1201) as claimed in any preceding claim, wherein each of the first to fourth wavelength-dependent weighting functions (601, 602, 603, 604) has a positive wavelength response.

12. The device (1201) as claimed in any preceding claim, wherein each of the first to fourth wavelength-dependent weighting functions (601 , 602, 603, 604) has a single passband.

13. The device (1201) as claimed in any preceding claim, wherein the device (1201) comprises an image sensor (1202) having one or more two-dimensional arrays of photosensitive elements, wherein the one or more two-dimensional arrays are configured to detect the intensity of light on each of the four channels.

14. The device (1201) as claimed in claim 13, wherein each of the elements comprises a color filter configured to filter light according to one of the first to fourth wavelength-dependent weighting functions (601 , 602, 603, 604).

15. An image signal processor (900) for transforming image sensor readings (901), the image signal processor (900) being configured to: receive the image sensor readings (901) corresponding to a scene of a source image in a source color space on a plurality of channels; perform a first transformation (904) to transform the image sensor readings (901) to a target color space to form a set of transformed readings; estimate an illumination color for the source image; and perform a second transformation (906) to transform the set of transformed readings based on the estimated illumination color to form an output image (907).

16. The image signal processor (900) as claimed in claim 15, wherein the first transformation (904) is performed by applying a predetermined transformation to the image sensor readings (901).

17. The image signal processor (900) as claimed in claim 15 or claim 16, wherein the first transformation is independent of the illumination color of the source image.

18. The image signal processor (900) as claimed in any of claims 15 to 17, wherein the image signal processor (900) is further configured to receive a set of spectral sensor readings (910).

19. The image signal processor (900) as claimed in claim 18 as dependent on claim 15, wherein the first transformation (904) is determined based on the set of spectral sensor readings (910).

20. The image signal processor (900) as claimed in claim 18 or claim 19, wherein the illumination color for the source image is estimated based on the set of spectral sensor readings (910).

21. The image signal processor (900) as claimed in any of claims 18 to 20, wherein one or more of the first (904) and second transformations (906) are further determined based on the image sensor readings (901).

22. The image signal processor (900) as claimed in any of claims 15 to 21 , wherein the first transformation (904) is performed by applying a matrix to the image sensor readings (901).

23. The image signal processor (900) as claimed in claim 22, wherein the first transformation (904) and the second transformation (906) are applied to the image sensor readings (901) as a single matrix.

24. The image signal processor (900) as claimed in any of claims 15 to 23, wherein the image sensor readings (901) received by the image signal processor (900) are on three or four channels.

25. The image signal processor (900) as claimed in any of claims 22 to 24, wherein the matrix is a 3x3 matrix or a 3x4 matrix.

26. The image signal processor (900) as claimed in any of claims 15 to 25, wherein the image sensor readings (901) received by the image signal processor (900) are output by the device (1201) of any of claims 1 to 14.

27. The image signal processor (900) as claimed in any of claims 15 to 26, wherein the target color space is one of the sRGB color space, the Adobe RGB color space, the Rec2020 color space or the DCI-P3 color space.

28. The image signal processor (900) as claimed in any of claims 15 to 27, wherein the output (907, 909) of the image signal processor (900) is image data in an RGB color space or a YUV color space.

29. The image signal processor (900) as claimed in any of claims 15 to 28, wherein the illumination color for the source image is estimated subsequent to performing the first transformation (904).

30. A method (1100) for transforming image sensor readings (901), the method comprising: receiving (1101) image sensor readings corresponding to a scene of the source image in a source color space on a plurality of channels; performing (1102) a first transformation to transform the image sensor readings to a target color space to form a set of transformed readings; estimating (1103) an illumination color for the source image; and performing (1104) a second transformation to transform the set of transformed readings based on the estimated illumination color to form an output image (907).