WO2008037812A2

WO2008037812A2 - Image enhancement method and apparatus therefor

Info

Publication number: WO2008037812A2
Application number: PCT/EP2007/060396
Authority: WO
Inventors: Vyacheslav Chesnokov; Vladimir Dolzhenko; Konstantin Gorokhovskiy
Original assignee: Apical Limited
Priority date: 2006-09-29
Filing date: 2007-10-01
Publication date: 2008-04-03
Also published as: GB0619284D0; GB2442257A; WO2008037812A3

Abstract

An image processing method comprises controlling the gain of pixels of an input image captured at ISO value A or equivalent according to a higher applied ISO value B or equivalent. The gain of each pixel is determined by a spatially- varying image transform, which is restricted such that the maximum gain depends on a relationship between the applied ISO value B or equivalent and the ISO value A or equivalent at which the image was captured.

Description

Image Enhancement Method and Apparatus Therefor

Field of the Invention

The invention relates to image processing, and more particularly to a method of effecting image improvement.

Background to the Invention

The setting of optimal ISO value for given shooting conditions is a familiar procedure for digital camera users. This may be done manually by the user or automatically by an appropriate algorithm in the camera hardware or firmware. The ISO value represents the gain applied to the image signal generated by the camera sensor. The user sets a higher value when shooting in conditions where some or all of the scene is in low light and it is not possible to extend exposure time due to lack of a suitable stabilizer such as a tripod or internal image stabilizer, or when the scene contains moving objects which would be blurred in a long exposure.

Traditionally in a digital camera, gain is applied in the analogue domain before analogue-to-digital conversion (ADC): an ISO value of 1600 has typically four times the gain of an ISO value of 400. Gain may also be applied in the digital domain.

The benefits to the user from increasing ISO value are offset by an increase in noise level, since the gain which is applied to the sensor data increases the level of image and noise equally. In addition, linear or non-linear gain tends to over- saturate bright regions ("highlights") in an image when dark parts ("lowlights") are brightened.

As a result, the expert user or in-camera algorithm must make a decision on optimal ISO level to achieve a balance between positive effects, namely the increase in brightness of lowlights which would otherwise be rendered too darkly, and negative effects, namely the increase in noise level globally across the image and the over-saturation of any highlights present in the image. There is therefore a need to provide a method which combines the advantages of higher ISO in improving visibility of lowlights when exposure time is limited, while minimizing the total noise of the image and preventing over-saturation of highlights.

It is well known that the ISO value in a digital camera may be composed of an analogue component (analogue gain) and a digital component (digital gain). Analogue gain is achieved by an electronic amplifier which increases the voltage of the sensor signal. Digital gain is achieved by multiplying the digital values of the image pixels by a given quantity using a digital processor. The quoted ISO value relates to the total gain applied to the image data emerging from the image sensor. By combining analogue and digital gain it is well known that higher ISO values can be achieved than via purely analogue gain. In the digital domain, different algorithms for applying gain are known. The simplest form of digital gain is the multiplication of the intensity of each input pixel, I₁ by a fixed value to produce an output intensity I₀, for example, I₀ = I₁ * G. Another form of digital gain is the multiplication by a non-linear function, such that low intensity pixels experience a higher amount of gain than high intensity pixels, for example, I₀ = I₁ * G(I) where G(I) is a non- linear function of intensity /.

There are also known various algorithms for the application of a digital gain which varies from pixel to pixel according to the spatial position of the pixel in the input image. These are frequently known as spatially varying gain methods, and include algorithms such as local histogram equalization, Retinex (as described in US Patent 5,991,456), gradient domain optimization and ORMIT

(as described in international patent application publication number WO

02/089060) amongst others. In addition, such algorithms are usually adaptive, meaning that the actual gain values depend on the distribution of pixel intensities in the original image.

Summary of the Invention

The present invention provides an image processing method comprising controlling the gain of pixels of an input image captured at a specified ISO value A or equivalent according to a higher applied ISO value B or equivalent, wherein the gain of each pixel is determined by a spatially- varying image transform which is restricted such that the maximum gain depends on a relationship between the applied ISO value B or equivalent and the ISO value A or equivalent at which the image was captured. Preferably this is achieved by restricting the maximum gain to B/A and the minimum gain to unity. For example, if the spatially- varying gain method is expressed in terms of a nonlinear tone curve for each pixel in the input image, this restriction may be achieved by limiting the maximum gradient of any curve to B/A and the minimum gradient to unity.

In accordance with a second aspect of the invention, there is provided an image processing device for carrying out the method described above. Further features and advantages of the invention will become apparent from the following description of preferred embodiments of the invention, given by way of example only.

Detailed Description of the Invention The present invention sets out to improve the appearance of digital images where a non-zero gain is applied to raw pixel intensities to compensate for low signal intensities produced by the source of the image.

A photographer may traditionally select ISO 400 to capture a well-illuminated scene with a short exposure time, but increase to ISO 1600 to capture a scene with dark details using the same exposure time. Using the invention described herein, the photographer may select a spatially variable ISO, and preferably an adaptive spatially variable ISO, in the defined range A-B, such as 400-1600 where A = 400 and B = 1600. In this case, the image is captured at ISO 400 in the traditional manner. Subsequently, digital gain is applied using a spatially- varying gain method, such as ORMIT. There is a minimum gain of unity in the brightest regions of the input image, and a maximum gain of B/A, which is 1600/400=4 in the darkest regions. In the resultant image, the darkest details have intensity characteristics B, of i.e. ISO 1600, while the brightest details have intensity characteristics of A, i.e. ISO 400, and details in the intermediate range have an effective ISO value between A and B, i.e. between 400 and 1600. Provided the signal-to-noise level of the signal produced by the analogue amplifier is significantly higher than the additional noise introduced by the analogue-to-digital converter (ADC), then the noise level of the resultant image has the following characteristics: the signal-to-noise level in dark regions is equal to or approximately equal to the noise level characteristic of B, i.e. ISO 1600, and the signal-to-noise level in bright regions is equal to or approximately equal to the noise level characteristic of A, i.e. ISO 400. On average, the noise level of the image obtained by the method in accordance with the invention is lower than that of an image of the same scene captured at the upper ISO value in the range.

In addition, it can be demonstrated that the dynamic range of the resultant image is up to four times higher than that of the same scene captured using traditional ISO 1600. The appearance of such an image is such that details which are present in bright areas, which would have lost contrast due to over-saturation at ISO 1600, are well preserved and show the same appearance as if captured at ISO 400. The above can be clarified by the following calculation. The dynamic range D is defined as the ratio of maximum recorded intensity to minimum recorded intensity, defined by level of noise at zero intensity. Capturing a scene at ISO 400 produces an image with value of D given

approximately by D = . = , where R is the resolution of the ADC, σ is the

V^{σ 2 +σ} _^ ² characteristic noise produced by the sensor and analogue gain at ISO400 and

σ _rf is the digitizing noise contributed by the ADC. Usually σ »G_d . Capturing

D the same scene at ISO 1600 gives D = which gives D ~ R/4 if

σ »G _j . Thus the dynamic range captured at ISO 1600 is approximately four

times lower than that at ISO 400.

With the present invention, a scene captured at ISO 400 has space-variant adaptive digital gain applied as described above. In bright regions, noise level is essentially unchanged, due to the nature of the adaptive space- variant transform.

16-Jo ² +G_j In dark areas, noise is increased by a factor , which approaches

unity when σ » σ _d .

As a result, an image captured at ISO 400 using the present invention shows the

same degree of visibility of dark details as an image captured at ISO 1600 but with a noise level characteristic of the lower ISO value, leading to an increase in dynamic range by up to approximately four times. In the case of an image captured at ISO 100, the increase in dynamic range would be sixteen times.

Other embodiments of the invention can also be envisaged. For example, the input image can be accessed from a storage medium.

Claims

1. An image processing method comprising controlling the gain of pixels of an input image captured at ISO value A or equivalent according to a higher applied ISO value B or equivalent, wherein the gain of each pixel is determined by a spatially-varying image transform which is restricted such that the maximum gain depends on a relationship between the applied ISO value B or equivalent and the ISO value A or equivalent at which the image was captured.

2. As claimed in Claim 1, wherein the relationship between the applied ISO value B and the ISO value A is defined as B/A.

3. As claimed in Claim 1 or 2, wherein the spatially- varying image transform is further restricted such that the minimum gain is substantially equal to unity.

4. As claimed in any one of the preceding claims, wherein the pixel intensity corresponds to that of the higher applied ISO value B or equivalent in the darkest regions of an output image and corresponds to that of the input image captured at ISO value A in the brightest regions of the output image.

5. An image processing method according to any one of the preceding claims, wherein the spatially-varying image transform is an adaptive spatially- varying image transform.

6. An image processing method according to any one of the preceding claims, wherein the dynamic range of the output image is maximised by adjusting the strength of analogue gain and adjusting the strength of digital gain such that the output signal-to-noise level is greater than or substantially equal to that produced by the analogue gain while the dynamic range is substantially equal to that of the ISO value B.

7. An image processing method according to any one of the preceding claims, wherein the ISO value A is achieved by an analogue amplifier and the differential gain to provide different effective ISO values for each pixel between A and B is achieved via a digital processor.

8. An image processing method according to any one of the preceding claims, wherein the analogue amplifier provides gain A_A and a digital processor provides gain A_D such that the ISO value A corresponds to the gain A_A* A_D.

9. An image processing method according to any one of Claims 1 to 7, wherein the ISO value A is achieved by an analogue amplifier and the differential gain to provide different effective ISO values for each pixel between A and B is achieved via an analogue processor.

10. An image processing device for carrying out the method claimed in any one of the preceding claims.