WO2008086886A1

WO2008086886A1 - Image enhancing device, process for image enhancing and computer-program

Info

Publication number: WO2008086886A1
Application number: PCT/EP2007/050396
Authority: WO
Inventors: Jan Klijn; Johan Schirris; Rien Eradus
Original assignee: Robert Bosch Gmbh
Priority date: 2007-01-16
Filing date: 2007-01-16
Publication date: 2008-07-24
Also published as: EP2105009A1

Abstract

Digital cameras are common in the everyday life and are used as web-cams, as handy-cams, as a substitute of the more traditional celluloid-based cameras, as measurement units and the like. Although digital cameras represent a well-established technology, many technologic challenges are still present. One problem is the well known current leakage in the pixels of the image sensors of the digital camera, which leads to different amplitudes for each pixel. This effect - also known as FPN fixed pattern noise - is dominant especially in CMOS sensors, but also present in CCD sensors. An image enhancing device (1) is proposed comprising a correction module (9) operable to correct the image by applying a correction term to the pixel values of the image and an estimating module (11, 12) operable to estimate the correction term, whereby the estimating module (11, 12) is operable to estimate and/or update the correction term on basis of a residue value from the image and/or a preceding image, whereby the residue value is defined as the sum of differences of the pixels and/or pre-corrected pixels with respect to their neighbours.

Description

Title

Image enhancing device, process for image enhancing and computer-program

State of the art

This invention relates to an image enhancing device, a process for image enhancing and a respective computer-program and more specifically to an image enhancing device, a process for image enhancing and a respective computer-program for enhancing the image from a digital image sensor.

Cameras with digital image sensors are common in the everyday life and are used as web- cams, as handy-cams, as a substitute of the more traditional celluloid-based cameras, as measurement units and the like. Although digital cameras represent a well-established technology, many technologic challenges are still present.

One problem is the well known leakage in the pixels of the image sensors, which leads to different amplitudes for each pixel. This effect - also known as FPN fixed pattern noise - is dominant especially in CMOS sensors, but also present in CCD sensors. In order to compensate this effect storing a reference image under "no light" condition and subtracting the values of the reference image from images taken during normal operation is often performed. But this procedure does not compensate the disturbing effect in a satisfying manner, because the effect is additional temperature dependent, so that the leakage doubles approximately each eight degrees centigrade. A common way to compensate also the temperature dependency is to calculate a correction factor in dependence to a measured temperature of the image sensor and thus to obtain temperature compensated correction values. But also this approach does not solve the problem in its entirety as the measuring of the temperature in a reliable way is difficult. The document WO 99/03262 discloses a method and a circuit configuration for compensating variations in CMOS image sensors resulting from temperature, voltage and production, whereby images of the CMOS image sensor, which shall be compensated, are compared with the images of two further CMOS image sensors at the same temperature, whereby one of the further sensors provides a dark reference image and the other provides a light reference image. On the basis of the dark and light reference image correction values for each pixel of the CMOS image sensor to be compensated are derived.

The document DE 102 39 994 Al, which seems to represent the closest prior art, discloses a process for correction of inhomogeneities of an image sensor system. According to the document, a correction term is generated to correct inhomogeneities in the image from the image sensor system, which are due to the illumination level.

Disclosure of the invention

The invention proposes an image enhancing device with the features of claim 1 , a process for image enhancing with the features of claim 10 and a computer program with the features of claim 12. Preferred embodiments are claimed by the dependent claims and/or disclosed by the following description and the attached figures.

The image enhancing device is preferably connected and /or connectable to an image sensor, especially a CMOS image sensor or a CCD image sensor, which provides a one- dimensional (line) or two-dimensional (field) image. In a preferred realisation the image enhancing device is integrated in the image sensor or in the evaluation electronics of the image sensor.

The image enhancing device according to the invention comprises a correction module, which is realised and/or adapted to correct an image by applying a correction term to the pixel values of the image. The correction term is preferably embodied as a field, for example a two- or multi-dimensional field.

In order to generate the correction term an estimating module is provided, which estimates or generates the correction term. Estimating preferably covers the possibilities that the correction term is generated by an iterative process, by a calculating process, by a look-up process, by a control process etc..

According to the invention it is proposed that the estimating module is realised and/or adapted to estimate and/or update the correction term on basis of a residue value from the image and/or a preceding image.

The residue value is defined as the sum of differences of the pixels with respect to their neighbours. In a preferred realisation, for each pixel considered by generating the residue value, the difference from the value of this pixel to the value of at least one neighbour pixel is calculated and the calculated values are accumulated to form the residue value.

Optionally the image and/or the preceding image is (are) pre-processed before deriving the residue value, especially the image and/or the preceding image is pre-processed by applying a prior correction term to the pixel values and the resulting residue value is used as a parameter to optimise and thus estimate and/or update the actual correction term.

The underlying idea of the invention is to propose a new approach for estimating and/or updating the correction term. In contrast to the techniques known from the state of the art, which are using a temperature measurement, which is difficult to carry out in a sufficient reliable manner, or using two further image sensors for reference measurements, which is nearly impossible as soon as the image sensor is delivered to the end-user, the present invention is based on the assumption that averaged over a complete image the sum of differences of the pixels with respect to their neighbours, is zero. In fact already some tens of pixels are sufficient to fulfil the assumption. Relying on this assumption leads to the consequence that in case the residue is not zero, the correction term is not optimal and should be amended. For this purpose it is possible to employ look-up-tables, open or closed loop control or higher sophisticated control algorithm like neural networks, fuzzy logic or adaptive filtering.

In a practical realisation the estimating module is realised and/or adapted to generate the residue value as a global value preferably for the whole image and/or for a section of the image with connected or adjacent lying pixels. - A -

In a preferred embodiment the summing up or accumulating of the differences is implemented, so that only the differences in a horizontal direction (line direction) or only in a vertical direction (row direction) are considered. In general it is possible to use the differences in vertical and horizontal direction and optional also in the diagonal direction for generating the residue value. But in order to safe processing time and further to allow a fast or real-time calculation a reduced set of differences is selected.

In a preferred development the correction term comprises a field of noise values scaled by a correction factor, whereby the correction factor is derived on basis of the residue value. The correction term is especially embodied as the mathematical product of the two- dimensional field of noise values and the correction factor.

The correction factor is preferably retrieved iteratively, especially by a closed loop control, whereby the image is pre-processed with a stored correction term and the correction factor is calculated on basis of the residue value of the pre-processed image which is then used to generate a new correction term. This procedure is repeated iteratively so that after some iterations the residue is zero and/or the correction factor is stable.

The image enhancing device optionally comprises a referencing module, which is realised and/or adapted to create the field of noise values. The field of noise values may in general be produce by any known technique, so for example by techniques as described in the introductory portion. The field of noise value preferably represents the temperature, voltage, production and/or the fixed pattern noise characteristics or distribution of the image sensor. The size of the field of noise is preferably equal to the size of the image.

In a further preferred embodiment the referencing module is realised and/or adapted to generate the field of noise values by evaluating a dark reference image. The dark reference image is captured when the image sensor is not or not substantially illuminated, especially darkened and or covered. For example using a digital camera the dark reference image is captured by completely closing the camera lens. As the main reason for the noise values is the so-called fixed pattern noise (FPN) and the FPN scales linearly with time a large integration time for measuring or capturing the dark reference image is preferred, so that large FPN values are achieved, which can be measured accurately and have relatively little random noise. In a preferred development, the referencing module is realised and/or adapted to allow only noise value entries unequal from zero in the field of noise values, which are larger than a low threshold and/or smaller than a high threshold. It is noted that the expressions low and high are set for the case that the noise values are positive and increase with time.

In case another sign convention is used, the absolute values have to be applied or the thresholds have to be renamed accordingly.

The advantage of using a low threshold is that not all pixels are corrected but only those over the threshold, which are the severely leaking pixels, so that the number of pixels to be corrected is minimised. As all pixels usually have some random noise, otherwise all or nearly all pixels would contribute to the field of noise values and the corrected image would be deteriorated more than necessary.

The advantage of using a high threshold level is to exclude pixel with a leaking beyond the certain level defined by the high threshold. These pixel are part of areas, which are called white blemishes, and which are preferably removed by an optional blemish correction circuit to exclude them from the above described image enhancement. This exclusion is advantageous as some blemishes are so large that at a certain temperature and image content these areas are clipped in a sensor output amplifier thus being not temperature dependent anymore and so causing a large error.

In a further preferred embodiment of the invention a counter module is employed counting all noise value entries in the field of noise values. Preferably the correction module is only activated in case the number of noise value entries is larger than a predefined relevant number of pixels.

Another subject-matter of the invention is a process for enhancing an image with the features of claim 10, which is preferably carried out on the image enhancing device according to one of the preceding claims and/or as described above.

The process comprises the steps of applying a correction term, which is preferably realised with one, some or all features as described above, to the pixel values of the image and estimating the correction term on basis of the residue value of the image and/or a preceding image, whereby the residue is preferably defined with one, some or all features as described above. In a preferred embodiment, the correction term comprises the field of noise values and a correction factor, which is applied on the field of noise values in order to generate the correction term.

The process is preferably carried out as a closed loop control and/or an iterative control, whereby as a pre-processing step the entries of the correction term are subtracted from the value of the corresponding pixel in the image to be enhanced. In a next step the residue of the pre-processed image is calculated, whereby the differences from the neighbours is measured and summed up over the complete image or frame. A control algorithm, preferably implemented as software, calculates a new correction factor, which is used to calculate a new correction term, so that after some iterations the residue is zero.

The field of noise is preferably generated in a measurement and storage step, whereby a dark reference image is used, which is an image captured when the image sensor is not or not substantially illuminated, for example when the image sensor is covered. The entries of the dark reference image are compared with the low and high thresholds and only entries between these two thresholds are stored in the field of noise. Preferably the remaining empty entries in the field of noise are filled up with zeros.

A further subject-matter of the invention is a computer program comprising program- code means for performing all the steps of the process as described above, when said program is run on a computer and/or on the device as described above.

Short descriptions of the drawings

Further advantages, features and effects of the present invention are disclosed in the following description and drawings of a preferred embodiment of the invention, whereby the figures show:

Figure 1 a block diagram illustrating the image enhancing device as an embodiment of the invention;

Figure 2 a schematic view on a section of an image illustrating a first alternative of the calculation of the residue value; Figure 3 a schematic view on a section of an image sensor illustrating a second alternative of the calculation of the residue value;

Figure 4 a schematic view on a section of an image sensor illustrating the calculation of the residue value by means of examples.

Embodiment of the invention

Figure 1 shows a block diagram of an embodiment of the invention realised as an image enhancing device 1, which functions as a fixed pattern noise correction part.

The image enhancing device 1 comprises an input interface 2 connectable or connected to an unit (not displayed) with an image sensor, for example a CMOS or a CCD image sensor. The input interface 2 is adapted to receive incoming images for example as a video. For outputting the enhanced image an output interface 3 is provided, which is connectable or connected for example with a recorder or a display.

The image enhancing device 1 comprises two functional blocks, a measurement and storage block 4 - also called referencing module - and a correction block 5. From a functional view, the measurement and storage block 4 is operable to generate and store a field of noise values and the correction block 5 is operable to iteratively optimise a correction factor and to correct the incoming images on basis of the field of noise values and the correction factor, so that the correction block 5 comprises a correction and an estimating module.

During initialisation of the image enhancing device 1 the measurement and storage block 4 receives a dark reference image from the input interface 2. The dark reference image is taken by the image sensor from which the further images to be image- enhanced are provided. For the purpose of taking the dark reference image the image sensor is darkened, for example by completely closing a corresponding camera lens. As the main noise source of the image sensor is fixed pattern noise (FPN), which increases linearly with time, the dark reference image is captured with a large integration time, for example more than a second, resulting in large noise or FPN values which can be measured accurately and have relatively little random noise. The dark reference image is guided as an input into a frame memory 6, where selected entries of the dark reference image are stored as a two-dimensional field of noise values. The selection of the entries to be stored is performed by a threshold unit 7, where the pixel values of the dark reference image are compared with a low threshold Threshold low and with a high threshold Threshold high, whereby only entries of the dark reference image are stored in the field of noise values in the frame memory 6 having pixel values between the two thresholds. The remaining empty entries are filled up, for example with zeros.

The underlying idea of using the two thresholds is that pixels with values below the low threshold are not severely leaking pixels. As nearly every pixel has random noise storing without using the low threshold limit would result in that nearly every pixel would contribute to the field of noise values, which would finally lead to a non necessarily deteriorating of the enhanced image. On the other hand side are pixels with values above the high threshold better handled as white blemishes and can be removed by special white blemish correction circuits or algorithms, which are known in the state of the art.

A counter unit 8 is employed to count all pixels with a pixel value between the two thresholds to be certain that a relevant number of pixels contributes to the field of noise values. For example, in case the number of counted pixels are below a first pre-defined number, the correction set forth hereunder is deactivated, in case the number exceeds a second pre-defined number, the thresholds are adjusted or a warning signal is emitted.

After the measurement and storage procedure, the frame memory 6 holds the field of noise values and the normal operation mode of the image sensor is active, whereby the image sensor captures images of real objects etc..

The image captured during normal operation mode is guided to a subtract unit 9, which subtracts a correction term from the image. The correction term is a two-dimensional field and is calculated by multiplying in a multiply unit 10 the field of noise values retrieved from the frame memory 6 with a correction factor. The processed images is sent as the enhanced image fpn corrected video to the output interface 3.

In order to steadily improve the quality of the image enhancement, the correction factor is iteratively improved. For that purpose a so called residue value of the processed image fpn corrected video is determined in a residue unit 11 , whereby first the differences of the pixels with respect to their neighbours are calculated and next the differences are summed up for the complete image to form the residue value.

In order to improve the understanding of the calculation of the residue value reference is made to figures 2 and 3. Figure 2 shows in a schematic view an image to be evaluated line by line with a plurality of pixels p. The differences of the pixels in each line with respect to their neighbours are indicated by the sign "dp". These differences dp are summed up for the complete image to form the residue value. Figure 3 illustrates in the same schematic view as figure 2 another way of calculating the differences, whereby in figure 3 the differences are calculated row by row and summed up as the residue value in a next step. In alternative embodiments of the invention the differences in horizontal and in vertical direction or in further directions are considered to form the residue value.

Figure 4 shows in a schematic view a section of a further image to be evaluated in order to explain the calculation of the difference of a pixel with respect to his neighbours by means of example. The neighbour pixels around a pixel P(0,0) to be currently evaluated are referenced as shown in figure 4. The calculation of the difference may be performed by filters (algorithms) involving no line delay or involving line delay.

A preferred example for a filter without involving a line delay are

Difference = 2*(P0,0) - P(-l,0) - P(I₅O) or

Difference = 2*(P0,0) - P(-l,0) - P(1, 0) - P(-2,0) - P(2,0) + P(3,0) + P(-3,0)

In case line delays are affordable the filter could look like:

Difference = 4*P(0,0) - P(-l,0) - P(l,0) - P(O,- 1) - P(0, 1) or

Difference=8*P(0,0)-P(-l,0)-P(l,0) - P(0,-l) - P(0,l) - P(-l,l) - P(I₅-I) - P(-l,-l) - P(I₅I)

Returning again to figure 1 the block diagram shows that the calculating of the differences and the summing up of the differences is performed in the residue unit 11. The output of the residue unit 11 is output to a software unit 12, which estimates an updated correction factor, which will be used in enhancing the next incoming image. As already explained, it is assumed that the residue value has to be zero for a optimally enhanced image. So the iteration procedure as shown by the block diagram in figure 1 is performed as long as the residue is larger than zero or larger than a pre-defined limit. As soon as the residue value is zero or below the pre-defined value the correction factor will be hold constant. Although the figure 1 shows a special distribution into a hardware and a software section of the image enhancing device 11, the distribution may be made in another way, so it is for example possible to implement the image enhancing device 11 as a software program or as a hard- wired electric or electronic circuit.

Claims

Claims:

1. Image enhancing device (1) comprising

a correction module (9) operable to correct the image by applying a correction term to the pixel values of the image and an estimating module (11, 12) operable to estimate the correction term,

characterised in that

the estimating module (11, 12) is operable to estimate and/or update the correction term on basis of a residue value from the image and/or a preceding image, whereby the residue value is defined as the sum of differences of the pixels and/or pre-corrected pixels with respect to their neighbours.

2. Image enhancing device (1) according to claim 1, characterised in that the residue is calculated for a connected section of the image and/or the complete image.

3. Image enhancing device (1) according to claim 1 or 2, characterised in that the estimating module (11, 12) is operable to calculate the residue by accumulating only the horizontal or only the vertical differences of the pixel and/or pre- corrected pixels with respect to their neighbours.

4. Image enhancing device (1) according to one of the preceding claims, characterised in that the correction term comprises a field of noise values scaled by a correction factor, whereby the correction factor is derived on basis of the residue value.

5. Image enhancing device (1) according to claim 4, characterised in that the estimating module (11, 12) is operable to derive the correction factor by closed loop control.

6. Image enhancing device (1) according to claim 4 or 5, characterised by a referencing module (4) operable to generate the field of noise values.

7. Image enhancing device (1) according to claim 6, characterised in that the referencing module (4) is operable to generate the field of noise values by evaluating a dark reference image.

8. Image enhancing device (1) according to claim 7, characterised in that the referencing module (4) is operable to allow only noise value entries in the field of noise values, which are larger than a low threshold (threshold low) value and/or smaller than a high threshold (threshold_high) value.

9. Image enhancing device (1) according to claim 8, characterised by a counter module (8) counting all noise value entries in the field of noise values.

10. Process for enhancing an image, preferably by using the image enhancing device (1) according to one of the preceding claims, characterised by the steps

applying a correction term to the pixel values of the image

estimating the correction term on basis of the residue value of the and/or a preceding image, whereby the residue value is defined as the sum of differences of the pixels and/or pre-processed pixels with respect to their neighbours.

11. Process according to claim 10, characterised in that the correction term comprises a field of noise values and a correction factor, whereby the field of noise values is generated by evaluating a dark reference image and the correction factor is derived on basis of the residue value.

12. Computer program comprising program-code means for performing all the steps of the process according to claim 10 or 11, when said program is run on a computer and/or on the device (1) according to one of the claims 1 to 9.