WO2010149591A1

WO2010149591A1 - Image recording device and method for generating a recording

Info

Publication number: WO2010149591A1
Application number: PCT/EP2010/058643
Authority: WO
Inventors: Michael Schoeberl; André KAUP
Original assignee: Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.
Priority date: 2009-06-22
Filing date: 2010-06-18
Publication date: 2010-12-29
Also published as: DE102009029868B4; DE102009029868A1

Abstract

The invention relates to an image recording device designed for generating a recording in an illumination time window, and comprising an optoelectric sensor, a reader for substantially continuously reading out and resetting the optoelectric sensor at least once within the illumination time window in order to obtain at least two readout values for the illumination time window, and a combiner for generating the recording by forming a sum of the at least two readout values and reducing the sum by a factor less than 1, so that a sensitivity of the digital camera is reduced in a neutral density manner.

Description

An image picking device and method for producing a recording

description

The present invention relates to image pickup devices, e.g. Digital cameras, such as turn digital still or video cameras and a method for generating a recording.

In digital imaging, correct exposure plays a crucial role during image acquisition. The received light quantity or light energy Q _v should be within a certain range, because electronic circuits only allow a certain minimum value (min) and a maximum value (max), which is due to the total noise in the system or the saturation charge (fill well Capacity) of a sensor are determined. Objects below a certain level will not be visible in the camera noise, and for objects above the maximum, the corresponding pixels will provide clipped or truncated values. In both cases, all details of these image areas will be lost and no image postprocessing will be possible. The ratio between the dynamic range 900 of a digital camera, the upper limit max and the lower limit min thereof, and the consequences for areas of an image where the corresponding pixels are too bright or too dark in the case of conventional exposure over a shutter speed window of a particular one Time duration or exposure time with respect to the light energy Q _v is shown once again in FIG. 12. The region 900 between minimum and maximum light energy, which leads neither to saturation of the energy store or charge store of the sensor nor to a non-visible, because in the noise sinking rash leads is called dynamic range 900 of a camera system.

For taking a picture, a sensor integrates the light flux F during the exposure time. In other words, the sensor accumulates an amount of energy corresponding to the incident luminous flux F, e.g. a charge amount. The amount of light thus accumulated becomes too

Q _v = \ Fdt (1) If the exposure time is fixed, the dynamic range limits max and min immediately set the dynamic range limits with respect to the light flux. FIG. 13 shows the luminous flux dynamic range limits or the dynamic range with respect to the luminous flux F using the example of a short and a long exposure time. Consequently, a correct exposure can be controlled by the exposure time for a given scene. The light flux is controllable by suitably adjusting the scene, by additional optical filtering or by adjusting the aperture of the lens.

Unfortunately, in the case of a high-end moving picture image, many of these factors are mostly fixed: namely, the lens aperture simultaneously adjusts the depth of focus range, which in turn is important to the scene composition. Conversely, the exposure time determines the motion blur required for a natural looking scene in motion picture recording. An "ISO setting" of a camera will produce images of different brightness, but only using an internal gain that resizes the image data values.The underlying sensor sensitivity will persist, or at least the dynamic range will remain unchanged, because lost information will be lost due to over-resolution. and underexposure will continue to be lost.

So all that remains is the amount of light from the scene that would be adjustable for correct exposure. In a studio, the brightness of the light may be adjustable. For outdoor shots, this is already no longer possible. Most often, additional optical filtering is used to reduce the sensitivity of the digital camera. Specific neutral density (ND) filters can be used to remove a certain amount of incident light. However, true color neutral and homogeneous optical filters are expensive and are a delicate or fragile piece of equipment. Such filters increase the weight and size of the camera system. So far, there is no electronic option for adjusting the sensitivity in a neutral density fashion.

Apart from optical filtering, there are several other options for capturing scenes with a higher dynamic range. For example, a combination of frames is used:

In MAN, Steve; MANDERS, Corey; FUNG, James: Painting with looks: Photography images from video using quantimetric processing. In: Proceedings ofthe tenth ACM International Conference on Multimedia, New York, NY, USA: ACM, 2002, pp. 117-126, for example, the combination of several linear recordings for the production of stand images with a higher dynamic range. However, this procedure works only for static scenes, since each moving object is otherwise recorded at different positions. This in turn would result in image artifacts if images of moving scenes were to be created in this way.

In Liu, Xinqiao; Gamal, Abbas E .: Synthosis of high dynamic wrestling motion blur free imprint from multiple captures. In: IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications 50 (2003), April, No. 4, pp. 530-539 and KANG, S. B .; UYTTENDAELE, M .; WINDER, S .; SZELISKI, R .: High dynamic ranks video. In: ACM Transactions on Graphics 22 (2003), No. 3, pp. 319-325, for example, a post-processing of video sequences to improve or increase the dynamic range is described, whereby these methods are aimed at a reconstruction without motion blurring. However, this is not desirable for moving pictures because the correct generation of motion blur is essential to the perceived naturalness of a scene.

In WANG, Hongcheng; RASKAR, Ramesh; AHUJA, Narendra: High dynamic ranks video using split aperture camera. In: IEEE 6 ^ι Workshop on Omnidirectional Vision, Camera Networks and Non-classical Cameras OMNIVIS, 2005 In turn, solutions are described that preserve motion blur but require at least two cameras, additional optics, and alignment.

Special architectures for image sensors for detecting a higher dynamic range of a scene are also already known.

In KAVUSI, Sam; GAMAL, Abbas E .: Quantitative study of high-dynamic-range image sensor architectures. For example, specialized sensors for detecting increased dynamic range are compared, including special logarithmic image sensors, in SPIE Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications F Vol. 5301, SPIE, 2004, pp. 264-275. which can capture a scene directly with a large dynamic range. Unfortunately, dark sensors and fixed pattern noise (FPN) are difficult to compensate for such sensors.

The setting of the photo-side charge capacity, as described in DECKER, S .; MCGRATH, D .; BREHMER, K .; SODINI, CG .: A 256x256 CMOS imaging array with wide dynamic ranks pixels and column-parallel digital output. In: IEEE Journal of Solid-State Circuits 33 (1995), Dec, No. 12, pp. 2081-2091, produces a similar behavior. Such an approach requires modified sensors and, because of the uneven response to light for different exposure times, may lead to additional artifacts on moving objects.

Spatially varying exposures, as described in NAYAR, S.K .; MITSUNAGA, T .: High dynamic ranks imaging: spatially varying pixel exposures. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Vol. 1, 2000, pp. 472-479 and US Pat. No. 5,789,737 reduces the sensitivity for some pixels. This, in turn, reduces the resolution for all operating conditions and in turn requires specialized sensor hardware and image reconstruction algorithm settings.

It would therefore be desirable to be able to reduce sensitivity or adjust sensitivity, which would provide a broader range of applications in terms of sensor technology, applicable image processing and / or scene type, possibly even leading to better quality images, providing a sensitivity setting that is less or not at all dependent on aperture , Scene brightness and / or exposure time setting and it is easy to implement.

The object of the present invention is therefore to provide an image pickup device and a corresponding method for producing recordings that fulfill this desire.

This object is achieved by an image recording device according to claim 1 and a method according to claim 12.

A central idea of the present invention is that a reduction in sensitivity can be carried out relatively inexpensively, with dynamic range advantages and with little effect on remaining image acquisition settings, by reading readings by reading the optoelectrical sensor several times, namely reading out and resetting the optoelectric sensor of the image recording device at least once during an exposure time window and once at the end of the exposure time window, are used to produce a shot, a sum is formed over these at least two readout values and the sum is reduced by a factor of less than 1, so that sensitivity of the image pickup device is neutral density reduced. The gaplessness guarantees that the result of recording a conventional recording with continuous accumulation within the shutter speed window is substantially the same as essentially the whole Luminous flux during the exposure period contributes to the result. The FPN noise is reduced due to the shorter sub-exposure intervals between the read-out / reset times. It also remains correctable. The dynamic noise is also reduced due to the incoming averaging via the readings. The result is a reduced lower limit to the flow of light that still falls within the dynamic range. Conversely, the upper photocurrent barrier is increased due to the shorter Teilbelichtungsintervalle. The effort for implementation is simple, since only a few additions and one multiplication per pixel are sufficient.

Preferred embodiments of the present invention will be explained in more detail below with reference to the accompanying drawings. Show it:

1 is a schematic block diagram of an image pickup apparatus according to an embodiment;

FIG. 2 shows a schematic illustration for comparing a continuous exposure and a division of an exposure into exposure subintervals according to an exemplary embodiment; FIG.

3 is a schematic representation of the resulting dynamic ranges for continuous exposure over a long exposure time window, continuous exposure over a short exposure time window, and subdivision of the long exposure time window into shorter subintervals according to one embodiment;

4 is a block diagram of an image pickup device according to an embodiment;

FIG. 5 is a graph comparing the dynamic noise of continuous exposure and exposure with the division of the exposure time window into FIG

Sub-intervals;

6 is a graph comparing the FPN noise with continuous exposure and dividing the exposure time window into subintervals;

7, 8 and 9 each show a frame of test exposures with continuous exposure in long exposure time windows, continuous exposure in short exposure time. illumination time windows or with exposure in subintervals of an exposure time window;

10 is a schematic representation of a still camera according to an embodiment;

11 is a schematic representation of a video camera according to an embodiment;

Fig. 12 is a schematic diagram illustrating a dynamic range in continuous exposure; and

13 shows a schematic representation for comparing the dynamic ranges for long and short continuous exposure.

Referring first to Fig. 1, an image pickup apparatus according to an embodiment of the present invention will be described. The image pickup device of FIG. 1 includes an opto-electrical sensor 12, a readout 14, and a combiner 16. The image pickup device of FIG. 1 is for generating a pickup 18 in an exposure time window and outputting it at an output 20. For this purpose, the read-out 14 and the combiner 16 are connected in the order mentioned between the opto-electrical sensor 12 and the output 20.

The optoelectronic sensor 12 is, for example, an image sensor with an array of pixels 22. The optoelectric sensor 12 or its pixels or pixels 22 are designed to integrate light incident during an exposure time, by

Conversion of light into electrical energy and accumulation of the latter. For example, electron-hole pairs are generated in pn junctions of the pixels 22 and stored capacitively. The relationship between accumulated charge and amount of incident light may be linear or have another function. Preferably, the aforementioned

Relationship for all pixels 22 is the same. Due to the manufacturing process, however, there are differences between the individual pixels 22 which lead to a dark image or FPN noise typical for the optoelectric sensor 12. In addition, dynamic noise occurs, which is essentially statistically independent of each other during successive exposures.

The image pickup device of Fig. 1 is adapted to produce the photograph 18 in a predetermined exposure time window. The exposure time window can be For example, be set by a shutter 24 beyond the photoelectric sensor 12 as a period in which incident light 26 in the image pickup device can hit the optoelectric sensor 12, whereas the shutter 24 otherwise prevents this impact. The shutter 24 may be, for example, a mechanical shutter such as a mirror in the case of a digital SLR camera or a rotary shutter in the case of a video camera. However, such an external closure 24 to the optoelectric sensor 12 is not necessarily present. The optoelectronic sensor 12 itself can be controllable such that the pixels 22 have their photosensitivity during the exposure time window and are otherwise sensitive to light.

The read-out 14 is designed to subject the optoelectronic sensor to a substantially continuous readout and reset operation several times within the exposure time window in order to obtain at least two readout values per exposure time window and for each pixel 22. In other words, the read-out 14 is designed to subdivide the exposure time window into partial exposure intervals, and to read the optoelectrical sensor 12 at the end of each partial exposure interval and then reset it again. For readout and reset, the readout 14 requires a certain amount of time during which there is no accumulation of light-induced electrical energy. However, since the read-out 14 reads out and resets substantially completely, the ratio between a sum of the time duration of the partial exposure intervals and a duration of the entire exposure time interval extending from the beginning of the first partial exposure interval to the end of the last partial exposure interval is, for example, between 1 and 0 , 9 including.

The combiner 16 is configured to generate the image 18 and by forming a sum over the at least two readings per pixel 22 and reducing the sum by a factor of less than 1, so that a sensitivity of the image pickup device is neutral density reduced. It may be that the combiner 16 obtains the readout values from the readout 14 in analog form, digitized and then added to obtain the sum in digital form, reduced by a factor of less than 1 by, for example, subsequently multiplying by the factor. An example of this will be explained later with reference to FIG. 4. The combiner 16 may reduce the sum in the digital form by the aforementioned factor and output it at the output 20. Alternatively, the combiner 16 could be designed to form the addition or sum in analog form and make the digitization later. Due to the division of the exposure period into partial exposure intervals, as will be explained in more detail below, each readout value compared to an integration over the entire exposure period only later, ie at a higher luminous flux, to a saturation threshold, for example, by the Ana- log / digital converter for digitizing the readout values in the combiner 16 (not shown in FIG. 1), or by the size of the capacitance for accumulating the amount of electrical energy corresponding to the amount of incident light within the optoelectric sensor 12 or readout 14. On this occasion, it should be noted that the aforementioned energy storage can be located either in the optoelectric sensor 12 or in the reader 14, or that there is a corresponding energy storage in both, with a re-storage or transhipment to the read / return network operations takes place.

Due to the shorter partial exposure intervals, the FPN noise of the optoelectric sensor 12 is less noticeable. Further, due to the mean-valued calculation of the readings in the combiner 16, the dynamic noise is reduced. This, in turn, causes the resulting receptacle 18 to be more sensitive to faint signals. Overall, the sensitivity of the image recording device of FIG. 1 is reduced compared to a continuous exposure in the exposure time window. Such a neutral density reduction of sensitivity would otherwise have required one of the more complicated measures of the introduction to the description, e.g. the installation of a neutral optical density filter. Conversely, the other image properties, such as the motion blur, in the shot obtained from the split exposure values, is the same as in continuous exposure photography over the exposure time window because substantially the entire amount of light is used.

The image pickup device of FIG. 1 may be, for example, a still camera, an example of which is shown in FIG. 10, or a video camera, which will be described with reference to FIG. 11. However, the image pickup device of Fig. 1 may, of course, also be integrated with other electronic devices, e.g. in a cell phone, a laptop, a PDA, a game console, etc., or in optical or medical gauges.

It has not yet been discussed in FIG. 1 that the number of read-out processes per exposure time window could, of course, also be adjustable. In this way, it would be possible to realize an adjustability of their sensitivity for the image pickup apparatus of FIG. 1, wherein the Einstellbarkweit hardly requires additional measures since the required operations remain essentially the same, such as summation and reduction by a factor. The factor by which the sum of the read-out values is reduced falls preferably monotonically with the number of read-out processes. For example, in one example, the factor is 1 / x, where x = N ± 5% of N or x = VΪV ± 5% of Λ [N, where N is the number of readings.

Having now described an exemplary embodiment of an image recording apparatus, the following explains the underlying principle, the resulting advantages, and the underlying physical conditions, which facilitate an understanding of the advantages and the exemplary embodiments described below.

As described with reference to Fig. 1, it is possible to realize neutral density reduction electronically, e.g. even purely digital, if one assumes that the digitization of the readings in image recording devices is usually already provided for. The subdivision of an exposure time window into partial exposure intervals with subsequent summation and reduction of the sum described in FIG. 1 makes it possible to reduce the sensitivity of the image acquisition device while the dynamic range is shifted and possibly even increased, as will be discussed in the following. In the following it is briefly explained why a shift of the upper and the lower limit of the dynamic range results. In particular, it follows from the following explanations that

a reduced sensor sensitivity,

a reduced dynamic noise with consequently increased dynamic range,

a reduced FPN noise with correspondingly increased dynamic range,

a correct reproduction of a movement smearing of moving objects in the recording,

a uniform sensitivity for different exposure times,

essentially no additional artifacts, as well

Realize possibilities of realization using normal, conventional image sensors. All of these advantages can be achieved by setting the sensor read pattern and performing additional pixel based image processing, as described above and below. To explain this, reference is made to FIG. In a), the recording of a single frame with uninterrupted exposure in the exposure time window 30 is shown in FIG. The exposure starts at the time t _\ and ends after a period of time τ _exp at the time t ₂ . That is, before the time t}, for example, the image shutter 24 of the image pickup device is closed, opened between the times tj and t ₂ and closed again after the time t ₂ . During the exposure time window 30, the light integration gives the pixel value I.

At b), a procedure is shown in Fig. 2 whereby four substantially gapless readout / reset operations 32r32 _{4 are} performed during the exposure time window 30 to be exactly three within the exposure time window 30 and one at the end at time t ₂ , whereby the exposure time window 30 is subdivided into four partial exposure intervals τ _N. As can be seen, the last process may be limited to reading and at the beginning of the exposure time window at time t _\ , or the sensor is still reset. The resetting causes a presetting of the aforementioned accumulation memory of the opto-electrical sensor, which is then discharged or charged at a rate corresponding to the instantaneous incident luminous flux. The read results in a readout of the current state of charge of the accumulation memory. The reading out can be destructive, ie change the state of charge, since then again a reset takes place.

In FIG. 2, the exposure time window 30 is subdivided by way of example into four partial exposure intervals 34 _r 34 ₄ of length τ _N , but of course a finer or coarser subdivision with N> 2 is also possible.

Between the partial exposures 34r34 ₄ , the readout / reset operation 32r 32 _{3 is} performed. The readings can be destructive, ie destroy the contents of the aforementioned energy storage or charge storage. The readout / reset operations 32r32 ₃ inside the exposure time window 30 have, as shown in Fig. 2, a readout operation and a reset operation, respectively, which take place substantially directly on each other. Between them no light accumulation occurs. For example, the ratio between the sum of these time gaps and the time duration t _exp is equal to or less than 9/10 or even more preferably less than 99/100. As further shown in FIG. 2, the sub-exposure intervals are 34p 34 ₄ preferably the same length, namely τ _N = l / iV -r _exp . The final value / for the current frame then results according to averaging on a per-pixel basis, for example

_ -i N-I i N-I

The adjustment of N advantageously requires no adjustment of any camera mechanics. No optical properties of the cameras are changed by an adjustment of N. This applies both to the depth of field defined by the lens and the aperture of the image recording device and to the color characteristic and resolution of the camera or the image recording device. Consequently, the interval division of the shutter speed window 30, with summation and reduction, acts as described above, like a neutral neutral density digital density filter.

In the following, a brief description will be made of physical fundamentals in image acquisition that contribute to understanding the advantages of the embodiments of the present invention. The process of image capture is non-ideal and different types of image noise will inevitably degrade the signal. In particular, a distinction can be made between two types of noise which influence the detection of a pixel value I:

¹ = \ _rav h * dt + N _FPN + N _dynamic (3)

The actual image information is detected, for example, by integration of a photocurrent i _ph corresponding to the incident luminous flux over the exposure time r _exp . Moving objects will appear smeared, which is essential for a natural scene appearance. The emergence or lifelike reproduction of the correct Bewegungsverschmierung is essential for high quality motion picture recordings.

The static or FPN noise N _FPN describes systematic interference that can be predicted within a certain range and subsequently removed or subtracted again, as for example in the MALUEC, RM: Detector Array Fixed-Pattern Noise Compensation. April 6, 1976. - U.S. Patent 3,949,162. This noise consists mainly of offset noise N _o ff _S et and dark current noise Ndunk _e i- The exposure time dependency can be modulated with a constant value for dark current i _dark i for each pixel, as described for example in US 7,092,017. The linear dependence of the dark current on the exposure time results in significant image disturbances in the case of long exposure times. Even with compensation of the FPN noise and FPN compensation according to equation (4) for each pixel with pixel-individual dark current idunk _e i two problems remain: even with a perfect FPN noise estimation and removal, the dynamic range of the image will be reduced , This in turn limits the maximum allowed exposure time. Second, for a true camera, the estimate of i _dunke i is unlikely to be perfect for each pixel, which further causes increased noise to occur at long exposure times. In practice, therefore, the FPN noise will increase with the length of the exposure time despite FPN compensation.

Theoretically, therefore, the described measure of the interval division of the exposure time window combined with the sum of the resulting read values and the reduction of the sum results in an image acquisition with a reduced sensitivity, reduced dynamic noise and reduced FPN noise. This is also shown in Table (1), which is explained below. A shorter exposure time by the factor N causes only 1 / N of the incident light to be effective during the exposure. This effectively reduces the sensitivity by a factor of 1 / N. Averaging over the samples does not change the amount of light. If all image information of all exposure subintervals 34 _r 34 _{4 are} utilized or included in the summation and thereby weighted equally, this guarantees a uniform behavior over the exposure time. In particular, the generated motion blur will be equal to that of a single uninterrupted long exposure.

For example, referring to the luminous flux F falling on the optoelectric sensor during the exposure time window, FIG. 3 shows the dynamic range 40 in the case of a continuous long exposure, the dynamic range 42 in the case of a continuous short exposure, and compared to the dynamic range 44 in FIG Case of a dynamic range, as reflected by the digital ND filter described, ie by dividing the exposure time window into several partial exposure intervals with summation over the partial exposure values and decreasing the sum by the number of partial exposure intervals.

For example, the dynamic noise with continuous exposure over the exposure window would be σ _a . If an averaging over N uncorrelated

Random variable a is performed, the mean value α becomes the value

accept. This directly reduces the dynamic noise around VN ^~ as shown in Table 1.

The FPΝ noise for a short exposure time results in a reduction of the dark current FPΝ noise Ν _du nkei by a factor of 1 / Ν. The offset noise N _o ffset remains constant, which only limits the overall reduction of the FPN noise. Averaging over the resulting same amount of FPN noise at each partial exposure value does not affect the noise level.

The result of these theoretical considerations is shown in Table (1).

Table 1: Expected behavior

interrupted or durchge¬

Digital ND stop exposure

Sensitivity N 1 Dynamic Noise N _dyn 4N Dark Current FPN iVd _U "no N

That is, dividing the exposure time window into N equal-sized partial exposure intervals with forming a sum over the partial exposure values and reducing the sum by a factor of l / N, with N the number of partial exposure intervals, results in a continuous exposure with the exposure period N times the partial exposure periods reduced to l / N sensitivity, by 1 / - // V reduced dynamic noise N _dyn and a reduced to l / N dark current FPN noise N _dunke i-

The applicability of this approach is unlimited even for large values of N. However, the increase in image quality is limited by the reduction in dynamic noise due to the constant offset FPN noise. With reference to FIG. 1, a few possibilities of integration into an existing camera system have already been discussed. Basically, supplementing an existing camera system with a device according to FIG. 1 is not difficult. The division of the exposure time window into partial exposure intervals with subsequent average processing of the partial exposure values requires only a very low computational complexity and can be easily integrated into an existing camera system. For example, NI additions and a single multiplication per pixel per generated frame or per shot are sufficient. Integration into an existing camera system requires only an internally higher frame rate than the actual camera frame rate, and the sensor must be capable of a substantially gapless read / reset mode.

For example, all components in the processing direction must be preceded by the cumulative image, such as, e.g. the sensor and the analog-to-digital converter are capable of processing at a rate n times higher than the frame rate. In a typical camera system, an increased internal frame rate is immediately possible: a camera system does not always work at the highest possible frame rate. This is especially true if a regular camera is used for time-lapse recording or if a high-speed camera is used for regular frame rates. Second, some camera systems produce an output frame rate that is lower than the maximum internal sensor readout rate. This is used, for example, in rotary-shutter cameras where read-out is performed as quickly as possible. This reduces shutter artifacts without increasing the required processing power and memory data rate.

The manner in which a gapless and interleaved readout / reset sensor readout mode is implemented depends on the type of sensor with respect to its details. However, typical high-end sensors support this mode. For some sensors, gapless reading requires a mode of operation that results in higher dynamic noise. Even in these cases, however, the division of the exposure time window described here into several partial exposure intervals with subsequent average-value processing of the partial exposure values is advantageous, since an equivalent noise reduction can also be achieved at N = 2.

For proper motion blur reconstruction, the averaging, or more generally the midspecific, because not necessarily the sum division by N performed processing of the partial exposure values is preferably performed in a linear data space. This can for example be accomplished by the Partial exposure images are added shortly after the A / D conversion or subjected to the summation. One possible embodiment is shown in FIG.

FIG. 4 shows an image capture device having a sensor 52, an analog-to-digital converter (ADC) 54, a fractional exposure value buffer 56, such as a sub-pixel. a RAM, as shown by way of example in FIG. 4, an adder 58, a multiplier 60 and a post-processing device 62. Components corresponding to one another from FIG. 4 and FIG. 1 are identified accordingly in FIG. 4. Accordingly, the sensor 52 of FIG. 4 internally comprises an opto-electrical sensor 12 and a corresponding read-out device 14. The partial exposure values 64 output sequentially by the read-out 14 per pixel and exposure time interval are digitized by the A / D converter 54. The partial exposure values 66 thus digitized are successively calculated or summed by the adder 58 with the intermediate result buffered in the intermediate memory 56, the sum result being stored again in the intermediate storage 56 or the contents of the intermediate storage 56 with the result of the current one Addition is updated, as indicated by the arrow 68. Once the N partial exposure values have been summed, the sum, as shown by arrow 70, passes to multiplier 60. There, the sum is multiplied by the factor I / N 72. The result of the multiplication represents the corresponding pixel value 74 which, together with the other pixel values, then represents the raw image which is fed to the further processing 62 in order, for example, to perform an FPN compensation with pixel-individual FPN correction function on the raw image, to perform a color correction and / or perform a color interpolation. Thus, according to Fig. 4, any further processing such as e.g. the FPN removal or a Bayer color reconstruction, performed only when the summation of the partial exposure values and the reduction has been performed with a factor of less than 1 for each pixel. Thus, the operations of the device 62 need only be performed once for the raw image, or in integrating the device of FIG. 4 into an existing camera system, these operations still need to be performed only once per generated frame and not about N times. The integration therefore remains low in terms of its complexity.

For the simulation or experimental test of the digital ND filter outlined above, test sequences with different frame rates and camera modes were recorded. The test scene included a rotating disc with a narrow slot to check motion artifacts and check for gap-free reading. To preserve the original camera noise, uncompressed sequences in VoIl AD resolution, ie 1920 x 1080 pixels, were recorded by a high-end camera system. In order to evaluate the true performance of the digital ND filter, FPN compensation according to US Pat. No. 7,092,017, demosaicing or color interpolation was performed and visual gamma correction was applied.

In particular, an image sequence with r _exp = 16.6 milliseconds was continuous

Exposure over the shutter speed windows recorded per frame. Subsequently, the exposure time was doubled, whereby equivalent light reduction was performed in the optical path. For the digital ND sequences, a single image sequence with gapless readout / reset was recorded with exposure intervals τ _N = 16.6 milliseconds, whereby from the images of this image sequence digital ND sequences with N = 2, 3, 4,. .., 8 were generated.

In all recorded and generated sequences, FPN noise and dynamic noise were analyzed in a homogeneous area of the image. The result of dynamic noise analysis is shown in FIG. It can be seen that the conventional continuous exposure readout during the exposure time window shows substantially constant dynamic noise. For seamless reading, the sensor used requires about 2 dB more dynamic noise. As N increases, there is a direct reduction in noise levels close to the expected ideal curve. The result of the analysis of FPN noise is shown in FIG. For the conventional exposure with continuous exposure during the exposure time window, an increase in the FPN noise with increasing exposure time can be seen, despite the linear FPN compensation used. One in SCHÖBERL, Michael; SENEL, Cihan; BLOSS, Hans; FOESSEL, Siegfried; BUY, Andre ;: Non-linear Dark Current Fixed Pattern Noise Compensation for Variable Frame Rate Moving Picture Cameras. In: 17 th European Signal Processing Conference EUSIPCO 2009, Glasgow, Scotland, United Kingdom, 8 postulated inaccuracy of the linear model is therefore confirmed here. As shown in FIG. 6, with digital ND image capture, subdividing the exposure time window into sub-exposure intervals, and then averaging the partial exposure values, the FPN noise will remain both low and constant. Again, the noise level of the generated sequences closely follows the ideal curve.

The examples in Figs. 7, 8 and 9 show a single image of the sequence for r _exp = milliseconds with N = 4. The upper images of these figures respectively show the complete recorded scene, whereas in the respective figures the lower images are enlarged Show the region of the test template from the left side of the scene. The conventional image exposure with continuous exposure in the exposure time window is shown in FIG. 7 shown. It shows a lot of noise. Optical filtering was used to reduce the amount of light by a factor of 4. In Fig. 8 the result is shown with a single short exposure. The picture pickup has a lower FPN noise. In Fig. 9, the result is shown using the division into four partial exposure intervals with averaging over the partial exposure values. As can be seen in FIG. 9, in addition to the case of FIG. 8, the dynamic noise is also reduced, in addition to the reduction of the FPN noise. The generated image also has the same motion blur as the conventional continuous exposure exposure in the exposure time window. The scene brightness in the case of the digital ND filtering of FIG. 9 is identical to the result by the optical filtering as used in FIG. That is, the sensitivity could be reduced by a factor of 4, with both dynamic and static noise improved by 4 dB and 9 dB, respectively. This reduction in noise corresponds to an increase in the dynamic range, as described above.

Thus, the above-described ND digital filtering has many advantages over the conventional long continuous exposure: no optical filter is needed to reduce the amount of light. The handling, the weight of the equipment and finally the cost of a camera package can be improved. The digital implementation is also computationally simple and can be easily integrated into a camera system and requires no specialized sensors. The described readout can also be used for still camera cameras in the consumer sector and allows still another option for adjusting the amount of light in a scene. This could directly enable long exposure photography during daylight where a typical aperture stop still lets in too much light.

The image noise levels are also reduced instead of increasing as in the conventional exposure. This eventually provides a higher dynamic range. An FPN compensation algorithm is required only for a smaller range of exposure times. The dark current calibration and compensation requirements are consequently lower than with respect to the wide bandwidth of exposure times. The successive short exposures do not suffer from the dynamic range limits due to dark current and exposure times even over the maximum useful exposure of the sensor can be generated.

The described embodiments can thus be realized by combining a gap-free sensor readout mode with the image processing, with reduction of an image sensor sensitivity. The generated images can increase the image quality. quality and the correct Bewegungsverschmierung show. This shift of the dynamic range is possible without the need for additional mechanics. The generated images also provide unique features, such as preserving motion blur, reduced sensitivity without hardware modification, and / or improved image quality.

An image pickup device according to Fig. 1 or 4 with the modifications as described above can be used for both motion pictures and still pictures. For video cameras, the exemplary embodiments provide a way for a user-selectable operating mode to reduce the sensitivity of the camera. The implementation is just firmware-based and does not interfere with regular camera operation. This allows the recording of time-lapse scenes even during sunlight. The described image acquisition can also be used for high speed cameras operating at lower frame rates. Optical filtering is not necessary and nevertheless any exposure times and lens apertures can be used.

For digital still cameras, the presented digital ND filtering can also be used. It provides yet another way to determine the best exposure setting, in addition to the lens aperture and exposure time setting. For example, if the sensitivity range adjustable by the lens aperture and exposure time is insufficient, the variation of the number of the above-described subintervals may be used to adjust the exposure. The range of aperture stops for a lens could even be reduced, which would save costs, and the digital sensitivity reduction, as described above, could be used instead.

FIG. 10 shows by way of example a digital still camera 80 in which the device of FIG. 1 or 4 is to be integrated, these components not being shown in FIG. 10 for the sake of clarity. The camera 80 may be a single lens reflex camera or a viewfinder camera. It has a lens 82, a shutter 84, such as a lens. a

Slat diaphragm, and a controller for manual and / or automatic adjustment of the aperture or f-number, the exposure time τ _exp and the number N of

Partial exposure intervals and thus to adjust the sensitivity. In the case of Fig. 10, the controller exemplifies three user-operable rotary encoders

86, 88 and 90 as input interfaces. Via the controller 86 and 88, the user could, for example, manually adjust aperture and exposure time r _exp, where it would be possible via the controller 90, regardless of these settings Set camera sensitivity by setting the number N via the controller 90. Additionally or alternatively, the camera 80 could include one or more of the following automatic programs. According to a first type of automatic program, it is the

Users can use a rotary control or an alternative input to set one of the parameters Aperture, Exposure time r _exp, and Number of subexposure intervals N, with the controller automatically adjusting the other two parameters. According to a second type of automatic program, the ratio between automatically set parameters and manually set parameter is reversed, ie, the user sets two manually and the remaining one is set automatically by the controller. According to fully automatic, all three parameters would be set automatically by the controller. As further shown in Fig. 10, the camera 80 has a trigger 92, upon actuation of which the user initiates the exposure or sets the beginning of the exposure time window.

Of course, there are various alternatives to the rotary encoders shown in Fig. 10, such as e.g. Toggle switch or other user input interface. In particular, in a fully automatic camera, the controller may be completely absent and the controller included only a not shown in detail in Fig. 10 processor for automatically setting the three parameters aperture, exposure time and number of Teilbelichtungsinter-.

FIG. 11 shows a video camera 100, into which a device according to FIG. 1 or FIG. 9 is integrated, again the components of these figures in FIG. 11 are not shown for the sake of clarity. In Fig. 11, however, the same reference numerals as those of Fig. 10 have been used for elements performing the same function. Thus, the digital camera 100 comprises a lens 82, a shutter 84, and a controller for controlling the aperture, exposure time r _exp and the number N, to which the controller in turn may have user input interfaces 86, 88 and 90.

In FIGS. 10 and 11, dashed lines indicate that diaphragm aperture, exposure time duration r _exp and number of partial exposure intervals N could be manually adjustable. As mentioned above, automatic or mixed auto-manual adjustability of these three parameters would also be possible. In the case of FIG. 11, it is shown that the exposure time duration r _exp can be smaller than the image repetition time duration tfr _at e _>, so that there are dark phases or exposure-free phases 102 between the exposure time windows. Possibly. In the case of the camera 100, the image repetition time or the frame rate or image frequency can also be set. Referring to both embodiments of Figs. 10 and 11, the camera may, for example, make an automatic adjustment to the brightness of the scene. For this purpose, a fully automatic system, for example, not only adjusts the exposure time and aperture suitably, but also the exposure time, aperture and number N or the factor 1 / x with which the sum of the partial exposure values is multiplied.

When changing one of the values manually, such as changing the exposure time or frame rate, the controller (not shown) could adjust the other values so that the image brightness remains the same. At a speed ramp when filming with the video camera of FIG. 11, ie a continuous change of the frame rate, the image brightness could thus be kept constant. If, for example, the frame rate is reduced and thus the exposure time r _{exp is} increased in an inherent manner, the controller (not shown), for example, ensures automatic adjustment of the diaphragm. It is also conceivable that in a camera, such as that of Fig. 10, a parameter is selectable by the user, such as exposure time or aperture, the others then being adjusted by the controller. An automatic adaptation of N and 1 / x depending on the manually selected other parameters is also conceivable. Other combinations, such as manually selecting one of the parameters and adjusting the others.

With reference to Figs. 10 and 11, it should be noted that some of the settings described above as being variable may also be fixed, such as e.g. Aperture and / or aperture time.

Comparing the above-presented embodiments with optical ND filters, the following results. Typical optical ND filters have densities of d = 0.3 and greater. These achieve a light attenuation by a factor of 10 ^d . The light reduction is also sometimes measured in f-stops. Each reduction by a factor of 2 corresponds to a reduction by one f-stop. Table (2) shows in the case of digital LP filtering with division of the exposure time window into N equal partial exposure intervals and averaging of the resulting partial exposure values, which N is necessary for a digital ND-V equivalent to certain optical ND filters.

In addition to these standard optical densities, it is possible to achieve a reduction by any integer factor. For example, an optical density of 0.48 results when N = 3 is set in the digital ND filter. In addition, it is possible to obtain intermediate values of the attenuation: the multiplication of the partial exposure value sum by a factor of 1 / N, as in the preceding embodiments has been mostly used, for example, can be replaced with a multiplication of l / VN. This results in a constant noise level and only VN attenuation. The dynamic range gain is lost due to clipping, but this approach would allow 1.41, 1.73, 2.33, 2.45, and so on. The following table (2) summarizes the typical optical densities and the corresponding required settings of Ν for the digital ND filter.

The above embodiments thus provide many advantages: they can be installed digitally in existing systems and do not require additional hardware or optics. In addition, the realized sensitivity reduction can be switched on and off. The color or brightness dependence is purely neutral: unlike optical filters that can produce a shift in color from the infrared light, the digital filter has none of these problems, as described above. Finally, the above embodiments can be used for any type of integrating sensor, so also for applications without optical imaging, provided that complete reading is possible.

In the foregoing, it has been assumed without separate mention that the readings are linear readings, that is, those which depend linearly on the integrated amount of incident luminous flux. The optoelectronic sensor 12, the read-out 14 and, if present, the ADC within the combiner therefore preferably have substantially linear characteristics, ie a linear relationship between input variables, namely luminous flux, integrated light quantity or analog read-out value, and output variable, namely analog electrical readout value or digitized value, at least as far as the value range is concerned, which lies in the resulting dynamic range. They may also have non-linear characteristics, which which compensates for a linear overall characteristic which maps an integrated luminous flux onto the read-out value to be summed up. The combiner could also include a dedicated nonlinear linearization module, such as an imager (not shown) between ADC 54 and adder 58 in FIG. 4. It could be in the form of a lookup table or analytically linearized. The linearization could be identical for all pixels, whereas the further processing device, for example, still performs a pixel-individual and τ> j-dependent FPN correction, as has already been described. However, it is not absolutely necessary that linear readings are available in the entire dynamic range. The application could ensure externally, such as by the user, that the amount of incident light does not vary only to an extent such that, within this variation, the resulting characteristic of the digital ND filter imaging device is approximately linear.

In particular, it is pointed out that, depending on the circumstances, the inventive scheme can also be implemented in software. The implementation may be on a digital storage medium, in particular a floppy disk or a CD with electronically readable control signals, which may interact with a programmable computer system such that the corresponding method is executed. In general, the invention thus also consists in a computer program product with program code stored on a machine-readable carrier for carrying out the method according to the invention when the computer program product runs on a computer. In other words, the invention can thus be realized as a computer program with a program code for carrying out the method when the computer program runs on a computer.

Claims

claims

An image pickup device, which is designed to produce a recording in an exposure time window, with

an opto-electrical sensor (12);

a readout (14) for reading and resetting the optoelectric sensor (12) substantially at least once within the exposure time window (30) and reading the optoelectric sensor at the end of the exposure time window (30) to obtain at least two exposure time window readings; and

a combiner (16) for generating the image by forming a sum over the at least two readout values and decreasing the sum by a factor of less than 1 so that a sensitivity of the image pickup device is reduced to a neutral density,

wherein the readout is adapted to perform substantially gapless read / reset and read at the end of the exposure time window (30) at equal intervals (T _N ) therebetween and at the beginning of the exposure time window (30), and

wherein a number of the at least two readout values is adjustable and the combiner is configured such that the factor is 1 / x, where x is dependent on and monotonically increasing with the number.

2. Imaging device according to claim 1, wherein the read-out device (14) is designed to output at least two readout values in analogue form, and the combiner (16) is designed to digitize (54) and then to add at least two readout values (58 ) to obtain the sum in digital form and to reduce the sum in the digital form by the factor (60).

3. Image recording apparatus according to claim 1 or 2, wherein the opto-electrical sensor (12), the read-out (14) and the combiner (16) are formed so that the sum of the at least two readout values substantially linearly from one to the opto-electrical sensor incident luminous flux depends.

4. An image picking device according to any one of claims 1 to 3, further comprising a processor operable to receive at the receptacle obtained by the reduced sum.

an FPN compensation,

a color interpolation, and / or

to perform a color correction.

5. An image pickup device according to any one of claims 1 to 4, wherein the combiner is formed such that the factor is 1 / x, where x is in a range of

± 5% around a number of at least two readings around or ± 5% around the root of the number of at least two readings around.

The image pickup device according to any one of claims 1 to 5, wherein said image pickup device is a still camera (80) and has a controller for manually and / or automatically adjusting at least one of an aperture of said image pickup device and a shutter speed window duration is

to allow manual adjustment of the number of at least two readings by a user of the image pickup device independent of the manual and / or automatic adjustment of the at least one aperture and the duration of the exposure time window, or

to perform an automatic adjustment of the number of at least two readout values.

7. An image pickup device according to any one of claims 1 to 5, wherein the image pickup device is a video camera (100) and has a control for manually and / or automatically adjusting at least one of an aperture of the image pickup device and a shutter speed window in frames of the video camera is trained, a manual and / or automatic adjustment of the at least one of the aperture and the duration of the exposure time window independent adjustment of the number of at least two readings by a user of the image pickup device to allow, or

to automatically adjust the number of at least two readings.

8. An image pickup device according to any one of claims 1 to 5 and 7, wherein the image pickup device is a video camera (100) which is formed

Producing a sequence of frames in a sequence of frames, wherein a time duration of the exposure time window in each frame is less than a frame repeat period, such that a dark phase (102) is present between the exposure time window of the frames.

An image pickup apparatus according to any one of the preceding claims, wherein a ratio between a sum of the time duration of partial exposure intervals between the at least one substantially gapless readout and reset of the optoelectric sensor within the exposure time window and the beginning and end of the exposure time window relative to a period of the exposure time window between 1 and 9/10 inclusive.

10. Image recording device according to one of the preceding claims, with a shutter (24) for fixing the exposure time window.

11. Computer or mobile telephone with an image recording device according to one of claims 1 to 10.

12. A method of producing a shot in an exposure time window, comprising the steps of:

essentially gap-free reading and resetting of the optoelectric sensor at least once within the exposure time window and readout of the optoelectronic sensor at the end of the exposure time window by at least two

To obtain readings for the exposure time window; and Generating the image by forming a sum over the at least two read-out values and reducing the sum by a factor of less than 1, so that a sensitivity of the recording production is reduced in neutral density,

wherein the substantially gapless reading and resetting and reading at the end of the exposure time window (30) are performed at equal intervals (T _N ) therebetween and at a beginning of the exposure time window, and

a number of the at least two readings is adjustable and the factor is 1 / x, where x is dependent on and monotonically increasing with the number.

13. Computer program with a program code for carrying out the method according to claim 12, when the computer program runs on a computer.