WO2010103109A1

WO2010103109A1 - Method for generating an hdr video image sequence

Info

Publication number: WO2010103109A1
Application number: PCT/EP2010/053200
Authority: WO
Inventors: Kilian Jacob; Peter Rieder; Markus Schu
Original assignee: Trident Microsystems, Inc.
Priority date: 2009-03-12
Filing date: 2010-03-12
Publication date: 2010-09-16
Also published as: DE102009001521A1; DE102009001521B4

Abstract

Described is a method for generating a sequence of high dynamic range video images, comprising the steps of: generating multiple image sequences, each of which includes at least one image; and generating at least one image from the individual images of an image sequence, wherein a movement in at least one earlier second image sequence is evaluated and a movement value is determined before generating the images of a first image sequence, and wherein the exposures of the individual images of the first image sequence are adjusted in accordance with the movement value.

Description

METHOD FOR GENERATING A HDR VIDEO LIVE RESULT

The present invention relates to a method for generating an HDR video image sequence.

HDR video image sequences are video image sequences in which the individual images have a high dynamic range (HDR) compared to conventional video images. Such HDR images can be generated by taking a plurality of images with different exposure times from a given camera position, the image information of which is then superimposed. In a picture with a long exposure time, for example, dark details of the photographed environment are displayed in high contrast, whereas in pictures with low exposure time, for example, bright details of the imaged surroundings appear rich in contrast. By combining the individual images created with different exposure times, a picture emerges in which both bright and dark details of the photographed environment are shown in high contrast.

A method for generating an HDR video image sequence is described, for example, in US Pat. No. 6,993,2200 B2.

The object of the present invention is to provide an improved method for generating an HDR video signal sequence and a device suitable for carrying out this method.

This object is achieved by a method according to claim 1.

The method according to the invention for generating a video image sequence with a high dynamic range comprises: generating a plurality of image sequences each having a plurality of images with different exposures; and generating a respective one of the individual images of a sequence of images. In this Method is evaluated prior to generating the images of a first image sequence movement in at least one temporally preceding second image sequence and a movement value is determined. The exposures of the individual images of the first image sequence are adjusted depending on the movement value.

The exposure of an image is affected by both the exposure time, which is exposed when the image is taken, and the size of the aperture. A different exposure of the individual images can thus be achieved by a variation of the exposure time and / or by a variation of the aperture. A "higher exposure" of one image compared to another image is present if a higher exposure time and / or a larger aperture were used to capture one image than for the other image.

Basically, enlarging the aperture results in a reduction in depth of field, and increasing the exposure time results in motion blur. By the exposure can be varied over both parameters, there is a possibility of optimization with regard to adaptation or change of the exposure with regard to the least possible motion blur or the greatest possible depth of field.

Embodiments of the present invention will be explained below with reference to figures. The figures serve to illustrate the basic principle of the invention. The figures therefore only show the features necessary for understanding this basic principle. In the figures, unless otherwise indicated, like reference numerals designate like features having the same meaning.

FIG. 1 illustrates three video image sequences recorded with different exposure times. FIG. 2 schematically illustrates the generation of a

HDR video image sequence from the three video image sequences according to FIG. 1.

Figure 3 illustrates a video image sequence comprising temporally successive images generated at different exposure times.

FIG. 4 illustrates the generation of a plurality of video signal sequences with the same phase of motion from the video signal sequence according to FIG. 3.

FIG. 5 illustrates a method for adjusting the exposure time for the generation of a video image.

FIG. 6 illustrates an example of a device for generating a plurality of video image sequences with motion-dependent exposure times.

FIG. 7 illustrates a dependence of the exposure time on a movement value according to a first example.

FIG. 8 illustrates a dependence of the exposure time on a movement value according to a second example.

Figure 9 illustrates an example of an apparatus for generating a video image sequence having images with motion-dependent exposure times.

FIG. 10 illustrates a dependence of the exposure time on a movement value during the duration of a picture sequence. FIG. 1 diagrammatically illustrates a plurality of-in the example three-video image sequences S1, S2, S3 which each have a number of temporally successive video images. These individual video image sequences form the basis for the generation of an HDR video image sequence. The video images of the different video sequences are recorded with different exposures. For purposes of explanation, for example, it is assumed that video images of a first video image sequence S1 with a first exposure, video images of a second video image sequence S2 with a second exposure, and video images of a third video image sequence S3 with a third exposure are captured, the first exposure being greater than second exposure and wherein the second exposure is greater than the third exposure.

Exposure can be varied by one or more of the following parameters: exposure time, aperture and auxiliary lighting. Exposure is increased if, with the same aperture and the same additional illumination, the exposure time is increased, if the aperture is increased with the same exposure time and the same additional illumination, or if the intensity of the additional illumination is increased with the same exposure time and the same aperture. The additional lighting can be realized for example by a lamp with two or more brightness levels. In order to be able to vary the brightness from image to image, the luminaire should have low delay times or persistence times. This can be achieved for example by a luminaire based on light emitting diodes (LEDs). Such a lamp includes, for example, one or more LEDs, which are turned on or off depending on the desired brightness of the additional lighting.

B is the exposure referred to below. This exposure B is a function of the exposure time explained T _B , the aperture A and the brightness Z of an additional illumination, so that in general:

B = f (T _B , A, Z).

A variation of an exposure B ₀ toward another exposure value B ₀ 'can be achieved by varying any one of the three parameters. The video image sequences shown schematically in FIG. 1 each have the same motion phases. This is illustrated in FIG. 1 in that in each case one image of each of the video image sequences has the same recording time. These recording times are designated in FIG. 1 by to, to ~ T, to ~ 2T. T in this case denotes a time interval between the recording times of the individual images of each of the video image sequences. Pictures of the individual video image sequences S1, S2, S3, which have the same recording time, are referred to below as an image sequence with the same recording time.

The images of an image sequence with the same recording time, which have different exposure times, can be superimposed in a well-known manner to an HDR image. If one carries out such an overlay method for all image sequences with the same recording time, one obtains an HDR video image sequence. Such a procedure is shown schematically in FIG.

1, an overlay unit is shown in FIG. 2, to which the individual video image sequences S1, S2, S3 are fed and which is designed to superimpose the image information of the image sequences with the same recording time and thus to generate an HDR video image sequence S. A method for superimposing individual images of an image sequence with the same recording time into an HDR image and devices for carrying out such a process are basically knows, so that can be dispensed with further explanations.

Image sequences with the same recording time can be produced in that a plurality of receiving devices, such as cameras, are provided, which generate from the same camera Posi tion ^¬ images with different exposure times to the same recording time points. However, such a procedure is complicated insofar as several cameras are needed.

Image sequences with the same recording time can also be generated by taking the individual images of a sequence chronologically from a camera position and then performing a motion-compensated interpolation, by means of which images with the same motion phase from the temporally successive images with different exposure time be generated .

FIG. 4 schematically shows a device for generating video image sequences S1, S2, S3 having the same motion phases from the video image sequence S 'shown in FIG. 3, which comprises temporally successive images with different exposure times. Such a video image sequence S 'with temporally successive images having different exposure times can be generated using a single imaging device, wherein the exposure time is changed from image to image. The frame rate of this video image sequence S 'is higher than the frame rate of the video image sequences S1, S2, S3 shown in FIG. This frame rate of the video image sequence S 'corresponds to the product of the desired frame rate of the later HDR video image sequence and the number of images whose image information is to be superposed onto an HDR image. The individual pictures of in

Figure 3 illustrated sequence S 'can be evenly spaced in time. In this case, based on the Representation according to Figure 3 ki = k ₂ = k ₃ = 1/3. In this context, it should be noted that the superimposition of three images into one HDR image is only an example. Of course, only two or even more than three images that were generated with different exposure times can be superimposed to an HDR image.

FIG. 4 schematically shows a device for generating video image sequences S1, S2, S3 having the same motion phases from the video image sequence S 'shown in FIG. This device comprises a motion estimation and interpolation unit 21, 22, 23 for each of the video image sequences S1, S2, S3 to be generated. Each of these motion estimation and interpolation units is supplied with a subimage sequence of the image sequence S '. These partial image sequences are formed by a multiplexer 20 in time division multiplexing the images of the input image sequence S 'alternately with the individual motion estimation and interpolation units 21, 22, 23, such that the first motion estimation and interpolation unit 21 only has the images, for example supplied with the first exposure time are supplied; the second motion estimation and interpolation unit 22 is supplied with only the images generated at the second exposure time; and that the third motion estimation and interpolation unit 23 is supplied only with the images generated at the third exposure time. The individual motion estimation and interpolation units 21, 22, 23 are designed to perform a motion estimation and, using this motion estimation, to generate image sequences that have the same phase of motion. Such motion estimation and interpolation methods and devices for carrying out such motion estimation and interpolation methods are known in principle, so that further explanations on this can be dispensed with.

In the method according to the invention, the exposures of the images of the image sequence resulting in an HDR image are provided be superimposed, depending on a movement value. The motion value at a given point in time at which an image is generated or recorded is dependent on movement that is present in images of the image sequence that were taken before this point in time. Such a method will be explained below with reference to FIG. 5, including FIGS. 1 and 3.

For the following explanation, let it be assumed that to in FIGS. 1 and 3 denotes a time at which at least one image of an image sequence is to be recorded, the images of which are later superimposed to form an HDR image. In the case illustrated in FIG. 1, in which a plurality of image sequences are generated simultaneously with the same movement phase, a plurality of images are generated at this time to. In the case illustrated in FIG. 3, in which the image sequences with images to be overlaid are generated in temporal succession, only one image is generated at the instant to.

In a first method step 101, it is provided in the method to generate a movement value which is dependent on a movement value of the video image sequence before the time t0. The determination of the movement value comprises the comparison of at least two previously recorded temporally successive images. If, in accordance with the example according to FIG. 1, there are several image sequences S1, S2, S3 with the same motion phases, one of these image sequences can be selected to determine the motion value and the motion value can be determined by comparing two temporally successive images of the selected image sequence. The temporally successive images can follow each other directly in time. However, there may be more pictures between the two pictures to compare.

It is also possible to compare temporally successive images from different image sequences S1, S2, S3 shown in FIG. However, in In this case, different exposures affect the comparison result.

In order to determine motion in an image sequence according to FIG. 3, it is possible to generate several image sequences with the same motion phases from this image sequence S ', to select one of these image sequences and to move by comparing two temporally successive images of the selected image sequence determine.

Moreover, it is also possible to directly compare images of the image sequence S 'with each other, and in particular such temporally successive images of the image sequence S', the same phase of motion with respect to the

Have pictures of the later HDR video sequences. With respect to the example according to FIG. 3, images with the same phase of motion are, for example, images whose time of recording is: to ~ ki ^• Ti ^• T, where i ≥ 0 or images for their recording time: t _o -k ₂ -TiT, with i > 0 or images for their time of admission applies: to ~ k3 ^• Ti ^• T, with i ≥ 0th

The individual images of all previously explained video image sequences each have a number of matrix-like arranged pixels (pixels), to each of which at least one image information value is assigned. In an RGB representation, each pixel has associated therewith three pixel values: a first pixel value for the color red (R); a second pixel value for the color green (G); and a third pixel value for the color blue (B). In a YUV representation, each pixel value is also associated with three pixel values: a luminance value (Y); and two chrominance values (U, V).

A comparison of two temporally successive images for determining the movement value includes, for example, the determination of a distance measure. This distance measure is a measure of the difference between the individual pixels of the associated pixel values. The determination of the distance measure comprises, for example, the determination of the difference between the pixel values of those pixels which are located in the same positions in the two images, the magnitude of this difference and the summation of these differences. The result obtained is also called the sum of the absolute values of the differences (SAD, Sum of Absolute Differences). The distance measure in this case is the higher, the more the image contents of the two images to be compared differ, the more moving contents are thus present in the two images to be compared. Of course, any other measures of distance, such as even powers of the pixel differences, may be used instead of the amounts of the differences.

The comparison of two temporally successive images may comprise the evaluation of only one pixel value associated with each pixel, e.g. only the red value, the green value or the blue value or the y value, the u-value or the v-value. Of course, it is also possible to evaluate all pixel values associated with a pixel and to add the distance measurements thus obtained in order to obtain an overall distance measure.

In the method according to the invention, it is provided in particular to set a difference between the exposures of the individual pictures of a picture sequence depending on the movement value such that the more moving contents the previously generated pictures contain, the lower this difference. This is based on the consideration that with moving content a high depth of field or high dynamics of the displayed images are hardly perceived by the viewer anyway. When generating the images to be superimposed from an image sequence which was generated by a single recording device, for example the image sequence S 'according to FIG. 3, and in which the generation of the images to be superimposed comprises a motion estimation (cf. Figure 4) In addition, there is the danger that the overlaying of images with different exposure times greatly falsifies the image impression in the case of an erroneous motion estimation.

Generally, it is possible to vary the dispersion of the exposures applied to the images of an image sequence, depending on the detected motion. The scattering is determined by the difference between the highest and the lowest exposure, for example a longest and a shortest exposure time or a largest and smallest aperture. For example, Bi, B _2, ..., B _n, Bi ₂ <... _<Bn <B, the exposures with which the images of an image sequence are received and is D = Bi - B _n, the variation range of the exposure, so that D gets smaller with increasing movement.

In another example, it is intended to reduce the number of different exposures used with increasing movement. In extreme cases, if a high degree of motion is present in the previously recorded images, the same exposure can be used for all images of the image sequence.

As an alternative or in addition to a variation of the exposure as a function of the detected movement, it is provided in another example to vary the number of images of an image sequence as a function of the detected movement. For example, it is intended to increase the number of images of a sequence of images when there is a high degree of movement. The exposures of these individual images may differ, and the variation of the exposure may be achieved by varying one of the aforementioned parameters. If the individual images of a sequence are recorded chronologically in succession and the exposure is varied by varying the exposure times for the individual images, then the maximum permissible exposure time pro reduces as the number of images per image sequence increases Image, if a given time is available for taking pictures of a sequence. In order to be able to set a high scattering of the exposure even with an increasing number of images, it is provided in one example to set the exposure over the aperture.

In a further example, it is provided in this case to generate only a single image during a time unit and not several images to be superimposed with the same exposure time.

An example of a device for generating a plurality of video image sequences with the same motion phase and motion-dependent exposure time is shown in FIG. This device comprises three recording devices or cameras 31, 32, 33, which are located at a common camera position. Each of these recording devices has an adjustable exposure time that is dependent on one of the recording device 31, 32, 33 supplied exposure signal El, E2, E3. This exposure time can be adjusted frame by frame. Each of the recording devices 31, 32, 33 has an output at which the image sequence generated by the respective recording device is available. These image sequences can be supplied to an overlay unit 1 (shown in dashed lines), which generates an HDR image sequence from these image sequences S1, S2, S3.

At least one of the image sequences S1, S2, S3 generated by the recording devices 31, 32, 33 is supplied with a motion detection unit which is designed to detect motion in the already recorded images of the image sequence, for example by comparing two temporally successive at least one of the images image sequences. The movement value S41 is available at the output of this movement determination unit 41. This movement value S41 is supplied to an exposure setting unit 43 which, depending on this movement value, provides exposure signals E1, E2, E3 which Exposure times of the individual recording devices 31, 32, 33 generates.

For example, the exposure setting unit 43 is configured to set fixed predetermined different exposure values when the motion value S41 is below a predetermined threshold value S41o and to set equal exposure values when the motion value S41 is above a predetermined threshold value. Such a procedure is shown schematically in FIG. 7, in which the exposure values E1, E2, E3 are shown as a function of the movement value S41. For purposes of explanation, it is assumed here that the greater the motion components S41, the larger the motion components S41 have compared to each other for determining the motion value S41.

In another example, it is provided to set the exposure values E1, E2, E3 so that they approach each other with increasing movement value S41. Such a procedure is shown in FIG. S44 ₀ here denotes a threshold value above which the same exposure values are provided for all display devices 31, 32, 33.

FIG. 9 shows an example of a device for generating a sequence of images with temporally successive images with different exposure time. This device comprises a recording device 34 which has an exposure time which can be set by an exposure signal E. The image signal S 'is available at an output of this device. This image signal S 'is fed to a motion detection unit 44, which is designed to detect motion in the previously recorded image sequence and to generate a motion value S44 taking into account this movement. This movement value S44 is supplied to an exposure setting unit 42. This exposure setting unit 42 is configured to generate the exposure signal E. Unlike the device explained with reference to FIG. 9, the movement setting unit 42 according to FIG. 9 generates time-sequentially different exposure values E, and in each case one exposure value for each image of a picture sequence to be generated by the recording device 34. The timing of the exposure values generated by this exposure setting unit 42 is schematically shown in FIG. T in FIG. 10 designates the time interval between two images of the later HDR image sequence. The multiple images with different exposure times are generated within this period of time T. El in Fig. 10 denotes a first exposure value, E2 a second exposure value, and E3 a third exposure value each applied to an image of an image sequence, the individual images of that image sequence then being superposed on an HDR image. In the illustrated example, the image with the highest exposure time El is taken first, then the image with the next lower exposure time E2 and then the image with the lowest exposure time E3. However, this order is basically arbitrary.

The individual exposure values E1, E2, E3 depend on the movement value S44. The dependence of the exposure values E1, E2, E3 on the movement value S44 corresponds, for example, to the dependencies of the exposure values E1, E2, E3 of the movement value S41 explained above with reference to FIGS.

Claims

claims

A method of generating a high dynamic range video image sequence comprising:

Generating a plurality of image sequences each having at least one image; and

Generating in each case at least one image from the individual images of an image sequence;

wherein, before generating the images of a first image sequence, motion is evaluated in at least one temporally preceding second image sequence and a motion value is determined,

and wherein the exposures of the individual images of the first image sequence and / or the number of images of the first image sequence are adjusted depending on the motion value.

2. Method according to claim 1, in which, depending on the movement value, a scattering of the exposure for the individual images of the first image sequence is varied.

3. The method of claim 2, wherein the scattering decreases with increasing movement indicated by the movement value.

4. The method of claim 1, wherein the number of different exposures used for the images of the first image sequence is dependent on the motion value.

A method according to claim 4, wherein the number of different exposures used decreases as the movement indicated by the motion value increases.

A method according to any one of the preceding claims, wherein the number of images of the first image sequence increases with increasing movement indicated by the motion value.

7. The method according to any one of the preceding claims, wherein the movement is evaluated within a temporally preceding second image sequence.

8. The method according to any one of claims 1 to 6, wherein the movement between at least two temporally preceding second image sequences is evaluated.

9. The method of claim 1, wherein the exposure depends on at least one of the following parameters: duration of the exposure time, size of the aperture, brightness of an additional illumination.