DE202016008887U1 - Image processing unit and computer program for the detection and correction of image errors - Google Patents

Abstract

Image processing unit (13) for recognizing and correcting image errors caused by flickering light information (1,2) in images (S _i , VS _i ) generated by an image sensor (10) with a data memory (12) for storing image data and a data processing unit (11 ), the data processing unit (11)
- for reading in an exposure series which comprises at least one reference image (L _i ) and at least one comparison image (S _i , VS _i ), the images of the exposure series showing the same scene (4) and the exposure time of the reference image (L _i ) being longer than the exposure time of the comparison image (S _i , VS _i ),
and for determining local deviations of the image data of the comparison image (S _i , VS _i ) from the image data of the reference image (L _i ) by locally comparing the image data of the comparison image (S _i , VS _i ) with the image data of the reference image (L _i ),
- and for generating an error mask (E (Si), E (VSi)) as a function of the determined local deviations of the image data, wherein the error filter (E (S _i), E (VS _i)), the defect positions of image defects within the comparison image ( S _i , VS _i ) indicates
- and for determining local correction values for the image data of the comparison image (S _i , VS _i ), each local correction value being assigned to a correction position within the comparison image (S _i , VS _i ) and for determining the local correction values for the image data of the comparison image (Si , VSi) an estimate is applied which is based on the image data of a comparison image time series (S'i N, ..., S'i) held in a data memory (12, 121, 122), the comparison image time series (S'i N, ..., S'i) comprises a number of stored images (S'i N, ..., S'i 1), which have the same exposure time as that Comparison image (Si) at times other than the comparison image (Si) were generated by the image sensor (10),
- And for generating a corrected comparison image (Si, cor, VSi, cor) by inserting the local correction values at the error positions of the comparison image (S _i , VS _i ) indicated by the error mask (E (Si), E (VSi)) by access on image data stored in the data memory (12).

Description

The invention relates to an image processing unit and a computer program for recognizing and correcting image errors caused by flickering light information in images generated by an image sensor.

A large number of light sources are operated in pulsed mode today. If the light source is switched on, the light source changes from an on state to an off state at high frequency or the brightness of the light source changes at high frequency. Such pulsed light sources can in particular be pulsed light-emitting diodes (LEDs for short), which have become extremely widespread in many areas in recent years. Due to its high frequency, the flickering associated with the pulsed operation of the light source, which is also referred to as flicker, is not visible to the human eye. If the light source is switched on, the human viewer therefore has the desired impression that the light source is continuously switched on.

For the acquisition and recording of images or image sequences which contain light emitted by LEDs or other pulsed light sources, image sensors are usually used, the scanning frequencies of which are significantly lower than the pulse frequencies of the pulsed light sources which are usually used. The latter can be a multiple of the sampling frequency. In addition, the exposure times of the image sensors are generally significantly shorter than the period of the pulsed light sources. This is particularly true in the automotive sector, where it is desired to equip vehicles with simple and inexpensive image sensors.

This can result in the image sensor capturing an image which shows the off state of the pulsed light source even though the light source is switched on. When capturing image sequences, it can happen that the image sensor captures long sequences of images that only show the off state of the light source, even though it is switched on. The reproduction of the image sequence recorded by the image sensor therefore creates the impression that the recorded light source is switched off, although it is actually switched on. The captured images or image sequences thus contain image errors which are caused by flickering light information.

Flickering light information in the sense of the present invention means that the light source periodically changes its brightness, as is the case when driving LED light sources or other pulsed light sources. Such flickering of the light source or the light information it emits is also referred to as a flicker. The terms flicker and flicker are therefore used synonymously for the purposes of this application.

For the purposes of the present application, the term exposure time is used synonymously with the term integration time, which is customary in particular in the field of semiconductor image sensors.

The problem described above is of particular practical importance in the acquisition and recording of images or image sequences which contain light emitted by pulsed LEDs. Since the brake and flashing lights of current vehicles are also operated with pulsed LEDs, image errors of the type described above can cause unacceptable safety risks when using image sensors in the automotive sector. This is particularly critical if an electronic camera takes over the function of the rearview mirror in a motor vehicle and the recorded image is shown on a display after the image processing. Then, for example, it can happen that e.g. a flashing light of a vehicle in the area is not reproduced or is not reproduced in time.

The avoidance of image errors caused by pulsed LEDs is made more difficult by the fact that commercially available pulsed LEDs are operated with a wide variety of pulse frequencies. In particular in the vicinity of a vehicle, pulsed light sources of a wide spectrum of different pulse frequencies can be observed today, all light sources within an image having to be captured by the image sensor. A synchronization of the scanning intervals of the image sensor with the switch-on phases of the LEDs is therefore out of the question for solving the problem.

Out US 2004/0012692 A1 A method for flicker detection is known in which images are taken with two or more different exposure times, the longer of the exposure times in particular being long enough that no flicker-related image errors occur. A comparison of the images taken with different exposure times determines whether flicker is present or not. If flicker has been detected, flicker compensation is carried out by choosing the exposure time so that it is an integral multiple of the flicker period, ie the reciprocal of the pulse frequency of the pulsed light source.

The end US 2004/0012692 A1 Known methods deliver good results if, on the one hand, the Exposure time can be chosen sufficiently long and on the other hand the pulse frequency of the pulsed light sources is known and uniform. However, these requirements are not met if images or image sequences that contain light emitted by pulsed LEDs are to be acquired using simple and inexpensive image sensors. On the one hand, the exposure times of such image sensors are short and can only be extended to a limited extent without overexposure occurring. The reason for this is that simple and inexpensive image sensors, such as those used in particular in the automotive sector, do not have an aperture with which the light transmission can be adapted to the exposure time. On the other hand, the pulse frequencies of the pulsed LEDs in the surroundings of the image sensor, in particular in the surroundings of a vehicle, are not known and, moreover, are not uniform, but can have completely different values in a wide spectrum. This can happen when capturing and recording images or image sequences that contain light emitted by pulsed LEDs US 2004/0012692 A1 known methods therefore do not provide satisfactory results.

Out US 2014/0153839 A1 A device and a method for recognizing and correcting flicker are known, which are based on the evaluation of the image data of images with long and short exposure times. In this case, an average pixel value is determined from two temporally successive images with a long exposure time. Whether flicker is present is judged on the basis of a difference between this mean pixel value of the image data of the image with a long exposure time and the associated pixel value of the image data of the image with a short exposure time. If flicker has been detected, a synthetic pixel value is used for correction, which is calculated as a mixture of the pixel values of the image with a long exposure time and the image with a short exposure time with a predetermined mixing ratio.

Proceeding from this, it is an object of the present invention to provide an improved possibility for detecting and correcting image errors caused by flickering light information, which is suitable for images with a short exposure time and which permits detection and correction of image errors caused by pulsed LEDs.

The object is achieved by the image processing unit with the features of claim 1 and the computer program with the features of claim 17. Advantageous embodiments are described in the subclaims.

To detect and correct image errors, which are caused by flickering light information, in images generated by an image sensor, an image processing unit with a data memory for storing image data and a data processing unit is proposed. A computer program with program code means is also proposed for this purpose.

The data processing unit is set up to read in an exposure series which contains at least one reference image and at least one comparison image, i. H. at least two pictures.

In a corresponding manner, the program code means are set up for reading in the exposure series when the computer program is executed with a data processing unit.

According to the invention, the exposure time with which the at least one reference image was recorded is longer than the exposure time with which the at least one comparison image was recorded, i. H. the exposure time of the reference image is longer than the exposure time of the comparison image.

The exposure series can, for example, be recorded by an image sensor and temporarily stored in the data memory. Such an image sensor can be, for example, a video camera, in particular a digital video camera, another type of camera or a component of such a camera.

According to the invention, in particular a plurality of comparison images can be read in, which were generated with different exposure times, which are each shorter than the exposure time of the reference image. For example, it is conceivable that the exposure time of the reference image is long, while the exposure time of a comparison image is short and the exposure time of another comparison image is very short. The number of comparison images of different or the same exposure time is fundamentally unlimited.

According to the invention, it is necessary for at least one reference image to be read in, the exposure time of which is longer than the exposure time of the at least one comparison image. The image of the read exposure series that was generated with the longest exposure time, this time being longer than the exposure time of at least one further image, is a reference image in the sense of the present invention. The image of the read exposure series that was generated with the shortest exposure time, this time being shorter than the exposure time of at least one further image, is a comparison image in the sense of the present invention.

According to the invention, the images of the exposure series show the same scene. A scene is understood to be a section of the visible surroundings of the image sensor at a specific point in time. The same scene is understood to mean an identical or approximately identical section of the visible environment at an identical or approximately identical point in time.

According to the invention, the different images of a series of exposures can either have been recorded simultaneously, for example with the aid of a split pixel sensor, or they can have been recorded in very short time intervals.

The data processing unit is further configured according to the invention to determine local deviations of the image data of the comparison image from the image data of the reference image by locally comparing the image data of the comparison image with the image data of the reference image. Local deviations are understood to mean deviations at a specific position in the image.

Correspondingly, according to the invention, when the computer program is executed with a data processing unit, the program code means are set up to determine the local deviations of the image data of the comparison image from the image data of the reference image by comparing the image data of the comparison image locally with the image data of the reference image.

Local comparison is understood to mean comparing the image data at a specific position of the image. The local comparison and / or the resulting local deviations of the image data can relate, for example, to the positions of individual pixels in the image when individual pixels are compared. In this way, a particularly high level of accuracy is achieved. However, larger areas of the image, for example image blocks, can also be compared with one another, the local comparison and / or the resulting local deviations of the image data in this case relating to the positions of image blocks in the image. This reduces the computing and storage effort associated with determining the local deviations. In principle, entire images can also be compared with one another, as a result of which the computing and storage effort can be further reduced. The local deviations can thus be determined, for example, in pixel-precise, block-precise or image-precise resolution.

Due to the longer exposure time of the reference image, the image data of the reference image fundamentally differ from the image data of the comparison image, although both images show the same scene. In particular, because of its longer exposure time, the reference image generally has a greater brightness than the comparison image. In order to compare the image data of the images taken with different exposure times, the image data can be normalized, which can relate, for example, to an absolute brightness and / or a specific exposure time. This will be discussed in more detail later.

According to the invention, by determining local deviations on the basis of a comparison of the images taken with different exposure times, the presence of flickering light information, i. H. the presence of flicker, recognized. The flickering light information can in particular be in the form of light which is generated by pulsed light sources, e.g. B. LEDs is emitted. The likelihood that a captured image shows the off state of a pulsed light source, even though the light source is switched on, decreases with increasing exposure time. The longer exposure time of the reference image can therefore result in the reference image at least partially showing the on state of the pulsed light source, while the comparison image recorded with a shorter exposure time shows the off state of the same pulsed light source. On the basis of the local deviations caused thereby between the image data of the reference image and the image data of the comparison image, flickering light information and image errors caused thereby are recognized.

In order to reliably detect flickering light information and / or image errors caused thereby, the exposure time of the at least one reference image can in particular be selected to be significantly longer than the exposure time of the at least one comparison image. In this way, it is advantageously achieved that the reference image recorded with a longer exposure time with a sufficient probability shows at least partially the on state of the pulsed light source, while the comparison image shows the off state of the same pulsed light source.

In order to reliably detect flickering light information and / or image errors caused thereby, the exposure time of the reference image can advantageously be selected in particular so long that the on-state of the switched-on pulsed light sources, e.g. B. switched on LEDs, with high probability - - or even in any case - falls completely or at least partially in the exposure time and is thus detected by the reference image.

For example, the exposure time of the reference image can be selected so that it corresponds at least to the reciprocal of the smallest possible pulse frequency of the pulsed light sources to be expected in the scene. Then the on state of the switched-on pulsed light sources falls completely in the exposure time of the reference image in any case and is thus reliably detected by the reference image.

Furthermore, the exposure time of the reference image can be selected so that it corresponds to at least 50% of the reciprocal of the image acquisition frequency (e.g. 10 ms at 50 Hz). Then the on-state of the switched-on pulsed light sources falls in part in any case (e.g. at least every second pulse at 100 Hz pulse light frequency) in the exposure time of the reference image and is thus reliably detected by the reference image.

A comparison image recorded at the same time or approximately at the same time, which was recorded with a shorter exposure time, on the other hand, can show the off-state of a pulsed light source due to the problem explained above, even though the light source is switched on.

In such cases, the reference image at the positions of the image which show a switched-on pulsed light source or the light emitted by it has a much greater brightness than the shorter-exposed comparison image which shows the switched-on pulsed light source in its off state. Therefore, the local comparison of the image data of the comparison image with the image data of the reference image reveals clear deviations which go beyond the deviations to be expected solely on the basis of the different exposure times. Accordingly, the normalized image data also have local deviations which can be recognized by comparing the standardized image data and which indicate the presence of flickering light information.

By determining local deviations, it is thus determined whether, to what extent and at which positions there are deviations between the image data of the comparison image and the image data of the reference image.

The data processing unit is also set up to generate an error mask as a function of the determined local deviations of the image data.

Correspondingly, according to the invention, when the computer program is executed with a data processing unit, the program code means are set up to generate the error mask as a function of the determined local deviations of the image data.

The error mask shows the error positions of image errors within the comparison image.

The error positions of image errors within the comparison image can in turn relate, for example, to the positions of individual pixels in the image, as a result of which a particularly high accuracy of the error mask is achieved. The error positions can also refer to the positions of larger areas of the image, for example the positions of image blocks in the image. This reduces the computation and storage effort associated with generating the error mask. The error mask can basically only indicate whether the entire image contains errors or not. In this way, the computing and storage effort can be reduced further. The error mask can thus be generated, for example, in pixel-precise, block-accurate or image-precise resolution.

In order to generate the error mask based on the determined local deviations, it can be checked in particular whether the determined local deviations are so large that it can be assumed that they were caused by a pulsed light source. This is particularly the case if the deviations are significantly larger than would be expected solely on the basis of the different exposure times or light sensitivities of the reference image and the comparison image. If the local deviations are so large that it can be assumed that they were caused by a pulsed light source, the position of the local deviation is noted in the error mask as the error position. In addition to the exposure time, the setting of the sensor sensitivity also plays a role. In the case of sensors with different cells for the reference image and the comparison image, their relative sensitivity plays a role. H. the relationship of their sensitivities to each other, a role.

In this way it is achieved that the error mask shows the error positions of image errors within the comparison image. If the error mask is generated, for example, in pixel-precise resolution, which advantageously achieves a particularly high accuracy of the error mask, the error mask can indicate the position of each detected defective pixel. In this case, the error mask can, for example, be constructed in such a way that it comprises one entry for all pixel positions within the comparison image, a status indicator (flag) for each entry indicating whether the pixel is faulty or not.

After generating the error mask according to the invention, it is thus known at which positions the at least one comparison image has image errors which have been caused by flickering light information. On this basis it is possible to correct the image errors of the comparison image.

For this purpose, it is provided according to the invention that the data processing unit is set up to determine local correction values for the image data of the comparison image. This can advantageously be done in particular by estimation. Each local correction value is assigned to a correction position within the comparison image, wherein the correction positions can have a pixel-precise, block-exact or image-exact resolution. The assignment of the local correction value to a correction position within the comparison image allows the correction of an image error of the comparison image at a specific error position that corresponds to the correction position.

Correspondingly, according to the invention, when the computer program is executed with a data processing unit, the program code means are set up to determine the local correction values for the image data of the comparison image.

In particular, the image data of the reference image can be used to determine local correction values by estimation. Use of the image data of the reference image is expedient in this regard, since it can be assumed that the comparison image shows the on-state of the pulsed light source in the event of an image error which was caused by a pulsed light source detected in the off state. This will be discussed in more detail later.

As an alternative or in addition to this, the image data of other comparison images that were generated at other times, in particular at earlier times than the current comparison image whose image errors are to be corrected, can be used in particular to determine local correction values by estimation. Use of the image data of other comparison images is expedient in this regard because comparison images generated at other times may have accidentally detected the on-state of the pulsed light source despite their shorter exposure time compared to the reference image. This will also be discussed in more detail later.

After determining local correction values according to the invention, local correction values are thus known, each of which is assigned to a correction position within the comparison image and, if necessary, i. H. if the error mask indicates the presence of image errors within the comparison image, can be used to correct the comparison image.

For this purpose, it is provided according to the invention that the data processing unit is set up to generate a corrected comparison image by inserting the local correction values at the error positions indicated by the error mask. The local correction value whose assigned correction position corresponds to the error position is used for the respective error position indicated by the error mask. In this way, the respective incorrect image data of the comparison image can be replaced by the correction value and the image error can be corrected in this way.

Correspondingly, according to the invention, the program code means, when the computer program is executed with a data processing unit, are set up to generate the corrected comparison image by inserting the local correction values at the error positions indicated by the error mask.

As a result, a corrected comparison image is available, the image errors caused by flickering light information selectively, i. H. at the error positions affected by image errors have been corrected by correction values. The image processing unit according to the invention and the computer program according to the invention can thus provide a flicker-reduced or even a completely flicker-free image despite the presence of flickering light information.

In an advantageous development of the invention, the image data are brightness values. In this case, brightness values are used for the local comparison of the image data and / or the local deviations of the image data relate to brightness values and / or the local correction values relate to brightness values. A brightness value in this sense can be, for example, the luminance or luma value of digital or analog image data, preferably digital image data.

In this sense, brightness values can in particular be the direct color components (RGGB) of a Bayer sensor. Brightness values in this sense can also be the value Y value in the YCbCr color model or a Y value in the YUV color model. Brightness values in this sense can also be the values of the color components R, G and B after debayering a Bayer sensor. A brightness value in this sense can also be another brightness value. The brightness values can also be used for the purposes of the invention from the Raw data of the images of the read exposure series are calculated, for example from such raw data as is provided by an image sensor at its output interface.

The use of brightness values as image data for the purposes of the invention offers the advantage that, based on the brightness values, it is particularly easy to identify and correct image errors caused by flickering light information. The reason for this is that the change in a light source from an on state to an off state or a change in the brightness of the light source is directly reflected in the brightness values and thus in the image data.

In a further advantageous development of the invention, the image sensor is a high dynamic range image sensor, in short HDR image sensor, and the exposure row is a high dynamic range exposure row, in short HDR exposure row, generated by the HDR image sensor.

HDR image sensors are able to generate an HDR exposure series by multiple exposure, which several simultaneously, e.g. B. when using a split pixel sensor, or includes images generated at very short time intervals, so that the images show the same scene in the sense of the present invention. The images of an HDR exposure series are recorded by the HDR image sensor with exposure times of different lengths in order to be able to provide greater brightness dynamics compared to a single image.

This functionality of an HDR image sensor can advantageously be used to provide the exposure series required for the inventive detection and correction of image errors caused by flickering light information. The use of an HDR image sensor that generates an HDR exposure series offers the advantage that several images of the same scene with exposure times of different lengths can be provided quickly and automatically and used as a reference image or comparison image in the sense of the present application.

In a further advantageous development of the invention, the data processing unit is set up to normalize the image data to determine the local deviations and to compare the normalized image data at a position of the comparison image with the normalized image data at the same position of the reference image. In order to compare the normalized image data, a difference can in particular be determined between the normalized image data at a position of the comparison image and the normalized image data at the same position of the reference image.

Correspondingly, in a further advantageous development of the invention, when the computer program is executed with a data processing unit, the program code means are set up to normalize the image data for determining the local deviations and to normalize the normalized image data at a position of the comparison image with the normalized image data on the same Compare the position of the reference image.

Due to the longer exposure time of the reference image, the image data of the reference image fundamentally differ from the image data of the comparison image, although both images show the same scene. In particular, because of its longer exposure time, the reference image has a greater brightness than the comparison image. In order to determine local deviations between the image data of the comparison image and the image data of the reference image, normalization of the image data is therefore expedient.

The normalization of the image data can relate, for example, to an absolute brightness and / or a specific exposure time. When normalizing the image data, the exposure parameters and operating parameters of the image sensor selected during the generation of the respective image, for example the exposure time, the sensitivity, the f-number and / or the response curve (camera response curve) of the image sensor, can be taken into account.

If, for example, the exposure parameters and operating parameters of the image sensor are the same for all images in the exposure series with the exception of the exposure time, the images of the exposure series can be normalized to an arbitrarily selectable reference exposure time on the basis of the respective exposure time. If the exposure series includes, for example, a reference image L with an exposure time T _L , a comparison image S with an exposure time Ts = T _L / 4 and a further comparison image VS with an exposure time 7 _VS = T _L / 16 and the exposure time T _{L is} selected as the reference exposure time, In this way, image data available as brightness values can be standardized to the reference exposure time by multiplying the brightness values of the comparison image S by four and the brightness values of the comparison image VS by sixteen.

After the normalization of the image data, these are immediately comparable, so that in particular a difference can be determined between the normalized image data at a position of the comparison image and to determine the local deviations the normalized image data at the same position of the reference image. In this way, the difference between the normalized image data at a specific position can quantify the local deviations.

The positions of the standardized image data can in turn be determined in pixel-precise, block-precise or image-precise resolution. A difference between the normalized image data of the reference image and the comparison image can advantageously be determined on the basis of pixel-precise positions for each individual pixel of the images. In this way, a particularly high accuracy of the local deviations determined is achieved. The difference in the image data at a specific pixel position can therefore be used to exactly quantify the local deviation at the pixel position.

In a further advantageous development of the invention, the data processing unit is set up to use a relative light sensitivity ratio between the light sensitivity of the reference image and the light sensitivity of the comparison image for normalizing the image data. As an alternative or in addition to this, an exposure time ratio between the exposure time of the reference image and the exposure time of the comparison image can be used to normalize the image data. A relative light sensitivity ratio is understood to mean a ratio of a light sensitivity of one image to a light sensitivity of another image. An exposure time ratio is understood to mean a ratio of an exposure time of one image to an exposure time of another image.

Correspondingly, in a further advantageous development of the invention, the program code means, when the computer program is executed with a data processing unit, are set up to normalize the image data by a relative light sensitivity ratio between the light sensitivity of the reference image and the light sensitivity of the comparison image and / or an exposure time ratio between the Exposure time of the reference image and the exposure time of the comparison image.

In a further advantageous development of the invention, the local deviations of the image data determined are compared with a threshold value in order to generate the error mask. If the local deviations exceed the threshold value, it can be assumed that there is an image error caused by flickering light information at the position of the local deviation. In this case, the position of the local deviation can be entered or marked as faulty in the fault mask, so that the fault position of the image fault within the comparison image is displayed in this way.

The comparison with a threshold value offers the advantage that the error mask can be generated in this way with little computation effort.

The image data of the reference image can be used, for example, to determine local correction values for the image data of the comparison image by estimation. To correct image errors caused by a pulsed light source detected in the off state, use of the image data of the reference image is expedient, since it can be assumed that the reference image shows the on state of the pulsed light source due to its longer exposure time. Because of the longer exposure time, however, the image data of the reference image generally have a greater brightness than the image data of the comparison image, so that the image data of the reference image cannot be used directly as correction values for the image data of the comparison image.

In a further advantageous development of the invention, it is therefore provided in particular that the data processing unit is set up to use a simple estimate for determining the local correction values for the image data of the comparison image, based on the image data of the reference image and the relationship between the exposure time of the reference image and the exposure time of the comparison picture is based.

Correspondingly, in a further advantageous development of the invention, when the computer program is executed with a data processing unit, the program code means are set up to use a simple estimate for determining the local correction values for the image data of the comparison image, based on the image data of the reference image and the ratio between the exposure time of the reference image and the exposure time of the comparison image.

To determine a local correction value that is assigned to a specific correction position within the comparison image, the image data, for example a brightness value, is used at the position of the reference image that corresponds to the correction position. If the relationship between the exposure time of the reference image and the exposure time of the comparison image is known, then an estimate of the corresponding brightness value of the shorter exposure comparison image can be calculated from the brightness value of the reference image and used to use this as a correction value. For example, includes the bracketing series Reference image L with an exposure time T _L and a comparison image S with an exposure time Ts = T _L / 4, the brightness value of the reference image can be divided by four and this value can be used as a local correction value for the image data of the comparison image at the assigned correction position.

It is problematic, however, that in the bright areas of the scene, the image data of the reference image may already have reached the saturation range due to the long exposure time. H. The image data of the reference image can have undesired clipping effects due to an overload of the image sensor. This applies in particular to those areas of the scene which contain flickering light information emitted by a pulsed light source, since these areas are very bright. An additional correction factor can therefore advantageously be used when estimating the local correction value from the ratio between the exposure time of the reference image and the exposure time of the comparison image. The value of the additional correction factor can be estimated, for example, on the basis of the image data from the surroundings of the reference image surrounding the correction position. If the image data of the reference image has already reached the saturation range in a large environment around the correction position, it makes sense to choose a large additional correction factor.

The method of simple estimation, which is based on the relationship between the exposure time of the reference image and the exposure time of the comparison image, offers the advantage that the local correction values can be determined with very little computation and storage effort.

According to the invention, it is further provided that the data processing unit is set up to use an estimate to determine the local correction values for the image data of the comparison image, which is based on the image data of a comparison image time series held in a data memory. The comparison image time series comprises a number of stored images that were generated by the image sensor with the same exposure time as the comparison image at different times than the comparison image. The stored images of the comparison image time series can in particular have been generated by the image sensor at earlier times than the comparison image. Furthermore, the times at which the various stored images of the comparison time series were generated by the image sensor can advantageously differ from one another. The data memory can in particular be a ring memory in which, when the data memory is full and / or a defined number of stored images is reached, the oldest stored image is replaced by a new stored image.

Correspondingly, according to the invention, when the computer program is executed with a data processing unit, the program code means are set up to use an estimate for determining the local correction values for the image data of the comparison image, which is based on the image data of the comparison image time series held in the data memory, in particular a ring memory .

The comparison image time series, which is used to determine the local correction values for the image data of the comparison image, can also be an unchanged copy or a modified copy of the current, i.e. H. last included comparison image itself.

The comparison image time series can comprise 30 stored images, for example. The points in time at which the stored images were generated by the image sensor can be points in time which are different from one another, in particular points in time which are immediately or indirectly successive. In particular, time steps of a certain length can lie between the times.

The stored images of the comparison image time series can, in particular, be copies of the comparison images from exposure series read in at unchanged or changed form at earlier times for the detection and correction of image errors according to the invention. In particular, an unchanged or changed copy of the comparison image of the read exposure series after reading in the exposure series can be stored in the data memory. In the case of a ring buffer, this newly added comparison image can then replace the oldest previously stored image of the comparison image time series when the data memory is full and / or a specified number of stored images has been reached. In this way, a comparison image time series is provided in the data memory, which comprises temporally successive images that were generated by the image sensor with the same exposure time as a current comparison image, the image errors of which are to be corrected with the aid of the local correction values.

The use of an estimate, which is based on the image data of such a comparison image time series held in a data memory, is useful for determining the local correction values for the image data of the comparison image, since comparison images generated at other times happen to have the on state despite their shorter exposure time compared to the reference image the pulsed light source may have captured. If the number of stored images of the comparison image time series is chosen large enough, it can be assumed that at least one of the stored images of the comparison image time series has detected the on state of the pulsed light source.

A suitable local correction value for the image data of the comparison image can therefore be determined from the image data of one or more stored images of the comparison image time series which has or have detected the on state of the pulsed light source. If the image data are available, for example, in the form of brightness values, the brightness value of a stored image, for example, which has detected the on state of the pulsed light source, can be adopted as a correction value.

The use of an estimate for determining the local correction values for the image data of the comparison image, which is based on the image data of a comparison image time series of the type described held in a data memory, offers the advantage that a particularly precise determination of the local correction values is possible. The reason for this is that the stored images of the comparison image time series were generated with the same exposure time as the comparison image, so that the image data of the stored images have a brightness that matches the exposure time of the comparison image.

In a further advantageous development of the invention, it is provided that the data processing unit is set up to subject the images of the comparison image time series to a dilation before being stored in the data memory and / or to reduce and / or compress them by downscaling. The operations mentioned can be used in any order and several times. For example, downscaling can be carried out first, then dilation and then downscaling again. As a result, both the computational complexity associated with the dilatation and the size of the data memory required for storing the images can advantageously be reduced.

In a corresponding manner, in a further advantageous development of the invention, when the computer program is executed with a data processing unit, the program code means are set up to subject the images of the comparison image time series to dilation before saving them in the data memory and / or to reduce them by downscaling and / or to compress.

In these cases, therefore, not unchanged copies are saved, but copies of images changed by dilation and / or downscaling and / or compression that were generated by the image sensor at times other than the comparison image. In this sense, the images are saved in a modified form. In particular, the stored images of the comparison image time series can advantageously be provided in that, after the exposure series has been read in, a copy of the comparison image of the read exposure series that has been modified in the manner mentioned is stored in the data memory.

The term dilation is to be understood in the sense of image processing. The dilation expands certain areas of the image, i.e. H. dilated. According to the invention, it is advantageously possible in particular to expand bright areas of the image by dilation. The expansion of light areas offers the advantage that the quality of the local correction values, which are determined on the basis of the stored images of the comparison image time series, can be improved in this way, since image data with high brightness are of particular relevance for the correction of image errors caused by flickering light information is caused.

This advantage comes into play in particular if a pulsed light source or the flickering light information it emits move relative to the image sensor over time, for example by moving the pulsed light source and / or the image sensor. If, under these circumstances, a local correction value assigned to a specific correction position is sought for the image data of the comparison image in the image data of the stored images, it can happen that due to the movement the suitable (bright) correction value within the stored image is not at the correction position, but can be found in the vicinity of this position. If the bright areas within the stored images relevant for the correction of the image errors are expanded by dilation, this effect can be compensated for.

The images of the comparison image time series can furthermore advantageously be downscaled before being stored in the data memory. Downscaling can in particular include subsampling and / or downsampling and / or subsampling. Downscaling the images before saving them has the advantage that the storage space required for holding the comparison images can be reduced in a simple manner.

The images of the comparison image time series can also be advantageously stored in the Data storage can be compressed. Compressing the images before saving them has the advantage that the storage space required for holding the comparison images can be reduced particularly effectively.

In a further advantageous development of the invention, it is provided that the image data of the stored images of the comparison image time series are motion-compensated with respect to the comparison image. The data processing unit is accordingly set up to store the images of the comparison image time series in motion-compensated form in the data memory. As an alternative or in addition to this, the data processing unit can advantageously be set up to perform motion compensation of the image data of the stored images of the comparison image time series with respect to the comparison image before the local correction values for the image data of the comparison image are determined. The motion compensation of the image data of the images of the comparison image time series can thus take place before and / or after the images have been stored in the data memory. In an analogous manner, the comparison image can also be motion-compensated before the local correction values are determined with respect to the images of the comparison image time series. The motion compensation can be based in particular on a motion estimation based on motion vectors.

Correspondingly, in a further advantageous development of the invention, when the computer program is executed with a data processing unit, the program code means are set up to store the images of the comparison image time series in motion-compensated form in the data memory and / or before determining the local correction values for the image data perform a motion compensation of the image of the stored images of the comparison image time series with respect to the comparison image of the comparison image.

The use of motion compensation is expedient because the times at which the stored images of the comparison image time series were generated by the image sensor differ from the time at which the at least one comparison image of the read exposure series was generated. In addition, the times at which the various stored images of the comparison time series were generated by the image sensor can differ from one another.

Relative movements of an object between the scenes captured with the different images can lead to the fact that the at least one comparison image and the different images of the comparison image time series show the moving object in different positions due to the different image generation times. Such a moving object can be, for example, a pulsed light source that shows the comparison image in the off state, even though the pulsed light source is actually switched on. In this case, with the aid of the invention, a number of image errors at assigned error positions are recognized within the comparison image, which must be corrected using local correction values. The accuracy of the local correction values that are determined from the stored images of the comparison image time series can be impaired by the relative movement of the pulsed light source, since it can happen that the stored images of the comparison image time series do not move the pulsed light source at the error position due to the movement but point to a position offset from the error position.

The use of motion compensation between the stored images of the comparison image time series on the one hand and the comparison image whose image errors are to be corrected, on the other hand, therefore has the advantage that the negative effects of relative movements can be compensated for by the motion compensation and the accuracy of the local correction values based on an estimate can be determined from the stored images of the comparison image time series, can be improved.

In a further advantageous development of the invention it is provided that the data processing unit is set up to determine the local correction values for the image data of the comparison image such that a maximum local brightness value is determined from the image data of the images of the comparison image time series and the maximum local brightness value is used as the local correction value becomes.

Correspondingly, in a further advantageous development of the invention, when the computer program is executed with a data processing unit, the program code means are set up to determine the local correction values for the image data of the comparison image such that a maximum local brightness value is determined from the image data of the images of the comparison image time series and the maximum local brightness value is used as the local correction value.

A local brightness value is understood to mean a brightness value at a specific position in the image. The position of the local brightness value can in turn affect the position of a single pixel in the image, for example relate, whereby a particularly high accuracy of the local correction value is achieved. However, the position of the local brightness value can also relate to the position of a larger area of the image, for example to the position of an image block in the image. This reduces the computing and storage effort associated with determining the local correction values. The local brightness value can in principle also refer to the entire image. In this way, the computing and storage effort can be reduced further. A local brightness value, in particular a maximum local brightness value, can thus have, for example, a pixel-precise, block-accurate or image-precise resolution.

In order to determine a local correction value that is assigned to a specific correction position within the comparison image, it can accordingly be provided that by comparing the brightness values of the images of the comparison image time series at the correction position, the maximum local brightness value, i. H. the image date with the highest brightness for this correction position is determined. This maximum local brightness value can be used as a local correction value for the correction position.

The use of maximum local brightness values as local correction values is advantageous since it can be assumed that one or more of the stored images of the comparison image time series has captured the flickering light information that has caused an image error to be corrected in the on state or in a state of high brightness . By determining maximum local brightness values, the image data from the comparison image time series can be determined in a simple and efficient manner that shows the pulsed light source or the flickering light information it emits in the on state or in the state of high brightness.

In particular, in an advantageous development of the invention, all maximum local brightness values for all possible correction positions can be determined and stored from the image data of the comparison image time series. For example, the maximum brightness value can be determined for each individual pixel position from the image data of the comparison image time series. In this way, a maximum brightness image is created, the image data of which have the maximum brightness value of the comparison image time series for each correction position. If necessary, d. H. if the generated error mask indicates an image error of the comparison image at at least one error position, the correction value for the comparison image can then be taken from the maximum brightness image at the correction position corresponding to the error position.

The image data of the maximum brightness image can also advantageously be subjected to a dilation in which the bright areas of the maximum brightness image are expanded. The dilation can advantageously compensate for relative movements of the pulsed light sources or the flickering light information they emit, so that an improved accuracy of the local correction values taken from the maximum brightness image is achieved.

In a further advantageous development of the invention, it is provided that HDR postprocessing (high dynamic range postprocessing) using at least one corrected comparison image results from the exposure series, which can in particular be an HDR exposure series generated by an HDR image sensor Output image is generated for the viewer. First, one or more comparison images that are part of the read exposure series are corrected by inserting local correction values at the error positions indicated by the error mask. This produces a corrected exposure series which contains images of different exposure times, namely at least one reference image and at least one corrected comparison image. An output image for the viewer can be generated from the image data of the images of this corrected exposure series by HDR postprocessing in a manner known to those skilled in the art.

The generation of such an output image from the (corrected) exposure series by HDR postprocessing has the advantage, on the one hand, that due to the different exposure times of the images of the exposure series, greater brightness dynamics are provided compared to a single image, so that the quality of the output image can be improved in this regard . On the other hand, this has the advantage that image errors caused by short exposure times are avoided in the output image, since the corrections made in the respective comparison image can be adopted in the output image. In this way, a flicker-reduced or even a flicker-free output image with improved brightness dynamics can be provided.

In a further advantageous development of the invention, it is provided that the read exposure series is part of an exposure series sequence generated by the image sensor, the exposure series sequence comprising a series of exposure series. The sequence of exposure series shows successive scenes that can contain flickering light information.

Such an exposure sequence can in particular be one generated by the image sensor Video sequence. Each of the temporally successive scenes is not only represented by a single image, as is the case with conventional image sequences for displaying moving images, but each of the successive scenes is represented by an exposure series which comprises several images with different exposure times.

In this way, the data processing unit can apply the detection and correction of image errors caused by flickering light information to the individual temporally successive exposure series of an exposure series sequence, in particular a video sequence, so that image errors in the exposure series sequence caused by flickering light information can advantageously be recognized and corrected. In a corresponding manner, the program code means can, when the computer program is executed with a data processing unit, apply the detection and correction of the image errors to the individual time-sequential exposure series of the exposure series sequence.

The invention is therefore particularly suitable for the detection and correction of image errors in video sequences caused by flickering light information.

In a further advantageous development of the invention it is provided that the exposure time of at least one reference image is between 50% and 100% of the image duration of the exposure sequence. For a frame rate of 60 frames per second, ie a frame duration of 1/60 s (~ 16.67ms), it is provided that the exposure time of at least one reference image is between 1/120 s (-8.33 ms) and 1/60 s (-16.67 ms) is. The exposure time of at least one comparison image can in particular be 1 ms or less.

In the case of a conventional image sequence in which a sequence of temporally successive images recorded and reproduced at a specific image frequency (frame rate) is used to display moving images, the image duration (frame duration) denotes the reciprocal of the image frequency, i. H. the time interval between two images that follow one another in time. In analogy to this, the duration of the image of the exposure series sequence is understood to mean the time interval between two exposure series immediately following one another.

The image sensor can, for example, generate exposure series with a frequency of 60 Hz. In this case, the image duration of the exposure series sequence generated by the image sensor is 1000/60 ms = 50/3 ms ≈ 16.67 ms. If in this case the exposure time of at least one reference image is to be between 50% and 100% of the image duration of the exposure sequence, exposure times between 25/3 ms and 50/3 ms result. However, the image sensor can also use a series of exposures with a higher or lower frequency, e.g. B. with a frequency of 50 Hz, 30 Hz, 25 Hz, 100 Hz or with another frequency. The invention is fundamentally independent of the frequency with which the image sensor generates the exposure series of the exposure series sequence, and can be used for exposure series sequences of any image duration.

If the exposure time of at least one reference image is chosen such that it is between 50% and 100% of the image duration of the exposure series sequence, this offers the advantage that the on-state of the switched-on pulsed light sources, e.g. B. switched on LEDs, with high probability falls completely or at least partially in the exposure time and is thus detected by the reference image. If the exposure time of at least one reference image is chosen so that it corresponds to 100% of the image duration of the exposure sequence, it can be assumed that the on-state of the switched-on pulsed light sources, e.g. B. switched on LEDs is detected with certainty by the reference image. If the exposure time of at least one reference image is chosen in such a way that it is between 50% and 100% of the image duration of the exposure series sequence, it can accordingly advantageously be ensured that image errors in the image data of the comparison image, which were caused by flickering light information, by a comparison with the Image data of the reference image can be reliably recognized.

If the exposure time of at least one comparison image is 1 ms or less, this offers the advantage, on the one hand, that the comparison image has sufficient image sharpness even if the imaged scene contains rapid relative movements. When using a longer exposure time, however, the parts of the scene affected by the relative movement could be reproduced out of focus. On the other hand, this short exposure time of the comparison image offers the advantage that even very bright areas of the scene can be imaged without errors, since the image sensor does not reach saturation even in the very bright areas due to the short exposure time. Undesired clipping effects in the image data of the comparison image, which are caused by an overload of the image sensor, are avoided.

In a further advantageous development of the invention, it is provided that the data processing unit is set up to carry out the previously explained measures for detection and correction to repeat image errors caused by flickering light information. The detection and correction according to the invention of image errors caused by flickering light information are applied to a plurality of exposure series of the exposure series sequence or to each exposure series of the exposure series sequence. As a result, a corrected exposure series sequence is generated, which comprises corrected comparison images.

Correspondingly, in a further advantageous development of the invention, when the computer program is executed with a data processing unit, the program code means are set up to repeat the detection and correction of the image errors caused by flickering light information and to a plurality of exposure series of the exposure series sequence or to each exposure series of the exposure series sequence to generate a corrected exposure series sequence comprising corrected comparison images.

The exposure sequence can in particular be a video sequence. In this way, the invention can advantageously be applied, for example, to a video sequence which contains image errors caused by flickering light information in order to recognize and correct the image errors of the video sequence, so that a flicker-reduced or even a flicker-free video sequence can be provided.

In a further advantageous development of the invention, it is provided that the data processing unit is set up to apply a correction method for correcting remaining image errors caused by flickering light information to the image data of the images of the corrected exposure sequence. The corrective measures can be based on a motion estimation and a motion compensation of the image data of the images of the corrected exposure sequence. In addition or as an alternative to this, the corrective measures can be based on a mixing of the image data of different images of the corrected exposure sequence.

Correspondingly, in a further advantageous development of the invention, the program code means, when the computer program is executed with a data processing unit, are set up to apply corrective measures to the image data of the images of the corrected exposure sequence to correct remaining image errors caused by flickering light information, which are related to a Base motion estimation and motion compensation of the image data of the images of the corrected exposure series sequence and / or on a mixing of the image data of different images of the corrected exposure series sequence.

Even after the above-described correction of image errors by inserting local correction values has been applied, the exposure series sequence, which in particular can be a video sequence, can still have remaining image errors which were caused by flickering light information. By mixing the image data of different images of the corrected exposure series sequence, the images of the corrected exposure series sequence are quasi interpolated. If flickering light information was correctly recorded in one of the images whose image data is mixed, but was not recorded in another image due to an image error, this correctly recorded light information is retained after the mixing. The light information is weakened by the mixing, but is still there. The images of the exposure series sequence can be motion-compensated for the mixing after a motion estimation, so that image contents shifted by an intermediate relative motion are brought together again for the quasi-interpolation. The methods known to those skilled in the art for motion estimation and motion compensation can thus be used.

In this way, remaining image errors caused by flickering light information in the images of the corrected exposure sequence can advantageously be corrected.

• 1 - Schematische Darstellung einer beispielhaften Belichtungsreihe mit einem Referenzbild und zwei Vergleichsbildern sowie zwei daraus generierten Fehlermasken;
• 2 - Schematische Darstellung eines ersten Ausführungsbeispiels der Erfindung;
• 3 - Blockdiagramm eines zweiten Ausführungsbeispiels der Erfindung;
• 4 - Blockdiagramm eines Ausführungsbeispiels einer erfindungsgemäßen Fehlerkorrektur unter Verwendung maximaler Helligkeitswerte;
• 5 - Blockdiagramm einer Bildverarbeitungseinheit.

The invention is explained in more detail below using exemplary embodiments with the accompanying drawings. Show it:

• 1 - Schematic representation of an exemplary exposure series with a reference image and two comparison images and two error masks generated therefrom;
• 2nd - Schematic representation of a first embodiment of the invention;
• 3rd - Block diagram of a second embodiment of the invention;
• 4th - Block diagram of an embodiment of an error correction according to the invention using maximum brightness values;
• 5 - Block diagram of an image processing unit.

1 shows a schematic representation of an exemplary exposure series, as it is read in by the data processing unit, and two error masks E (S _i ) and E (VS _i ) generated therefrom. The exposure series consists of a reference image L _i , a first comparison image S _i and a second comparison image VS _i . The exposure series is part of a video sequence generated by an image sensor, which comprises successive exposure series generated at a frequency (frame rate) of 60 Hz. The image sensor generates an exposure series in each time step, the length of which corresponds to the image duration of the video sequence of -16.67 ms. The index i indicates that this is an exposure series generated by the image sensor in time step i, ie the i-th exposure series of the video sequence.

The images L _i , S _i and VS _{i of} the exposure series show the same scene 4th with a vehicle 3rd whose headlights 1 , 2nd use pulsed LEDs and therefore emit flickering light information. The images of the exposure series are taken with different exposure times. The exposure time of the reference image L _i is T _L = 16 ms, the exposure time of the first comparison image S _i is Ts = T _L / 4 = 4 ms and the exposure time of the second comparison image VS _i is Tvs = Ts / 4 = T _L / 16 = 1 ms. The images therefore have different brightness. While the image S _i reproduces the brightness, which is predominantly realistic, the image VS _i is underexposed due to its short exposure time and the image L _i is overexposed due to its long exposure time.

Both headlights 1 , 2nd of the vehicle are switched on. All images, ie both the reference image L _i and the comparison images S _i and VS _i therefore show the on state of the first headlight 1 . Due to their short exposure times, however, the comparison images S _i and VS _i do not show the on state, but the off state of the headlight 2nd . This is due to the fact that the exposure times of the comparison images S _i and VS _i completely in the off phase of the pulsed LEDs of the headlight 2nd fall. This effect occurs quasi-randomly, ie an immediately preceding comparison image S _i-1 or VS _i-1 or an immediately following comparison image S _{i + 1} or VS _{i + 1} with the same exposure time can already indicate the on state of the headlight 2nd demonstrate. The two comparison images S _i and VS _i thus have image errors caused by those of the headlight 2nd emitted flickering light information.

Due to its long exposure time, however, the reference image L _i shows both headlights 1 , 2nd in its on state, ie the reference image L _i has no image errors caused by flickering light information. This fact can be used to identify and correct the image errors caused by flickering light information in the comparison images.

For this purpose, the image data of the respective comparison image S _i or VS _{i are} compared locally with the image data of the reference image L _i , ie position-wise, and local deviations between the image data are determined in this way. In the example shown, the comparison is carried out pixel by pixel, ie the image data are compared for each individual pixel of the images. The image data are in the form of brightness values with an exemplary resolution of 8 bits, ie in a value range from 0 to 255.

The image data of the reference image L _i have due to the long exposure time in the area of the two headlights 1 , 2nd reached the range of saturation. As a result of an overload of the image sensor, the associated pixels of the reference image accordingly point at the positions x ₁ , y _{1 of} the first headlight 1 and the positions x ₂ , y _{2 of} the second headlamp 2nd the maximum brightness values L _i (x ₁ , y ₁ ) = L _i (x ₂ , y ₂ ) = 255. Based on an absolute brightness scale, the reference image in the area of the headlights is much too dark due to the saturation.

In order to be able to compare the image data of the images taken with different exposure times, they are first standardized to a uniform reference variable. The reference size of the standardized image data can in principle be chosen freely. In the in 1 In the example shown, the brightness value at an exposure time of T _L = 16 ms is defined as the normalized image data, ie as the normalized brightness. The image data L _i (x, y) of the comparative image L _i, the MS was taken with a time exposure of T _L = 16, are thus already in normalized form, so that no normalization is necessary.

für das erste Vergleichsbild S_i und $${\text{k}}_{\text{V}}={\text{T}}_{\text{L}}/{\text{T}}_{\text{VS}}=16\text{ ms}/1\text{ ms}=16$$

für das zweite Vergleichsbild VS_i. Es ergeben sich hieraus die normierten Helligkeitswerte ks - S_i(x,y) und kv VS_i(x,y), die unmittelbar mit den Bilddaten L_i(x,y) des Referenzbildes verglichen werden können, um örtliche Abweichungen zwischen den Bilddaten zu ermitteln.The image data S _i (x, y) and VS _i (x, y) of the first and second comparison image, on the other hand, are based on a shorter exposure time and are therefore standardized for comparison purposes to the exposure time T _{L of} the reference image by the brightness values having a factor ks or . kv multiplied, which results from the ratio of the exposure times. The following applies $${\text{k}}_{\text{S}}={\text{T}}_{\text{L}}/{\text{T}}_{\text{S}}=16\text{ms}/\mathrm{4th}\text{ms}=\mathrm{4th}$$

for the first comparison image S _i and $${\text{k}}_{\text{V}}={\text{T}}_{\text{L}}/{\text{T}}_{\text{VS}}=16\text{ms}/1\text{ms}=16$$

for the second comparison image VS _i . This results in the standardized brightness values ks-S _i (x, y) and kv VS _i (x, y), which can be compared directly with the image data L _i (x, y) of the reference image, by local deviations between the image data to investigate.

It is initially apparent that the reference image at the pixel positions that the first headlight 1 show, due to the saturation, a clearly low brightness L _i (x ₁ , y ₁ ) = 255 than the first comparison image and the second comparison image, the normalized brightness of these positions of the image ks - S _i (x ₁ , y ₁ ) = 4 · 200 = 800 or kv VS _i (x ₁ , y ₁ ) = 16- 50 = 800.

In addition, it can be seen from the local comparison of the image data that the reference image at the pixel positions that the second headlamp 2nd show a significantly higher brightness L _i (x ₂ , y ₂ ) = 255 than the first comparison image and the second comparison image, the normalized brightness at these positions of the image only ks - S _i (x ₂ , y ₂ ) = 4 20 = 80 or kv VS _i (x ₂ , y ₂ ) = 16- 5 = 80.

für das erste Vergleichsbild S_i und $${\Delta }_{\text{V}}\left(x_2,y_2\right)={\text{L}}_{\text{i}}\left({\text{x}}_2,y_2\right)-{\text{k}}_{\text{SV}}\cdot {\text{VS}}_{\text{i}}\left(\text{x2},\text{y2}\right)=255-80=175$$

für das zweite Vergleichsbild VS_i.The normalized brightness of the comparison images is therefore in the area of the headlight 2nd significantly lower than the brightness of the reference image. This shows local deviations in the positions of the comparison images, the pixels of which indicate the off state of the pulsed LEDs of the headlamp 2nd show, although this is actually switched on. These deviations can be quantified, for example, by a difference Δs (x, y) or Δv (x, y) between the normalized image data at the pixel position (x, y). In the example shown, this results in $${\Delta }_{\text{S}}\left(x_{\mathrm{2nd}},y_{\mathrm{2nd}}\right)={\text{L}}_{\text{i}}\left({\text{x}}_{\mathrm{2nd}},y_{\mathrm{2nd}}\right)-{\text{k}}_{\text{S}}\cdot {\text{S}}_{\text{i}}\left({\text{x}}_{\text{2nd}},{\text{y}}_{\text{2nd}}\right)=255-80=175$$

for the first comparison image S _i and $${\Delta }_{\text{V}}\left(x_{\mathrm{2nd}},y_{\mathrm{2nd}}\right)={\text{L}}_{\text{i}}\left({\text{x}}_{\mathrm{2nd}},y_{\mathrm{2nd}}\right)-{\text{k}}_{\text{SV}}\cdot {\text{VS}}_{\text{i}}\left(\text{x2},\text{y2}\right)=255-80=175$$

for the second comparison image VS _i .

These local deviations between the image data of the reference image L _i and the image data of the comparison images S _i and VS _i indicate that the image data of the reference images in the region of the second headlight contain image errors caused by flickering light information. Depending on the local deviations, error masks E (S _i ) and E (VS _i ) can therefore be generated, which display the error positions of image errors within the comparison images S _i and VS _i .

$$\text{E}\left({\text{S}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\mathrm{0,}\text{ sonst}$$

oder $$\begin{array}{ccc}\text{E}\left({\text{S}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\text{1},& \text{wenn}& {\text{L}}_{\text{i}}\left(\text{x},\text{y}\right)>>{\text{k}}_{\text{S}}\cdot {\text{S}}_{\text{i}}\left(\text{x},\text{y}\right)\end{array},$$

$$\text{E}\left({\text{S}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\mathrm{0,}\text{ sonst}$$

angewendet werden. Für das zweite Vergleichsbild VS_i können entsprechend die Bedingungen $$\begin{array}{ccc}\text{E}\left({\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\text{1},& \text{wenn}& {\text{L}}_{\text{i}}\left(\text{x},\text{y}\right)>{\text{k}}_{\text{S}}\cdot {\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\end{array},$$

$$\text{E}\left({\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\mathrm{0,}\text{ sonst}$$

oder $$\begin{array}{ccc}\text{E}\left({\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\text{1},& \text{wenn}& {\text{L}}_{\text{i}}\left(\text{x},\text{y}\right)>>{\text{k}}_{\text{S}}\cdot {\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\end{array},$$

$$\text{E}\left({\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\mathrm{0,}\text{ sonst}$$

angewendet werden.For the generation of such an error mask E (S _i (x, y)) or E (VS _i (x, y)), for example, the conditions for the first comparison image S _i $$\begin{array}{ccc}\text{E}\left({\text{S}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\text{1},& \text{if}& {\text{L}}_{\text{i}}\left(\text{x},\text{y}\right)>{\text{k}}_{\text{S}}\cdot {\text{S}}_{\text{i}}\left(\text{x},\text{y}\right)\end{array},$$

$$\text{E}\left({\text{S}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\mathrm{0,}\text{otherwise}$$

or $$\begin{array}{ccc}\text{E}\left({\text{S}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\text{1},& \text{if}& {\text{L}}_{\text{i}}\left(\text{x},\text{y}\right)>>{\text{k}}_{\text{S}}\cdot {\text{S}}_{\text{i}}\left(\text{x},\text{y}\right)\end{array},$$

$$\text{E}\left({\text{S}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\mathrm{0,}\text{otherwise}$$

be applied. The conditions for the second comparison image VS _i can accordingly $$\begin{array}{ccc}\text{E}\left({\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\text{1},& \text{if}& {\text{L}}_{\text{i}}\left(\text{x},\text{y}\right)>{\text{k}}_{\text{S}}\cdot {\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\end{array},$$

$$\text{E}\left({\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\mathrm{0,}\text{otherwise}$$

or $$\begin{array}{ccc}\text{E}\left({\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\text{1},& \text{if}& {\text{L}}_{\text{i}}\left(\text{x},\text{y}\right)>>{\text{k}}_{\text{S}}\cdot {\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\end{array},$$

$$\text{E}\left({\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\mathrm{0,}\text{otherwise}$$

be applied.

$$\begin{array}{ccc}\text{E}\left({\text{S}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\mathrm{0,}& \text{wenn}& {\Delta }_{\text{S}}\left(\text{x},\text{y}\right)<{\text{E}}_{\text{th}}\end{array}$$

für das erste Vergleichsbild S_i und $$\begin{array}{ccc}\text{E}\left({\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\text{1},& \text{wenn}& {\Delta }_{\text{V}}\left(\text{x},\text{y}\right)\ge {\text{E}}_{\text{th}}\end{array},$$

$$\begin{array}{ccc}\text{E}\left({\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\mathrm{0,}& \text{wenn}& {\Delta }_{\text{V}}\left(\text{x},\text{y}\right)<{\text{E}}_{\text{th}}\end{array}$$

für das zweite Vergleichsbild VS_i.Alternatively, to generate the error mask, the local deviations quantified by the differences Δs (x, y) and Δv (x, y) can be compared with a threshold value E _th . In this case, an image error at the error position (x, y) is assumed when the threshold value is reached or exceeded, i.e. the conditions result $$\begin{array}{ccc}\text{E}\left({\text{S}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\text{1},& \text{if}& {\Delta }_{\text{S}}\left(\text{x},\text{y}\right)\ge {\text{E}}_{\text{th}}\end{array},$$

$$\begin{array}{ccc}\text{E}\left({\text{S}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\mathrm{0,}& \text{if}& {\Delta }_{\text{S}}\left(\text{x},\text{y}\right)<{\text{E}}_{\text{th}}\end{array}$$

for the first comparison image S _i and $$\begin{array}{ccc}\text{E}\left({\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\text{1},& \text{if}& {\Delta }_{\text{V}}\left(\text{x},\text{y}\right)\ge {\text{E}}_{\text{th}}\end{array},$$

$$\begin{array}{ccc}\text{E}\left({\text{VS}}_{\text{i}}\left(\text{x},\text{y}\right)\right)=\mathrm{0,}& \text{if}& {\Delta }_{\text{V}}\left(\text{x},\text{y}\right)<{\text{E}}_{\text{th}}\end{array}$$

for the second comparison image VS _i .

In this way, an error mask can be generated which displays the error positions of image errors within the respective comparison image, which were caused by flickering light information.

In the example shown the 1 shows the resulting error mask E (S _i ) for the first Comparison image S _i (consistent regardless of which of the above conditions is used to generate the error mask) for the pixel positions (x ₂ , y ₂ ) that the second headlamp 2nd indicate an image error, ie E (S _i (x ₂ , y ₂ )) = 1. For all other pixel positions E (S _i (x, y)) = 0. The resulting error mask E (VS _i ) for the second comparison image VS _i accordingly contains the value E (VS _i (x ₂ , y ₂ )) = 1 for the area of the second headlight 2nd and E (VS _i (x, y)) = 0 for all remaining pixel positions.

Using the generated error masks, it is now possible to correct the image errors of the comparison images at the error positions indicated by the error mask by means of correction values. The determination of local correction values according to the invention and the generation of a corrected comparison image according to the invention are illustrated by the following 2nd to 5 explained.

In a departure from the simplified example shown here, in addition to the exposure time, other exposure parameters and operating parameters of the image sensor can also be taken into account when normalizing the image data. These include, for example, the sensor sensitivity, the analog gain in the sensor-A / D converter, the f-number and / or the sensitivity curve (camera response curve) of the image sensor.

2nd shows a schematic representation of a first embodiment of the detection and correction of image errors according to the invention. In accordance with the in 1 The example shown again shows a reference image L _i and a comparison image S _i , which are components of an exposure series read by the data processing unit. The images L _i and S _{i of} the exposure series show the same scene 4th with a vehicle 3rd whose headlights 1 , 2nd use pulsed LEDs and therefore emit flickering light information. The exposure time of the reference image L _i is T _L = 16 ms, the exposure time of the comparison image S _i is Ts = T _L / 4 = 4 ms. The image data of the images are again in the form of brightness values.

The read-in exposure series, which comprises the images L _i and S _i , is analogous to the previous explanations, the i-th exposure series of a video sequence generated by an image sensor, ie an exposure series sequence which comprises exposure series generated in successive time steps.

Both headlights 1 , 2nd of the vehicle are switched on. While the reference image L _i both headlights due to its long exposure time 1 , 2nd shows in its on state, the comparison image S _i does not show the on state, but the off state of the headlights 1 , 2nd . The reason for this is that the short exposure times of the comparison image S _i completely in the off phase of the pulsed LEDs of the headlights 1 and 2nd fall. The comparison image S _i thus has image errors caused by those of the headlights 1 , 2nd emitted flickering light information.

Analogous to the explanations for the in 1 The example shown is determined by local, pixel-by-pixel comparison of the image data of the reference image L _i and the comparison image S _i, local deviations between the image data of the two images. Local deviations arise here at those pixel positions of the images which are the headlights emitting the flickering light information 1 , 2nd of the vehicle 3rd demonstrate. Depending on the local deviations determined, analogous to the explanations for the in 1 shown an error mask E (S _i ) are generated. The error mask E (S _i ) shows the error positions of the image errors caused by flickering light information within the comparison image S _i . These are just those pixel positions where the images are either the spotlight 1 or the headlight 2nd show that the reference image L _i has a significantly greater brightness at these positions than the comparison image S _i due to the pulsed light emitted by the LEDs.

The positions are thus known at which the comparison image S _i has image errors caused by flickering light information which have to be corrected. Using the generated error masks, it is now possible to correct the image errors of the comparison images at the error positions indicated by the error mask by means of local correction values.

To determine the local correction values, the in 2nd shown an applied an estimate based on a comparison _{image time series} S ' _iN , ..., S' _i-1 , S ' _i stored in a data memory. The comparison image time series contains N + 1 stored images and can, for example, contain a total of 30 stored images. The stored images S ' _iN , ... S' _i-1 are modified copies of comparison images S _iN , ... S _i-1 , which belong to immediately preceding exposure series of the video sequence which the image sensor used in earlier, the current time step i immediately preceding time steps iN, ..., i-1. The stored images S ' _iN , ... S' _i-1 were generated with the same exposure time as the currently read comparison image S. In addition, the comparison _{image time series} S ' _iN , ..., S' _i-1 , S ' _{i contains} one modified copy of the currently read comparison image S _i . The stored images S ' _iN , ... S' _i-1 , S ' _i represent modified copies of the comparison images S _iN , ... S _i-1 , S _i in such a way that they are saved in the Data storage was downsized. In this way, the size of the data memory required for storing the comparison _{image time series} S ' _iN , ... S' _i-1 , S ' _i can be reduced. The stored images S ' _iN ,... S' _i-1 , S ' _{i can} also advantageously be stored in a motion-compensated form, in relation to relative movements of the vehicle 3rd to compensate for the image sensor.

In order to determine the local correction values for the image data of the comparison image Si, use is made of the fact that image errors caused by flickering light information occur quasi-randomly. The stored images S ' _iN ,... S' _i-1 recorded at earlier times can therefore, in contrast to the comparison image S _i, the on state of the headlight 1 or 2nd show when the on-phase of the pulsed LEDs of the headlamp has at least partially fallen within the exposure time of one or more of the stored images S ' _iN , ... S' _i-1 . In the embodiment of the 2nd for example, the image S ' _{iN shows} the on state of the headlight 1 and the image _{S'i-1 shows} the on state of the headlamp 2nd .

To determine the local correction values for the image data of the comparison image, according to the in 2nd The exemplary embodiment shown now determines and stores the maximum local brightness value from the image data of the comparison image time series S ' _iN , ... S' _i-1 , S ' _i for each individual pixel position, ie for all possible correction positions. In this way, a maximum brightness image max (S ' _i ) can be generated, the image data of which have the maximum brightness value of the comparison image time series S' _iN , ... S ' _i-1 , S' _i for each correction position. The image data of the maximum brightness image max (S ' _i ) are also subjected to a dilation in which the bright areas of the maximum brightness image are expanded.

The maximum local brightness values of the generated maximum brightness image can now be used as local correction values for the image errors of the comparison image S _i at the associated error positions. As a result of the dilation of the maximum brightness image, the downscaling of the stored images and relative movements of the vehicle 3rd For the image sensor between the time steps iN, ..., i-1, i of the video sequence, the maximum local brightness values of the maximum brightness image max (S ' _i ) do not have a high positional accuracy. However, high accuracy is not necessary in this regard either, since the highly precise error positions of the image errors present in the comparison image can be found in the error mask E (S _i ).

Using the comparison image S _i , the error mask E (S _i ) and the maximum brightness values of the maximum brightness image max (S ' _i ), a corrected comparison image S _{i, cor} is now generated in which the image errors which were corrected by the pulsed LEDs of the Headlights 1 , 2nd were caused. For this purpose, the brightness value of the comparison image S _{i is} replaced by the maximum brightness value of the maximum brightness image max (S ' _i ) at each error position indicated by the error mask E (S _i ), ie the maximum brightness values are inserted into the comparison image as local correction values. In the embodiment of the 2nd this has the consequence that the brightness values of all pixels of the comparison image S _i , which are the headlights operated with pulsed LEDs 1 , 2nd show by which the maximum brightness values from the maximum brightness image max (S ' _i ) are replaced. The corrected comparison image Si, cor accordingly shows both headlights in the on state and thus no longer has any image errors.

By repeating the in 2nd Detection and correction of image errors according to the invention on a plurality of successive exposure rows of the video sequence, it is possible according to the invention to generate a corrected video sequence with corrected comparison images Si, cor.

3rd shows a block diagram of a second embodiment of the detection and correction of image errors according to the invention.

An image sensor is shown 10th which is an HDR image sensor and has produced an HDR exposure series. Notwithstanding that in 2nd The first exemplary embodiment shown comprises the one shown in 3rd Shown exposure series in addition to the reference image L and the comparison image S, another comparison image VS. To simplify the illustration, the time index i of the images L _i , S _i and VS _i , which indicates the time step in which the image was taken, in the 3rd omitted. With regard to the properties of the images L, S and VS, in particular with regard to their different exposure times, reference can be made to the corresponding explanations for the in 1 shown example. In this exemplary embodiment, too, the image data are in the form of brightness values.

The block diagram of the 3rd also shows a flicker detection unit 21 A flicker correction unit for the detection of image errors caused by flickering image information 22 for the correction of image errors caused by flickering image information and an HDR postprocessing unit 24th which generates an output image through HDR postprocessing. The flicker detection unit 21 , the flicker correction unit 22 and the HDR postprocessing unit 24th can be implemented in software and / or hardware components of a Data processing unit 11 be the in 3rd is not shown, but in 5 is shown.

The image data of the exposure series with the three images L, S and VS are from the image sensor 10th via appropriate interfaces of both the flicker detection unit 21 as well as the flicker correction unit 22 provided. The image data of the reference image L are also processed via an interface of the HDR postprocessing unit without further processing 24th provided.

The flicker detection unit 21 leads in the function block 25th First, multiply the brightness values of the reference image L by the quotient 1 / ks in order to take into account the different exposure times and sensor sensitivities of the images L and S and to enable the image data of these two images to be compared directly. ks denotes the in connection with the 1 explained factor, which results from the ratio of the exposure times of both images as ks = T _L / Ts assuming the same sensor sensitivities. The image data S _CMP resulting from the multiplication are then in the function block 27 compared with the image data of the comparison image S and in this way local deviations between the image data of both images are determined. Depending on the determined local deviations of the image data, an error mask E (S) is generated, which displays the error positions of image errors within the comparison image S.

The flicker detection unit performs in a corresponding manner 21 in the function block 26 a multiplication of the brightness values of the reference image L by the quotient 1 / kv in order to take into account the different exposure times of the images L and VS and to enable direct comparison of the image data of these two images. Kv results from the ratio of the exposure times as kv = T _L / T _V. Alternatively, the function block 26 carry out the comparison with the already corrected image Scor and the quotient kvs. Instead of the reference image L, the already corrected image Scor is used and the value 1 / kvs with kvs = Ts / Tv instead of 1 / kv. The image data VS _CMP resulting from the multiplication are then in the function block 28 compared with the image data of the comparison image VS and in this way local deviations between the image data of both images determined. Depending on the determined local deviations of the image data, an error mask E (VS) is generated, which displays the error positions of image errors within the comparison image VS.

The generated error masks E (S) and E (VS) are sent to the flicker correction unit 22 passed on.

The flicker correction unit 22 has a function block 231 on, with the help of a data memory assigned to it 121 performs a maximum value-based image error correction of the image data of the comparison image S. The flicker correction unit 22 also has a function block 232 on, with the help of a data memory assigned to it 122 performs a maximum value-based image error correction of the image data of the comparison image VS. The data store 121 and the data store 122 can store areas in a common memory 12th be the one in the 3rd is not shown, but in the 5 is shown. In order to be able to carry out the image error correction, the function block 231 the image data of the comparison image S and the error mask E (S) and the function block 232 the image data of the comparison image VS and the error mask E (VS) are provided. The flicker correction unit generates on this basis 22 a corrected comparison image S _cor and a corrected comparison image VScor, their image data via corresponding interfaces of the HDR postprocessing unit 24th to be provided.

The HDR postprocessing unit 24th finally generates an output image HDR-OUT, which is an HDR output image, by HDR postprocessing using the reference image L, the corrected comparison image S _cor and the corrected comparison image VS _cor . The output image HDR-OUT can be directly displayed to a viewer, stored, transmitted over a network and / or further processed in further image processing steps.

4th shows a block diagram of an embodiment of an error correction according to the invention using maximum brightness values. It is a detailed representation of the already in the 3rd function blocks shown 231 and the data memory assigned to it 121 .

The image data of the comparison image S _i read in with the exposure series form the input data IN of the function block responsible for the maximum value-based image error correction of the image data 231 . The image data are available in the form of brightness values.

The image data can in particular also be present as values of the RGGB color components, such as are output by a Bayer sensor, for example. The blocks 31 , 32 , 33 , 121 , 35 and 34 are available once for each color component in this case. It is advantageous to process only the maximum of the two G values in the Bayer pattern and not to separate the two G values. Behind block 34 this processed G-value is via the multiplexer 36 used for both G values of the output signal Si, cor if the error mask E (S _i ) indicates an image error at the image position.

The function block 231 the data of the error mask E (S _i ) are provided, which indicate the error positions of image errors within the comparison image S _i . In the detailed representation of the 4th it can be seen that the image data of the comparison image S _i in the function block 31 first be scaled down by downscaling. This downscaling can advantageously take place, for example, in the form of a 1: 4 subsampling. Then the image data in the function block 32 subjected to a dilation, the bright areas of the comparison image S _{i being} widened. After dilation, the image data are stored in the function block 33 downsized again. This downscaling can advantageously take place, for example, in the form of a 1: 8 subsampling.

In this way, a modified copy S ' _{i of} the comparison image S _{i is} generated, which in the data memory 121 is saved. The data store 121 is a ring buffer that is configured so that it can record N + 1 stored images S ' _iN , ... S' _i-1 , S ' _i . In this case, when the modified copy S ' _{i of} the current, ie the last, read comparison image S _i in the ring memory 121 is saved, the oldest image S ' _iN-1 stored therein is deleted, so that the ring buffer always contains the most current N + 1 images stored. If the image data have values of the color components R, G and B of an RGB color space, the individual values of the color components R, G and B are stored in the ring memory.

In the function block 35 the maximum local brightness value is then determined and stored for all possible correction positions from the image data of all images of the comparison image time series S ' _iN , ... S' _i-1 , S ' _i . In this way, a maximum brightness image is generated, which is in the function block 34 is enlarged by upscaling. The image data of the resulting maximum brightness image have the maximum brightness value of the comparison image time series S ' _iN , ... S' _i-1 , S ' _i for each correction position.

In the function block 36 Finally, using the error mask E (S _i ), the image data of the by upscaling (reversal of function block 31 and 33 , Upscaling factor for example, 32: 1) maximum brightness image obtained and the image data of the comparison image S _i a corrected comparison image Si, cor generated by the maximum luminance values are inserted at the positions of the comparison image S _i for which the error mask E (S _i) Image error. The image data of the corrected comparison image S _{i, cor} form the output data OUT of the function block 231 .

This way, within the function block 231 the image errors of the comparison image S _i caused by flickering light information are corrected by a maximum value-based image error correction.

The comments on the embodiment of 4th apply accordingly to the in 3rd Function block shown and responsible for the image error correction of the comparison image VS. 232 and the data storage assigned to it 122 .

5 shows a block diagram of an image processing unit according to the invention. The image processing unit has a data processing unit 11 and a data store 12th for storing image data. The data processing unit can be, for example, a suitably programmed microprocessor, a digital signal processor (DSP), a specialized image processing processor, an FPGA (Field Programmable Gate Array), an ASIC (user-specific integrated circuit) or the like. The data processing unit 11 accesses the data storage read and write 12th to. This can be fragmented or consist of several storage units.

From an image sensor 10th , which can be a video camera, in particular a digital video camera, another type of camera or part of such a camera, images and / or exposure series comprising images are generated. The image sensor 10th is directly or indirectly with the image processing unit 13 connected so that the image data of the images or the exposure series in this way from the image processing unit 13 can be read. The image data can then be stored in the data memory 12th be cached.

After the processing of the image data in the manner described above by the data processing unit 11 can output images to a display unit 14 which can be, for example, a screen or other display, are transmitted for playback. The display unit 14 is for this purpose directly or indirectly with the image processing unit 13 connected.

However, it is also conceivable for the output image data to be stored on a suitable data carrier and / or to be transmitted further via a data connection.

The image processing unit 13 is preferably integrated in a vehicle, in particular in a motor vehicle. The image sensor 10th and the display unit 14 can in particular replace or supplement conventional rear-view mirrors. The image sensor 10th is aligned with the field of view to be observed and the display unit 14 arranged in a suitable viewing area for the driver.

QUOTES INCLUDE IN THE DESCRIPTION

This list of documents listed by the applicant has been generated automatically and is only included for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 2004/0012692 A1 [0009, 0010]
• US 2014/0153839 A1 [0011]

Claims (32)

Image processing unit (13) for recognizing and correcting image errors caused by flickering light information (1,2) in images (S _i , VS _i ) generated by an image sensor (10) with a data memory (12) for storing image data and a data processing unit (11 ), the data processing unit (11) - for reading in an exposure series, which comprises at least one reference image (L _i ) and at least one comparison image (S _i , VS _i ), the images of the exposure series showing the same scene (4) and the exposure time of the Reference image (L _i ) is longer than the exposure time of the comparison image (S _i , VS _i ), - and to determine local deviations of the image data of the comparison image (S _i , VS _i ) from the image data of the reference image (L _i ) by comparing the Image data of the comparison image (S _i , VS _i ) with the image data of the reference image (L _i ), and for generating an error mask (E (Si), E (VSi)) depending on the determined local n deviations of the image data, the error mask (E (S _i ), E (VS _i )) indicating the error positions of image errors within the comparison image (S _i , VS _i ), and for determining local correction values for the image data of the comparison image (S _i , VS _i ), each local correction value being assigned to a correction position within the comparison image (S _i , VS _i ) and an estimate being used to determine the local correction values for the image data of the comparison image (Si, VSi), which is based on the image data of a based on a data memory (12, 121, 122) comparison image time series (S'i N, ..., S'i), the comparison image time series (S'i N, ..., S'i) a number of stored images ( S'i N, ..., S'i 1), which were generated with the same exposure time as the comparison image (Si) at different times from the comparison image (Si) by the image sensor (10), and to generate one corrected comparison image (Si, cor, VSi, cor) by inserting the local one Correction values at the error positions of the comparison image (S _i , VS _i ) indicated by the error mask (E (Si), E (VSi)) are set up by accessing image data stored in the data memory (12). Image processing unit (13) after Claim 1 , characterized in that the image data are brightness values. Image processing unit (13) after Claim 1 or 2nd , characterized in that the image sensor (10) is an HDR image sensor and the exposure series is an HDR exposure series generated by the HDR image sensor. Image processing unit (13) according to one of the preceding claims, characterized in that the data processing unit (11) is set up to determine the local deviations in such a way that the image data are normalized and the normalized image data at a position of the comparison image (S _i , VS _i ) the normalized image data at the same position of the reference image (L _i ) are compared, in particular by determining a difference between the normalized image data at a position of the comparison image (S _i , VS _i ) and the normalized image data at the same position of the reference image (L _i ) . Image processing unit (13) after Claim 4 , characterized in that the data processing unit (11) is set up to normalize the image data, a relative light sensitivity ratio between the light sensitivity of the reference image (L _i ) and the light sensitivity of the comparison image (S _i , VS _i ) and / or an exposure time ratio between the exposure time of the reference image (L _i ) and the exposure time of the comparison image (S _i , VS _i ). Image processing unit (13) according to one of the preceding claims, characterized in that the data processing unit (11) is set up to compare the determined local deviations of the image data with a threshold value in order to generate the error mask (E (Si), E (VSi)). Image processing unit (13) according to one of the preceding claims, characterized in that the data processing unit (11) is set up to use an estimate for determining the local correction values for the image data of the comparison image (S _i , VS _i ), which is based on the image data of the reference image (L _i ) and the ratio between the exposure time of the reference image (L _i ) and the exposure time of the comparison image (S _i , VS _i ). Image processing unit (13) according to one of the preceding claims, characterized in that the data memory (12, 121, 122) in which the comparison image time series (S ' _iN , ..., S' _i ) is held is a ring memory and / or the comparison image time series (S _', ..., S' _i) a plurality of stored images (S _', ..., S' _i-1), which with the same exposure time as the comparison image (Si) at earlier time points when the comparison image (Si) was generated by the image sensor (10). Image processing unit (13) according to one of the preceding claims, characterized in that the data processing unit (11) is set up to save the images of the comparison image time series (S ' _iN , ..., S' _i ) before being stored in the data memory (12, 121 , 122) undergo dilation and / or downsize and / or compress. Image processing unit (13) according to one of the preceding claims, characterized in that the data processing unit (11) is set up to _display the images of the comparison image time series (S ' _iN , ..., S' _i ) in a manner related to the comparison image (S _i ) to store motion-compensated form and / or before determining the local correction values for the image data of the comparison image, a motion compensation of the image data of the stored images of the comparison image time series (S ' _iN , ..., S' _i ) in relation to the comparison image (S _i ) perform. Image processing unit (13) according to one of the preceding claims, characterized in that the data processing unit (11) for determining the local correction values for the image data of the comparison image (Si) is set up such that a maximum local brightness value from the image data of the images of the comparison image time series (S ' _iN , ..., S' _i ) is determined and the maximum local brightness value is used as the local correction value. Image processing unit (13) according to one of the preceding claims, characterized in that the data processing unit (11) is set up _{to produce} an output image from the exposure series by HDR postprocessing using at least one corrected comparison image (S _{i, cor} , VS _{i, cor} ) to generate the viewer. Image processing unit (13) according to one of the preceding claims, characterized in that the exposure series is part of an exposure series sequence generated by the image sensor (10), the exposure series sequence comprising a sequence of exposure series and the sequence of exposure series showing successive scenes (4) which may contain flickering light information (1,2). Image processing unit (13) after Claim 13 , characterized in that the exposure time of at least one reference image (L _i ) is between 50% and 100% of the image duration of the exposure series sequence, in particular between 25/3 ms and 50/3 ms, and / or the exposure time of at least one comparison image (Si) is 1 ms or less. Image processing unit (13) after Claim 13 or 14 , characterized in that the data processing unit (11) is set up to repeat the detection and correction of image errors caused by flickering light information (1, 2) and to apply them to a plurality of exposure series of the exposure series sequence or to each exposure series of the exposure series sequence in order to correct a correction Generate exposure sequence that includes corrected comparison images (Si, cor, VSi, cor). Image processing unit (13) according to one of the Claims 13 to 15 , characterized in that the data processing unit (11) is set up to apply corrective measures for correcting remaining image errors caused by flickering light information (1, 2) to the image data of the images of the corrected exposure series sequence, the corrective measures being based on motion estimation and motion compensation of the Support image data of the images of the corrected exposure series sequence and / or a mixing of the image data of different images of the corrected exposure series sequence. Computer program for the detection and correction of image errors caused by flickering light information (1,2) in images (S _i , VS _i ) generated by an image sensor (10) with program code means which - for reading in an exposure series, the at least one reference image (L _i ) and comprises at least one comparison image (S _i , VS _i ), the images of the exposure series showing the same scene (4) and the exposure time of the reference image (L _i ) being longer than the exposure time of the comparison image (S _i , VS _i ), - and for determining local deviations of the image data of the comparison image (S _i , VS _i ) from the image data of the reference image (L _i ) by locally comparing the image data of the comparison image (S _i , VS _i ) with the image data of the reference image (L _i ), - and for generating an error mask (e (Si), e (VSi)) as a function of the determined local deviations of the image data, wherein the error filter (e (S _i), e (VS _i)), the defect positions of image defects within the V comparative image (S _i , VS _i ) displays, and for determining local correction values for the image data of the comparison image (S _i , VS _i ), each local correction value being assigned to a correction position within the comparison image (S _i , VS _i ) and for determining of the local correction values for the image data of the comparison image (Si, VSi), an estimate is applied which is based on the image data of a comparison image time series (S'i N, ..., S'i) held in a data memory (12, 121, 122) , the comparison image time series (S'i N, ..., S'i) comprising a number of stored images (S'i N, ..., S'i 1), which have the same exposure time as the comparison image (Si) at times other than the comparison image (Si) generated by the image sensor (10), and for generating a corrected comparison image (Si, cor, VSi, cor) by inserting the local correction values on the error mask (E (Si), E (VSi)) displayed error positions of the comparison image (S _i , VS _i ) ei are directed when the computer program is executed with a data processing unit (11). Computer program after Claim 17 , characterized in that the image data are brightness values. Computer program after Claim 17 or 18th , characterized in that the image sensor (10) is an HDR image sensor and the exposure series is an HDR exposure series generated by the HDR image sensor. Computer program according to one of the preceding claims, characterized in that the program code means, when the computer program is executed with a data processing unit (11), are set up to determine the local deviations in such a way that the image data are standardized and the standardized image data at a position of the comparison image ( S _i , VS _i ) are compared with the normalized image data at the same position of the reference image (L _i ), in particular by determining a difference between the normalized image data at a position of the comparison image (S _i , VS _i ) and the normalized image data at the same Position of the reference image (L _i ). Computer program after Claim 20 , characterized in that the program code means, when the computer program is executed with a data processing unit (11), are set up to normalize the image data, a relative light sensitivity ratio between the light sensitivity of the reference image (L _i ) and the light sensitivity of the comparison image (S _i , VS _i ) and / or to use an exposure time ratio between the exposure time of the reference image (L _i ) and the exposure time of the comparison image (S _i , VS _i ). Computer program according to one of the preceding claims, characterized in that the program code means, when the computer program is executed with a data processing unit (11), are set up to generate the determined local deviations of the error mask (E (Si), E (VSi)) Compare image data with a threshold. Computer program according to one of the preceding claims, characterized in that the program code means, when the computer program is executed with a data processing unit (11), are set up to use an estimate to determine the local correction values for the image data of the comparison image (S _i , VS _i ) , which is based on the image data of the reference image (L _i ) and the ratio between the exposure time of the reference image (L _i ) and the exposure time of the comparison image (S _i , VS _i ). Computer program according to one of the preceding claims, characterized in that the data memory (12, 121, 122) in which the comparison _{image time series} (S ' _iN , ..., S' _i ) is held is a ring memory and / or the comparison image time series ( S ' _iN , ..., S' _i ) comprises a number of stored images (S ' _iN , ..., S' _i-1 ) which have the same exposure time as the comparison image (Si) at earlier times than the comparison image (Si) were generated by the image sensor (10). Computer program according to one of the preceding claims, characterized in that the program code means, when the computer program is executed with a data processing unit (11), are set up to save the images of the comparison image time series (S ' _iN , ..., S' _i ) undergo dilatation in the data memory (12, 121, 122) and / or downsize and / or compress it by downscaling. Computer program according to one of the preceding claims, characterized in that the program code means, when the computer program is executed with a data processing unit (11), are set up to display the images of the comparison image time series (S ' _iN , ..., S' _i ) in one With respect to the comparison image (S _i ) to store motion-compensated form and / or before determining the local correction values for the image data of the comparison image, a motion compensation of the image data of the stored images of the comparison image time series (S ' _iN , ..., S' _i ) in relation to perform on the comparison image (S _i ). Computer program according to one of the preceding claims, characterized in that the program code means, when the computer program is executed with a data processing unit (11), are set up to determine the local correction values for the image data of the comparison image (Si) in such a way that a maximum local brightness value from the Image data of the images of the comparison image time series (S ' _iN , ..., S' _i ) are determined and the maximum local brightness value is used as the local correction value. Image processing unit (13) according to one of the preceding claims, characterized in that the program code means, when the computer program is executed with a data processing unit (11), are set up to remove from the exposure series by HDR postprocessing using at least one corrected comparison image (Si, cor , VSi, cor) to generate an output image for the viewer. Computer program according to one of the preceding claims, characterized in that the exposure series is part of an exposure series sequence generated by the image sensor (10), the exposure series sequence comprising a sequence of exposure series and the sequence of exposure series sequentially sequential scenes (4) shows that flickering light information (1,2) may contain. Computer program after Claim 29 , characterized in that the exposure time of at least one reference image (L _i ) is between 50% and 100% of the image duration of the exposure series sequence, in particular between 25/3 ms and 50/3 ms and / or the exposure time of at least one comparison image (S _i ) 1 ms or less. Computer program after Claim 29 or 30th , characterized in that the program code means, when the computer program is executed with a data processing unit (11), are set up to repeat the detection and correction of image errors caused by flickering light information (1, 2) and to a plurality of exposure series of the exposure series sequence or to be applied to each exposure series of the exposure series sequence in order to generate a corrected exposure series sequence which comprises corrected comparison images (Si, cor, VSi, cor). Computer program according to one of the Claims 29 to 31 , characterized in that the program code means, when the computer program is executed with a data processing unit (11), are set up to apply a correction method to the image data of the images of the corrected exposure series sequence for correcting remaining image errors caused by flickering light information (1, 2), wherein the correction method is based on a motion estimation and a motion compensation of the image data of the images of the corrected exposure series sequence and / or a mixing of the image data of different images of the corrected exposure series sequence.

Images