DE102018115991B4 - DIGITAL CIRCUIT FOR CORRECTING A VIGNETTING EFFECT IN PIXEL VALUES OF AN ELECTRONIC CAMERA IMAGE - Google Patents

Abstract

Correction method for correcting a vignetting effect in values of pixels of an image (I1) of an electronic camera (10), comprising: - determining the value of a correction function at the position of a pixel iteratively by multiplying one or more factors with an already determined value of the correction function at the position of another pixel, and correcting the value of the pixel, the correcting comprising multiplying the value of the pixel by the value of the correction function at the position of the pixel, the correcting correcting the vignetting effect in the value of the pixel, the correction function being a Exponential function to a base, preferably either Euler's number e or the number 2 or 10, the argument of which is a one- or multi-dimensional polynomial of degree n ≥ 2, where the value of the correction function at the position of the pixel based on a cascading of feedback multiplications, which are carried out by a circuit of at least two cascaded feedback multipliers, are determined such that the values of the correction function at the positions of the pixels are continuously calculated, multiplying the value of the pixel by the value of the correction function at the position of the Pixel is carried out with a multiplier connected downstream of the at least two cascaded feedback multipliers.

Description

The invention relates to a correction method for correcting a vignetting effect in pixel values of an image from an electronic camera. The invention further relates to a corresponding correction device and an electronic camera which includes an image sensor and the correction device. The invention further relates to a computer device and a computer program product.

BACKGROUND OF THE INVENTION

Electronic cameras are used, among other things, in industrial applications, such as those in the EP patent EP 2 929 503 B1 are described in detail.

1 shows an electronic camera in a simplified application. There, a lamp 16 usually illuminates a scene 17 with objects 18. This scene is recorded by a camera 10. The scene is imaged onto an image sensor 12 through a lens 11. From its raw image, an image processing unit 13 generates a result image, which is transmitted via an interface 14 with an output 15 for further use.

In an industrial application, the camera is often part of a machine. For reasons of profitability, it is often desired that the machine runs quickly. This also requires the camera to record images at high speed from the image sensor 12, process them quickly in the image processing unit 13 and then transmit them quickly. As in EP 2 929 503 B1 explained, for the high speed of the machine it is fundamentally important to keep the latency between the recording of the raw image and the transmission as low as possible.

Image sensors generally consist of a large number of pixels. These can be arranged, for example, in the form of a one-dimensional matrix, as in 2 (a) shown. Then we speak of a line sensor or “line scan sensor”. The pixels are particularly common in the form of a two-dimensional matrix, as in 2 B) arranged. Then one also speaks of a matrix sensor. Matrix sensors have a very large number of pixels, usually in the order of millions.

The image sensor can be monochrome, as in 2 (a) and (b) is indicated. Then the pixels all have identical or at least very similar spectral sensitivity. Color sensors, in turn, often have an arrangement of color filters of different colors, which is referred to as a mosaic filter. 2(c) shows a mosaic filter according to the US patent US 3,971,065 B1 , which is called the “Bayer Pattern” after its inventor and is made up of filters in the colors red R, green G and blue B.

The raw image is typically read out from the image sensor 12 progressively. 3 shows a schematic representation of a progressive readout process, which is carried out here line by line. The signal of the first pixel 30 is first output in the line direction 32, then the signal of the pixel 31 adjacent to it to the right and so on in the reading direction. When the last pixel 33 of a line is reached, the line-by-line reading carries out a jump 34 to the next line 35 and begins there again with the first pixel.

In 4 (a) An image sensor is shown as an example, the pixel matrix of which is divided into two blocks 40 and 41 and in which the reading in these two blocks takes place in mirror image of one another. Such an image sensor is, for example, from the US patent US 7,362,358 B2 known. There are numerous other ways to divide the pixel matrix of an image sensor into blocks, for example into strips or quadrants. According to common usage, this process is referred to here in English as “split readout”.

Furthermore, image sensors are known which transmit more than one pixel value in each cycle of a readout process, for example the PB-MV13 image sensor from Photobit Corporation. This is based on the 4(b) explained. Here the pixels of the first line are transmitted in the clock cycles t ₀₀ , t ₀₁ and t ₀₂ . In each of these clock cycles, two pixel values of adjacent pixels are transmitted as an example. This also applies when reading out the second line in the clock cycles t ₁₀ , t ₁₁ and t ₁₂ and so on. The pixel values transmitted at the same time are viewed here as a vector and this readout scheme is therefore also referred to as “vectorized”. More than two pixels can also be combined into a vector when reading out, e.g. 4, 8, 10, 16, etc.

A certain type of reading leads to a data format in which the image data of the raw image is available. It is common to convert data formats into other data formats. For example, image data that was read out with “split readout” can be converted into progressive data by buffering the image data of a line and then progressively reading it out at double the frequency. It is also possible to increase or decrease vector widths, for example by Clock frequency is reduced or increased by the corresponding factor.

As in EP 2 929 503 B1 explained, it is common practice to carry out image corrections in image processing means. These image corrections usually have to be calculated in real time. In principle, a computer can be used for this, e.g. a computer with a processor (CPU), microcontroller (MCU) or graphics processor (GPU). However, the high speed of the image processing unit 13 required for the operation of the machine and the simultaneous cost pressure often require the use of special electronic circuits, for example field-programmable gate array (FPGA), system-on-chip (SOC) or image signal processor (ISP). Depending on the architecture, it is desirable to use digital resources sparingly, as these consume, for example, computing time, computing power, logic resources, chip area or energy and their use is therefore fundamentally associated with costs. The heat development associated with high power consumption is also often a problem in practice.

In the FPGA, SOC and ISP circuits mentioned, the calculation takes place in special digital hardware provided by configuration or chip design. In principle, it is still possible to carry out such calculations in analog circuits, although these have steadily become less important for image processing in recent decades.

In digital circuits, it is comparatively easy, resource-saving and cost-effective to carry out additions, subtractions and multiplications. For example, additions and subtractions can be carried out in the FPGA using logic elements. Alternatively, dedicated multipliers are also available for multiplications. Therefore, these operations are referred to here as hardware-friendly operations and their use in hardware is considered beneficial. Other operations, such as division, taking roots or calculating transcendental functions such as cosine, hyperbolic cosine, exponential function and logarithm, require considerable effort or costs in hardware. These operations are referred to herein as hardware-unfriendly operations and their use in hardware is considered disadvantageous.

As in EP 2 929 503 B1 explained, in industrial applications there is often a computer that takes over the transmitted data 15 and is able to parameterize the camera 10. Depending on the application, it may be possible to offload extensive or complicated calculations to the computer. This is particularly true if these calculations are not time-critical and are, for example, only used to configure the camera 10, which then works independently in subsequent operation. In principle, hardware-unfriendly operations can be performed better on computers than in hardware. It is therefore often advantageous in terms of resource requirements and costs to move such operations to the computer, but this requires more computing time.

Electronic cameras are often used for measurement tasks in industrial applications. For example, workpieces to be tested are recorded with a camera and the brightness is compared pixel by pixel by the computer with a target value. Lower and/or upper threshold values are often used to make a decision about whether the workpiece is defective. Therefore, it is fundamentally desirable that equally bright parts of the scene 17 appear equally bright in the resulting image, because an inhomogeneous brightness distribution could disrupt this decision-making process.

It is known that lenses generally do not produce equally bright images. They usually create a so-called edge light fall-off, which is often also called edge shading. Sometimes this effect is also generally referred to as vignetting. In English we speak of “vignetting” or “lens roll-off”. The edge light falloff causes outer parts of the image in particular to appear too dark compared to other, especially central, parts. So it is basically an intensity error. A good description and presentation of this effect can be found in B. Jähne et al. (Ed.), “Handbook of Computer Vision and Applications”, Academic Press, 1999.

It is desirable to correct these intensity errors to achieve a uniformly bright image. The resulting image should correctly reproduce the intensity distribution on the object 18, i.e. as it would result without the intensity error generated by the lens 11.

STATE OF THE ART

There are a considerable number of publications on correcting the edge light fall-off, for example the US and EP patents US 9,414,052 B2 , US 7,453,502 B2 , EP 1 700 268 B1 , US 7,391,450 B2 , EP 1 711 880 B1 , EP 2 278 788 B1 , US 7,692,700 B2 , US 7,834,921 B1 , US 7,634,152 B2 , US 7,548,262 B2 , US 7,920,171 B2 , US 7,907,195 B2 , US 7,388,610 B2 , US 7,408,576 B2 , US 8,130,292 B2 , US 8,218 , 037 B2 , US 7,961,973 B2 , US 8,472,712 B2 , US 8,712,183 B2 , US 8,577,140 B2 and US 7,817,196 B1 .

5 explains a widely used, basic method for correcting edge shading. This method is used by most procedures, e.g. in JC Russ, “The Image Processing Handbook”, ^2nd Edition, CRC Press, 1994. At setup time 50, in an industrial context e.g. during the production of a camera or during the commissioning of a machine , in which the camera is used, a reference image I ₀ is recorded. This reference image I ₀ is often designed in such a way that it is easy to evaluate, for example in that the recorded scene 17 consists only of a homogeneous surface. For example, the scene 17 can show a white reference map or it can be completely empty with no objects 18 in it. Then a mathematical model M is applied and with its help parameters P are determined. At runtime 51, a correction K is made for a raw image I ₁ using these parameters P. This results in a corrected result image I ₂ .

6 explains one way in which such a correction K according to JC Russ can be carried out. This is done by applying the same model 53 at runtime 51 that was also used as model 52 to determine the parameters P during setup time 50. Since it can be assumed that edge shading is a multiplicative process, the correction then consists of dividing 54 by the model value M of the edge shading, because division is the inverse operation of multiplication. This eliminates the effect of edge shading. The disadvantage here is that the division is hardware-unfriendly and has to be carried out 51 times for each pixel at runtime and therefore very frequently, which means that it cannot be easily relocated to a computer, for example.

A different basic procedure is used, for example, in the US patent US 9,414,052 B2 described and is in 7 shown. Here the required division 60 is carried out on the image data of the reference image I ₀ , for example by calculating its reciprocal value pixel by pixel. A model 61 suitable for the reciprocal values is then applied to the reciprocal image values and parameters 62 are obtained for this. In this case, the correction K can be carried out by determining the output values of the model 63, which is identical to the model 61, at runtime 51 and multiplying them to the pixel values of the raw image I ₁ . Compared to the previously described division, the multiplication 64 carried out at runtime 51 is more hardware-friendly.

Numerous methods for correcting edge light falloff work radially and refer to a center point, for example the US and EP patents US 7,453,502 B2 , EP 1 700 268 B1 , EP 2 278 788 B1 , US 7,692,700 B2 , US 7,834,921 B1 , US 7,634,152 B2 , US 7,548 262 B2 , US 7,920,171 B2 , US 7,907,195 B2 , US 7,408 , 576 B2 , US 8,130,292 B2 , US 7,961,973 B2 , US 8,472,712 B2 and US 8,577,140 B2 . 8 (a) explains the difference between the geometric center 71 and the optical center 72 of an image. This difference is also, for example, in US 7,548,262 B2 and US 7,453,502 B2 described. The geometric center 71 is located in the middle of the pixel array 70 of the image sensor and can, for example, in US 7,920,171 B2 can generally be used for vignetting correction. It results directly from the mechanical properties of the image sensor and is therefore easy to determine.

The optical center 72, on the other hand, describes the center of the effects emanating from the lens. It typically has an offset from the geometric center 71 and often coincides with the position of the optical axis of the lens on the image sensor. Since the effect of edge shading or vignetting usually occurs in a good approximation with radial symmetry around the optical center 72, a correction based on the optical center 72 instead of the geometric center 71 usually provides better results. On the other hand, the correct determination of the optical center 72 is fundamentally associated with technical difficulties and sometimes requires extensive calculations, as shown, for example EP 1 700 268 B1 , EP 2 278 788 B1 and US 7,634,152 B2 emerges. In US 7,692,700 B2 There is an indication that it is advantageous to carry out a correction in such a way that the image brightness in the optical center 72 is not changed by the correction.

In principle, it is possible to treat the entire image as a uniform zone 73 in which the correction is carried out in a uniform manner, ie in particular with identical parameters P. This will be in 8 (a) indicated and is, for example, in US 7,45,3502 B2 , US 7,634,152 B2 , US 7,548,262 B2 and US 7,920,171 B2 described.

Alternatively, the image can also be as in 8(b) shown, are divided into concentric zones 82, 83 and 84 around a center point 81, with special parameters for the respective zone being selected from the entire parameters P depending on the zone be chosen. The use of a radial correction function, which is defined as a function of the radius starting from a center 81 and divided into areas, also leads to such a situation. Such a subdivision can be carried out, for example, using support points between which interpolation takes place. Methods that can be described by such radial zones are described, among others, in EP 1 700 268 B1 , US 7,834,921 B1 , US 7,907,195 B2 , US 7,408,576 B2 and US 7,961,973 B2 called.

8(c) shows another widely used way to divide the image 90 into zones, often referred to as blocks, along row or column directions. A basic idea is to select specific parameters for the respective block from the entire parameters P. Such a division can be made using case distinctions, for example. The use of grid-shaped distributed support points, between which interpolation takes place, and the use of two-dimensional spline elements ultimately lead to such a division into blocks, with the special parameters then resulting from the values of the support points adjacent to a pixel to be corrected. Such an approach is used, for example, in US 9,414,052 B2 , EP 1 711 880 B1 , US 7,388,610 B2 , US 8,218,037 B2 , US 8,472,712 B2 , US 8,712,183 B2 and US 7,817,196 B1 described.

• In US 8,472,712 B2 , US 8,218,037 B2 und US 7,961,973 B2 werden z.B. lineare Funktionen genannt, zu denen auch die lineare Interpolation gehört. Da eine lineare Funktion die Form einer Randabschattung oder Vignettierung nur sehr schlecht beschreibt, werden gegebenenfalls lineare Korrekturen mit jeweils einem festen Parametersatz nur kleinräumig im Bereich einer Zone angewandt, an die sich üblicherweise jeweils weitere Zonen mit linearen Korrekturen mit anderen Parametersätzen anschließen. Somit werden für eine lineare Korrektur der Randabschattung oder Vignettierung eine große Anzahl an linearen Funktionen benötigt, die in zahlreichen Zonen angewandt werden. Dies führt zu einem hohen Speicherbedarf für die Parameter, wie z.B. Koeffizienten oder Stützstellen.

The mathematical model M is fundamentally based on a mathematical function that depends on the radius or a power of the radius or on coordinates or differences of coordinates or another function of coordinates. A wide variety of functions are used in the state of the art:

• In US 8,472,712 B2 , US 8,218,037 B2 and US 7,961,973 B2 For example, linear functions are called, which also include linear interpolation. Since a linear function only describes the shape of edge shading or vignetting very poorly, linear corrections with a fixed set of parameters are only applied on a small scale in the area of a zone, which is usually followed by further zones with linear corrections with different sets of parameters. Therefore, for a linear correction of edge shading or vignetting, a large number of linear functions are required, which are applied in numerous zones. This leads to a high memory requirement for the parameters, such as coefficients or support points.

Linear functions form first degree polynomials. This leads to the idea of improving the correction by using higher degree polynomial functions instead, which are able to better approximate the function actually required. Such polynomial functions are often used as a function of radius, such as in US 7,453,502 B2 , EP 1 700 268 B1 , US 7,548,262 B2 , US 7,907,195 B2 and US 7,408,576 B2 . Two-dimensional polynomials can also be used, which are defined on surfaces and usually take two coordinate values, e.g. x and y, as input values. Corresponding suggestions or solutions can be found, for example, in Russ and in US 7,692 , 700 B2 and US 8,712,183 B2 described.

Another approach uses surface curves based on two-dimensional spline or Bezier functions and is described, for example, in US 9,414,052 B2 used. It should be noted at this point that Bezier or spline curves can also be converted into equivalent polynomials or expressed using them. A distinction between spline, Bezier and polynomial functions and curves is therefore omitted here and in the context of the invention they are summarized under the generic term “polynomial functions”.

Finally, another possibility is to use trigonometric functions and functions derived from them, such as cos ⁴ . This will also be in US 9,414,052 B2 described. The cosine hyperbolicus function cosh is also used as a basis for a solution to the vignetting problem, such as in US 7,920,171 B2 explained. The fundamental disadvantage of such a solution is that calculating the values of a trigonometric function is always a hardware-unfriendly operation.

A. Kordecki et al., “Fast Vignetting Reduction Method for Digital Still Camera,” IEEE, International Conference on Methods and Models in Automation and Robotics (MMAR), 2015, pp. 1145-1150 , discloses a new method for reducing vignetting effects based on two images captured with different camera/lens settings. Changing the lens opening also changes the vignetting effect in the images. These differences are used to estimate a vignetting function used for vignetting reduction.

AA Sawchuk, “Real-Time Correction of Intensity Nonlinearities in Imaging Systems,” IEEE, Transactions on Computers, 1977, pp. 34-39 , presents an analysis of two-dimensional imaging systems with general brightness nonli nearities and discusses various methods for real-time correction. Exact corrections for measurement purposes and approximate corrections for the human observer are described and implementations in hardware, in particular using programmable memory components (PROMs), are presented. In addition, calibration procedures for such systems as well as an assessment of the system complexity are presented.

US 6,175,613 B1 discloses that the blur of radiological images is estimated by four independent runs that recursively implement the same exponential function, and the blur of the previous image is subtracted from the current image.

It is desirable to be able to carry out the correction in an image processing unit, preferably in a camera, at high speed. It is desirable to save digital resources for the digital operations that are carried out at high speed, to use hardware-friendly operations, to reduce their number to a minimum if possible and to avoid hardware-unfriendly operations as much as possible.

SUMMARY OF THE INVENTION

The invention is based on the object of providing a correction method for correcting a vignetting effect in values of pixels of an image from an electronic camera, which is suitable for meeting one or more, preferably as many as possible, of the requirements mentioned above. Furthermore, the invention is based on the object of providing a corresponding correction device as well as a computer device and a computer program product. Finally, the invention is based on the object of providing an electronic camera which includes the correction device according to the invention.

• - Bestimmen des Wertes einer Korrekturfunktion an der Position eines Pixels iterativ durch Multiplikation eines oder mehrerer Faktoren mit einem bereits bestimmten Wert der Korrekturfunktion an der Position eines anderen Pixels, und
• - Korrigieren des Wertes des Pixels, wobei das Korrigieren ein Multiplizieren des Wertes des Pixels mit dem Wert der Korrekturfunktion an der Position des Pixels umfasst, wobei das Korrigieren den Vignettierungseffekt in dem Wert des Pixels korrigiert,
  • wobei die Korrekturfunktion eine Exponentialfunktion zu einer Basis, vorzugsweise entweder der Eulerschen Zahl e oder der Zahl 2 oder 10, umfasst, deren Argument ein ein- oder mehrdimensionales Polynom des Grades n ≥ 2 ist,
  • wobei der Wert der Korrekturfunktion an der Position des Pixels basierend auf einer Kaskadierung von rückgekoppelten Multiplikationen, die von einer Schaltung von mindestens zwei kaskadierten rückgekoppelten Multiplizierwerken durchgeführt werden, so bestimmt wird, dass die Werte der Korrekturfunktion an den Positionen der Pixel fortlaufend berechnet werden,
  • wobei das Multiplizieren des Wertes des Pixels mit dem Wert der Korrekturfunktion an der Position des Pixels mit einem den mindestens zwei kaskadierten rückgekoppelten Multiplizierwerken nachgeschalteten Multiplikator durchgeführt wird.

According to a first aspect of the invention, there is provided a correction method for correcting a vignetting effect in values of pixels of an electronic camera image, comprising:

• - Determining the value of a correction function at the position of a pixel iteratively by multiplying one or more factors with an already determined value of the correction function at the position of another pixel, and
• - correcting the value of the pixel, the correcting comprising multiplying the value of the pixel by the value of the correction function at the position of the pixel, the correcting correcting the vignetting effect in the value of the pixel,
  • wherein the correction function comprises an exponential function to a base, preferably either Euler's number e or the number 2 or 10, the argument of which is a one- or multi-dimensional polynomial of degree n ≥ 2,
  • wherein the value of the correction function at the position of the pixel is determined based on a cascading of feedback multiplications carried out by a circuit of at least two cascaded feedback multipliers, such that the values of the correction function at the positions of the pixels are continuously calculated,
  • wherein the multiplication of the value of the pixel by the value of the correction function at the position of the pixel is carried out with a multiplier connected downstream of the at least two cascaded feedback multipliers.

It is assumed here that typical vignetting effects can be described well or sufficiently well by the so-called cos ⁴ law. However, as mentioned above, calculating the values of the cos ⁴ function is a hardware-unfriendly operation. The inventor has now recognized that the cos ⁴ function can be described very well by an exponential function to a basis, the argument of which is a one- or multi-dimensional polynomial of degree n ≥ 2, for example a Gaussian normal distribution. According to the inventor's knowledge, such a function has many properties that predestine it for use as a correction function to correct a vignetting effect in pixel values of an image from an electronic camera. For example, a negation of its argument gives the reciprocal value, so that the function can be converted with a very simple operation to correct the values of the pixels either by dividing or multiplying the values of the pixels by the values of the function at the positions of the pixels. The use of the claimed exponential function as a correction function thus makes it possible to provide a correction method for correcting a vignetting effect in values of pixels of an image from an electronic camera that largely meets the above-mentioned requirements.

The inventor has recognized that for a correction function that includes an exponential function to a base, preferably either Euler's number e or the number 2 or 10, the argument of which is a one- or multi-dimensional polynomial of degree n ≥ 2, the value the correction function The position of a pixel can be determined very easily and based on hardware-friendly operations iteratively or recursively by multiplying one or more factors with an already determined value of the correction function at the position of another pixel. This makes it possible to carry out the correction even with a large number of pixels at high speed, preferably in a camera, either in hardware or in software.

In addition, a structure based on a cascading of feedback multiplications can be implemented very easily and efficiently in both hardware and software.

According to a further advantageous development of the invention, factors of the feedback multiplications are each determined as the value of the exponential function to the base, the argument of which is a weighted sum of one or more of the coefficients of the polynomial.

According to a further advantageous development of the invention, the image is a two-dimensional image and the cascading of the feedback multiplications includes first feedback multiplications in a first direction, for example in the row direction, of the image and second feedback multiplications in a second direction, for example in the column direction Image, wherein an output value of the second feedback multiplications provides an input value of the first feedback multiplications.

• - jeweils separat voneinander verarbeitet werden, oder
• - so verarbeitet werden, dass in einem Zeitschritt das Bestimmen der Werte der Korrekturfunktion an den Positionen von Pixeln mindestens zweier verschie dener Kanäle auf Grundlage eines gemeinsamen bereits bestimmten Wertes der Korrekturfunktion an der Position eines anderen Pixels erfolgt.

According to an advantageous development of the invention, the determination of the values of the correction function at the positions of the pixels is parallelized in such a way that the pixels are divided into a number of channels which:

• - each processed separately from each other, or
• - be processed in such a way that the values of the correction function at the positions of pixels of at least two different channels are determined in one time step on the basis of a common already determined value of the correction function at the position of another pixel.

By parallelizing the determination of the values of the correction function at the positions of the pixels, the speed of the correction can be further increased. The division of the pixels into a number of channels, each of which is processed separately from one another, offers the advantage of a very simple structure. However, this can have the disadvantage that calculation errors, e.g. rounding errors, can accumulate in different ways in different channels, which can lead to visible differences in pixels of different channels and thus to undesirable visible stripes in the resulting image. This can be avoided by processing the pixels divided into the number of channels in such a way that in one time step the values of the correction function at the positions of pixels of at least two different channels are determined based on a common already determined value of the correction function at the position of one another pixel.

According to an advantageous development of the invention, the image is a color image which is present as an image of a mosaic filter, with the value of each pixel being assigned to a color channel.

• - Korrigieren der Werte der Pixel mit einer eigenen Korrekturfunktion für jeden Farbkanal, vorzugsweise dergestalt, dass sich die Korrekturfunktionen für jeweils zwei unterschiedliche Kanäle nur in einem konstanten Offset der Werte der Korrekturfunktion unterscheiden, oder
• - Korrigieren der Werte der Pixel mit einer für alle Farbkanäle gemeinsamen Korrekturfunktion.

According to an advantageous development of the invention, the values of the pixels of the image are color values of two or more color channels and correcting the values of the pixels includes:

• - Correcting the values of the pixels with a separate correction function for each color channel, preferably in such a way that the correction functions for two different channels only differ in a constant offset of the values of the correction function, or
• - Correcting the values of the pixels with a correction function common to all color channels.

In this way, color images can also be processed in a suitable manner with the correction according to the invention.

According to an advantageous development of the invention, only values of pixels of a section of the image are corrected and a shift in the origin of the section relative to the origin of the image by a corresponding shift in the correction function is taken into account in such a way that at least some of the parameters of the correction function are represented by shifted parameters the correction function can be replaced. In this way, the correction can be applied very efficiently to a section of the image, e.g. an “area of interest” or “region of interest”.

According to an advantageous development of the invention, several zones are defined in the image and the values of the pixels for each zone are corrected with parameters of the correction function specific to this zone. With the help of such zoning it may be possible to achieve even greater accuracy of the correction. This is particularly the case if the vignetting effect does not apply to the entire image with sufficient precision with a single Set of parameters of the correction function can be described.

According to an advantageous development of the invention, the respective parameters of the correction function for two adjacent zones are such that the transition of the correction function between the two zones is continuous, and preferably continuously differentiable. In this way, visible artifacts can be avoided when transitioning from one zone to the next.

According to another advantageous development of the invention, the zones relate to a center, preferably the optical center, of the image. This is advantageous because the center, in particular the optical center, of the image usually also represents the center of the vignetting effect and in many cases this is also radially symmetrical.

• - Erzeugen eines eindimensionalen Datenvektors in der einen Dimension aus den Werten der Pixel des Bildes,
• - iteratives Fitten einer eindimensionalen Ausgleichsparabel an den eindimensionalen Datenvektor,

und dass für den Fall, dass das Bild mit einem zweidimensionalen Bildsensor erzeugt wurde, das Schätzen einer Koordinate des optischen Mittelpunkts in einer ersten Dimension umfasst:

• - Mitteln der Werte der Pixel des Bilder in der zweiten Dimension zum Erzeugen eines eindimensionalen Datenvektors von gemittelten Werten in der ersten Dimension, und
• - iteratives Fitten einer eindimensionalen Ausgleichsparabel an den eindimensionalen Datenvektor.

According to an advantageous development of the invention, the correction method includes estimating the optical center of an image, which preferably shows a homogeneous scene, such that in the event that the image was generated with a one-dimensional image sensor, estimating the coordinate of the optical center in the one dimension includes:

• - Generating a one-dimensional data vector in one dimension from the values of the pixels of the image,
• - iterative fitting of a one-dimensional balancing parabola to the one-dimensional data vector,

and in the event that the image was generated with a two-dimensional image sensor, estimating a coordinate of the optical center in a first dimension comprises:

• - averaging the values of the pixels of the images in the second dimension to produce a one-dimensional data vector of averaged values in the first dimension, and
• - iterative fitting of a one-dimensional balancing parabola to the one-dimensional data vector.

This method offers a fairly simple but robust way to estimate the optical center of the image if this is required, for example, to perform zoning of the image. Both in the case that the image was recorded with a one-dimensional image sensor and in the case that the image was recorded with a two-dimensional image sensor, generating the one-dimensional data vector can also include averaging the values of the pixels of images recorded one after the other in time .

According to another advantageous development of the invention, the iterative fitting takes place in each iteration in a data area of the one-dimensional data vector around a currently estimated coordinate of the optical center in the one or first dimension, the data area preferably being symmetrical to the currently estimated coordinate of the optical Center point in the one or first dimension and / or the iterative fitting is initialized with the coordinate of the geometric center of the image in the one or first dimension as the currently estimated coordinate of the optical center in the one or first dimension. The updated estimated coordinate usually converges very quickly towards the actual optical center, so that a good estimate can usually be obtained with just a few iterations.

According to a further advantageous development, the correction method includes determining parameters of the correction function based on a reference image, which preferably shows a homogeneous scene, wherein determining the parameters of the correction function includes normalizing the correction function such that the correction function has a predetermined value at its apex, preferably has the value 1. On the one hand, the determination of the parameters of the correction function ideally only needs to be carried out once at the setup time, e.g. during the production of a camera or during the commissioning of a machine in which a camera is used. In addition, through a suitable choice of the reference image, for example so that the image shows a homogeneous scene, the evaluation of the image and thus the determination of the parameters of the correction function can be carried out in a comparatively simple manner. In addition, normalization can ensure that the correction does not change the brightness at the center of the vignetting effect.

According to yet another advantageous development of the invention, the normalization is carried out in such a way that the zero-order coefficient of the polynomial is replaced by a zero-order normalizing coefficient, which is calculated from the higher-order coefficients of the polynomial. This means that normalization can be carried out in a simple manner without the need to determine the optical center.

• - Bestimmen des Wertes einer Korrekturfunktion an der Position eines Pixels iterativ durch Multiplikation eines oder mehrerer Faktoren mit einem bereits bestimmten Wert der Korrekturfunktion an der Position eines anderen Pixels, und
• - Korrigieren des Wertes des Pixels, wobei das Korrigieren ein Multiplizieren des Wertes des Pixels mit dem Wert der Korrekturfunktion an der Position des Pixels umfasst, wobei das Korrigieren den Vignettierungseffekt in dem Wert des Pixels korrigiert,
  • wobei die Korrekturfunktion eine Exponentialfunktion zu einer Basis, vorzugsweise entweder der Eulerschen Zahl e oder der Zahl 2 oder 10, umfasst, deren Argument ein ein- oder mehrdimensionales Polynom des Grades n ≥ 2 ist,
  • wobei der Wert der Korrekturfunktion an der Position des Pixels basierend auf einer Kaskadierung von rückgekoppelten Multiplikationen, die von der Schaltung von mindestens zwei kaskadierten rückgekoppelten Multiplizierwerken durchgeführt werden, so bestimmt wird, dass die Werte der Korrekturfunktion an den Positionen der Pixel fortlaufend berechnet werden,
  • wobei das Multiplizieren des Wertes des Pixels mit dem Wert der Korrekturfunktion an der Position des Pixels mit den mindestens zwei kaskadierten rückgekoppelten Multiplizierwerken nachgeschalteten Multiplikator durchgeführt wird.

According to a further aspect of the invention, a correction device for correcting a Vignetting effect provided in values of pixels of an image of an electronic camera, the correction device comprising a circuit of at least two cascaded feedback multipliers and a multiplier connected downstream of the at least two cascaded feedback multipliers and is designed to:

• - Determining the value of a correction function at the position of a pixel iteratively by multiplying one or more factors with an already determined value of the correction function at the position of another pixel, and
• - correcting the value of the pixel, the correcting comprising multiplying the value of the pixel by the value of the correction function at the position of the pixel, the correcting correcting the vignetting effect in the value of the pixel,
  • wherein the correction function comprises an exponential function to a base, preferably either Euler's number e or the number 2 or 10, the argument of which is a one- or multi-dimensional polynomial of degree n ≥ 2,
  • wherein the value of the correction function at the position of the pixel is determined based on a cascading of feedback multiplications carried out by the circuit of at least two cascaded feedback multipliers, such that the values of the correction function at the positions of the pixels are continuously calculated,
  • wherein the multiplication of the value of the pixel by the value of the correction function at the position of the pixel is carried out with the multiplier connected downstream of at least two cascaded feedback multipliers.

• - einen Bildsensor mit Pixeln zur Erzeugung eines Bildes; und
• - die Korrektureinrichtung nach Anspruch 15 zur Korrektur eines Vignettierungseffekts in Werten von Pixeln des Bildes.

According to a further aspect of the invention, an electronic camera (10) is provided, the electronic camera comprising:

• - an image sensor with pixels to generate an image; and
• - The correction device according to claim 15 for correcting a vignetting effect in values of pixels of the image.

According to a further aspect of the invention, a computer device is provided, wherein the computer device comprises a computing unit which is designed to carry out the correction method according to one of claims 1 to 14.

According to a further aspect of the invention, a computer program product is provided, wherein the computer program product comprises code means for causing a computing device to execute the correction method according to any one of claims 1 to 14 when the computer program product is executed on the computing device.

It is understood that the correction method according to claim 1, the correction device according to claim 15, the electronic camera according to claim 16, the computer device according to claim 17 and the computer program product according to claim 18 are similar and / or identical preferred embodiments, in particular as in the dependent ones defined requirements.

It is understood that a preferred embodiment of the invention can also be any combination of the dependent claims with the corresponding independent claim.

BRIEF DESCRIPTION OF THE DRAWINGS

• 1 schematisch und exemplarisch die Anwendung und den Aufbau einer elektronischen Kamera zeigt, und
• 2 schematisch und exemplarisch einen Bildsensor zeigt, der z.B. als (a) eindimensionale Matrix, (b) zweidimensionale Matrix oder (c) zweidimensionale Matrix mit einer Anordnung aus Farbfiltern ausgebildet sein kann,
• 3 schematisch und exemplarisch ein progressives Auslesen zeigt,
• 4 schematisch und exemplarisch ein (a) blockweise geteiltes und (b) vektorisiertes Auslesen eines Bildsensors zeigt,
• 5 schematisch und exemplarisch den grundsätzlichen Ablauf der Korrektur zeigt,
• 6 schematisch und exemplarisch die Durchführung einer Korrektur mit Division zeigt,
• 7 schematisch und exemplarisch die Durchführung der Korrektur durch Multiplikation eines Models, das mit dem reziproken Referenzbild ermittelt wurde, zeigt,
• 8 schematisch und exemplarisch (a) den geometrischen und optischen Mittelpunkt eines Bildes, (b) eine Aufteilung eines Bildes in radial Zonen und (c) eine Aufteilung des Bildes in kartesische Zonen zeigt,
• 9 schematisch und exemplarisch den Fit einer (a-c) cos⁴-Funktion und (d-f) cos^-4-Funktion durch (a,d) eine Parabel, (b,e) eine cash-Funktion und (c,f) eine gaußverteilungsförmige bzw. eine reziproke gaußverteilungsförmige Funktion zeigt,
• 10 schematisch und exemplarisch den Ersatz der hardwareunfreundlichen Division durch eine Transformation zeigt,
• 11 schematisch und exemplarisch Polynomexponentialfunktionen in (a) x-Richtung, (b) in y-Richtung und (c) eine zweidimensionale Polynomexponentialfunktion zeigt,
• 12 schematisch und exemplarisch Beispiele der Auswahl von Pixeln für die Bestimmung der Parameter zeigt,
• 13 schematisch und exemplarisch ein rückgekoppeltes Multiplizierwerk in zwei alternativen Ausführungsformen zeigt,
• 14 schematisch und exemplarisch eine Kaskadierung zweier rückgekoppelter Multiplizierwerke mit nachfolgender Korrektur eines Pixelwertes zeigt,
• 15 schematisch und exemplarisch eine Kaskadierung von n rückgekoppelten Multiplizierwerken mit nachgeschaltetem Multiplikator zur Korrektur von Pixelwerten zeigt,
• 16 schematisch und exemplarisch zeigt, wie zwei kaskadierte rückgekoppelte Multiplizierwerke zu einem zweidimensional kaskadierten rückgekoppelten Multiplizierwerk verbunden werden,
• 17 schematisch und exemplarisch zeigt, wie eine Mehrzahl kaskadierter rückgekoppelter Multiplizierwerke zu einem zweidimensional kaskadierten rückgekoppelten Multiplizierwerk zur Korrektur mit einer zweidimensionalen Polynomexponentialfunktion des Grades n × m verbunden werden,
• 18 schematisch und exemplarisch zeigt, dass verschiedene Farbkanäle unterschiedliche Helligkeiten aufweisen können,
• 19 schematisch und exemplarisch zeigt, dass logarithmierte Farbwerte verschiedener Farbkanäle einen unterschiedlichen Offset aufweisen und (a) getrennt oder (b) gemeinsam beschrieben werden können.
• 20 schematisch und exemplarisch einen Bildausschnitt zeigt,
• 21 schematisch und exemplarisch eine serielle Korrektur von Pixeln durch ein kaskadiertes rückgekoppeltes Multiplizierwerk zeigt,
• 22 schematisch und exemplarisch eine Teilparallelisierung der Berechnung durch mehrere kaskadierte rückgekoppelte Multiplizierwerke zeigt,
• 23 schematisch und exemplarisch eine Stabilisierung der teilparallelisierten Berechnung durch einen gemeinsamen Rückbezug von Korrekturwerten zeigt,
• 24 schematisch und exemplarisch verschiedene Formen der Zonierung zeigt,
• 25 schematisch und exemplarisch einen Überblick über einen Algorithmus zum Finden des optischen Mittelpunkts zeigt,
• 26 schematisch und exemplarisch die Wirkung einer zeilen- oder spaltenweisen Mittelung zeigt,
• 27 schematisch und exemplarisch die erste Iteration der Bestimmung der Koordinate des optischen Mittelpunkts zeigt, und
• 28 schematisch und exemplarisch die zweite Iteration der Bestimmung der Koordinate des optischen Mittelpunkts zeigt.

Preferred embodiments of the invention are described in more detail below with reference to the attached figures, where:

• 1 shows schematically and as an example the application and structure of an electronic camera, and
• 2 shows schematically and by way of example an image sensor which can be designed, for example, as a (a) one-dimensional matrix, (b) two-dimensional matrix or (c) two-dimensional matrix with an arrangement of color filters,
• 3 shows schematically and exemplarily a progressive readout,
• 4 shows schematically and as an example a (a) block-by-block and (b) vectorized readout of an image sensor,
• 5 shows schematically and as an example the basic process of correction,
• 6 shows schematically and as an example how to carry out a correction with division,
• 7 shows schematically and as an example how the correction is carried out by multiplying a model that was determined with the reciprocal reference image,
• 8th shows schematically and as an example (a) the geometric and optical center of an image, (b) a division of an image into radial zones and (c) a division of the image into Cartesian zones,
• 9 schematically and exemplarily the fit of an (ac) cos ⁴ function and (df) cos ^-4 function tion by (a,d) a parabola, (b,e) a cash function and (c,f) a Gaussian distribution or a reciprocal Gaussian distribution function,
• 10 shows schematically and as an example the replacement of the hardware-unfriendly division by a transformation,
• 11 shows schematically and exemplarily polynomial exponential functions in (a) x-direction, (b) in y-direction and (c) a two-dimensional polynomial exponential function,
• 12 shows schematically and exemplary examples of the selection of pixels for determining the parameters,
• 13 shows schematically and as an example a feedback multiplier in two alternative embodiments,
• 14 shows schematically and as an example a cascading of two feedback multipliers with subsequent correction of a pixel value,
• 15 shows schematically and as an example a cascading of n feedback multipliers with a downstream multiplier for correcting pixel values,
• 16 shows schematically and as an example how two cascaded feedback multipliers are connected to form a two-dimensional cascaded feedback multiplier,
• 17 shows schematically and as an example how a plurality of cascaded feedback multipliers are connected to a two-dimensional cascaded feedback multiplier for correction with a two-dimensional polynomial exponential function of degree n × m,
• 18 shows schematically and as an example that different color channels can have different brightnesses,
• 19 shows schematically and as an example that logarithmic color values of different color channels have a different offset and can be described (a) separately or (b) together.
• 20 shows a section of the image schematically and as an example,
• 21 shows schematically and as an example a serial correction of pixels by a cascaded feedback multiplier,
• 22 shows schematically and as an example a partial parallelization of the calculation by several cascaded feedback multipliers,
• 23 shows schematically and as an example a stabilization of the partially parallelized calculation by a common reference of correction values,
• 24 shows schematically and exemplary different forms of zoning,
• 25 shows schematically and as an example an overview of an algorithm for finding the optical center,
• 26 shows schematically and as an example the effect of row or column averaging,
• 27 shows schematically and as an example the first iteration of determining the coordinate of the optical center, and
• 28 shows schematically and as an example the second iteration of determining the coordinate of the optical center.

DETAILED DESCRIPTION OF THE EMBODIMENTS

In the figures, identical or corresponding elements or units are each provided with identical or corresponding reference numerals. If an element or unit has already been described in connection with a figure, a detailed description may be omitted in connection with another figure. The terms “left”, “right”, “top”, “bottom”, etc., which may be used within the description of the figures, refer to the corresponding figure in an orientation with normally readable figure names and reference symbols.

As described, according to the invention, a correction method for correcting a vignetting effect in values of pixels of an image of an electronic camera can be provided, which is suitable for meeting one or more, preferably as many, requirements as to high speed, saving resources, using hardware-friendly operations, etc. to fulfill. This is done by correcting the values of the pixels based on a correction function comprising an exponential function to a base, preferably either Euler's number e or the number 2 or 10, the argument of which is a one- or multi-dimensional polynomial of degree n ≥ 2, or which is mathematically convertible into such a function, wherein correcting the value of a pixel comprises dividing or preferably multiplying the value of the pixel by or with the value of the correction function at the position of the pixel.

How this can be done in various embodiments of the invention is explained below by way of example in numbers 1 to 11. In this context, it should be noted that the inventive ideas described in these numbers can be combined in a suitable manner by the person skilled in the art and these combinations each represent embodiments of the invention:

1. Correction function

It is assumed that the edge light falloff generated by the lens is described well or sufficiently well by the so-called cos ⁴ law. This law is also referred to in US 7,453,502 B2 , US 7,692,700 B2 , US 7,634,152 B2 , US 7,920,171 B2 , US 8,472,712 B2 and US 8,577,140 B2 referred.

A cos ⁴ function is difficult to describe using a linear function or a polynomial function. This is done using 9 explained. The function values f for different functions of a free variable z are shown there. The 9 (a) to (c) each show a cos ⁴ -shaped function 100 in a range of z as a dashed line. The range is chosen so that the function drops to around 25% of the maximum value at the edge. This corresponds to the observation that the drop in peripheral light that occurs in practice leads to a darkening of around 75% in the worst case, so that at least 25% of the light intensity is still available without any drop in edge light. In the illustration, f and all functions shown are scaled linearly to a value range of 0 to 255, as it is common practice to specify brightnesses in digital images in this range.

In 9 (a) the cos ⁴ function 100 is approximated with a parabolic function 101 while minimizing the integral squared deviation. It can be seen that a parabolic function is not able to approximate the cos ⁴ function with sufficient accuracy. This is believed to be a reason why in numerous prior art documents the parabolic function is expanded as a second degree polynomial function by further degrees, such as in US 7,453,502 B2 and US 7,548 , 262 B2 . Higher degree polynomial functions require more computational effort. Furthermore, they require greater precision in the calculation, which in turn requires more effort in hardware.

In 9(b) the same cos ⁴ -shaped function 100 is approximated in the same way with a cash function 102 plus a constant offset, as in, for example US 7,920,171 B2 is proposed. This approximation also shows a clearly suboptimal result. In addition, a hyperbolic cosine is a hardware-unfriendly function.

The invention is based on the inventor's knowledge that the form of the cos ⁴ -shaped function is very similar to a Gaussian distribution, which is also referred to as a Gaussian normal distribution: $$\text{f}\left(\text{x}\right)=\frac{1}{\mathrm{\sigma }\sqrt{2\mathrm{\pi }}}{\text{e}}^{-\frac{{\left(\text{x}-\mathrm{\mu }\right)}^2}{2{\mathrm{\sigma }}^2}}$$

und einer Exponentialfunktion, die auf das Argument -(x - µ)²/(2σ²) angewandt ist. Um für die Vignettierung eine geeignete Beschreibung oder Korrektur zu erhalten, kann auf den Normalisierungsfaktor verzichtet werden, da hier keine entsprechende integrale Normierungsbedingung vorliegt. Weiterhin kann das Argument unter Verwendung der binomischen Formel ausmultipliziert werden. Dabei wird ein quadratisches Polynom erhalten. Der Mittelwert der Gaußverteilung µ kann dann ebenso in die Koeffizienten hinein gezogen werden, wie das führende Minus und die Division durch 2σ². Dadurch wird beispielhaft eine Exponentialfunktion eines Polynoms erhalten, die hier als Polynomexponentialfunktion zweiten Grades g mit Koeffizienten c₀, c₁ und c₂ bezeichnet wird. Dabei folgt aus den Rechenregeln für die Exponentialfunktion die Tatsache, dass ein eventuell benötigter Faktor ebenfalls mit in den Koeffizient c₀ hineingezogen werden kann. Ersetzt man schließlich die freie Variable x durch eine freie Variable z, so stellt sich die Polynomexponentialfunktion zweiten Grades folgendermaßen dar: $$\text{g}\left(\text{z}\right)={\text{e}}^{{\text{c}}_0+{\text{c}}_1\text{z}+{\text{c}}_2{\text{z}}^2}$$

This equation of the Gaussian distribution consists of a normalization factor $1/\left(\mathrm{\sigma }\sqrt{2\mathrm{\pi }}\right)$

and an exponential function applied to the argument -(x - µ) ² /(2σ ² ). In order to obtain a suitable description or correction for the vignetting, the normalization factor can be omitted since there is no corresponding integral normalization condition here. Furthermore, the argument can be multiplied out using the binomial formula. A quadratic polynomial is obtained. The mean value of the Gaussian distribution µ can then be included in the coefficients, as can the leading minus and division by 2σ ² . This gives an example of an exponential function of a polynomial, which is referred to here as a polynomial exponential function of the second degree g with coefficients c ₀ , c ₁ and c ₂ . The calculation rules for the exponential function result in the fact that any factor that may be required can also be included in the coefficient c ₀ . If you finally replace the free variable x with a free variable z, the polynomial exponential function of the second degree is represented as follows: $$\text{G}\left(\text{e.g}\right)={\text{<figure-callout id="0" label="e c" filenames="DE102018115991B4\_0067.png,DE102018115991B4\_0076.png" state="{{state}}">e</figure-callout>}}^{{\text{c}}_{}}$$

This gives rise to the idea of describing the cos ⁴ function of vignetting using a polynomial exponential function. In 9 (c) becomes the same cos ⁴ -shaped function as in 9 (a) and (b) approximated again in the same way with a polynomial exponential function. It is clearly visible that this function is able to describe the cos ⁴ -shaped function much better than the aforementioned functions in 9 (a) and (b). It is therefore proposed that the correction of edge light fall-off or vignetting be based on such a function.

Since, as explained above, it is fundamentally possible to use a reciprocal function for correction, which describes a reciprocal brightness distribution, approximations of a reciprocal cos ⁴ -shaped function are presented below, i.e. a 1/cos ⁴ - or cos ^{- 4} - shaped function 104. This function is also scaled proportionally to the value range of 0 to 255 that is common in image processing. It is in turn shown over a free variable z, the display area of which is chosen exactly as in 9 (a) to (c).

9 (d) shows an approximation of a cos ^-4 -shaped function 104 by a parabolic function 105. It can be seen that in this case too the parabola is not able to describe the function 104 well. The same applies to the approximation with a cosh-shaped function 106, which is in 9 (e) is shown. 9 (f) finally shows the approximation of the cos ^-4 -shaped function 104 by a polynomial exponential function of the second degree 107, which can also describe this function very well.

This shows, on the one hand, that the polynomial exponential function is well suited to describing vignetting as a mathematical model M, where the coefficients c represent parameters P in the sense of the above description. A correction K can then be made using division. On the other hand, the polynomial exponential function is also suitable for describing the reciprocal vignetting as a reciprocal model M, so that the correction K can take place as a multiplication.

Furthermore, the exponential function has the favorable property that a negation of the argument provides the reciprocal value: $${\text{e}}^{-\text{e.g}}=\frac{1}{{\text{e}}^{\text{e.g}}}$$

This is the basis of the inventor's idea of replacing the described hardware-unfriendly divisions with hardware-friendly multiplications. The method based on this is shown in 10 . Here, at setup time, the reference image I ₀ is described as a mathematical model M using a first polynomial exponential function. Parameters P are determined, which are converted into transformed parameters P ₂ using a transformation N. This transformation can preferably consist of a negation. The transformed parameters P ₂ can now be used to correct the image data of the image I ₀ by a multiplication 112 with the aid of a polynomial exponential function 111. The transformation N can be used to replace hardware-unfriendly division with hardware-friendly multiplication. The additional effort for the transformation is low because in the example mentioned this only consists of a hardware-friendly negation of the parameters and because the number of parameters is preferably in the order of magnitude of, for example, 1 to 10, while the number of pixels is usually in the order of millions and therefore the computational effort for transforming a few parameters is very small, especially compared to dividing the values of pixels.

The negation can also be performed by applying a negation operator N to the coefficients of the polynomial exponential function summarized in a column vector. The negation operator can be formulated in matrix notation and consists of a unity matrix multiplied by -1. For the case mentioned in equation (1.2), this matrix is as follows: $$\text{N}=\left[\begin{array}{ccc}-1& 0& 0\\ 0& -1& 0\\ 0& 0& -1\end{array}\right].$$

The one-dimensional second degree polynomial exponential function mentioned according to equation (1.2) can now be used directly to correct images from line sensors in the manner mentioned.

However, the majority of electronic cameras have two-dimensional matrix sensors in which vignetting occurs in two dimensions. The idea is therefore further developed to describe the vignetting preferably by a two-dimensional polynomial exponential function, which can be represented as a function of the two coordinates x and y. It is assumed that vignetting is a purely multiplicative approach, so that the idea arises from this, according to the two-dimensional polynomial exponential function 11(c) by multiplying a horizontal polynomial exponential function 11 (a) with a vertical polynomial exponential function according to 11(b) to create. The horizontal polynomial exponential function is obtained by replacing the free variable z with a horizontal coordinate x and the vertical polynomial exponential function by replacing z with the vertical coordinate y. This is shown here as an example of a second-order polynomial exponential function and can be extended to higher orders by a person skilled in the art: $$\text{G}\left(\text{x},\text{y}\right)={\text{e}}^{{\text{c}}_{\text{x}.0}+{\text{c}}_{\text{x}.1}\text{x}+{\text{c}}_{\text{x}.2}{\text{x}}^2}{\text{e}}^{{\text{c}}_{\text{y}.0}+{\text{c}}_{\text{y}.1}+{\text{c}}_{\text{y}.2}{\text{x}}^2}$$

Here c _x.0 , c _x.1 and c _x.2 are the coefficients for the horizontal polynomial exponential function and c _y.0 , c _y.1 and c _y.2 are the coefficients for the vertical polynomial exponential function. By applying the calculation rules for the exponential function, both exponential functions can be converted into an exponential function can be combined by adding the arguments: $$\text{G}\left(\text{x},\text{y}\right)={\text{e}}^{{\text{c}}_{\text{x}.0}+{\text{c}}_{\text{x}.1}\text{x}+{\text{c}}_{\text{x}.2}{\text{x}}^2+{\text{c}}_{\text{y}.0}+{\text{c}}_{\text{y}.1}\text{y}+{\text{c}}_{\text{y}.2}{\text{y}}^2}$$

The sum of c _x.0 and c _y.0 appears in the argument. Both coefficients can be summed to form a single coefficient c _0.0 = c _x.0 + c _y.0 . This exemplifies how mathematical transformations can be used to generate numerous equivalent representations of the polynomial exponential function.

The resulting two-dimensional polynomial exponential function according to 11(c) has the advantage that it can assume a radially symmetrical shape and is therefore very well suited to correcting radially symmetrical vignetting. This is particularly the case when the coefficients c _x.2 and c _y.2 assume the same values c ₂ .

It can also be shown by mathematical proof that the polynomial exponential function is the only class of functions in which a radially symmetric function can advantageously be generated by multiplying a horizontal and an equal vertical function.

The correction of edge light fall-off or vignetting with a polynomial exponential function can in principle be done by multiplying uncorrected pixel values p by the function value of the polynomial exponential function g or dividing by the function value of the polynomial exponential function g. As already explained, both cases only differ in the choice of sign of the coefficients and, from a mathematical point of view, both are always possible. However, due to its greater hardware friendliness, multiplication is usually preferable.

In general, a one-dimensional polynomial exponential function of nth degree g can be described by an exponential function applied to a one-dimensional polynomial of nth degree: $$\text{G}\left(\text{e.g}\right)={\text{e}}^{{\displaystyle {\sum }_{\text{i}=0}^{\text{n}}{\text{c}}_{\text{i}}{\text{e.g}}^{\text{i}}}}.$$

The advantage of using higher order terms and coefficients is that the accuracy can be further increased. However, this may also increase the calculation effort. This also leads to the idea that in the two-dimensional case the accuracy can be further increased by using a polynomial exponential function g with the degree m × n. Here, n denotes the horizontal degree of the two-dimensional polynomial exponential function and m denotes the vertical degree. Then g can be calculated, for example, as follows: $$\text{G}\left(\text{x}\text{, y}\right)={\text{e}}^{{\displaystyle {\sum }_{\text{j}=0}^{\text{m}}{\displaystyle {\sum }_{\text{i}=0}^{\text{n}}{\text{c}}_{\text{j}\text{.i}}{\text{x}}^{\text{i}}{\text{y}}^{\text{j}}}}}.$$

Since the underlying vignetting problem is usually radially symmetrical and therefore has no preferred direction, it makes sense to choose the same horizontal degree n and vertical degree m. As a result, neither of these two directions is given preferential treatment in the correction.

Of course, numerous equivalent forms of representation can also be found here, for example by changing the order of summation. Furthermore, the solution is given here as an exponential function, i.e. as a power function with Euler's number e as the base. Almost any other number can also be used as a base, for example the numbers 2 or 10, which are particularly often used for such purposes. The corresponding conversion follows the following well-known formula. This means that a corresponding change in the basis only requires a corresponding adjustment of the coefficients: $${\text{a}}^{\text{x}}={\text{b}}^{{\text{xlog}}_{\text{b}}\left(\text{a}\right)}.$$

In this equation, unlike the rest of the description, a and b denote two different bases of a respective exponential function and x is any free variable.

2. Determination of parameters

Types of determining the coefficients of the polynomial exponential function g as parameters P for the correction according to the invention are described below by way of example.

First, pixel values p _i are selected. Such a selection can be made in various ways and taking into account the circumstances with regard to the severity of the vignetting and the available storage and computing resources. Some examples are in 12 shown. For example, it is possible to select all pixels, as in 12 (a) shown. Alternatively, it is also possible to use a regularly arranged subset of pixels, as in 12(b) . It is also possible to select pixels along preferred lines or directions, as in 12(c) . Finally, it is also possible to summarize pixels, for example by adding or averaging their values, as in 12 (d) indicated. Such a process is also known as “binning”.

The parameters can then be determined based on the pixel values p _i . According to the invention, this can be done, for example, as a fit by applying a fitting algorithm to the input data. The pixel values of the reference image I ₀ or the reciprocal pixel values of the reference image I ₀ can serve as input data. This minimizes the amount of error between the input data and the function to be fitted. An example of such a fitting algorithm is the so-called “Levenberg-Marquardt ordinary least-squares method”.

A preferred alternative when determining the parameters is to use a division 60 (cf. 7 ) and instead calculate the logarithm of the input data to a base. For example, Euler's number e, 2 or 10 or another number can be used as a basis. By taking the logarithm of the input data, the determination of the parameters can be converted into a linear problem, which can be solved using simpler methods, less computing effort and also in less computing time.

This can be accomplished, for example, by combining the input data, which consists of logarithmic values of pixels p _i with coordinates x _i and y _i , into a target vector d. The following formula uses the natural logarithm with base e as an example: $$\text{d}=\left[\begin{array}{c}\text{ln}\left({\text{p}}_0\right)\\ \text{ln}\left({\text{p}}_1\right)\\ \text{ln}\left({\text{p}}_2\right)\\ ⋮\end{array}\right],\text{x}=\left[\begin{array}{c}{\text{x}}_0\\ {\text{x}}_1\\ {\text{x}}_2\\ ⋮\end{array}\right]\text{and y}=\left[\begin{array}{c}{\text{y}}_0\\ {\text{y}}_1\\ {\text{y}}_2\\ ⋮\end{array}\right].$$

Furthermore, the parameters can be combined in an exemplary order to form a parameter vector c. As described, c _0.0 includes both the x-offset c _x.0 and the y-offset c _y.0 : $$\text{c}=\left[\begin{array}{l}{\text{c}}_{0.0}\hfill \\ {\text{c}}_{\text{x}.1}\hfill \\ {\text{c}}_{\text{x}.2}\hfill \\ {\text{c}}_{\text{y}.1}\hfill \\ {\text{c}}_{\text{y}.2}\hfill \end{array}\right].$$

Finally, a matrix can be formed that contains the different powers of the coordinates x _i and y _i corresponding to the coefficients in columns: $$\text{M}=\left[\begin{array}{lllll}1\hfill & {\text{x}}_0\hfill & {\text{x}}_0^2\hfill & {\text{y}}_0\hfill & {\text{<figure-callout id="0" label="y" filenames="DE102018115991B4\_0067.png,DE102018115991B4\_0076.png" state="{{state}}">y</figure-callout>}}_0^2\hfill \\ 1\hfill & {\text{x}}_1\hfill & {\text{x}}_1^2\hfill & {\text{y}}_1\hfill & {\text{<figure-callout id="1" label="y" filenames="DE102018115991B4\_0063.png,DE102018115991B4\_0064.png" state="{{state}}">y</figure-callout>}}_1^2\hfill \\ 1\hfill & {\text{x}}_2\hfill & {\text{x}}_2^2\hfill & {\text{y}}_2\hfill & {\text{y}}_2^2\hfill \\ ⋮\hfill & ⋮\hfill & ⋮\hfill & ⋮\hfill & ⋮\hfill \end{array}\right].$$

Then, using a squared error to be minimized, the overdetermined optimization problem can be written as follows: $${\left(\text{d}-\text{Mc}\right)}^2\to \text{min}.$$

A simple, well-known way to solve this problem is to use the Moore-Penrose pseudoinverse, also called the Moore-Penrose inverse: $$\text{c}={\left({\text{M}}^{\text{T}}\text{M}\right)}^{-1}{\text{M}}^{\text{T}}\text{d}.$$

In this way, the coefficients of the polynomial exponential function can be calculated easily and with little effort. In the event that a specific search is being made for coefficients for a radially symmetric correction, an additional secondary condition can be introduced, for example c _x.2 = c _y.2 . The determination of the parameters can then be sought using the least square error method with constraints.

Another alternative is to first determine the two parameters to be equated independently of each other and then equate them both to their common mean.

Finally, it is also possible to replace both parameters c _x.2 and c _y.2 with a parameter c ₂ and write the two-dimensional second-order polynomial exponential function as: $$\text{G}\left(\text{x},\text{y}\right)={\text{<figure-callout id="0" label="e c" filenames="DE102018115991B4\_0067.png,DE102018115991B4\_0076.png" state="{{state}}">e</figure-callout>}}^{{\text{c}}_{}}.$$

enthält. Daneben gibt es noch weitere äquivalente Möglichkeiten, die sich hieraus durch Anwendung bekannter Rechenregeln ableiten lassen, z.B. durch Linearkombination.Accordingly, the parameter vector c can then be constructed from the parameters c ₀ , c _x.1 , c _y.1 and c ₂ , in which case the column of M assigned to c ₂ contains the values ${\text{x}}_{\text{i}}^2+{\text{y}}_{\text{i}}^2$

contains. There are also other equivalent possibilities that can be derived from this by applying known calculation rules, for example through linear combination.

This method of determining the parameters offers the advantage that it is possible to easily determine the parameters P without this would require a difficult determination of an optical center.

Furthermore, this method can also be easily extended to the determination of the parameters of higher-order polynomial exponential functions by including the corresponding coefficients in the parameter vector c and expanding the matrix M to include the corresponding columns of the powers or power products of the coordinate values associated with the parameters.

3. Normalization

It is advantageous if the brightness at the optical center remains constant through a correction. Since previously the determination of an optical center point was deliberately avoided in the course of simplification, the question now arises as to how such a standardization can be carried out without determining an optical center point.

One possibility of such normalization according to the invention is to determine the value of the function of the mathematical model for all pixels. Using a suitable extreme value, i.e. the maximum or the minimum, the function can subsequently be divided by this value or multiplied by its inverse or by a number divided by this value. The disadvantage of this approach is that it requires a relatively high amount of computing effort.

In order to take a simpler approach to normalization, the application of the polynomial exponential function is first considered. Since this function is used to multiply or divide by it, it should preferably assume the neutral value 1 at the desired point of constant brightness. For this purpose, its definition according to equation (1.2) is also considered. The exponential function takes the value 1 if and only if its argument, consisting of a second degree polynomial, takes the value 0. The idea is to choose the parameter c ₀ so that the argument of the exponential function becomes 0 exactly at the extreme value of the function g. For this purpose, g is differentiated in the derivation, set equal to 0, solved for a location x, its value for x is inserted into the argument of the function g, this is also set equal to 0 and solved for c ₀ . This results in the following normalizing value ĉ ₀ for c ₀ : $${\widehat{\text{c}}}_0=\frac{1}{4}\frac{{\text{<figure-callout id="1" label="c" filenames="DE102018115991B4\_0063.png,DE102018115991B4\_0064.png" state="{{state}}">c</figure-callout>}}_1^2}{{\text{c}}_2}.$$

The calculation steps explained in the derivation only serve to make it easier to understand how equation (3.1) came about. They do not have to be reproduced at runtime. In the polynomial exponential function, the normalizing value ĉ ₀ then replaces the previous value c ₀ . This means that the polynomial exponential function is normalized.

In principle, you can proceed in the same way for the two-dimensional case. Instead of differentiation, a gradient is formed according to a vector from the values x and y, which, after being set to zero, provides a two-dimensional coordinate value for this vector. Just as before, the elements corresponding to x and y of this value for x and y are inserted into the argument of the function g, which in turn is set equal to 0 and solved for c _0.0 . Alternatively, the lines of the vector equation mentioned here can be calculated individually and then used. This results in the following normalizing value ĉ _0.0 for c _0.0 : $${\widehat{\text{c}}}_{0.0}=\frac{1}{4}\frac{{\text{c}}_{\text{x}.1}^2}{{\text{c}}_{\text{x}.2}}+\frac{1}{1}\frac{{\text{c}}_{\text{y}.1}^2}{{\text{c}}_{\text{y}.2}}.$$

This exemplary expression can be easily converted into various equivalent forms of representation. As an example, the following representation is given for the case where c _x.2 = c ₂ = c _y.2 is applicable: $${\text{c}}_{0.0}=\frac{{\text{c}}_{\text{x}.1}^2+{\text{c}}_{\text{y}.1}^2}{4{\text{c}}_2}.$$

Equations (3.1) to (3.3) each represent a simple normalizing transformation N of the parameters P 10 This transformation is elegant and can be calculated with very little effort. In particular, no knowledge of an optical center is required to carry it out, so that the effort involved in calculating this optical center can be dispensed with.

In advance, a simplifying notation should be introduced at this point. Equations (3.1) to (3.3) are non-linear calculations. These calculations can be written somewhat simplified by first introducing the vector of polynomial exponential function coefficients c and the vector of normalized polynomial exponential function coefficients ĉ. For the one-dimensional case these are as follows: $$\text{c}=\left[\begin{array}{c}{\text{c}}_0\\ {\text{c}}_1\\ {\text{c}}_2\end{array}\right],\text{and}\widehat{\text{c}}=\left[\begin{array}{c}{\widehat{\text{c}}}_0\\ {\widehat{\text{c}}}_1\\ {\widehat{\text{c}}}_2\end{array}\right].$$

With these vectors one can define a nonlinear normalization operator in the parameter space F: $$\widehat{\text{c}}=\text{FC}.$$

Vectors and a normalization operator F can also be defined for the two-dimensional case.

4th iteration

Until now, the fundamental technical problem when using a polynomial exponential function was that the exponential function is a hardware-unfriendly function, the pixel-by-pixel calculation of which requires a lot of computing effort or extensive hardware resources. Therefore, a detailed analysis was carried out on how the calculation of the polynomial exponential function can be reduced to hardware-friendly operations.

The underlying insight follows the mathematical idea of the definition of the hyper operator. With the hyper operator, an operation of a certain level results from multiple execution of operations of a level one lower. The exponential function is a power function and therefore an operation of the third level of the hyper operator. According to the definition of the hyper-operator, this can be calculated by repeatedly executing operations of the second stage of the hyper-operator, which also includes multiplication.

Following this idea, the operation of a feedback multiplier was analyzed, which is available in two alternative embodiments 13 is shown. A feedback multiplier contains a variable or a register, for example h ₀ . Using an initialization signal r, a start value can be read in at a starting time using a selection 200 via input d ₀ and an initialization can thus be carried out. In the case shown, this input d ₀ is assigned a value a ₀ . The selection can be carried out, for example, with a multiplexer. After the start time, the initialization signal r can assume a different value, which indicates that there is no longer a start time. The selection can then select a different value, for example the result of a multiplication, which can be carried out using a multiplier 202, for example. Such a multiplier can be built, for example, through logical operations or through logic elements of an FPGA. Alternatively, for example, special multiplier elements can be used in an FPGA, which can be part of a DSP block, for example. For example, a processor can use a multiplier contained in its ALU. The multiplier 202 multiplies the variable h ₀ by the value present at the data input d. In the two examples shown, this is the value a ₁ . The value of the variable h ₀ is output via an output q. A delay element 201 ensures that this operation is carried out once within a time step. Such a delay can be implemented, for example, by a register that is made up of flip-flops, each with a clock input and optionally with a so-called enable input. The latter is also known as “enable” input in English. For example, the frequency at which the pixels, rows or columns of the image are calculated, or a multiple thereof, can be used as the clock. It is also possible for individual clocks to be selected as time steps in a digital circuit using release signals. This makes it possible, for example, to carry out this multiplication exactly once per pixel, per row or per column.

There are basically different ways to carry out the order of multiplication 202, selection 200, delay 201 and storage in the variable h ₀ . For example, it is as in 13 (a) shown possible to carry out the delay 201 before storage in the variable h ₀ . However, it is also possible, as in 13 (b) shown to make the storage in the variable h ₀ before the delay 201. A key difference between these two embodiments is how quickly the signal from the input d ₀ is present at the output q. So after a first time step in 13 (a) at the output the value a ₀ , while in 13 (b) After a first time step the value a ₀ a ₁ is already present. This difference can be compensated for by choosing the values. For example, if you choose 13 (b) compared to the value for a ₀ by a factor of 1/a ₁ 13 (a) , the same initial values result after the first time step. It is therefore fundamentally possible to implement different embodiments and to achieve equivalent behavior in certain cases by appropriately selecting the values at the inputs. Only the embodiment is shown below as an example 13 (a) discussed. If necessary, the technical teachings of this embodiment can also be transferred to other embodiments.

The mathematical behavior of feedback multipliers is discussed below. A time step z and a count value for time steps i are used. Without restrictions on space It is generally assumed that an initialization signal r is set at a time step z = 0. A set signal means that it assumes a value that causes the selection 200 to select the starting value d ₀ regardless of whether a logical value of 0 or 1 is required for the initialization signal r. Furthermore, it is assumed that the initialization signal r is not set for all time steps z > 0 or i > 0, with an unset initialization signal r causing the selection 200 to select the result of the multiplier 202. Under these assumptions, after a time step z = 0, the value a ₀ is present at the output q. Without limiting the generality, it is assumed that during the time step z = 0, the signal d ₀ selected by the selection 200 with the value a ₀ is present at the delay 201, so that after the time step z = 0 this value is from the delay 201 is passed on to the variable h ₀ and to the output q. It is possible that after the time step z = 0 the time step z = 1 already exists, but this is not absolutely necessary. After each additional time step, a value is present at the output q, which has been multiplied by the value a ₁ present at the input d. This results in the following recursive equation for the value at output q after time step z: $${\text{H}}_0\left(\text{e.g}\right)=\{\begin{array}{ll}{\text{a}}_0\hfill & \text{if e.g}=0\hfill \\ {\text{a}}_1{\text{H}}_0\left(\mathrm{e.g}-1\right)\hfill & \text{otherwise}\hfill \end{array}.$$

This recursion can be converted into the following iteration formula and, assuming that a ₀ and a ₁ are positive, can be further transformed as follows: $$\begin{array}{c}{\text{H}}_0\left(\text{e.g}\right)={\text{a}}_0{\displaystyle {\prod }_{\text{i}=1}^{\text{e.g}}{\text{a}}_1}\\ ={\text{a}}_0{\text{a}}_1^{\text{e.g}}\\ ={\text{e}}^{{\text{ln a}}_0{\text{a}}_1^{\text{e.g}}}\\ ={\text{e}}^{{\text{ln a}}_0+{\text{ln a}}_1^{\text{e.g}}}\\ =e^{\stackrel{{\mathrm{<figure-callout\; id="0"\; label="c"\; filenames="DE102018115991B4\_0067.png,DE102018115991B4\_0076.png"\; state="\{\{state\}\}">c</figure-callout>}}_0}{\overbrace{{\text{ln a}}_0}}}{}^{+\text{e.g}\stackrel{{\text{<figure-callout id="1" label="c" filenames="DE102018115991B4\_0063.png,DE102018115991B4\_0064.png" state="{{state}}">c</figure-callout>}}_1}{\overbrace{{\text{ln a}}_1}}}\\ ={\text{<figure-callout id="0" label="e c" filenames="DE102018115991B4\_0067.png,DE102018115991B4\_0076.png" state="{{state}}">e</figure-callout>}}^{{\text{c}}_{}}\end{array}$$

Coefficients c ₀ and c ₁ are used here, which in the case mentioned result in the values a ₀ and a ₁ according to the following formula: $${\text{a}}_0={\text{e}}^{{\text{c}}_0}{\text{and a}}_1={\text{<figure-callout id="1" label="e c" filenames="DE102018115991B4\_0063.png,DE102018115991B4\_0064.png" state="{{state}}">e</figure-callout>}}^{{\text{c}}_{}}.$$

Through this transformation and in this notation it can be seen that a feedback multiplier is able to continuously calculate the values of a polynomial exponential function of the first degree with coefficients b ₀ and b ₁ for an integer free variable z.

Based on the knowledge that a feedback multiplier can continuously calculate a polynomial exponential function of the first degree, the question arises as to how a polynomial exponential function of the second degree can be continuously calculated by feedback multipliers and how a pixel can thus be corrected for vignetting.

For this purpose, two feedback multipliers are connected according to 14 considered. The same reference numbers correspond to those from 13 (a) . A second feedback multiplier 221 is connected in front of a first feedback multiplier 222. This causes the two feedback multipliers to be cascaded. The resulting circuit consisting of the two feedback multipliers 221 and 222 is referred to here as a cascaded feedback multiplier of the second degree. Since the input of the left feedback multiplier cannot be left open, a constant value 220 is switched as the starting element of the cascading chain in front of this second feedback multiplier. A multiplier 224 for correcting the pixel value can be connected behind the output of the first feedback multiplier 222.

The cascading is carried out in detail in such a way that the output q of a feedback multiplier, for example W ₁ , is connected to the input d of the subsequent feedback multiplier, for example W ₀ . Just like in 10 A constant value is applied to the input d of the feedback multiplier W ₁ , which is at the forefront in the cascading, for example a ₂ . The output of the last-positioned feedback multiplier of the cascaded feedback multiplier can subsequently be used to correct the pixel value p _z by multiplying its value with the pixel value p _i using a multiplier 224. This multiplication corresponds, among other things, to the multiplication 112 10 . A corrected pixel value p̂ _i is obtained in each case.

The decreasing numbering of the feedback multipliers in the course of cascading, eg W ₁ , then W ₀ is motivated at this point by the usual mathematical notation of the mathematical effect achieved by cascading. To explain this, the previously selected time steps and their numbering are retained in the same way.

$${\text{h}}_{\text{1}}\left(\text{z}\right)=\{\begin{array}{ll}{\text{a}}_1\hfill & \text{falls z}=0\hfill \\ {\text{a}}_2{\text{h}}_1\left(\text{z}-1\right)\hfill & \text{sonst}\hfill \end{array}$$

The mathematical effect of the feedback multiplier can first be described recursively. The number of the time step is shown in round brackets specified according to which this value is taken: $${\text{H}}_0\left(\text{e.g}\right)=\{\begin{array}{ll}{\text{a}}_0\hfill & \text{if e.g}=0\hfill \\ {\text{H}}_0\left(\text{e.g}-1\right){\text{H}}_1\left(\text{e.g}-1\right)\hfill & \text{otherwise}\hfill \end{array}$$

$${\text{H}}_{\text{1}}\left(\text{e.g}\right)=\{\begin{array}{ll}{\text{a}}_1\hfill & \text{if e.g}=0\hfill \\ {\text{a}}_2{\text{H}}_1\left(\text{e.g}-1\right)\hfill & \text{otherwise}\hfill \end{array}$$

$$\begin{array}{cc}{\text{h}}_0\left(1\right)={\text{h}}_0\left(0\right){\text{h}}_1\left(0\right)={\text{a}}_0{\text{a}}_1& {\text{h}}_1\left(1\right)={\text{h}}_1\left(0\right){\text{a}}_2={\text{a}}_1{\text{a}}_2\end{array}$$

$$\begin{array}{cc}{\text{h}}_0\left(2\right)={\text{h}}_0\left(1\right){\text{h}}_1\left(1\right)={\text{a}}_0{\text{a}}_1^2{\text{a}}_2^3& {\text{h}}_1\left(2\right)={\text{h}}_1\left(1\right){\text{a}}_2={\text{a}}_1{\text{a}}_2^2\end{array}$$

$$\begin{array}{cc}{\text{h}}_0\left(3\right)={\text{h}}_0\left(2\right){\text{h}}_1\left(2\right)={\text{a}}_0{\text{a}}_1^3{\text{a}}_2^3& {\text{h}}_1\left(3\right)={\text{h}}_1\left(2\right){\text{a}}_2={\text{a}}_1{\text{a}}_2^3\end{array}$$

$$\begin{array}{cc}{\text{h}}_0\left(4\right)={\text{h}}_0\left(3\right){\text{h}}_1\left(3\right)={\text{a}}_0{\text{a}}_1^4{\text{a}}_2^6& {\text{h}}_1\left(4\right)={\text{h}}_1\left(3\right){\text{a}}_2={\text{a}}_1{\text{a}}_2^4\end{array}$$

The output value q of the first feedback multiplier 222 assumes the value h ₀ (z) after time z. For better understanding, the values resulting from times z = 0 to z = 4 are explicitly stated here: $$\begin{array}{cc}{\text{H}}_0\left(0\right)={\text{a}}_0& {\text{H}}_1\left(0\right)={\text{a}}_1\end{array}$$

$$\begin{array}{cc}{\text{H}}_0\left(1\right)={\text{H}}_0\left(0\right){\text{H}}_1\left(0\right)={\text{a}}_0{\text{a}}_1& {\text{H}}_1\left(1\right)={\text{H}}_1\left(0\right){\text{a}}_2={\text{a}}_1{\text{a}}_2\end{array}$$

$$\begin{array}{cc}{\text{H}}_0\left(2\right)={\text{H}}_0\left(1\right){\text{H}}_1\left(1\right)={\text{a}}_0{\text{a}}_1^2{\text{a}}_2^3& {\text{H}}_1\left(2\right)={\text{H}}_1\left(1\right){\text{a}}_2={\text{a}}_1{\text{a}}_2^2\end{array}$$

$$\begin{array}{cc}{\text{H}}_0\left(3\right)={\text{H}}_0\left(2\right){\text{H}}_1\left(2\right)={\text{a}}_0{\text{a}}_1^3{\text{a}}_2^3& {\text{H}}_1\left(3\right)={\text{H}}_1\left(2\right){\text{a}}_2={\text{a}}_1{\text{a}}_2^3\end{array}$$

$$\begin{array}{cc}{\text{H}}_0\left(4\right)={\text{H}}_0\left(3\right){\text{H}}_1\left(3\right)={\text{a}}_0{\text{a}}_1^4{\text{a}}_2^6& {\text{H}}_1\left(4\right)={\text{H}}_1\left(3\right){\text{a}}_2={\text{a}}_1{\text{a}}_2^4\end{array}$$

Generally speaking, the value of h ₀ (z) can be expressed by replacing the value a ₁ , which represents the signal present at input d, for all previous time steps i in the first step of equation (4.2) for the simple feedback multiplier by the respective output value q of the upstream feedback multiplier 221. This results in the following iteration formula, which can subsequently be further transformed into a polynomial exponential function: $$\begin{array}{c}{\text{H}}_0\left(\text{e.g}\right)={\text{a}}_0{\displaystyle {\prod }_{\text{i}=1}^{\text{e.g}}\left({\text{a}}_1{\displaystyle {\prod }_{\text{j}=1}^{\text{i}-1}{\text{a}}_2}\right)}\\ ={\text{e}}^{\text{ln}\left({\text{a}}_0{\displaystyle {\sum }_{\text{i}=1}^{\text{e.g}}\left({\text{a}}_1{\displaystyle {\sum }_{\text{j}=1}^{\text{i}-1}{\text{a}}_2}\right)}\right)}\\ ={\text{e}}^{\stackrel{{\text{<figure-callout id="0" label="b" filenames="DE102018115991B4\_0067.png,DE102018115991B4\_0076.png" state="{{state}}">b</figure-callout>}}_0}{\overbrace{\text{ln}{a}_0}}+\stackrel{{\text{<figure-callout id="1" label="b" filenames="DE102018115991B4\_0063.png,DE102018115991B4\_0064.png" state="{{state}}">b</figure-callout>}}_1}{\overbrace{{\text{ln a}}_1}}{\displaystyle {\sum }_{\text{i}=1}^{\text{e.g}}1+\stackrel{{\text{b}}_2}{\overbrace{{\text{ln a}}_2}}{\displaystyle {\sum }_{\text{i}=1}^{\text{e.g}}{\displaystyle {\sum }_{\text{j}=1}^{\text{i}=1}1}}}}\\ ={\text{e}}^{\stackrel{{\text{<figure-callout id="0" label="c" filenames="DE102018115991B4\_0067.png,DE102018115991B4\_0076.png" state="{{state}}">c</figure-callout>}}_0}{\overbrace{{\text{b}}_0}}+\stackrel{{\text{<figure-callout id="1" label="c" filenames="DE102018115991B4\_0063.png,DE102018115991B4\_0064.png" state="{{state}}">c</figure-callout>}}_1}{\overbrace{\left({\text{<figure-callout id="1" label="b" filenames="DE102018115991B4\_0063.png,DE102018115991B4\_0064.png" state="{{state}}">b</figure-callout>}}_1-\frac{1}{2}{\text{b}}_2\right)}}\text{e.g}+\stackrel{{\text{c}}_2}{\overbrace{\frac{1}{2}{\text{b}}_2}}{\text{e.g}}^2}\\ ={\text{<figure-callout id="0" label="e c" filenames="DE102018115991B4\_0067.png,DE102018115991B4\_0076.png" state="{{state}}">e</figure-callout>}}^{{\text{c}}_{}}\end{array}.$$

For better understanding and simplified notation, a substitution with values b ₀ , b ₁ and b ₂ and c ₀ , c ₁ and c ₂ was made here. It turns out that by cascading two feedback multipliers, a polynomial exponential function of the second degree can basically be calculated.

To calculate the values a ₀ , a ₁ and a ₂ , b ₀ , b ₁ and b ₂ from the coefficients c ₀ , c ₁ and c ₂ from the polynomial exponential function, three vectors a, b and c are defined: $$\text{a}=\left[\begin{array}{c}{\text{a}}_0\\ {\text{a}}_1\\ {\text{a}}_2\end{array}\right],\text{b}=\left[\begin{array}{c}{\text{b}}_0\\ {\text{b}}_1\\ {\text{b}}_2\end{array}\right]\text{and c}=\left[\begin{array}{c}{\text{c}}_0\\ {\text{c}}_1\\ {\text{c}}_2\end{array}\right].$$

The vectors a and b are connected to each other by the exponential function, whereby the application of the exponential function to a vector is understood here as the respective application of the exponential function to the respective elements of the vector: $$\text{a}={\text{e}}^{\text{b}}.$$

The relationship between b and c can be taken from the second to last step of equation (4.11) and described by a linear transformation, which can be represented by an inverse transformation matrix T ^-1 : $$\text{c}={\text{T}}^{-1}\text{b}.$$

The values for T ^-1 can be taken directly from equation (4.11): $${\text{T}}^{-1}=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& -\frac{1}{2}\\ 0& 0& \frac{1}{2}\end{array}\right].$$

This linear transformation is defined here as an inverse matrix because in the application the calculation is usually done in the reverse order, i.e. b is calculated from c: $$\text{b}=\text{Tc}.$$

Then T is the inverse of the inverse transformation matrix: $$\text{T}=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 1\\ 0& 0& 2\end{array}\right].$$

In summary, the vector of values a for the feedback multiplier can be as follows 14 can be calculated from the vector c of the coefficients of the polynomial exponential function using the following formula: $$\text{a}={\text{e}}^{\text{Tc}}.$$

It can therefore be done with a cascading of two feedback multipliers 221 and 222 and a downstream multiplier 224 according to 14 perform a vignetting correction with a continuously calculated polynomial exponential function of the second degree.

Continuing this idea, more than two feedback multipliers can also be cascaded. Is in 15 for n feedback multipliers W ₀ , W ₁ , W ₂ , ..., W _n-1 with a downstream multiplier 224 for correcting pixel values 224. This allows a continuous calculation of a polynomial exponential function of the nth degree to be carried out. The mathematical steps required for the calculation basically follow the ideas, mathematical laws and principles mentioned so far, which can then be applied accordingly to n feedback multipliers of an expanded form.

Until now, the calculation of a polynomial exponential function to correct vignetting was carried out one-dimensionally using cascaded feedback multipliers and was therefore particularly suitable for vignetting correction of images recorded with one-dimensional image sensors, e.g. line sensors. To correct two-dimensional image sensors, it is fundamentally advantageous to also be able to carry out a two-dimensional polynomial exponential function using cascaded feedback multipliers.

16 shows a two-dimensional chaining of a cascaded feedback multiplier of the second degree 231 with an upstream cascaded feedback multiplier of the second degree 230. The two-dimensional chaining takes place via a connection 225. This is in contrast to that in 14 The chaining shown is not carried out via a connection of an output q with an input d, but rather via the connection of an output q with an input d ₀ according to 13 . This does not result in a fourth-order cascaded feedback multiplier, but rather a second-order two-dimensional cascaded feedback multiplier.

This architecture is basically designed to calculate a polynomial exponential function in progressive order 3 to perform at the locations of the pixels in a two-dimensional matrix according to 2 B) are arranged. In this progressive order according to 3 is iteratively processed one row after the other and in each of these rows the pixels one column after the other. Therefore, the order is selected, for example, such that a vertical cascaded feedback multiplier 230 works iteratively in the vertical direction, i.e. in one line after the other, and that this is followed by a horizontal cascaded feedback multiplier 231, which works horizontally and within the respective line The values of the two-dimensional polynomial exponential function are calculated for their pixels in one column after the other, these values then being used as a correction K multiplicatively 112 according to 10 be applied.

The two cascaded feedback multipliers 230 and 231 basically carry out different time steps, because all columns are run through within a row. A distinction is therefore made between vertical time steps for the vertical cascaded feedback multiplier 230 and horizontal time steps for the horizontal cascaded feedback multiplier 231. For example, there are as many vertical time steps as the image I has ₁ lines. For each line, for example, there is a vertical time step and within this line, for example, there are as many horizontal time steps as the image I has ₁ columns. A value of the polynomial exponential function can thus be calculated for each pixel of the image I ₁ .

A separation of the vertical and horizontal time steps can be achieved, for example, by connecting the corresponding cascaded feedback multipliers 230 and 231 to different clock signals, which are in 16 are not shown separately. On the other hand, it can also be achieved in that although both cascaded feedback multipliers 230 and 231 are connected to the same clock signal, execution of the respective time step is made possible for the respective cascaded feedback multiplier at different time steps by an enable signal.

In the example mentioned, the two cascaded feedback multipliers 230 and 231 not only work at different time steps, but they are also initialized at different starting times and with different frequencies. Therefore they also need separate initialization signals, which are in 16 be referred to as the vertical time step signal r _v and as the horizontal time step signal r _h . While, for example, for progressive readout, the vertical time step signal r _v is set at the beginning of each image I ₁ in the sense of an initialization, the horizontal time step signal r _h is set at the beginning of each line in the sense of an initialization.

Through such an operation of the two-dimensionally cascaded feedback multiplier according to 16 The following value f is obtained at the output q of the multiplier W _ho for a choice of the coordinates x and y of the image point, for example starting with the values x = 0 and y = 0 for the upper left corner of the image: $$\text{f}\left(\text{x},\text{y}\right)={\text{a}}_{0.0}{\displaystyle {\prod }_{\text{i}=0}^{\text{y}}{\text{a}}_{1.0}}\left({\displaystyle {\prod }_{\text{j}=0}^{\text{i}-1}{\text{a}}_{2.0}}\right){\displaystyle {\prod }_{\text{k}=0}^{\text{x}}{\text{a}}_{0.1}}\left({\displaystyle {\prod }_{\text{i}=0}^{\text{k}-1}{\text{a}}_{0.2}}\right).$$

According to the calculation steps of equation (4.11) and equation (1.5) according to equation (1.6), equation (4.19) can be converted into the following calculation formula: $$\text{f}\left(\text{x},\text{y}\right)={\text{e}}^{{\text{c}}_{0.0}+{\text{c}}_{0.1}\text{x}+{\text{c}}_{0.2}{\text{x}}^2+{\text{c}}_{1.0}\text{y}+{\text{c}}_{2.0}{\text{y}}^2}.$$

Analogous to equation (4.18), a vector of the values a is obtained from the coefficients c by applying the exponential function element by element to the elements of a vector b, which I obtain by applying a suitable transformation T to a vector of coefficients c: $$\text{a}={\text{e}}^{\text{Tc}}.$$

As an example, the vectors a and c can be chosen as follows: $$\text{a}=\left[\begin{array}{c}{\text{a}}_{0.0}\\ {\text{a}}_{0.1}\\ {\text{a}}_{0.2}\\ {\text{a}}_{1.0}\\ {\text{a}}_{2.0}\end{array}\right],\text{and c}=\left[\begin{array}{c}{\text{c}}_{0.0}\\ {\text{c}}_{0.1}\\ {\text{c}}_{0.2}\\ {\text{c}}_{1.0}\\ {\text{c}}_{2.0}\end{array}\right]$$

Then the transformation takes the following form as an example and based on equation (4.17): $$\text{T}=\left[\begin{array}{ccccc}1& 0& 0& 0& 0\\ 0& 1& 1& 0& 0\\ 0& 0& 2& 0& 0\\ 0& 0& 0& 1& 1\\ 0& 0& 0& 0& 2\end{array}\right].$$

5. Color

The image to be corrected can be monochrome, i.e. monochrome, or it can be a color image, for example as an image of a mosaic filter (as described) or, for example, as an interpolated color image according to EP 2 929 503 B1 with one color vector per pixel. In the case of the mosaic filter, the value of each pixel is assigned to a color channel. In the case of a color vector, the pixel can also have several values, which then, for example, indicate the color of the pixel as elements of the color vector.

When describing the vignetting of color images, the fundamental problem arises that effects such as the color of the light source or the unequal light sensitivity of the different color channels may lead to the color channels having different brightnesses compared to each other. This deviation in brightness generally leads to difficulties when describing it with a mathematical model.

This is in 18 shown for the example of the image of a mosaic filter, in which the arrangement of the colors follows the so-called Bayer pattern US 3,971,065 B1 follows. Each line contains a sequence of two different colors, for example red and green. If these colors in the image have different values, then assuming a cos ⁴ -shaped brightness distribution, the situation shown as an example results. The values 130 and 131 of the two different colors follow each other alternately and differ noticeably from each other.

At this point it is assumed that the brightness deviation of the different color channels is basically a multiplicative effect. This multiplicative effect can be converted into an additive effect by logarithmizing the values according to equation (2.1) according to the invention, ie the color gradients then have a fundamentally the same progression and are only shifted relative to one another by different offsets in the value range. This is in 19 (a) and (b) easy to see. Alternating logarithmic color values 132 and 133 are shown here, which correspond to color values 130 and 131 18 are equivalent to. In 19 (a) It can be seen that the logarithmic color values of a color 130 or 131 can be well described by two curves 134 and 135, which are identical in shape and which are shifted relative to one another in the vertical direction by an offset.

In principle, a vignetting correction can be carried out on the one hand by considering the color channels separately from one another, determining separate correction parameters P for each of the color channels and the values of pixels assigned to them and correcting them with a separate correction K. This is in 19 (a) shown. Here, the logarithmic values of a first color 132 are described by a first second-degree polynomial function 134 and the logarithmic values of a second color 133 are described by a second second-degree polynomial function 135. These functions or their parameters can subsequently be used for separate correction of the different color channels. Such a procedure is possible in principle, but leads to a relatively high computational effort both in determining the parameters and in the correction. Nevertheless, it can prove to be advantageous to proceed in this way, for example with different forms of vignetting of different colors.

Another possibility is to describe the logarithmic color values channel by channel using a polynomial function, whereby the polynomial functions for the different color channels only differ from each other in their offset values and agree in all other coefficients. The offset associated with this color channel can then be used to describe and correct a specific color channel. For example, the brightness distributions of two color channels with values r and g, where the symbols here represent red and green, can be described in the one-dimensional case in the following way, analogous to equation (1.2): $$\text{G}\left(\text{e.g}\right)=\{\begin{array}{cc}{\text{<figure-callout id="0" label="e c" filenames="DE102018115991B4\_0067.png,DE102018115991B4\_0076.png" state="{{state}}">e</figure-callout>}}^{{\text{c}}_{}}& \text{if <figure-callout id="0" label="r e c" filenames="DE102018115991B4\_0067.png,DE102018115991B4\_0076.png" state="{{state}}">r</figure-callout>}& \\ {\text{e}}^{{\text{c}}_{}}{{}_{0.\text{G}}+{\text{<figure-callout id="1" label="c" filenames="DE102018115991B4\_0063.png,DE102018115991B4\_0064.png" state="{{state}}">c</figure-callout>}}_1\text{e.g}+{\text{c}}_2{\text{e.g}}^2}^{}& \text{if g}\end{array}.$$

r and g also denote the affiliation of an image point or a color value to the corresponding color. When determining the coefficients c, it is possible to combine the various coefficients into a vector c analogous to equation (2.2) and to formulate a corresponding matrix analogous to equation (2.3), in which then in each row only the one belonging to the corresponding color The value c _0.r or c _0.g is a 1 and the values that do not belong to the corresponding color are each a 0. Then the values of the coefficient vector c can be determined in the manner mentioned in equation (2.5). In the context of a subsequent normalization, the situation will usually arise that the standardized coefficients ĉ _ir and ĉ _ig generated from the coefficients c _0.r and c _0.g assume the same values. This means that the initially different polynomial functions for different color channels become identical after normalization and can therefore be described by a single polynomial function.

Based on this knowledge, a third possibility according to the invention is to consider all color channels together and to treat all color values without considering their membership in a color channel. Common correction parameters P can then be determined for the color channels and corrected with a common correction K. This will be through 19 (b) illustrated and offers the advantage of the lowest computational effort. Furthermore, it offers the advantage that monochrome images and color images can be treated identically in this way and there is no need to create expensive variants between monochrome and color cameras. The procedures mentioned in points 1 and 2 above can be applied to the color case without any changes.

The fact that such a joint observation of the color channels is possible is achieved through an interaction between the logarithm of the input values, the frequency distribution of the colors in the same spatial area, which is usually constant in the mosaic filter and also in the interpolated color image, and the least squares method.

When the error squares are considered together, parameters for a polynomial exponential function are found that are suitable for precisely describing the shape of the intensity distribution and its spatial position, but not its offset in the space of logarithmic values, which corresponds to a factor in the linear space of values. In relation to equation (1.2), this means that the parameters c ₁ and c ₂ can be determined exactly in this way, but not the parameter c ₀ . However, if standardization is subsequently applied in accordance with point 3, this does not represent a disadvantage because the standardization replaces the incorrectly or inaccurately determined parameter c ₀ with a standardized parameter c ₀ . No knowledge of the parameter c ₀ is required to calculate this standardized parameter c ₀ because, for example, according to equation (3.1), c ₀ is not included in the calculation of ĉ ₀ .

The formulas and relationships mentioned can be extended to further colors, e.g. blue, by further degrees and/or by a second dimension, using the mathematical principles, relationships and procedures mentioned in a way that is understandable to those skilled in the art. An extension to interpolated color images, where there are several color values, e.g. RGB, for each pixel, is also possible.

6. Displacement and AOI

Sometimes when operating an electronic camera, not an entire image 120 is required, but only an image section 121, which is also referred to in English as an “area of interest” (AOI) or as a “region of interest” (ROI). This situation is in 20 shown. In such a case, it is desirable not to redetermine parameters separately for each selectable image section not only to be able to use certain parameters for the entire image 120 but also to correct the image section 121.

Without loss of generality, it is assumed here that there is a progressive readout. The reading of the entire image 120 thus begins at a starting point 122, which is usually in its upper left corner, and the reading of the image section 121 begins at a shifted starting point 123, which is usually in its upper left corner. The horizontal displacement value is referred to as Δx and the vertical as Δy.

Then, when using a polynomial exponential function according to, for example, equation (1.2) or equation (1.6), the coordinates of the image section x _a and y _a can be converted into the coordinates of the entire image x and y by adding the corresponding shift values to the coordinates: $$\text{x}={\text{x}}_{\text{a}}+\mathrm{\Delta }\text{x},\text{and y}={\text{y}}_{\text{a}}+\mathrm{\Delta }\text{y}.$$

Both equations can also be easily solved for x _a and y _a . After such a conversion, the image section can be corrected with the parameters determined for the entire image using the calculated coordinates x and y.

However, if a cascaded feedback multiplier according to point 4 is used for a correction, the situation becomes somewhat more difficult. If such a cascaded feedback multiplier runs on the entire image 120, certain values are set in each of the feedback multipliers until, for example, the upper left corner of the image section 123 is reached, which has coordinates shifted by Δx and Δy. However, if you want to start at starting point 123, all of the feedback multipliers involved must be initialized with the corresponding values. This requires a respective recalculation of the value vector according to equation (4.12).

A simple way to make this recalculation from an already determined coefficient vector c according to equation (4.12) is to shift the polynomial in the argument of the polynomial exponential function. For reasons of simplicity, the underlying mathematical procedure is explained for a one-dimensional polynomial of the second degree depending on a general free variable z. For a shift in coordinate space by a shift value Δz, z is replaced in the argument from equation (1.2) by the difference z - Δz. This results in the following equation, which can be further transformed below: $$\begin{array}{c}\text{G}\left(\text{e.g}-\mathrm{\Delta }\text{e.g}\right)={\text{<figure-callout id="0" label="e c" filenames="DE102018115991B4\_0067.png,DE102018115991B4\_0076.png" state="{{state}}">e</figure-callout>}}^{{\text{c}}_{}}\\ ={\text{<figure-callout id="0" label="e c" filenames="DE102018115991B4\_0067.png,DE102018115991B4\_0076.png" state="{{state}}">e</figure-callout>}}^{\stackrel{}{\stackrel{}{{\text{c}}_{}}}}\\ ={\text{e}}^{{\tilde{\text{c}}}_0+{\tilde{\text{c}}}_1+\text{e.g}+{\tilde{\text{c}}}_2{\text{e.g}}^2}\end{array}.$$

It turns out that by shifting the polynomial exponential function with coefficients c ₀ , c ₁ and c ₂ by Δz, a polynomial exponential function with mostly different coefficients c̃ ₀ , c̃ ₁ and c̃ ₂ is created. The coefficients c̃ ₀ , c̃ ₁ and c ₂ are in turn combined into a vector c̃: $$\tilde{\text{c}}=\left[\begin{array}{c}{\tilde{\text{c}}}_0\\ {\tilde{\text{c}}}_1\\ {\tilde{\text{c}}}_2\end{array}\right].$$

Then the calculation of c̃ from c can be carried out using a linear shift operator S, which can be written down in matrix form according to the second step of equation (6.2): $$\text{S}\left(\mathrm{\Delta }\text{e.g}\right)=\left[\begin{array}{ccc}1& \mathrm{\Delta }\text{e.g}& \mathrm{\Delta }{\text{e.g}}^2\\ 0& 1& 2\mathrm{\Delta }\text{e.g}\\ 0& 0& 1\end{array}\right].$$

Consequently, c̃ is obtained from c by applying the shift operator S: $$\tilde{\text{c}}=\text{S}\left(\mathrm{\Delta }\text{e.g}\right)\text{c}.$$

If the shifted value vector c with its calculation formula according to equation (6.5) is now inserted into equation (4.15), the calculation of the value vector a for the coordinates shifted by Δz in the one-dimensional case results in: $$\text{a}={\text{e}}^{\text{TSc}}.$$

Using the same or similar mathematical concepts, laws and considerations, one skilled in the art can also determine shift operators for polynomial exponential functions of higher degree or multiple dimensions. As an example, the two-dimensional displacement matrix for a value vector c according to equation (4.19) by a horizontal displacement value Δx and a vertical displacement value Δy for a two-dimensional polynomial exponential function is given explicitly: $$\text{S}\left(\mathrm{\Delta }\text{x},\mathrm{\Delta }\text{y}\right)=\left[\begin{array}{ccccc}1& \mathrm{\Delta }\text{x}& \mathrm{\Delta }{\text{x}}^2& \mathrm{\Delta }\text{y}& \mathrm{\Delta }{\text{y}}^2\\ 0& 1& -2\mathrm{\Delta }\text{x}& 0& 0\\ 0& 0& 1& 0& 0\\ 0& 0& 0& 1& 2\mathrm{<figure-callout\; id="0"\; label="\Delta \; y"\; filenames="DE102018115991B4\_0067.png,DE102018115991B4\_0076.png"\; state="\{\{state\}\}">\Delta </figure-callout>}\text{y}\\ 0& 0& 0& 0& 1\end{array}\right]$$

By applying such a shift, starting values can be determined in a simple manner and with little computational effort as elements of the value vector a, with which a cascaded feedback multiplier can be initialized in order to directly start calculating correction values for the image section 121 in its starting point 123 without any previous calculation steps to start.

7. Mirroring and reversing the reading direction

In order to apply the correction according to the invention to a sensor with a "split readout" (as described above) using cascaded feedback multipliers, it may be advantageous to use several cascaded feedback multipliers, e.g. one for the left block and one for the right block. The right block is read out in a horizontally mirrored direction compared to the progressive reading of the left block.

This can happen when using a cascaded feedback multiplier, for example according to 16 lead to difficulties as this is initially designed for use with a block and with progressive readout. These difficulties can be overcome, for example, by using several, for example two, separate cascaded feedback multipliers for each block, for example for the left block and for the right block. For example, one can work in a progressive direction from left to right and another, for example, in a horizontally mirrored direction that reads progressively from right to left.

The fundamental problem here is to determine the vector a for the horizontally mirrored, cascaded feedback multiplier. Basically, the polynomial in the argument of the polynomial exponential function can be reflected on the line with the coordinate x = 0 by replacing x with -x everywhere. However, an equivalent mathematical effect can also be achieved by adjusting the coefficient vector c with a mirror operator M. A mirrored coefficient vector č is obtained: $$\stackrel{⌣}{\text{c}}=\text{Mc}.$$

For example, for a one-dimensional polynomial exponential function of the second degree, the mirror operator takes the following form: $$\text{M}=\left[\begin{array}{ccc}1& 0& 0\\ 0& -1& 0\\ 0& 0& 1\end{array}\right].$$

Now, in the rarest of cases, a reflection is required at the line x = 0, but usually at another line, for example at the dividing line between the left block and the right one. For this purpose, a first shift of the polynomial to the corresponding dividing line can take place with a shift operator S, the mirroring can be carried out there and subsequently a backward shift can take place, which can be carried out, for example, with the shift operator S ^-1, which is inverse to the first shift S, or with a second one Shift operator with an argument negated from the first shift operator. The matrices used for these operations can be unified using matrix multiplications.

8. Vectorization and parallelization

The way of iteratively calculating a polynomial exponential function suggested under point 4 basically works serially. One time step is required to correct each pixel.

This situation will be in 21 symbolized. Successive pixels 300, 301, 302 and 303 are corrected in successive time steps by calculating a correction value for pixel 301 from the correction value for pixel 300 in a first time step 304, and a correction value for pixel 301 from the correction value for pixel 301 in a second time step 305 Pixel 302 and in a third time step 306 from the correction value for pixel 302 a correction value for pixel 303.

Sometimes it may be necessary to perform a partially parallel calculation instead of the serial calculation. This makes it possible, for example, to achieve a higher correction bandwidth by correcting more pixels per unit of time.

This can be done, for example, by having several cascaded feedback multipliers working at the same time. This way of working is called parallel and is in 22 shown. Here, pixels 310 to 321 are alternately grouped into four channels. In the example shown, pixels 310, 314 and 318 form a first channel, pixels 311, 315 and 319 form a second channel, pixels 312, 316 and 320 form a third, and pixels 313, 317 and 321 a fourth. During a first time step, the calculation 322 of the correction value of the pixel 314 from the correction value of the pixel 310, the calculation 323 of the correction value of the pixel 315 from the correction value of the pixel 311, the calculation 324 of the correction value of the pixel 316 from the correction value of the pixel 312 and the calculation 325 of the correction value of the pixel 317 from the correction value of the pixel 313 simultaneously. During a second time step, based on this, the calculation 326 of the correction value of the pixel 318 takes place from the correction value of the pixel 314, the calculation 327 of the correction value of the pixel 319 from the correction value of the pixel 315, the calculation 328 of the correction value of the pixel 320 from the correction value of the pixel 316 and the calculation 329 of the correction value of the pixel 321 from the correction value of the pixel 317 simultaneously. This is achieved by four parallel cascaded feedback multipliers executing calculations in time steps simultaneously, with each cascaded feedback multiplier operating on one channel each and the pixels being assigned to the channels in alternating order. Basically, the parallel cascaded feedback multipliers work independently of each other. This can have the disadvantage that calculation errors, for example rounding errors, can accumulate in different ways in different channels, which can lead to visible differences in pixels of different channels and thus to undesirable visible stripes in the resulting image.

One way according to the invention to avoid this is, when using several parallel cascaded feedback multipliers, to have these calculation steps carried out in such a way that in a calculation step carried out by several parallel cascaded feedback multipliers, the calculations of the individual feedback multipliers are based on the same value of a predetermined channel is placed.

This is in 23 shown in which the same reference numbers are used for the pixels 22 and in which they are assigned to the same channels. Here, the calculations for correction values of pixels 314, 315, 316 and 317 are carried out simultaneously in parallel in calculation steps 330, 331, 332 and 333 based on the correction value of the same pixel 310 and the calculations for correction values of pixels 318, 319, 320 and 321 simultaneously in parallel in calculation steps 334, 335, 336 and 337 based on the correction value of the same pixel 314.

9. Zoning

In principle, it is possible to apply the proposed correction to the entire image in the same way. To achieve greater accuracy for the correction, it can also be advantageous to define several zones in the image. This can be done, for example, by defining radial zones or dividing the image into rectangular areas, as in 8(b) or (c) proposed. In order to achieve even higher precision, it is possible according to the invention to further increase the number of zones. In principle, it is possible to relate the zones to a center, for example to a geometric center 71 or an optical center 72 according to 8 (a) .

Practical observations show that the edge shading of real lenses sometimes does not fully satisfy the assumption of radial symmetry. In such cases, better results can be achieved if zones are divided not only in the radial direction but also in the circumferential direction, such as in 24 (a) shown. Here there is a subdivision in the circumferential direction into 8 zones per radial subdivision. Zones can also be combined, such as in 24 (b) shown. Since edge shading effects can occur primarily in peripheral areas, the calculation effort can be reduced by, for example, combining several zones in the central area. In the example shown, this creates a round zone in the central area.

Furthermore, the calculation effort for the division into zones can be reduced or made more hardware-friendly if the dividing lines between the zones are made straight. This then results in zones in the form of polygons, such as those shown in Figures 24 (c) to (e).

Finally, it is also possible to define zones in such a way that there is no reference to a center point. This makes the calculation easier because it eliminates the need to determine a center point. Such a classification can be made, for example, as in 8(c) shown. 24 (f) shows an alternative in which each polygon is connected to fewer neighboring polygons, thereby consuming fewer degrees of freedom in determining the coefficients for the polynomial exponential functions. In principle, there are numerous other possible ways to divide an area into uniform or uneven polygons.

Within each zone, the correction is made with a polynomial exponential function, whereby the parameters of the polynomial expo nential functions in different zones can be distinguished from one another. The various polynomial exponential functions defined on different zones then together form a correction function.

In order to avoid visible artifacts when moving from one zone to the next, it is important that in adjacent zones the polynomial exponential functions defined on these zones continuously merge into one another when forming the correction function. This can be ensured by ensuring that the polynomial exponential functions of two adjacent zones each assume the same value at the same point. Since changes in brightness gradients can also be clearly visible, visible artifacts are further avoided by the fact that, in adjacent zones, the polynomial exponential functions defined on these zones merge into one another in a continuously differentiable manner when forming the correction function. This can be achieved by the polynomial exponential functions of two adjacent zones each assuming the same derivative at the same point, for example by having the same gradient vector.

These conditions can be met when determining coefficients together, e.g. using the least squares method and on logarithmic image values, by formulating them mathematically as secondary conditions. The subsequent calculation of a solution with least square error can be determined using a so-called “constrained least squares” method.

10. Alternative Embodiments

In the present description, several exemplary calculation options for the value vector a and for the values that form its elements were mentioned, e.g. in number 1 the negation with the negation operator N, in number 3 the normalization with the normalization operator F, in number 4 the transformation with the transformation operator T, in number 6 the shift with the shift operator S and in number 8 the reflection with the reflection operator M. These operators can be combined with one another as required to calculate the value vector a, for example in the following form: $$\text{a}={\text{e}}^{\text{TSNFc}}.$$

In addition, it is also possible to swap the orders of the operators if they commute in individual cases. For example: $$\text{a}={\text{e}}^{\text{TSNFc}}={\text{e}}^{\text{TSFNC}}.$$

Furthermore, several operators can also be combined to form common operators. If the operators are linear operators in matrix notation, this can be done by applying matrix multiplication to the multiple operators. For example, you can define a matrix A with: $$\text{A}=\text{TSN}.$$

Then equation (10.1) can be rewritten as: $$\text{a}={\text{e}}^{\text{AFc}}.$$

Using similar mathematical ideas and procedures, numerous other equivalent representations of the calculation of parameters can also be achieved.

A calculation in digital hardware was described in point 4. It is also possible to carry out the specified operations partially or completely in analog electronics and, for example, to replace digital multipliers with analog multipliers. Further simplifications are also possible in analog electronics. For example, the operation of a feedback multiplier 221, whose inputs are each connected to constant values and whose mathematical operation is described in equation (4.2), can also be achieved using a so-called RC element, the voltage of which also increases exponentially over the course of linearly successive time steps falls off.

Furthermore, all calculation steps outlined for the calculation in digital hardware can in principle also be carried out in software. The feedback can be programmed, for example in the form of a loop, which then carries out the calculation for a time step with each loop run.

Furthermore, the calculation of correction values or the correction of pixels can also be carried out in a different order. For example, it is also possible to carry out the correction column by column and then within each column for the pixels of this column line by line. For this purpose, the coefficients or initialization values can be exchanged or adjusted accordingly.

It is also possible to carry out a preliminary calculation for correction values and to save the correction values until the correction is applied. The pre-calculation can be carried out, for example, at the setup time or at another time before the runtime become. Correction values can be saved for each individual pixel or for groups of pixels. This allows them to be retrieved at runtime and applied without the need to calculate them at runtime. This allows logic resources to be saved in hardware.

Horizontal correction values g(x) and vertical correction values g(y) can also be stored. A multiplication of horizontal and vertical correction values can then be carried out at runtime according to equation (1.5). This also results in a two-dimensional polynomial exponential function g(x, y), which can be used for vignetting correction. The calculation effort of g(x,y) that occurs at runtime consists of just one multiplication, which can be carried out by a multiplier, for example, and is low. At the same time, such a procedure requires much less storage space to store correction values than with pixel-by-pixel storage. This is important because in numerous circuits that can be used as image processing means 13, for example in an FPGA, in an MCU or in a DSP, only relatively little memory is available and that generally static memory, which has a fast Access allows, requires a lot of expensive chip space.

Finally, it is of course also possible to save parts of the correction values before runtime and to calculate parts of the correction values during runtime. For example, vertical correction values can be saved and used as initialization values for a horizontal cascaded feedback multiplier. This allows the aforementioned advantages to be combined accordingly.

11. Determination of the center point

Some of the aforementioned methods require the determination of an optical center. This can be determined based on the usually sufficiently accurate assumption that vignetting occurs radially symmetrically.

According to 25 According to the invention, to determine the optical center, one can start from image data 500 of an initial image that was recorded under homogeneous illumination, for example by recording a white or gray reference board or a white surface. The image data can be unprocessed as required, they can be corrected, for example with regard to brightness or color or other effects that occur in an electronic camera, but they can also be logarithmized in the sense of number 2 or reciprocal in the sense of 6 .

This image is now subjected either to horizontal averaging 501 to determine the vertical coordinate y _c of the optical center or to vertical averaging 502 to determine the horizontal coordinate x _c of the optical center. With horizontal averaging, an average or a multiple of the average of the pixel values of each line is calculated and with vertical averaging, an average or a multiple of the pixel values of each line is calculated. This averaging provides a data vector 503. As in, for example 26 As can be seen, this averaging has the advantage that even with a brightness distribution 520 that deviates greatly from a parabolic shape, due to the radial symmetry of these distributions, it usually delivers a result of the averaging 521 whose shape is already much closer to a parabola, which is the following Determination of a coordinate of the optical center is made much easier.

An estimate of the corresponding x or y coordinate of the center point 504 can now be made. This estimate can, for example, be based on a coordinate of the geometric center. This estimated value is adopted as a current midpoint 505 in the first step 510. A data range is selected based on this current center point. Such a selection is preferably carried out symmetrically, so that the same amount of data from the data vector 503 belongs to the data area on both sides of the current center point. A compensation parabola 507 with a minimum squared error is then determined on the data area. Choosing the data area symmetrically around the current center has the advantage that even if the data vector deviates significantly from a parabola, the higher moments occur symmetrically, so that they do not lead to a deviation in the coordinates of the vertex. Therefore, this coordinate of the vertex 508 can be determined with higher accuracy and results in an updated center 509, which replaces the previous current center in the subsequent steps 511.

In 27 this procedure is illustrated. Here, as a result of the estimate 504, the coordinate x _c.0 is obtained. It lies in the middle of the x area, so that the distance to the edge r _x.0 is the same to the left and to the right. In this data area, the data vector 530 is described by a parabola 531 with the least square error. The vertex of the parabola can now be determined using the following formula and adopted as the updated center x _c.1 : $${\text{X}}_{\text{c}.1}=\frac{-{\text{<figure-callout id="1" label="c" filenames="DE102018115991B4\_0063.png,DE102018115991B4\_0064.png" state="{{state}}">c</figure-callout>}}_1}{2{\text{c}}_2}.$$

c ₁ and c ₂ describe the corresponding coefficients of the parabola as a second degree polynomial.

The further procedure is based on 28 explained. The parabola 533 shown here is identical to the parabola 531 27 identical. Starting from its vertex x _c.1, a data area is now selected again. Since x _c.1 is closer to an image edge than x _c.0 , the data area can only be chosen slightly smaller this time and extends between the two arrowheads. In the second step, a parabola 534 with the least square error is now determined only on the selected data area of the data vector 532. Now the vertex of this parabola is determined again and a further updated center point x _c.2 is obtained. This process can now be repeated for further steps. The updated center point converges very quickly towards the actual optical center and the iteration can usually be successfully completed after the second or third step.

Further variations of the disclosed embodiments may be understood and carried out by one skilled in the art practicing the claimed invention from a review of the drawings, description and appended claims.

In the claims, the words "comprising" and "comprising" do not exclude other elements or steps and the indefinite article "a" does not exclude a plurality.

A single unit or device can perform the functions of several elements listed in the claims. The fact that individual functions and/or elements are listed in different dependent claims does not mean that a combination of these functions and/or elements could not also be used advantageously.

The reference numerals in the claims should not be understood as limiting the subject matter and scope of the claims by these reference numerals.

• - Korrigieren der Werte der Pixel basierend auf einer Korrekturfunktion, die eine Exponentialfunktion zu einer Basis, vorzugsweise entweder der Eulerschen Zahl e oder der Zahl 2 oder 10, umfasst, deren Argument ein ein- oder mehrdimensionales Polynom des Grades n ≥ 2 ist, oder die mathematisch in eine solche Funktion umformbar ist,
  • wobei das Korrigieren des Wertes eines Pixels ein Dividieren oder vorzugsweise ein Multiplizieren des Wertes des Pixels durch den bzw. mit dem Wert der Korrekturfunktion an der Position des Pixels umfasst.

In summary, a correction method for correcting a vignetting effect in values of pixels of an electronic camera image has been provided, the correction method comprising:

• - Correcting the values of the pixels based on a correction function comprising an exponential function to a base, preferably either Euler's number e or the number 2 or 10, the argument of which is a one- or multi-dimensional polynomial of degree n ≥ 2, or the can be mathematically transformed into such a function,
  • wherein correcting the value of a pixel includes dividing or preferably multiplying the value of the pixel by the value of the correction function at the position of the pixel.

Claims (18)

Correction method for correcting a vignetting effect in values of pixels of an image (I ₁ ) of an electronic camera (10), comprising: - determining the value of a correction function at the position of a pixel iteratively by multiplying one or more factors with an already determined value of the correction function the position of another pixel, and correcting the value of the pixel, wherein correcting comprises multiplying the value of the pixel by the value of the correction function at the position of the pixel, wherein correcting corrects the vignetting effect in the value of the pixel, wherein the correction function an exponential function to a base, preferably either Euler's number e or the number 2 or 10, the argument of which is a one- or multi-dimensional polynomial of degree n ≥ 2, the value of the correction function at the position of the pixel based on cascading of feedback multiplications carried out by a circuit of at least two cascaded feedback multipliers is determined such that the values of the correction function at the positions of the pixels are continuously calculated, multiplying the value of the pixel by the value of the correction function at the position of the pixel is carried out with a multiplier connected downstream of the at least two cascaded feedback multipliers. correction procedure Claim 1 , where factors of the feedback multiplications are each determined as the value of the exponential function to the base, the argument of which is a weighted sum of one or more of the coefficients of the polynomial. Correction procedure according to one of the Claims 1 or 2 , where the image (I ₁ ) is a two-dimensional image and the cascading of the feedback multiplications first feedback multiplications in a first direction, for example, in time direction, of the image (I ₁ ) and second feedback multiplications in a second direction, for example, in the column direction, of the image (I ₁ ), wherein an output value of the second feedback multiplications provides an input value of the first feedback multiplications. Correction procedure according to one of the Claims 1 until 3 , wherein the determination of the values of the correction function at the positions of the pixels is parallelized in such a way that the pixels are divided into a number of channels, which: - are each processed separately from one another, or - are processed in such a way that the determination is made in one time step the values of the correction function at the positions of pixels of at least two different channels are based on a common already determined value of the correction function at the position of another pixel. Correction procedure according to one of the Claims 1 until 4 , where the image is a color image which is present as an image of a mosaic filter, with the value of each pixel being assigned to a color channel. Correction procedure according to one of the Claims 1 until 5 , where the values of the pixels of the image (I ₁ ) are color values of two or more color channels and correcting the values of the pixels comprises: - Correcting the values of the pixels with a separate correction function for each color channel, preferably in such a way that the correction functions for two different channels only differ in a constant offset of the values of the correction function, or - correcting the values of the pixels with a correction function common to all color channels. Correction procedure according to one of the Claims 1 until 6 , whereby only values of pixels of a section (121) of the image (I ₁ ) are corrected and a shift of the origin (123) of the section relative to the origin (122) of the image is taken into account by a corresponding shift of the correction function in such a way that at least some of the parameters of the correction function are replaced by shifted parameters of the correction function. Correction procedure according to one of the Claims 1 until 7 , whereby several zones are defined in the image (I ₁ ) and the values of the pixels for each zone are corrected using the correction function parameters specific to this zone. correction procedure Claim 8 , whereby the respective parameters of the correction function for two adjacent zones are such that the transition of the correction function between the two zones is continuous, and preferably continuously differentiable. correction procedure Claim 8 or 9 , wherein the zones relate to a center (71, 72), preferably the optical center (72), of the image. correction procedure Claim 10 , wherein the correction method comprises estimating the optical center (72) of an image, which preferably shows a homogeneous scene, such that, in the event that the image was generated with a one-dimensional image sensor, estimating the coordinate of the optical center (72 ) in one dimension includes: - generating a one-dimensional data vector in one dimension from the values of the pixels of the image, - iteratively fitting a one-dimensional compensation parabola to the one-dimensional data vector, and that in the case that the image was generated with a two-dimensional image sensor , estimating a coordinate of the optical center (72) in a first dimension: - averaging the values of the pixels of the image in a second dimension to generate a one-dimensional data vector of averaged values in the first dimension, and - iteratively fitting a one-dimensional balancing parabola the one-dimensional data vector. correction procedure Claim 11 , wherein the iterative fitting takes place in each iteration in a data area of the one-dimensional data vector around a currently estimated coordinate of the optical center (72) in the one or first dimension, the data area preferably being symmetrical to the currently estimated coordinate of the optical center (72 ) is in the one or first dimension and/or the iterative fitting with the coordinate of the geometric center (71) of the image in the one or first dimension as the currently estimated coordinate of the optical center (72) in the one or first Dimension is initialized. Correction procedure according to one of the Claims 1 until 12 , comprising determining parameters of the correction function based on a reference image (I ₀ ), which preferably shows a homogeneous scene, wherein determining the parameters of the correction function includes normalizing the correction function such that the correction function has a predetermined value at its apex, preferably the Value 1. correction procedure Claim 13 , where the normalization is carried out such that the zero-order coefficient of the polynomial is replaced by a zero-order normalizing coefficient which is calculated from the higher-order coefficients of the polynomial. Correction device for correcting a vignetting effect in values of pixels of an image (I ₁ ) of an electronic camera (10), the correction device comprising a circuit of at least two cascaded feedback multipliers and a multiplier connected downstream of the at least two cascaded feedback multipliers and is designed to: - Determining the value of a correction function at the position of a pixel iteratively by multiplying one or more factors with an already determined value of the correction function at the position of another pixel, and - correcting the value of the pixel, wherein correcting involves multiplying the value of the pixel by the Value of the correction function at the position of the pixel, the correcting correcting the vignetting effect in the value of the pixel, the correction function comprising an exponential function to a base, preferably either the Euler number e or the number 2 or 10, the argument of which is a - or multidimensional polynomial of degree n ≥ 2, where the value of the correction function at the position of the pixel is determined based on a cascading of feedback multiplications carried out by the circuit of at least two cascaded feedback multipliers, such that the values of the Correction function at the positions of the pixels are continuously calculated, the multiplication of the value of the pixel by the value of the correction function at the position of the pixel being carried out with the multiplier connected downstream of the at least two cascaded feedback multipliers. Electronic camera (10), the electronic camera (10) comprising: - an image sensor (12) with pixels for generating an image (I ₁ ); and - the correction device Claim 15 to correct a vignetting effect in values of pixels of the image (I ₁ ). Computer device, comprising a computing unit which is used to carry out the correction method according to one of Claims 1 until 14 is designed. Computer program product comprising code means for causing a computer device to carry out the correction method according to one of the Claims 1 until 14 when the computer program product is running on the computing device.

Images