DE102016222548A1 - Method and control unit for determining a distortion characteristic of an optical medium during travel operation - Google Patents

Abstract

The invention relates to a method for determining the distortion characteristics of an optical medium (202) using an optical sensor (102) sensing through the optical medium (202) during the running operation, wherein in a step of determining at least one parameter of a distortion model for compensating an optical Distortion of the optical medium (202) is determined using an image sequence of the optical sensor (102) and a trajectory of the sensor to determine the distortion characteristic of the optical medium (202) during a driving operation, the trajectory extending at least over two imaging times and a translation and / or rotation of the sensor (102) between a first detection time of a first image of the image sequence and a second detection time of a second image of the image sequence represents.

Description

State of the art

The invention is based on a control device or a method according to the preamble of the independent claims. The subject of the present invention is also a computer program.

In an optical sensor which is arranged behind an obliquely oriented disk, an image of an environment detected by the sensor through the disk is recorded. To correct for this distortion, a correction rule is required which shifts sample pixels of the image opposite to the distortion.

Disclosure of the invention

Against this background, a method for determining a distortion characteristic of an optical medium during travel operation is presented with the approach presented here, furthermore a control unit which uses this method, and finally a corresponding computer program according to the main claims. The measures listed in the dependent claims advantageous refinements and improvements of the independent claim device are possible.

A distortion characteristic of an optical medium, such as a front or rear window, manifests itself in an image recorded through the optical medium. For example, straight lines and edges can be displayed bent by the distortion or distortion characteristic. In other words, the distortion results in a distorted image in which a scene point is imaged at a different position than in the unrecognized case.

With fixed relative position of the optical medium to the optical sensor, the distortion characteristic remains constant over time. This is the case with a behind a windshield, or windshield, arranged optical sensor.

From the displacement of the image of a scene point at different sampling times (also referred to as optical flow hereinafter), it is possible to draw conclusions about the movement of the sensor between the sampling instants on the distortion property of the optical medium. In addition, the inclusion of a distance information with respect to the scene point is possible.

The approach described here uses the relationships described above to determine the present distortion characteristics of the optical medium. It establishes a functional relationship between distortion characteristic, vehicle environment and their recorded image. The distortion characteristic is described by a model whose parameters, via the functional relationship, can be determined from an imaging sequence of the vehicle environment. The imaging sequence results from a trajectory of the moving vehicle over time, which results in a sensor trajectory. This consists of at least two individual poses of the sensor at the respective sampling times. The respective transformation consists of at least one rotational and / or one or more translatory components and can be described for example by a vector-valued translation and a rotation about the three main axes of the sensor.

A method is provided for determining a distortion characteristic of an optical medium during travel using an optical sensor sensing the medium, wherein in a step of determining at least one parameter of a distortion model for compensating for optical distortion of the optical medium using an image sequence of the optical Sensor and a trajectory of the optical sensor is determined to determine the distortion characteristic of the optical medium during a driving operation, wherein the trajectory translation and / or rotation of the optical sensor between a first detection time of a first image of the image sequence and a second detection time of a second image represents the image sequence.

The model describes the distortion characteristic of the optical medium at least approximately and contains at least one parameter. The distortion rule contained in the model allows the correction of a sensor image, thereby achieving at least a reduction in the distortion contained.

The information of the sensor trajectory can be determined or estimated as a further degree of freedom or parameters, via the functional. This is possible via the images of static scene objects, as will be explained in more detail below in a formula. Alternatively, the movement information from an additional sensor of the vehicle can be fed, such as an inertial sensor, which can be read via the data bus of the vehicle.

As further information, depth information regarding the vehicle environment may be included in the functional (environment sensing sensor). This makes it possible to dispense with the step of eliminating the depth as unknown, according to a formula described in more detail below. This results in a variation of the method based on a slightly different functional.

The method can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control unit.

The approach presented here also creates a control unit which is designed to execute, to control or to implement the steps of a variant of a method presented here in corresponding devices. Also by this embodiment of the invention in the form of a control device, the object underlying the invention can be achieved quickly and efficiently.

For this purpose, the control unit can have at least one arithmetic unit for processing signals or data, at least one memory unit for storing signals or data, at least one interface to a sensor or an actuator for reading sensor signals from the sensor or for outputting control signals to the actuator and / or or at least a communication interface for reading or outputting data embedded in a communication protocol. The arithmetic unit may be, for example, a signal processor, a microcontroller or the like, wherein the memory unit may be a flash memory, an EEPROM or a magnetic memory unit. The communication interface can be designed to read or output data wirelessly and / or by line, wherein a communication interface that can read or output line-bound data, for example, electrically or optically read this data from a corresponding data transmission line or output in a corresponding data transmission line.

In the present case, a control device can be understood as meaning an electrical device which processes sensor signals and outputs control and / or data signals in dependence thereon. The control unit may have an interface, which may be formed in hardware and / or software. In the case of a hardware-based embodiment, the interfaces can be part of a so-called system ASIC, for example, which contains various functions of the control unit. However, it is also possible that the interfaces are their own integrated circuits or at least partially consist of discrete components. In a software training, the interfaces may be software modules that are present, for example, on a microcontroller in addition to other software modules.

Also of advantage is a computer program product or computer program with program code which can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and for carrying out, implementing and / or controlling the steps of the method according to one of the embodiments described above is used, especially when the program product or program is executed on a computer or a device.

• 1 ein Blockschaltbild eines Steuergeräts zur Bestimmung einer Verzeichnungseigenschaft eines optischen Mediums während des Fahrtbetriebs unter Verwendung eines optischen Sensors gemäß einem Ausführungsbeispiel;
• 2 eine Darstellung eines optischen Pfads durch eine Frontscheibe;
• 3 eine Darstellung einer optischen Verzeichnung durch ein optisches Medium;
• 4 einen Verlauf einer optischen Verzeichnung eines optischen Mediums; und
• 5 ein Ablaufdiagramm eines Verfahrens zur Bestimmung einer Verzeichnungseigenschaft eines optischen Mediums während des Fahrtbetriebs unter Verwendung eines optischen Sensors gemäß einem Ausführungsbeispiel.

Embodiments of the approach presented here are shown in the drawings and explained in more detail in the following description. It shows:

• 1 a block diagram of a controller for determining a distortion characteristic of an optical medium during the driving operation using an optical sensor according to an embodiment;
• 2 a representation of an optical path through a windshield;
• 3 a representation of an optical distortion by an optical medium;
• 4 a course of optical distortion of an optical medium; and
• 5 a flowchart of a method for determining a distortion characteristic of an optical medium during the driving operation using an optical sensor according to an embodiment.

In the following description of exemplary embodiments of the present invention, the same or similar reference numerals are used for the elements shown in the various figures and similarly acting, wherein a repeated description of these elements is omitted.

1 shows a block diagram of a controller 100 for operating an optical sensor 102 according to an embodiment. The control unit 100 and the optical sensor 102 are connected to a data bus 104 of a vehicle 106 connected. The optical sensor 102 is disposed in an interior of the vehicle 106 and through a windshield of the vehicle 106 as an optical medium on an environment of the vehicle 106 directed. That of the optical sensor 102 captured image of the environment is recorded by the windscreen. As a result of this distortion, objects of the environment are depicted with different distortions depending on their position in the image. The distortion is done using a distortion model 108 compensated. The distortion model 108 allows to derive a correction vector for each pixel of the image. It is noted in this connection that, analogously, also a distortion engagement of a rear window is determined using the or another optical sensor. Thus, it will be appreciated that while the approach presented herein is described using a windshield of the vehicle herein, this approach may be used to determine the distortion characteristic of virtually any optical medium, regardless of use on a windshield.

The control unit 100 can be termed ECU and is designed to be in a facility 110 for estimating or determining at least one parameter 112 of the distortion model 108 using one of the optical sensor 102 recorded sequence of pictures 114 and a movement of the optical sensor 102 while capturing the picture sequence 114 to determine. The distortion model 108 may be referred to as the distortion characteristic of the windshield. In this case, the movement of at least one object image-forming pixel in the image sequence 114 with the movement of the optical sensor 102 compared. A proportion of the movement of the pixel, which does not result from the movement of the optical sensor, results either from a self-motion of the object or from the distortion by the front screen.

In one embodiment, the controller uses 100 further more on the data bus 104 provided sensor data 116 other sensors 118 of the vehicle 106 to the parameter 112 to determine. For example, from an on-the-field radar sensor 118 captured position information 116 of the object used to account for the inherent motion of the object.

Additional in the vehicle 106 built-in sensors 118 can be used to improve the estimate. For example, measurement data 116 a radar sensor 118 to be used over the CAN bus 104 be available.

By avoiding a separate calibration process for determining the disc distortion considerable additional expenses or additional costs in vehicle production and subsequent replacement of discs can be avoided.

In 1 a schematic structure of the directory estimation is shown. The optical sensor 102 For example, it may be an image sensor or a line sensor.

2 shows a representation of an optical path 200 through a windscreen 202. The windscreen 202 is for example the windscreen of in 1 shown vehicle. As in 1 is on an inside of the front window 202 in the interior 204 an optical sensor 102 , For example, a camera 102, arranged. As in 1 is the optical sensor 102 with a control unit 100 for processing the data of the optical sensor 102 connected.

In optical sensor 102 incident light first penetrates the windscreen 202. Due to the higher air density of the windscreen 202 the light is at the interfaces of the windscreen 202 Broken. The here as optical path 200 designated light beam 200 meets with an angle of incidence α on the windscreen 202 , At the transition from air to glass the light beam becomes 200 broken and refracted by an angle of incidence α dependent refraction angle. Inside the windscreen 202 points the light beam 200 so that a different angle than the angle of incidence α. At the transition from glass to air the light beam becomes 200 again refracted by an angle of refraction and leaves the windscreen with an exit angle β. The angle of refraction at the exit is the angle of the light beam 200 inside the windscreen 202 dependent.

The light beam 200 is when penetrating the windscreen 202 laterally offset and deflected by the difference between the angle of incidence α and the angle of reflection β. Because the windscreen 202 has a curvature, resulting for the whole of the optical sensor 102 incident light rays 200 a distortion. The distortion is a two-dimensional course of the displacement and the deflection.

The distortion becomes a through the windscreen 202 captured outer scene 206 distorted. The approach presented here allows a correction of this distortion.

It becomes an online estimate, in the sense of an estimate during operation, of the windshield distortion characteristics 202 presented.

Optical sensors 102 are an important data source in the field of driver assistance. Due to the low price, optical sensors 102 are often used, such as video imagers 102 , These are usually inside the vehicle cabin 204 installed and register the outdoor scene 206 through the windshield 202 ,

The disc 202 represents a disturbance of the optical path 200 which causes a distortion of the image. That's how light rays become 200 broken according to the Snellius law of refraction. The result is a misperception of the outer scenery 206 , For this reason, the vehicle environment is displayed correctly 206 a distortion correction. Are the distortion characteristics of the disc 202 known, a correction of the image can be done, creating an unadulterated perception of the scenery 206 is guaranteed. This can be done on the software side. Alternatively, an inverse optic can be used.

A method is presented which illustrates the distortion characteristics of a windshield 202 determined without using an explicit calibration procedure.

In 2 is the effect of windshield distortion on the optical path 200 a sensor installed in the vehicle 102 shown. Depending on angle α, the sight beam experiences 200 of a world point a distraction that results in a shifted image. In other words, the windshield distortion results in a distorted line of sight 200 ,

The distortion of the windshield 202 can also be determined in an initial calibration process. This requires an external calibration device or additional sensors. The calibration process means a considerable extra effort in vehicle production and every time the windshield is changed. In contrast, in the approach presented here, the distortion characteristics of the disc 202 determined in the finished vehicle and uses no external aids, sensors or equipment.

The distortion characteristics of the windshield 202 are based on the measured data of the behind the disc 202 built-in sensor 102 certainly. An estimate or determination is made in the final vehicle during operation and without the use of external aids. The estimate or determination can be referred to as an online estimate.

The online estimation ensures a necessary correction of the sensor image during operation.

The online estimate provides a functional relationship between a distortion model, the vehicle environment 206 and her distorted image.

3 shows a representation of an optical distortion 300 through an optical medium, such as the windscreen. The windscreen corresponds essentially to the windscreen in 2 , The distortion 300 is given for the pixels of an image captured using an optical sensor, such as those shown in FIG 2 is shown. Pixel coordinates in x and y are plotted in a feature space. Each pixel is a value of distortion 300 assigned.

Here pixels with the same value are connected by isolines. The value is plotted on the right edge of the feature space.

The distortion increases from top to bottom, the increase is uneven. At the margins the distortion is slightly lower than in the middle of the picture.

In other words, the vertical disk distortion 300 depending on the picture position.

4 shows a course 400 an optical distortion 300 the optical medium such as the windscreen. The course essentially corresponds to the course of the distortion in 3 , The gradient is shown in a diagram, the y axis on its abscissa a vertical image coordinate value y and on its ordinate the distortion 300 has applied. As in 3 takes the distortion 300 with increasing image coordinate value y.

Additionally are in 4 polynomials 402 . 404 shown the history 300 approximate. The polynomials 402 . 404 are part of the distortion model to correct the distortion 300 , The shape of the polynomials 402 . 404 is determined by changeable parameters. The first polynomial indicates this 402 a variable parameter while the second polynomial 404 has two variable parameters.

In the approach presented here, the parameters during operation of the optical sensor are optimized to the course of the polynomials 402 . 404 as close as possible to the actual course 400 the distortion 300 to adapt.

In 4 is an approximation of vertical disk distortion 300 represented by a ploynomial model. Here are the measured distortion 400 and a modeling 402 . 404 or approximation of the distortion 300 abraded by polynomial first and second degree. It is an example of a measured vertical windshield distortion 400 with associated polynomial 404 second degree representing the measured vertical distortion 400 approximated as a model.

In this example, the distortion 300 be described by means of three model parameters or the polynomial coefficients. In order to establish a functional relationship to the actually expected image of the environment, for example, one condition of the multiple-view geometry can be used. One such condition is, for example, epipolar geometry or homography. It is also possible to use other possibilities of a functional relationship, which can be derived or derived from the formula (5) of the coordinate transformation. Other measured variables that are frequently available on the CAN bus of the vehicle can be used. For example, as further information, depth information from a depth-measuring sensor, such as a radar sensor, may be used. The depth information can be used to set up a functional using projective geometry.

An embodiment of an algorithm according to the approach presented here is described below.

codiert den Vektor von A nach B im Koordinatensystem C. Siehe auch Formel 5, zur Verdeutlichung der Notation am Beispiel einer Koordinatentransformation in ℝ³. Gemessene Bildkoordinaten P¹ = [u¹,v¹]^T ∈ ℝ² werden in normalisierten Bildkoordinaten als $${\overline{\text{P}}}^{\text{C}}={\text{K}}^{-1}+\left[{}_1^{\text{P'}}\right]=\left[\begin{array}{c}\left(u\text{'}-u_0^{{}_{\text{'}}}\right)\frac{1}{f_u}\\ v\text{'}-v_0^{{}_{\text{'}}}\frac{1}{f_v}\\ 1\end{array}\right]=\left[\begin{array}{c}{\overline{x}}^C\\ {\overline{y}}^C\\ 1\end{array}\right]\text{,}$$

Matrices and vectors are given in bold capital letters and lowercase letters. A designation of a reference coordinate system is added as an upper index on the right side. The relative position or orientation is added as the upper index on the left and as the index on the right. A rotation ^A R _B ∈ ℝ ^{3 × 3} describes a rotation sequence R _{z (} ^A ψ̇ _B ) × R _y ( ^A θ _B ) × R _× ( ^A Φ × _B ) for transferring image A into the image B. A translation vector ${}^{\text{A}}{\text{t}}_{\text{B}}^{\text{C}}\in {\mathrm{<figure-callout\; id="3"\; label="R"\; filenames="DE102016222548A1\_0028.png"\; state="\{\{state\}\}">ℝ</figure-callout>}}^3$

encodes the vector from A to B in the coordinate system C. See also formula 5, for clarification of the notation using the example of a coordinate transformation in ℝ ³ . Measured image coordinates P ¹ = [u ¹ , v ¹ ] ^T ∈ ℝ ² are used in normalized image coordinates as $${\overline{\text{P}}}^{\text{C}}={\text{K}}^{-1}+\left[{}_1^{\text{P '}}\right]=\left[\begin{array}{c}\left(u\text{'}-u_0^{{}_{\text{'}}}\right)\frac{1}{f_u}\\ v\text{'}-v_0^{{}_{\text{'}}}\frac{1}{f_v}\\ 1\end{array}\right]=\left[\begin{array}{c}{\overline{x}}^C\\ {\overline{y}}^C\\ 1\end{array}\right]\text{.}$$

gegeben.Shown, where ^{K is} the intrinsic calibration matrix. For a corresponding pair of points, the point in the last and in the current image is as ${\overline{\text{P}}}^{\text{C1}}\text{and}{\overline{\text{P}}}^{\text{C2}}$

given.

All coordinate systems used are defined according to the Computer Vision Standard.

epipolar geometry

bezeichnet, sodass [v] _× x = v × x, ∀x ∈ ℝ³, so kann die Epipolarbedingung als $${\left({\overline{\text{P}}}^{\text{C2}}\right)}^T{+}^{\text{C2}}{\text{E}}_{\text{C1}}+{\overline{\text{P}}}^{\text{C1}}=\mathrm{0,}$$

formuliert werden. Dabei ist

die essentielle Matrix, welche sich aus der relativen Lage des Sensors zu zwei Aufnahmezeitpukten beziehungsweise aus zwei Aufnahmepositionen ergibt. Die Epipolarbedingung kann aus der Formel der Koordinatentransformation $${\text{P}}^{\text{C2}}{=}^{\text{C2}}{\text{R}}_{\text{C1}}+{\text{P}}^{\text{C1}}{+}^{\text{C2}}{\text{t}}_{\text{C1}}^{\text{C2}}$$

abgeleitet werden, die in normierten Bildkoordinaten als $$z^{\text{C2}}+{\overline{\text{p}}}^{\text{C2}}{=}^{\text{C2}}{\text{R}}_{\text{C}1}+z^{\text{C1}}+{\overline{\text{p}}}^{\text{C1}}{+}^{\text{C2}}{\text{t}}_{\text{C1}}^{\text{C}2}$$

geschrieben wird.Assuming intrinsically calibrated imaging sensors and [v] _x ∈ ℝ ^{3 × 3} the skew-symmetric matrix $${\left[\text{v}\right]}_{\text{x}}=\left[\begin{array}{ccc}0& -v_3& v_2\\ {\mathrm{<figure-callout\; id="3"\; label="v"\; filenames="DE102016222548A1\_0028.png"\; state="\{\{state\}\}">v</figure-callout>}}_3& 0& -v_1\\ -v_2& v_1& 0\end{array}\right]$$

so that [v] _× x = v × x, ∀x ∈ ℝ ³ , the epipolar condition can be considered as $${\left({\overline{\text{P}}}^{\text{C2}}\right)}^T{+}^{\text{C2}}{\text{e}}_{\text{C1}}+{\overline{\text{P}}}^{\text{C1}}=0$$

be formulated. It is

the essential matrix, which results from the relative position of the sensor to two Aufnahmezeitpukten or from two recording positions. The epipolar condition can be derived from the formula of coordinate transformation $${\text{P}}^{\text{C2}}{=}^{\text{C2}}{\text{R}}_{\text{C1}}+{\text{P}}^{\text{C1}}{+}^{\text{C2}}{\text{t}}_{\text{C1}}^{\text{C2}}$$

derived in normalized image coordinates as $$z^{\text{C2}}+{\overline{\text{p}}}^{\text{C2}}{=}^{\text{C2}}{\text{R}}_{\text{C}1}+z^{\text{C1}}+{\overline{\text{p}}}^{\text{C1}}{+}^{\text{C2}}{\text{<figure-callout id="2" label="t C1 C" filenames="DE102016222548A1\_0028.png" state="{{state}}">t </figure-callout>}}_{\text{C1}\text{C}}^2$$

is written.

multipliziert. Dies ergibt $$z^{\text{C2}}+{\left({\overline{\text{P}}}^{\text{C2}}\right)}^T+{\left[{}^{\text{C2}}{\text{t}}_{\text{C1}}^{\text{C2}}\right]}_{\text{x}}+{\overline{\text{P}}}^{\text{C2}}={\left({\overline{\text{P}}}^{\text{C2}}\right)}^T+{\left[{}^{\text{C2}}{\text{t}}_{\text{C1}}^{\text{C2}}\right]}_{\text{x}}{}^{\text{C2}}\text{R}{}_{\text{C1}}+z^{\text{C1}}+{\overline{\text{P}}}^{\text{C1}}$$

To get a formulation free of depth information, both sides are included ${\left[{\text{t}}_{\text{C2C1}}^{\text{C2}}\right]}_{\text{x}}\text{and}{\overline{\text{<figure-callout id="2" label="P ̄ C" filenames="DE102016222548A1\_0028.png" state="{{state}}">P </figure-callout>}}}^{\text{C}}$

multiplied. This results $$z^{\text{C2}}+{\left({\overline{\text{P}}}^{\text{C2}}\right)}^T+{\left[{}^{\text{C2}}{\text{t}}_{\text{C1}}^{\text{C2}}\right]}_{\text{x}}+{\overline{\text{P}}}^{\text{C2}}={\left({\overline{\text{P}}}^{\text{C2}}\right)}^T+{\left[{}^{\text{C2}}{\text{t}}_{\text{C1}}^{\text{C2}}\right]}_{\text{x}}{}^{\text{C2}}\text{R}{}_{\text{C1}}+z^{\text{C1}}+{\overline{\text{P}}}^{\text{C1}}$$

senkrecht zu dem Vektor ${\overline{\text{P}}}^{\text{C2}}$

steht, ist das innere Produkt ${\left({\overline{\text{P}}}^{\text{C2}}\right)}^T+{\left[{}^{\text{C2}}{\text{t}}_{\text{C1}}^{\text{C2}}\right]}_{\text{x}}+{\overline{\text{P}}}^{\text{C2}}$

Null. Dies bedeutet, dass der Term

ebenfalls gleich Null ist. Da z^c1 > 0 ist, erfüllen die beiden normalisierten Bildpunkte ${\overline{\text{P}}}^{\text{C1}}$

und ${\overline{\text{P}}}^{\text{C2}}$

die Epipolarbedingung.As the vector

perpendicular to the vector ${\overline{\text{P}}}^{\text{C2}}$

stands, is the inner product ${\left({\overline{\text{P}}}^{\text{C2}}\right)}^T+{\left[{}^{\text{C2}}{\text{t}}_{\text{C1}}^{\text{C2}}\right]}_{\text{x}}+{\overline{\text{P}}}^{\text{C2}}$

Zero. This means that the term

also equal to zero. Since z ^c1 > 0, the two normalized pixels meet ${\overline{\text{P}}}^{\text{C1}}$

and ${\overline{\text{P}}}^{\text{C2}}$

the epipolar condition.

Example algorithm based on epipolar geometry

als ein Polynom ersten Grades $${\overline{\phi }}_{\text{V}}^{\text{C}}={\overline{c}}_{\mathrm{1,}v}+{\overline{y}}^C+{\overline{c}}_{\mathrm{0,}v}\text{.}$$

beschrieben.The method presented here estimates the parameters of a windshield distortion model based on the epipolar condition together with optical flow information from successive images. The distortion is in the vertical direction ${\overline{\phi }}_{\text{V}}^{\text{C}}$

as a polynomial of the first degree $${\overline{\phi }}_{\text{V}}^{\text{C}}={\overline{c}}_{\mathrm{1,}v}+{\overline{y}}^C+{\overline{c}}_{0v}\text{,}$$

described.

In the horizontal direction this results $${\overline{\phi }}_{\text{H}}^{\text{C}}={\overline{c}}_{\mathrm{1,}H}+{\overline{x}}^{\text{C}}+{\overline{c}}_{0H}\text{.}$$

wird aus dem gemessenen verzerrten Punkt als $${\overline{\text{P}}}^{\text{C}}=\left[\begin{array}{c}\left(1-{\overline{c}}_{\mathrm{1,}h}\right)+{\overline{x}}^{\text{C}}-{\overline{c}}_{\mathrm{0,}h}\\ \left(1-{\overline{c}}_{\mathrm{1,}v}\right)+{\overline{y}}^{\text{C}}-{\overline{c}}_{\mathrm{0,}v}\end{array}\right]$$

berechnet. Formel 10 in Formel 3 eingesetzt ergibt eine Bedingung, die den funktionellen Zusammenhang von Sensorbewegung, optischem Fluss und den Windschutzscheibenverzerrungsparameter beschreibt. $$\begin{array}{l}0={\left[\begin{array}{c}\left(1-{\overline{c}}_{\mathrm{1,}h}\right)+{\overline{x}}^{\text{C2}}-{\overline{c}}_{\mathrm{0,}h}\\ \left(1-{\overline{c}}_{\mathrm{1,}v}\right)+{\overline{y}}^{\text{C2}}-{\overline{c}}_{\mathrm{0,}v}\end{array}\right]}^T+{\left[{}^{\text{C2}}{\text{t}}_{\text{C1}}^{\text{C2}}\right]}_{\text{x}}{+}^{\text{C2}}{\text{R}}_{\text{C1}}++++\\ \begin{array}{ccc}& +++& \left[\begin{array}{c}\left(1-{\overline{c}}_{\mathrm{1,}h}\right)+{\overline{x}}^{\text{C1}}-{\overline{c}}_{\mathrm{0,}h}\\ \left(1-{\overline{c}}_{\mathrm{1,}v}\right)+{\overline{y}}^{\text{C}1}-{\overline{c}}_{\mathrm{0,}v}\end{array}\right]\text{.}\end{array}\end{array}$$

Where c̅ _{1, v} , c̅ _{0, v} , c̅ _{1, h} , c̅ _{0, h} are the polynomial coefficients given in normalized image coordinates. The undistorted normalized pixel ${\overline{\text{P}}}^{\text{C}}$

is calculated from the measured distorted point as $${\overline{\text{P}}}^{\text{C}}=\left[\begin{array}{c}\left(1-{\overline{c}}_{\mathrm{1,}H}\right)+{\overline{x}}^{\text{C}}-{\overline{c}}_{0H}\\ \left(1-{\overline{c}}_{\mathrm{1,}v}\right)+{\overline{y}}^{\text{C}}-{\overline{c}}_{0v}\end{array}\right]$$

calculated. Formula 10, used in Formula 3, yields a condition that describes the functional relationship between sensor motion, optical flow, and the windshield distortion parameter. $$\begin{array}{l}0={\left[\begin{array}{c}\left(1-{\overline{c}}_{\mathrm{1,}H}\right)+{\overline{x}}^{\text{C2}}-{\overline{c}}_{0H}\\ \left(1-{\overline{c}}_{\mathrm{1,}v}\right)+{\overline{y}}^{\text{C2}}-{\overline{c}}_{0v}\end{array}\right]}^T+{\left[{}^{\text{C2}}{\text{t}}_{\text{C1}}^{\text{C2}}\right]}_{\text{x}}{+}^{\text{C2}}{\text{R}}_{\text{C1}}++++\\ \begin{array}{ccc}& +++& \left[\begin{array}{c}\left(1-{\overline{c}}_{\mathrm{1,}H}\right)+{\overline{x}}^{\text{C1}}-{\overline{c}}_{0H}\\ \left(1-{\overline{c}}_{\mathrm{1,}v}\right)+{\overline{y}}^{\text{C}1}-{\overline{c}}_{0v}\end{array}\right]\text{,}\end{array}\end{array}$$

werden, die unbekannten Polynomkoeffizienten mit nichtlinearen Optimierungsverfahren, wie beispielsweise Gauß-Newton oder Levenberg-Marquardth geschätzt. Die Abweichung des Terms der rechten Seite in Gleichung 11 beschreibt die Abweichung zu der in Gleichung 11 formulierten Bedingung. Die quadrierte Summe dieses Fehlers über alle Korrespondenzen wird iterativ minimiert. Jede weitere Abbildung des optischen Sensors ergibt eine neue Schätzung, aus der eine endgültige Schätzung abgeleitet werden kann, beispielsweise durch Filterung oder eine Majoritätsabschätzung (z.B. Histogramm) über die Zeit.For a given sensor movement $\left({}^{\text{C2}}{\text{R}}_{\text{C1}}{\text{.}}^{\text{C2}}{\text{t}}_{\text{C1}}^{\text{C2}}\right)$

The unknown polynomial coefficients are estimated using nonlinear optimization methods such as Gauss-Newton or Levenberg-Marquardth. The deviation of the term of the right-hand side in equation 11 describes the deviation from the condition formulated in equation 11. The squared sum of this error across all correspondences is minimized iteratively. Any further mapping of the optical sensor yields a new estimate from which a final estimate can be derived, for example, by filtering or a majority estimate (eg, histogram) over time.

The sensor movement can be retrieved from an inertial sensor or from existing ESP sensor data, such as the longitudinal speed and the yaw rate. Via a vehicle movement model (eg Ackerman model), the sensor movement can be derived from this information.

$${}^{\text{C2}}{\text{R}}_{\text{C1}}=\left[\begin{array}{ccc}0& {-}^{\text{C2}}{\psi }_{\text{C1}}& {}^{\text{C2}}{\theta }_{\text{C1}}\\ {}^{\text{C2}}{\psi }_{\text{C1}}& 0& {-}^{\text{C2}}{\varphi }_{\text{C1}}\\ {-}^{\text{C2}}{\theta }_{\text{C1}}& {}^{\text{C2}}{\varphi }_{\text{C1}}& 0\end{array}\right]$$

gewählt, wobei ${\overline{t}}_x^{\text{C}}\text{, }{\overline{t}}_y^{\text{C}}{\text{, }}^{\text{C2}}{\varphi }_{\text{C1}}{\text{, }}^{\text{C2}}{\theta }_{\text{C1}}{\text{, }}^{\text{C2}}{\psi }_{\text{C1}}$

die Sensorbewegungsparameter sind.If the sensor movement is not known, a minimal parameterization of the translation and rotation as $${}^{\text{C2}}{\text{t}}_{\text{C1}}^{\text{C2}}=\left[\begin{array}{c}{\overline{t}}_x^{\text{C}}\\ {\overline{t}}_y^{\text{C}}\\ 1\end{array}\right]$$

$${}^{\text{C2}}{\text{R}}_{\text{C1}}=\left[\begin{array}{ccc}0& {-}^{\text{C2}}{\psi }_{\text{C1}}& {}^{\text{C2}}{\theta }_{\text{C1}}\\ {}^{\text{C2}}{\psi }_{\text{C1}}& 0& {-}^{\text{C2}}{\phi }_{\text{C1}}\\ {-}^{\text{C2}}{\theta }_{\text{C1}}& {}^{\text{C2}}{\phi }_{\text{C1}}& 0\end{array}\right]$$

chosen, where ${\overline{t}}_x^{\text{C}}\text{.}{\overline{t}}_y^{\text{C}}{\text{.}}^{\text{C2}}{\phi }_{\text{C1}}{\text{.}}^{\text{C2}}{\theta }_{\text{C1}}{\text{.}}^{\text{C2}}{\psi }_{\text{C1}}$

are the sensor motion parameters.

In this way, the sensor movement can be estimated along with the distortion parameters. A prior knowledge of the variables contained in the functional is therefore not necessary.

The parameterization of the vertical windshield distortion in two parameters describes the measured distortion 400 good enough. Nevertheless, it is possible to use a polynomial of higher degree as a model. As another possibility, the relatively small horizontal distortion can be neglected to reduce the dimensionality of the problem.

5 shows a flowchart of a method 500 for determining the distortion characteristics of an optical medium using an optical sensor sensing optical sensor according to an embodiment. The procedure 500 For example, in a control device, as in 1 be executed. The procedure 500 has a step 502 of determining. In step 502 in determining, at least one parameter of a distortion model for compensating optical distortion of the optical medium is determined using an image sequence of the optical sensor and a trajectory of the sensor. The trajectory can consist of at least two individual positions of the sensor at the respective sampling instants. The respective transformation between the sampling instants may consist of at least one rotational component and, alternatively or additionally, at least one translational component. The online correction presented here has direct effects on the behavior of driver assistance functions. If the distortion characteristics of the optical medium change, the distortion model can be tracked during operation to ensure correct behavior of the driving assistance functions.

If an exemplary embodiment comprises a "and / or" link between a first feature and a second feature, then this is to be read so that the embodiment according to one embodiment, both the first feature and the second feature and according to another embodiment either only first feature or only the second feature.

Claims (10)

A method (500) for determining a distortion characteristic of an optical medium (202) using an optical sensor (102) sensing through the optical medium during a driving operation of a vehicle (106), wherein in a step (502) of determining at least one parameter (112 ) of a distortion model (108) for compensating optical distortion (300) of the optical medium (202) using an image sequence (114) of the optical sensor (102) and a trajectory of the optical sensor (102) to determine the distortion characteristic of the optical During a travel operation, the trajectory represents a translation and / or rotation of the optical sensor (102) between a first detection time of a first image of the image sequence (114) and a second detection time of a second image of the image sequence (114) , Method (500) according to Claim 1 in which the trajectory from the image sequence (114) is additionally determined in the step (502) of determining. Method (500) according to Claim 1 in which, in step (502) of determining, the trajectory is derived from a motion signal of a sensor of the vehicle (106). Method (500) according to one of the preceding claims, wherein in the step (502) of determining at least one further parameter of the distortion model (108) is determined using the image sequence (114) and the trajectory. Method (500) according to one of the preceding claims, in which, in the step (502) of determining, an imaging vector representing a translation of an image coordinate of an object imaged in the image sequence (114) between the first image and the second image is used to determine the parameter ( 112). Method (500) according to Claim 5 in which, in step (502) of determining, the parameter (112) is further determined using a signal (116) of environment detecting means (118) of the vehicle (106), wherein the signal (116) is at least uniaxial and undistorted translation of the imaged one Object between the first detection time and the second detection time represents. Method (500) according to one of the preceding claims, wherein in step (502) of determining a parameter (112) influencing a vertical distortion correction of the distortion model is determined. A controller (100) arranged to execute steps of the method (500) according to any one of the preceding claims in respective units. Computer program adapted to carry out the method according to one of the preceding claims. Machine-readable storage medium on which the computer program is based Claim 9 is stored.

Images