DE102014219423B4 - Dynamic model to compensate for windshield distortion - Google Patents

Abstract

Method for stereo rectifying images from a first camera (1) and a second camera (2) of a stereo camera (32) of a vehicle (30) to compensate for the influence of a window, the method comprising the following steps: providing a first calculation rule for rectifying images of the first Camera of the stereo camera of the vehicle (S1);Providing a second calculation rule for rectifying images of the second camera of the stereo camera of the vehicle (S2);Recording a first image of an environment of the vehicle by the first camera (S3);Recording a second image of an environment of the vehicle by the second camera (S4);Adjusting at least one of the first and second calculation rules during driving operation of the vehicle to compensate for image distortions of the first and/or the second image (S5);Rectifying the first image by applying the first calculation rule to the first image (S6); Rectifying the second image by applying the second calculation rule to the second image (S7); wherein at least one of the first and second calculation rules has a rotation component for compensating for a non-plane-parallel stereo geometry;at least one of the first and second calculation rules has a second component for modeling local distortions; andgenerating a window distortion model to compensate for window influence of the vehicle.

Description

Field of invention

The invention relates to the compensation of window influences on driver assistance cameras. In particular, the invention relates to a method and a driver assistance system for stereo rectification of images from a first camera and a second camera of a stereo camera of a vehicle.

Background of the invention

Many camera-based driver assistance systems are installed behind the windshield in a vehicle for practical reasons. The camera is protected from the weather, the field of vision is kept clear by the windshield wiper and it is oriented towards that of the driver.

In order to be able to implement a driver assistance function such as lane keeping assistance, adaptive cruise control, braking for pedestrians, etc. using camera recordings, an exact transformation of the image information is necessary so that objects detected in the image are located exactly in the world, for example in the area surrounding the vehicle can be. In this context, stereo camera systems or stereo cameras play a major role, as the stereo image information makes it particularly possible to assign image objects an exact 3D coordinate in the world. The accuracy of the assignment of 3D coordinates is significantly influenced by how precisely the transformation from image to world is known. An important part of this is the calibration of the stereo camera. This normally includes the intrinsic parameters of each camera in a vehicle, such as the orientation and position of a second camera relative to the first camera or the reference camera and the location of the reference camera in the world.

However, the significant influence of the windshield on the optical paths of the cameras is not taken into account. This influence plays a major role, especially in the case of a stereo camera system, since it is important to find features or objects from the image of one camera in the image of the other in order to calculate depth information and thus 3D coordinates of an object in the environment.

Out of US20120242806A1 A dynamic calibration system with a rectification module for rectifying raw data stereo images has become known.

US20130147948A1 relates to an image processing device by means of which a calibration can be carried out by determining a degree of coincidence between two images recorded by two different cameras.

Suggests for calibrating multiple image capture units DE102012001858A1 to carry out a calibration based on intrinsic and extrinsic parameters of the individual detection device using a low-pass filter and a correspondence analysis.

In US20050237385A1 In turn, a stereo camera system with a control device for setting an orientation or position of a stereo camera has been described.

Summary of the invention

It is an object of the present invention to provide an improved method for stereo rectification of images from a stereo camera of a driver assistance system.

This task is solved by the subject matter of the independent claims. Exemplary embodiments result from the dependent claims and the following description.

It should be noted that for all embodiments of the present invention relating to a method, that method may be carried out in the order of steps as explicitly described herein. However, this is not the only and essential order of the procedural steps. The person skilled in the art will understand from the present disclosure that other process step sequences are also part of the present invention and different combinations of step sequences are possible in the methods described, unless otherwise stated below.

According to the invention, a method for stereorectification of images from a first camera and a second camera of a stereo camera of a vehicle to compensate for the influence of a window is specified. A step of the method includes providing a first calculation rule for rectifying images from the first camera of the vehicle's stereo camera and providing a second calculation rule for rectifying images from the second camera of the vehicle's stereo camera. In a further step, the method includes recording a first image of an environment of the vehicle by the first camera and recording a second image of an environment of the vehicle by the second camera. It should be noted that the first and second images ideally appear at the same time should be generated, but a more or less slight time offset is also possible. In a further step, the method further comprises adapting at least one of the first and second calculation rules during driving operation of the vehicle to compensate for image distortions of the first and/or the second image. A further step of the method comprises rectifying the first image by applying the first calculation rule to the first image and rectifying the second image by applying the second calculation rule to the second image. In particular, image distortions in the first image are compensated for by applying the first calculation rule. Analogously, image distortions in the second image are compensated for by applying the second calculation rule.

In addition, according to the invention, one of the first and second calculation rules has a rotation component for compensating for a non-plane-parallel stereo geometry, with at least one of the first and second calculation rules having a second component for modeling local distortions. According to the method according to the invention, in a further step, a window distortion model is finally generated to compensate for the influence of the vehicle's window.

It can be provided that only one calculation rule is used to compensate for image distortions in the first and second images. However, in one embodiment, two different calculation rules are provided and two different models are used for the first and second cameras. These can be adjusted while the vehicle is moving. In the context of the present invention, we therefore speak of a dynamic adaptation. Details and further explanations in this regard will be further explained in the context of exemplary embodiments.

Through such a method, distortion influences caused by the windshield of the vehicle can be measured during normal vehicle operation, so that the windshield influence can be compensated for during driving operation of the vehicle using multiple measurements and by means of a provided windshield distortion model of the first and second cameras. According to specific exemplary embodiments, this can be achieved on the one hand through clever modeling of the problem and on the other hand through an efficient solution process. Details on this and exemplary embodiments will be explained in detail.

The compensation for the influence of the pane can be done by repeatedly changing or adapting the calculation rules of the first and second cameras. This repeated adjustment occurs while the vehicle is moving. This means that while driving, new objects or objects at different distances are always detected by the camera in the field of view of the respective camera, the coordinates of which can be calculated with respect to the cameras in such a way that image distortions of the first and second images caused by the influence of the windshield of the vehicle can be compensated for. The calculation rules can be adjusted online, i.e. while driving. In other words, there is a dynamic rectification process.

It is possible, for example, that the calculation rules can be constantly recalculated by accessing values stored in a database. Furthermore, the calculation rules can be adjusted during driving using newly obtained measurement data or newly calculated values, which in turn can be stored in a database. It is therefore possible to provide an online learning method for compensating for the influence of a vehicle window. The values for adapting the calculation rules can be constantly redetermined while the vehicle is traveling, for example using the image information coming from the first camera and the second camera. Parameters of the calculation rules can also be adjusted, with the parameters being subject to optimization, for example.

In particular, the driver assistance system of the present invention can perform the method described herein.

The expert understands rectification to be a process in which the so-called epipolar lines are transformed onto the pixel matrix of the image sensor of a camera. The epipolar lines are, for example, visual rays from an object into the respective cameras of the stereo camera. A first line of sight is generated from an object to the first camera and, in the case of a stereo camera, a second line of sight is generated from the same object to the second camera. Proper rectification can ensure that the object, also referred to as a feature, is imaged in the same position with respect to the first image as it is in the second image. In other words, it can be ensured that an object or feature that lies in a pixel line u in the first image lies on the same pixel line u in the second image must. A rectification error or image distortion refers to a deviation in the pixel line between two corresponding objects or two corresponding features. The optical distortion introduced into the optical beam path of a stereo camera through a vehicle windshield has a significant effect on rectification. Stereo rectification describes the rectification of the images from a stereo camera, i.e. an arrangement of two or more cameras. Due to the use of at least one calculation rule, which is compensated for the error generated by the windshield of the vehicle with regard to the rectification, a more precise and improved stereo rectification of image data from the vehicle stereo camera is made possible. In other words, this method for stereo rectification of stereo camera images in a driver assistance system teaches to use a calculation rule which is compensated for a calculated error of a location of an object in a stereo camera image caused by a windshield of the vehicle. By calculating an error in the location of an object in a stereo camera image caused by the windshield of the vehicle, image distortion can be compensated for by applying the calculation rule.

A calculation rule can be specified to rectify the images from the first camera and the second camera. This calculation rule can also be referred to as a rectification function or transformation function. The calculation rule can be a mathematical function that has several components or terms. These components include values or numerical values and/or parameters that can constantly change while the vehicle is being driven. In other words, the rectification function can be adapted to the situation, i.e. the orientation and position of the cameras in the vehicle, using parameters and/or numerical values. The rectification function models the transformation between the monocular rectified camera image and the stereo rectified camera image. In a monocularly rectified image, the distortions caused, for example, by the influence of the lens are compensated for. Accordingly, a monocularly rectified camera is also referred to as a de-scanned camera. With stereo-rectified cameras, distortions, for example due to the different arrangement of the cameras behind the windshield, are compensated for.

The first camera and the second camera of the stereo camera can be mounted in different positions in or on the vehicle. The stereo camera can be part of a driver assistance system. Such a driver assistance system is, for example, a lane keeping assistant, an object recognition assistant, a parking aid, a lane change assistant, etc. The first camera can also be referred to as a left camera and the second camera can also be referred to as a right camera. Conversely, the first camera can also be referred to as the right camera and the second camera can also be referred to as the left camera. Even if the method according to the invention is described primarily using two cameras, it is possible that a large number of cameras can be provided to which the same method steps apply, without departing from the scope of protection of the invention.

It can be provided that the first calculation rule for rectifying the images from the first camera differs from the second calculation rule for rectifying the images from the second camera. For example, the variable numerical values of the two calculation rules can differ from one another. If the first camera records a first image of the vehicle's surroundings, the recorded first image can be rectified using the first calculation rule. Analogously, the second image recorded by the second camera can be rectified using the second calculation rule. The distortion influences on the first image and the second image can thus be at least partially compensated for. The two cameras are designed to capture a sequence of images and thus image information that changes over a certain period of time. The calculation rules are constantly adjusted while the vehicle is driving, so that an adapted rectification of the images captured by the cameras is carried out continuously. The values or numerical values that are required for the calculation rules for carrying out the rectification of the images are therefore always recalculated or with a predefined frequency, so that the calculation rules also change.

For example, if an object is captured by the first and second cameras, the object is imaged in the first and second images, respectively. However, due to the influence of the vehicle's windshield, the object is imaged in a different position in relation to the respective image than it would be imaged if there were no window between the object and the cameras. This deviation in the position of the depicted object in the respective images creates a horizontal offset and/or a vertical one Offset of the object on the respective images is created. The vertical offset is called the rectification error and the horizontal offset is called the disparity error. This fact is explained in more detail in the description of the figures.

According to one embodiment of the invention, the adaptation of the first and/or the second calculation rule is carried out with a fixed frequency.

According to one embodiment of the invention, a rectification error and/or a disparity error is compensated for by adapting at least one of the first and second calculation rules.

Thus, the vertical offset and the horizontal offset of an imaged object in the first image and the second image can be reduced or compensated. Since the rectification error and the disparity error depend on the distance between the respective camera and the object located in the vehicle's surroundings, improved rectification can be achieved by constantly recalculating or adapting the associated calculation rules while driving. For example, the numerical values required by the calculation rule change depending on the distance of an object captured by the cameras. The values or numerical values that are provided for the calculation rules can, for example, be determined using a separate optimization process or stored in so-called lookup tables. A storage unit can be provided in the vehicle for this purpose. In addition to the distance, the adjustment of the calculation rules can also depend on the orientation and/or position of the cameras within the vehicle. The alignment and position of the cameras in relation to the window of the vehicle can also have an influence on the beam path, so that the rectification errors and/or disparity errors of the first image differ from those of the second image. The first calculation rule for rectifying the first image and the second calculation rule for rectifying the second image can therefore also differ. A calculation rule for the stereo rectification can also be provided so that the captured images are rectified taking into account the orientation and arrangement of the two cameras of the stereo camera.

According to one embodiment of the invention, the method further comprises determining at least one value for adapting at least one of the first and second calculation rules, the determination of the at least one value taking place during driving operation of the vehicle. To determine the at least one value, a rectification error is measured in one step based on the image from the first camera and the image from the second camera. Furthermore, a predetermined optimization function is evaluated based on the measured rectification error. The determination of the at least one value includes, as a further step, using a result of evaluating the optimization function to adapt at least one of the first and second calculation rules.

The optimization function is also called an evaluation function and can have different components that can be used to calculate an optimum. Various embodiments will be discussed in more detail herein. For example, the optimization function is preferably minimized as part of the optimization. However, it can also be maximized. The different components of the optimization function can, for example, be added together so that the sum of the components of the optimization function is then minimized. A component of the optimization function depends, for example, on the rectification error of the first and second cameras. To determine this rectification error, the first and second images are required. The rectification error is therefore measured and then provided directly or after a conversion to evaluate the optimization function. By minimizing the optimization function, a value can be determined that is used to adjust the calculation rule. Different values can be determined for the first and second calculation rules, through which the calculation rules are adapted and/or changed. However, it is also possible to adapt the first and second calculation rules with the same value. The value is, for example, a numerical value. The determination of the value for adjusting the calculation rules can take place repeatedly, so that a new value for adjusting the calculation rules can constantly be provided while driving. This means that the calculation rules can also be constantly adjusted and the rectification can also be carried out while driving based on the calculation rules.

The determination of the at least one value for adapting the first and second calculation rules can take place in a computing unit that is itself arranged in the vehicle. The computing unit is connected, for example, to the stereo camera and/or a storage unit so that data transmission can take place. The Computing unit can therefore also be connected to the driver assistance system. The computing unit is, for example, a processor on which the evaluation or minimization of the specified optimization function takes place. Furthermore, the calculation rules can be adjusted in the computing unit or in the processor. The measurement of the rectification error can be carried out using the image data recorded by the stereo camera in the stereo camera itself or also by the computing unit.

According to an embodiment of the invention, the measurement of the rectification error is carried out by determining a difference of a vertical component of an object in the first image and a vertical component of the same object in the second image.

The images recorded by the first and second cameras can be divided in their image dimensions into a horizontal component, which is also referred to as the x component, and a vertical component, which is also referred to as the y component. An object depicted in the first image therefore has a first y-coordinate. If the same object is mapped to the second image, this object has a second y-coordinate in the second image, which may be different from the first y-coordinate. This difference is called the difference in the vertical component of the object in the two images. The difference in the vertical component is, for example, a consequence of the distortion influence of the vehicle window through which the light rays from the object pass to the cameras. In other words, the difference in the vertical component indicates the rectification error that is needed to evaluate, i.e. minimize, the given optimization function. Minimizing the optimization function therefore also minimizes the rectification error. However, the optimization function includes further components that are also minimized, which means that the quality of the rectification can be increased using the calculation rules.

According to one embodiment of the invention, in addition to determining the difference in the vertical component, the measurement of the rectification error is also carried out in another second way, which can be used in addition to or instead of measuring the difference in the vertical component. This option for measuring the rectification error takes place in various steps. In one step, a translation vector is determined from the first camera to the second camera. In a further step, a first visual ray is determined from an object in the area surrounding the vehicle to the first camera and a second visual ray is determined from the same object to the second camera. Using the translation vector and the first visual ray, a first plane is determined, which is also referred to as the epipolar plane. Furthermore, an angle between the first plane or the epipolar plane and the second visual ray is determined.

From this, a value or a numerical value can be calculated, which is, for example, the cosine of the specific angle and is then provided to the optimization function as an input variable. The cosine of the specific angle therefore represents the rectification error, which can be minimized during optimization.

The rotation component of the calculation rule depends on parameters that describe a spatial orientation of the first camera to a spatial orientation of the second camera of the stereo camera in the vehicle. The second component of the calculation rule is determined by modeling distortions. This can be a modeling of local distortions.

In other words, the rotation component serves to compensate for a non-plane-parallel stereo geometry that is caused by the alignment of the first and second cameras to one another.

The modeling of local distortions can be done individually for both cameras and also for both image dimensions. The image dimensions are represented by the x component and the y component in the respective image. Examples of local distortion models include a lookup table (LUT) or a linear combination of basis functions such as radial basis functions, polynomials, splines (2D/3D) or wavelets. The modeling can be carried out by the computing unit.

The rotation component and the second component of the calculation rule can be combined with one another by concatenation and/or by addition.

According to one embodiment of the invention, the distortions are modeled in an x-direction and/or in a y-direction of the first image and the second image.

An object that is captured by the first and second cameras and is therefore imaged in both images can have a different y-coordinate in the first image than in the second image due to the distortion influence of the vehicle window. This is particularly the case if the Light rays on the pane are refracted to different degrees, for example by an asymmetrical arrangement of the cameras behind the pane. The x coordinate and the y coordinate are measured, for example, from an image edge of the respective image or image sensor. The y-coordinate describing the position of the object with respect to the first image may be different from the y-coordinate describing the position of the object with respect to the second image. The same applies to the difference in the x coordinate from the expected position on the respective image. This deviation or distortion can be modeled, for example, by the computing unit. Through this modeling, the second component of the first and second calculation rules can be provided, whereby the deviation can be removed or minimized or eliminated. This process is also called rectification, which is why the calculation rules are also called the rectification function. It is also possible for the first and second images to be rectified using only one calculation rule.

When modeling local distortions, a distortion vector is modeled by a linear combination of basis functions, such as radial basis functions.

However, image distortion can also have other causes. The distortion is determined, for example, during camera production, so that lens distortions caused by the camera lens can be compensated for.

According to an embodiment of the invention, the optimization function depends on a first component that describes the rectification error between the first camera and the second camera of the stereo camera in the vehicle.

Thus, in order to determine the first component of the optimization function, the rectification error is determined by determining a difference of a vertical component of an object in the first image and a vertical component of the same object in the second image. Additionally or alternatively, the first component of the optimization function can be carried out by determining the cosine of the angle between the first plane or the epipolar plane and the second visual ray.

According to an embodiment of the invention, the optimization function further depends on a second component that describes a deviation of distortion components from a locality requirement. In addition or as an alternative to the first and second components of the calculation rule, the calculation rule can depend on a third component that describes a deviation from a static model.

The distortion is determined by minimizing the optimization function, which is also referred to as the cost function or evaluation function, using an optimization method. The cost function is composed, among other things, of weighted basic functions or parameters. An optimizer, for example the computing unit, changes the weights of the individual basic functions in such a way that the costs are minimal and the distortion is optimally compensated for. Locality requirement means, for example, that the weights are as free of a mean as possible, that is, that the mean is as small as possible and this becomes part of the cost function to be optimized. This can prevent the iterative optimization process from drifting away or diverging.

A deviation from, for example, an already predetermined static model is determined as a third component of the calculation rule. The static model describes, for example, the rectification of images captured by the cameras. In addition to the static model, the dynamic model according to the invention can be used to optimize the rectification while driving.

According to the invention, a driver assistance system is specified for stereo rectification of images from a first camera and a second camera of a stereo camera of a vehicle to compensate for the influence of a window. The driver assistance system has a computing unit, a stereo camera with a first camera and a second camera and a storage unit. A first calculation rule and a second calculation rule are stored on the storage unit. The storage unit is designed to provide the first calculation rule for rectifying images from the first camera of the vehicle's stereo camera and to provide the second calculation rule for rectifying images from the second camera of the vehicle's stereo camera. The first camera is designed to capture a first image of an environment of the vehicle and the second camera is designed to capture a second image of an environment of the vehicle. The computing unit is designed to adapt at least one of the first and second calculation rules while the vehicle is driving. Furthermore, the computing unit is designed to rectify the first image by applying the first calculation rule to the first image and the second Rectify the image by applying the second calculation rule to the second image, wherein at least one of the first and second calculation rules has a rotation component for compensating for a non-plane-parallel stereo geometry, and wherein at least one of the first and second calculation rules has a second component for modeling local distortions. In addition, a windshield distortion model is created to compensate for the influence of the vehicle's windshield.

In addition, the driver assistance system can also include a plurality of cameras, so that each image from the plurality of cameras can be rectified by applying an associated calculation rule. It is also possible to provide a large number of stereo cameras in the vehicle, each of which provides image information to various driver assistance functions, such as lane departure warning, attention assistant, etc. It is possible for the images from each of these stereo cameras to be rectified by adjusting the associated calculation rules while driving. The images from the large number of cameras can also be rectified using one and the same calculation rule.

According to an embodiment of the invention, the method further comprises determining at least one new value for readjusting at least one of the first and second calculation rules. The at least one new value is determined while the vehicle is driving. Determining the at least one new value involves several process steps. In one step of the method, a new rectification error is measured based on a new image from the first camera and a new image from the second camera. In a further step, the optimization function is evaluated based on the measured new rectification error. In a further step of the method, a new result of evaluating the optimization function is used to readjust at least one of the first and second calculation rules.

The individual method steps for determining at least one further value can be repeated as often as desired, so that the at least one calculation rule can be readjusted at certain time intervals or continuously. Since the distortions or rectification errors in the images captured by the cameras vary while driving, such dynamic adjustment can improve the rectification for each newly captured image. An online learning method can therefore be provided which can compensate for the image distortions caused by the disk by continuously adapting the calculation rules.

Short description of the characters

• 1 shows beam paths through the windshield of a vehicle at different distances between the object and the stereo camera according to an embodiment of the invention.
• 2 shows a flowchart for a method for stereorectification of images from a first camera and a second camera of a stereo camera of a vehicle according to an embodiment of the invention.
• 3 shows a vehicle with a driver assistance system for stereo rectification of images from a first camera and a second camera of a stereo camera according to an embodiment of the invention.

Detailed description of exemplary embodiments

It should be noted that the representations in the figures are schematic and not to scale.

1 shows the influence of the windshield 10 on the optical paths of the first camera 1 and the second camera 2 of the stereo camera. In the case of a stereo camera system, this influence plays a major role, since it is important here to find features or objects from the image of the first camera 1 in the image of the second camera 2 in order to calculate depth information (3D information) from them. Based on the first object 5 is in 1 the difference between the beam paths with and without the influence of the windshield is shown. When the windshield 10 is present, the rays strike different areas of the first camera sensor 3 of the first camera 1 and the second camera sensor 4 of the second camera 2 in the beam path than without the influence of the window. Since the disk influence is not taken into account in the camera calibration, incorrect conclusions are drawn when transforming image coordinates into world coordinates. World coordinates describe the position of a detected object in the area surrounding the vehicle. As in 1 shown, the light rays without the influence of the disk 7 (dashed lines) from the first object 5 strike the camera sensors 3, 4 at different positions than the light rays with the influence of the disk 8 (solid lines). In the example shown, the stereo disparity, i.e. the horizontal offset, would be assumed to be too small and thus the first object 5 is detected as too far away. In addition, the windshield is usually not curved in just one direction. Therefore, the image is influenced not only in the horizontal direction, but also in the vertical direction.

It can be assumed that the curvature of the disk is not constant and that a camera is not exactly centered in the axis of symmetry of the disk. Therefore, the interference in the first camera 1 and the second camera 2 can vary in intensity. This becomes noticeable, for example, when the images need to be rectified to better estimate the stereo disparity. This means that the images are transformed in such a way that a line-by-line match of image features or objects imaged on the camera sensor is achieved. Line-by-line agreement means that the respective image feature is located on the same pixel lines of the first camera sensor 3 and the second camera sensor 4. Depending on the influence of the pane and the asymmetry of the pane and camera mounting or orientation, a certain error occurs in the rectification.

1 also shows the dependence of the horizontal offset or the vertical offset of the points of impact of the light rays on the camera sensors 3, 4 on the object position.

Depending on the position of the object, different beam paths and thus also different points of passage through the windshield 10 result. The light rays emanating from the second object 6 without the influence of the window 17 are marked by dashed lines and the light rays emanating from the second object 6 with the influence of the window 18 are marked by solid lines marked. Due to the locally changing pane influence, object position-dependent errors arise for the stereo camera. Comparing the beam paths of the first object 5 and the second object 6 shows a distance-dependent horizontal offset, i.e. disparity error. When asymmetries are taken into account, there is a position-dependent error in the vertical alignment of the image contents to one another in a direction orthogonal to the horizontal direction. This leads to a rectification error.

The illustrations make it clear that the windshield influence must be taken into account. A simple measurement of the windshield is hindered by the fact that the influence does not only come from the windshield itself, but the positioning and alignment of the camera to the windshield also plays a decisive role. Measuring every single camera-pane combination is not possible due to the effort involved. In addition, no complex recalibration or adjustment should be necessary when replacing the lens or camera. This can be compensated for by the dynamic model or method described here for compensating for distortions caused by the windshield 10.

Through the online learning method according to the invention, which describes a dynamic compensation model, the disk influence can compensate for the rectification error and/or the disparity error during driving. The beam path can be determined using so-called feature correspondences, for example in 1 shown, at least partially reconstructed. In this way, the passage points through the pane 10 can be determined down to a constant factor that describes the distance from camera to pane. An object or feature that is found or recognized in a camera image is also referred to here as a feature. If this object or feature is also found in the recordings of at least one other camera and it is clear that it is the same object or feature, this is referred to as feature correspondence.

For each point of passage through the pane 10, there is a specific distortion in the x and y directions in the image. This is the disk influence on the light beam. For example, the online learning method can be used to measure the composite influence in the form of a rectification error or a disparity error. The x and y distortion influences can be measured online, i.e. during normal driving, using feature correspondences, so that a disk distortion model is generated for each camera based on several measurements and thus compensates for the disk influence. This can be done, for example, by modeling the problem and, on the other hand, by an efficient solution process.

By introducing the windshield 10, the recorded image for the first camera 1 and the second camera 2 shifts and distorts to a certain extent equally, so that, for example, a distortion occurs in the image due to an offset in the y-direction. In addition, the insertion of the disk results in a proportion of uneven displacement and distortion. This means that the image or the image lines of the first camera 1 can be offset from the second camera 2. A subpixel-precise correspondence finder can be used to measure the disk influence in the y-direction. Correspondences are preferably distributed across the entire picture collected. Preferably, the y-offset of the correspondences is zero. Due to the disk 10, the method described above measures y offsets, i.e. rectification errors, distributed over the image. These rectification errors can each have the distortion contribution for both cameras. From this it is possible to create a compensation model for both cameras. So that the model can accurately reproduce the distortions for a specific three-dimensional scenario, 3D information can be specified by a line of sight of the second camera that corresponds to the line of sight of the first camera. Depending on the distance to the object, the field of view from the second camera penetrates different areas of the windshield, so different distortions can be expected. The compensation model therefore applies, for example, to the case that the 3D scenario is fixed and/or known, that is to say that each line of sight from the first camera corresponds to a specific line of sight from the second camera.

So that the compensation for the influence of the window can also take place while driving, a distortion model can be determined for each camera, which is subject to constant optimization. In other words, the measurement of the rectification error can be divided into two models. This means that a large number of scenarios are evaluated, so that several correspondences occur on the line of sight of the first camera with many different lines of sight of the second camera and vice versa. This procedure can be formulated, for example, as a system of equations that can be solved up to one degree of freedom. The degree of freedom, for example, describes the absolute amount of distortion. Furthermore, assumptions can be made here, for example that the absolute offset has a certain size on average, such as 0 pixels. Alternatively, additional conditions such as upper and lower limits for the absolute offset can also be introduced. Offset here means the distortion-related offset that the disk distortion model can correct. Constraints and restrictions are intended to ensure that the disk distortion model can carry out a correction with realistic values despite the underdetermined problem or system of equations.

In addition to the solution using a system of equations, modeling for stereorectification can also be done in other ways. For parametric pane modeling, i.e. modeling the image distortions caused by the pane, a rectification function can be used, which is also referred to as a calculation rule. This calculation rule can be adapted to the situation in the vehicle via parameters and/or values or numerical values. In other words, the rectification function models the transformation between the image of the monocularly rectified camera, i.e. the de-scanned camera, and the image of the stereo-rectified camera. In other words, monocular rectification refers to equalization, i.e. distortion correction related to a single camera, regardless of the camera's viewing direction. Stereo rectification refers to the transformation of the camera images as if they had been recorded with a pair of cameras that have optical axes aligned parallel to one another. The angle between the camera viewing directions is 0° when the optical axes are aligned parallel to one another. This ensures that the epipolar lines run along the pixel grid or the pixel matrix of the sensor.

zur Kompensation einer potentiell nicht planparallelen Stereogeometrie. Mit anderen Worten wird durch die Rotationskomponente die räumliche Ausrichtung der beiden Kameras zueinander beschrieben. Die zweite Komponente der Berechnungsvorschrift ${\delta }_{\theta R}\left(p_{\text{MonoRect}}^R\right)$

dient der Modellierung lokaler Verzeichnungen. Die Modellierung der Verzeichnung erfolgt individuell für beide Kameras, das heißt für die erste Kamera 1 und für die zweite Kamera 2 separat. Ferner erfolgt die Modellierung für beide Kameras in zwei Bilddimensionen, zum Beispiel in eine x-Richtung und in eine y-Richtung des jeweiligen Bildes. Modelle lokaler Verzeichnungen sind zum Beispiel eine Lookup Tabelle oder auch eine Linearkombination von Basisfunktionen ψ_k, wie beispielsweise radiale Basisfunktionen, Polynome, Splines (2D/3D) oder Wavelets. Die Rektifizierungsfunktion kann durch die folgende Gleichung beschrieben werden: $${\delta }_{{\theta }_R}\left(p\right)={\displaystyle \sum _k{w}_k{\psi }_k\left(p\right)}$$

The calculation rule has various components. The first component of the calculation rule is a rotation component $r_{\theta R}\left(p_{\text{MonoRect}}^R\right)$

to compensate for a potentially non-plane-parallel stereo geometry. In other words, the rotation component describes the spatial orientation of the two cameras to one another. The second component of the calculation rule ${\delta }_{\theta R}\left(p_{\text{MonoRect}}^R\right)$

is used to model local distortions. The distortion is modeled individually for both cameras, i.e. separately for the first camera 1 and for the second camera 2. Furthermore, the modeling for both cameras takes place in two image dimensions, for example in an x-direction and in a y-direction of the respective image. Models of local distortions are, for example, a lookup table or a linear combination of basis functions ψ _k , such as radial basis functions, polynomials, splines (2D/3D) or wavelets. The rectification function can be described by the following equation: $${\delta }_{{\theta }_R}\left(p\right)={\displaystyle \sum _k{w}_k{\psi }_k\left(p\right)}$$

$$p_{\text{rect}}^L=r_{{\theta }_L}\left({\delta }_{{\theta }_L}\left(p_{\text{MonoRect}}^L\right)\right)$$

The combination of the rotation component and the second component of the calculation rule can, for example, be chained, as shown below. $$p_{\text{rect}}^R=r_{{\theta }_R}\left({\delta }_{{\theta }_R}\left(p_{\text{MonoRect}}^R\right)\right)$$

$$p_{\text{rect}}^L=r_{{\theta }_L}\left({\delta }_{{\theta }_L}\left(p_{\text{MonoRect}}^L\right)\right)$$

$$p_{\text{rect}}^L=r_{{\theta }_L}\left(p_{\text{MonoRect}}^L\right)+{\delta }_{{\theta }_L}\left(p_{\text{MonoRect}}^L\right)$$

However, the combination of the rotation component and the second component of the calculation rule can also be done additively. $$p_{\text{rect}}^R=r_{{\theta }_R}\left(p_{\text{MonoRect}}^R\right)+{\delta }_{{\theta }_R}\left(p_{\text{MonoRect}}^R\right)$$

$$p_{\text{rect}}^L=r_{{\theta }_L}\left(p_{\text{MonoRect}}^L\right)+{\delta }_{{\theta }_L}\left(p_{\text{MonoRect}}^L\right)$$

einen monokular entzeichneten Punkt im ersten und im zweiten Bild und $p_{\text{rect}}^{R/L}$

das Ergebnis der Rektifizierung in ein rektifiziertes Stereosystem. $p_{\text{rect}}^{R/L}$

ist somit der stereo-rektifizierte Punkt. Das heißt dieser Punkt ist entzeichnet und die Achsen der Kameras sind parallel, sodass die Epipolarlinien auf dem Pixelgrid beziehungsweise der Pixelmatrix liegen und Bildmerkmale oder Objekte in denselben Bildzeilen der verschiedenen Kameras zu finden sind.A combination through concatenation and addition is also possible. It describes $p_{\text{MonoRect}}^{R/L}$

a monocularly delineated point in the first and second images and $p_{\text{rect}}^{R/L}$

the result of rectification into a rectified stereo system. $p_{\text{rect}}^{R/L}$

is therefore the stereo-rectified point. This means that this point is delineated and the axes of the cameras are parallel, so that the epipolar lines lie on the pixel grid or pixel matrix and image features or objects can be found in the same image lines of the different cameras.

ψ _k describes the k-th basis function for modeling the local distortion and w _k describes the weight of the k-th basis function. θ _R/L is a parameter vector which combines all parameters of the rectification components, i.e. the components of the calculation rule, for the first and second cameras.

The quality of a rectification is evaluated by an optimization function O(θ _R ,θ _L ). To determine optimal model parameters θ _R and θ _L . A parameter search maximizes the quality of the rectification or compensation model based on various possible search strategies. Such search strategies are, for example, local, non-linear optimization using Gauss-Newton, Levenberg-Marquardt, SteepestDescent, Newton's method, gradient-free optimization or global optimization methods.

zum Beispiel eines Objekts im ersten Bild, und einer Vertikalkomponente $p_{rect}^L,$

zum Beispiel desselben Objekts im zweiten Bild, bestimmt werden. Zusätzlich oder alternativ kann die erste Komponente der Optimierungsfunktion durch das Bestimmen des Cosinus des Winkels E(θ_R,θ_L) zwischen der ersten Ebene beziehungsweise der Epipolarebene und dem zweiten Sehstrahl erfolgen. Mit anderen Worten wird der Cosinus des Winkels zwischen Richtungsvektor des zweiten Korrespondenzfeatures, also dem zweiten Sehstrahl, und der Epipolar-Ebene, die durch den Richtungsvektor des ersten Korrespondenzfeatures und dem Translationsvektor von erster zu zweiter Kamera aufgespannt wurde, bestimmt. Der Rektifizierungsfehler wird als der Cosinus eines Winkels folgendermaßen ausgedrückt: $$E\left({\theta }_R,{\theta }_L\right)={\Vert q_{\text{rect}}^R\left[t\right]\times q_{\text{rect}}^L\Vert }_2^2\to \text{min}$$

Dabei beschreibt $q_{\text{MonoRect}}^{R/L}$

die rück-projizierte Darstellung des Bildpunkts $p_{\text{MonoRect}}^{R/L}.$

Projektion bezeichnet dabei die Abbildung eines Punktes, der auf einem Sehstrahl liegt, auf einen Punkt in einer Bildebene. Rückprojektion bezeichnet die Erzeugung eines Sehstrahls zu einem Punkt in der Bildebene.The optimization function O(θ _R ,θ _L ) can evaluate one or more target criteria of the rectification. An example of this is the rectification error between the first and second cameras. This can be done by determining a difference in a vertical component $p_{\text{rect}}^R,$

for example an object in the first image, and a vertical component $p_{rect}^L,$

for example the same object in the second image. Additionally or alternatively, the first component of the optimization function can be carried out by determining the cosine of the angle E (θ _R , θ _L ) between the first plane or the epipolar plane and the second visual ray. In other words, the cosine of the angle between the direction vector of the second correspondence feature, i.e. the second visual ray, and the epipolar plane, which was spanned by the direction vector of the first correspondence feature and the translation vector from the first to the second camera, is determined. The rectification error is expressed as the cosine of an angle as follows: $$E\left({\theta }_R,{\theta }_L\right)={\Vert q_{\text{rect}}^R\left[t\right]\times q_{\text{rect}}^L\Vert }_2^2\to \text{min}$$

It describes $q_{\text{MonoRect}}^{R/L}$

the back-projected representation of the pixel $p_{\text{MonoRect}}^{R/L}.$

Projection refers to the image of a point that lies on a visual ray onto a point in an image plane. Rear projection refers to the creation of a visual ray to a point in the image plane.

und ${\delta }_{{\theta }_L}$

von einer Lokalitätsanforderung gemäß $$L\left({\theta }_R,{\theta }_L\right)={\Vert {\displaystyle \sum _k{w}_k}\Vert }_2^2\to \text{min}$$

An example of another target criterion for rectification, which can be evaluated by the optimization function, is the deviation of the distortion components ${\delta }_{{\theta }_R}$

and ${\delta }_{{\theta }_L}$

of a locality requirement $$L\left({\theta }_R,{\theta }_L\right)={\Vert {\displaystyle \sum _k{w}_k}\Vert }_2^2\to \text{min}$$

The deviation from a previously determined static model as a further target criterion can be evaluated as follows: $$D\left({\theta }_R,{\theta }_L\right)={\Vert \text{Static}-{\displaystyle \sum _k{w}_k{\psi }_k\left(p\right)}\Vert }_2^2\to \text{min}$$

Furthermore, the exceeding of predefined limits as well as any external criteria such as the results of a squint angle calibration and/or the results of an external measurement of the lens distortion in other functions can be used as target criteria for the evaluation by the optimization function. This means that the dynamically learning distortion model can also be combined with other online calibration methods that, for example, determine a squint angle of the stereo camera, in that the pane distortion model takes such results into account in the optimization function.

An example of such an optimization function or combined evaluation function is: $$O\left({\theta }_R,{\theta }_L\right)=E\left({\theta }_R,{\theta }_L\right)+L\left({\theta }_R,{\theta }_L\right)+D\left({\theta }_R,{\theta }_L\right)\to \text{min}$$

The evaluation function can further evaluate influences of the parameters θ _R and θ _L on other properties of the camera calibration. With this approach it is possible to determine all stereo calibration parameters in a common parameter search. After the parameter search has been completed, a rectification instruction is calculated from the model instance found, for example in the form of a lookup table, so that a rectification is carried out based on this rectification instruction can take place. The initialization of the parameter search can, for example, be based on simulation results. The constant calculation of the rectification function is subject to a continuous optimization process. The disk influence can be compensated for regardless of the object distance, as a single disk distortion model can be provided for each camera, so that independence from the distance-dependent stereo disparity is achieved.

In 2 a flowchart for a method for stereo rectification of images from a first camera 1 and a second camera 2 of a stereo camera 32 of a vehicle 30 is shown. In a step S1 of the method, a first calculation rule for rectification of images from the first camera 1 of the stereo camera 32 of the vehicle 30 is provided by the storage unit 34. In a further step S2, a second calculation rule is provided for the rectification of images from the second camera 2 the stereo camera 32 of the vehicle 30 through the storage unit 34. In a further step S3, a first image of the surroundings of the vehicle 30 is recorded by the first camera 1. A further step S4 is the recording of a second image of the surroundings of the vehicle 30 by the second camera 2. In addition, in a further step S5, at least one of the first and second calculation rules is adjusted by the computing unit 33 while the vehicle 30 is driving. In a further step S6 of the method, the first image is rectified by applying the first calculation rule the first image by the computing unit 33 and in a further step S7 the second image is rectified by applying the second calculation rule to the second image by the computing unit 33.

3 shows a vehicle with a driver assistance system 31 for stereo rectification of images from a first camera 1 and a second camera 2 of a stereo camera 32 of a vehicle 30. The driver assistance system 31 can be designed, for example, as a system for object recognition, a system for road recognition, a system for traffic sign recognition, or as a high beam assistant but other versions are also possible. In principle, the driver assistance system (31) can be any system in which correct rectification is important. The first priority is road profile measurement, as this involves accuracies in the centimeter range. Functions that are intended to enable braking and steering interventions should also be mentioned, for example lane detection, object detection, traffic sign detection, and/or pedestrian detection. A high beam assistant can also be considered. The driver assistance system is part of the vehicle 30. The stereo camera 32 has a first camera 1 and a second camera 2. Likewise, the driver assistance system 31 has a computing unit 33 and a storage unit 34. A first calculation rule for stereo rectification of images from the first camera 1 and a second calculation rule for stereo rectification of images from the second camera 2 can be stored on the storage unit 34. One of the two calculation rules is compensated for an error in the location of an object in at least one image of the first camera 1 and the second camera 2 caused by a windshield 10 of the vehicle 30. The compensation takes place according to the method described here while the vehicle 30 is driving. Likewise, the computing unit 33 is designed both to rectify images from the first camera 1 using the first calculation rule and to rectify images from the second camera 2 according to the second calculation rule. In principle, the vehicle can be a car, a truck, a motorcycle or another land vehicle. The computing unit 33 and the storage unit 34 are connected to each other and to the stereo camera 32 via one or more data transmission lines 35.

A stereo camera 32 of a driver assistance system 31 is usually arranged in the vehicle 30 behind the windshield 10 and the first camera 1 and second camera 2 are arranged spatially next to one another. In 3 However, for reasons of illustration, the two cameras (1, 2) are shown arranged one above the other in order to be able to show both components in one figure.

In addition, it should be noted that “comprising” and “having” do not exclude other elements or steps and “a” or “an” does not exclude a plurality. Furthermore, it should be noted that features or steps that have been described with reference to one of the above-described exemplary embodiments can also be used in combination with other features or steps of other above-described embodiments. Reference symbols in the claims are not to be regarded as limitations.

Claims (11)

Method for stereo rectification of images from a first camera (1) and a second camera (2) of a stereo camera (32) of a vehicle (30) for compensating for a window influence, the method comprising the following steps: Providing a first calculation rule for rectifying images of the first Vehicle stereo camera camera (S1); Providing a second calculation rule for rectifying images from the second camera of the vehicle's stereo camera (S2); Recording a first image of an environment of the vehicle by the first camera (S3); Recording a second image of an area surrounding the vehicle by the second camera (S4); Adapting at least one of the first and second calculation rules while the vehicle is driving to compensate for image distortions of the first and/or the second image (S5); Rectifying the first image by applying the first calculation rule to the first image (S6); Rectifying the second image by applying the second calculation rule to the second image (S7); wherein at least one of the first and second calculation rules has a rotation component to compensate for a non-plane-parallel stereo geometry; wherein at least one of the first and second calculation rules has a second component for modeling local distortions; and generating a window distortion model to compensate for window influence of the vehicle. Procedure according to Claim 1 , wherein a rectification error and/or disparity error is compensated for by adapting at least one of the first and second calculation rules when applying the adapted calculation rules. Method according to one of the preceding claims, further comprising the steps: Determining at least one value for adapting at least one of the first and second calculation rules, wherein the at least one value is determined while the vehicle (30) is driving; wherein determining the at least one value comprises the following method steps: measuring a rectification error based on the image from the first camera (1) and the image from the second camera (2); Evaluating a given optimization function based on the measured rectification error; Using a result of evaluating the optimization function to adapt at least one of the first and second calculation rules. Procedure according to Claim 3 , wherein the optimization function depends on a first component that describes the rectification error between the first camera (1) and the second camera (2) of the stereo camera (32) in the vehicle (30). Procedure according to one of the Claims 3 or 4 , where the optimization function depends on: a second component that describes a deviation of distortion components from a locality requirement; and/or a third component that describes a deviation from a static model. Procedure according to one of the Claims 3 until 5 , wherein measuring the rectification error occurs by determining a difference of a vertical component of an object in the first image and a vertical component of the same object in the second image. Procedure according to one of the Claims 3 until 6 , wherein measuring the rectification error further comprises the steps of: determining a translation vector from the first camera (1) to the second camera (2); Determining a first line of sight from an object in the vicinity of the vehicle (30) to the first camera (1); determining a second line of sight from the same object to the second camera (2); determining a first plane from the translation vector and the first visual ray; Determining an angle between the first plane and the second line of sight. Method according to one of the preceding claims, wherein the rotation component of the calculation rule depends on parameters which describe a spatial orientation of the first camera (1) to a spatial orientation of the second camera (2) of the stereo camera in the vehicle (30); and wherein the second component of the calculation rule is determined by modeling distortions. Procedure according to Claim 8 , whereby the modeling of the distortions takes place in an x-direction and/or in a y-direction of the first image and the second image. Procedure according to one of the Claims 3 until 7 , determining at least one new value for a new adjustment of at least one of the first and second calculation rules, the determination of the at least one new value taking place during driving operation of the vehicle (30); wherein determining the at least one new value comprises the following method steps: measuring a new rectification error based on a new image from the first camera (1) and a new image from the second camera (2); Evaluating the optimization function based on the measured new rectification error; and Using a new result of evaluating the optimization function to readjust at least one of the first and second calculation rules. Driver assistance system for stereo rectification of images from a first camera (1) and a second camera (2) of a stereo camera (32) of a vehicle (30) to compensate for the influence of a window, comprising: a computing unit (33); the stereo camera (32) with a first camera (1) and a second camera (2); a storage unit (34) on which a first calculation rule and a second calculation rule are stored; wherein the storage unit (34) is designed to provide the first calculation rule for rectifying images from the first camera (1) of the stereo camera (32) of the vehicle (30); wherein the storage unit (34) is designed to provide the second calculation rule for rectifying images from the second camera (2) of the stereo camera (32) of the vehicle (30); wherein the first camera (1) is designed to record a first image of an environment of the vehicle (30); wherein the second camera (2) is designed to record a second image of an environment of the vehicle (30); wherein the computing unit (33) is designed to adapt at least one of the first and second calculation rules during driving operation of the vehicle (30) to compensate for image distortions of the first and/or the second image; wherein the computing unit (33) is designed to rectify the first image by applying the first calculation rule to the first image; and wherein the computing unit (33) is designed to rectify the second image by applying the second calculation rule to the second image; wherein at least one of the first and second calculation rules has a rotation component to compensate for a non-plane-parallel stereo geometry; and wherein at least one of the first and second calculation rules has a second component for modeling local distortions; and Generating a window distortion model to compensate for the influence of the vehicle's window.