WO2016146559A1

WO2016146559A1 - Method for determining a position of an object in a three-dimensional world coordinate system, computer program product, camera system and motor vehicle

Info

Publication number: WO2016146559A1
Application number: PCT/EP2016/055393
Authority: WO
Inventors: Duong-Van NGUYEN; Ciáran HUGHES; Jonathan Horgan
Original assignee: Connaught Electronics Ltd
Current assignee: Connaught Electronics Ltd
Priority date: 2015-03-18
Filing date: 2016-03-14
Publication date: 2016-09-22
Anticipated expiration: 2017-09-18
Also published as: DE102015104065A1

Abstract

The invention relates to a method for determining a position of an object (7) located in an environmental region (6) of a motor vehicle (1) in a three-dimensional world coordinate system (8), in which the following steps are performed: - providing a first image (9) having the object (7) and a second image (10) having the object (7) of an image sequence by means of a camera (3) of the motor vehicle (1), - determining a first characteristic pixel (I₁) of the object (7) in the first image (9) and a second characteristic pixel (I₂) of the object (7) in the second image (10), - transforming the first characteristic pixel (I₁) present in a two-dimensional image coordinate system into the three-dimensional world coordinate system (8) as a first ray (v₁) and the second characteristic pixel (I₂) present in a two-dimensional image coordinate system into the three-dimensional world coordinate system (8) as a second ray (v₂), - determining a connecting straight line (w) oriented perpendicularly to the first ray (v) and to the second ray (v₂), - determining a center (P) of the connecting straight line (w) as the position of the object (7) in the environmental region (6).

Description

Method for determining a position of an object in a three-dimensional world coordinate system, computer program product, camera system and motor vehicle

The invention relates to a method for determining a position of an object located in an environmental region of a motor vehicle in a three-dimensional world coordinate system. A first image having the object and a second image having the object of an image sequence are provided by means of a camera of the motor vehicle. Furthermore, a first

characteristic pixel of the object in the first image and a second characteristic pixel of the object in the second image are determined. The invention also relates to a computer program product, to a camera system for a motor vehicle as well as to a motor vehicle with a camera system.

Methods for determining a position of an object in a three-dimensional world coordinate system are known from the prior art. Thus, 3D camera systems such as for example TOF (time of flight) cameras can for example be used to provide the position of the object in the three-dimensional world coordinate system. Based on the position of the object in the three-dimensional world coordinate system, for example, a distance of the object to a motor vehicle, on which the camera system is disposed, can be determined in the three- dimensional world coordinate system. In the three-dimensional world coordinate system, the position of the object and/or of the motor vehicle is described in posture and height with respect to a reference surface such as for example the earth's surface.

Presently, the position of the object in the three-dimensional world coordinate system is preferably determined depending on the principle of stereoscopy. Herein, at least two images are captured from different sites. Thus, the object is presented in each of the images with at least slightly different views. Known methods for determining the position of the object in the three-dimensional world coordinate system according to this principle are usually computationally intensive and slow.

It is the object of the invention to provide a method, a computer program product, a camera system as well as a motor vehicle, by which or in which the determination of a position of an object located in an environmental region of a motor vehicle in a three- dimensional world coordinate system can be fast performed with little effort. According to the invention, this object is solved by a method, by a computer program product, by a camera system as well as by a motor vehicle having the features according to the respective independent claims.

In a method according to the invention, a position of an object located in an environmental region of the motor vehicle in a three-dimensional world coordinate system is determined. A first image having the object and a second image having the object of an image sequence are provided by means of a camera of the motor vehicle. A first position of the camera during the capture of the first image is in particular different from a second position of the camera during the capture of the second image. A first characteristic pixel of the object in the first image and a second characteristic pixel of the object in the second image are determined. As an essential ideal of the invention, the first characteristic pixel present in a two-dimensional image coordinate system is transformed into the three- dimensional world coordinate system as a first ray and the second characteristic pixel present in a two-dimensional image coordinate system is transformed into the three- dimensional world coordinate system as a second ray. A connecting straight line oriented perpendicularly to the first ray and perpendicularly to the second ray is determined.

Further, a center of the connecting straight line is determined as the position of the object in the environmental region.

By the method according to the invention, the determination of the position of the object located in the environmental region of the motor vehicle in the three-dimensional world coordinate system can be effected fast and with little computation. Thus, the position can be provided with little effort.

In particular, the position of the object is only determined based on two images of the image sequence, which are captured at different sites. The determination of the first characteristic pixel can for example be initially effected by an interest point operator.

However, the first characteristic pixel as well as the second characteristic pixel can be determined depending on a characteristic pixel determined for example at earlier time by means of a method for tracking the characteristic pixels. Thus, the first characteristic pixel and the second characteristic pixel can for example be determined by an optical flow method. The first characteristic pixel and the second characteristic pixel are in particular each present in a two-dimensional image coordinate system. In the two-dimensional image coordinate system, the characteristic pixel is for example described by two coordinates in the image plane. The transformation of the first characteristic pixel and the second characteristic pixel into the three-dimensional world coordinate system is in particular effected with the knowledge about calibration parameters of the camera and a determined position and orientation of the camera at the time of capture of the first image and/or of the second image. The transformation can then be performed with a translation vector and a rotation matrix. After transforming, the first characteristic pixel is present in the three-dimensional world coordinate system as a first ray, while the second characteristic pixel is present as a second ray. By the first ray as well as by the second ray, a positional blur is described, which is caused in that a two-dimensional coordinate is transformed into a three- dimensional coordinate system. The third dimension, which is in particular characterized by a distance from the motor vehicle or the camera to the object, cannot be determined based on a single image. In a rarely occurring ideal case, the first ray and the second ray intersect in the position of the object in the three-dimensional world coordinate system. In reality, however, the first ray and the second ray mostly do not intersect. Therefore, the connecting straight line is determined perpendicularly to the first ray and perpendicularly to the second ray. The connecting straight line is in particular the shortest connection between the first ray and the second ray, which is perpendicularly oriented to the two rays. The connecting straight line thus is particularly located where the first ray and the second ray can be connected by the shortest possible connecting straight line, which is preferably perpendicular to the two rays. The connecting straight line is in particular formed as a line segment having a start point on the first ray or the second ray and an end point on the second ray or on the first ray. The line segment is in particular formed straight and not curved. The center is determined on the connecting straight line. The intersection of the connecting straight line with the first ray is equally distant from the center as the intersection of the connecting straight line with the second ray. The position of the object in the environmental region is determined by the center. The position of the object is therefore described by the center in the three-dimensional world coordinate system.

By the position of the object in the three-dimensional world coordinate system, for example, a distance from the object to the motor vehicle or to the camera can be determined. Furthermore, a 3D reconstruction of the object can for example be

determined. Thus, the motor vehicle can for example be assisted in a parking procedure.

In particular, it is provided that the transformation from the two-dimensional image coordinate system into the three-dimensional world coordinate system is performed depending on orientation parameters of the motor vehicle determined by odometry and/or visual odometry. The odometry denotes a method of estimating position and orientation of a mobile system based on the data of its propulsion. The data of the odometry can for example be provided by a CAN bus of the motor vehicle. The visual odometry can for example be performed based on the first image and/or the second image and/or further images of cameras of the motor vehicle. It is also advantageous that the odometry and the visual odometry can be combined. Thus, it can for example be that the odometry is supplemented or improved by the visual odometry. The orientation parameters of the motor vehicle can therefore be accurately and reliably determined. Thus, the

transformation can also be accurately and reliably effected.

Furthermore, it can be provided that yaw information and/or pitch information and/or roll information of the motor vehicle are described by the orientation parameters of the motor vehicle. In particular, a rotation around a vertical axis of the motor vehicle is described by the yaw information. In particular, a rotation around the transverse axis of the motor vehicle is described by the pitch information. A rotation around the longitudinal axis of the motor vehicle is described by the roll information. By uniquely defining the orientation parameters, the transformation can be uniquely performed and thus the position of the object can be reliably determined.

In particular, it is provided that a ray pair is determined by the first ray and the second ray and the center of the connecting straight line is determined only depending on the ray pair. This means that the center and thus the position of the object are in particular determined exclusively by the first characteristic pixel and the second characteristic pixel. This further means that the center and thus the position of the object is in particular determined exclusively based on two images of the image sequence. By the ray pair, the position of the object can be determined in fast, effective and low-computation manner. Thus, it is in particular not required to transform more characteristic pixels than the first characteristic pixel and the second characteristic pixel into the three-dimensional world coordinate system. Thus, the first ray and the second ray are in particular sufficient to determine the position of the object in the three-dimensional world coordinate system.

In a further embodiment, it can be provided that the first image and the second image are provided as images of the image sequence immediately consecutive in time. Thus, the position of the object can be determined within a short period of time.

In particular, it is provided that the position of the object in the environmental region characterized by the center is checked by an error checking method. By the error checking method, it can be determined how reliable the determined position of the object is. Thus, by the error checking method, it can be determined that the center is incorrect. An incorrect center incorrectly describing the determined position of the object in the environmental region can for example be excluded from the further procedure. Thus, it can for example be that the incorrect center is not taken into account in further processing of the information. Thus, the incorrect center cannot be taken into account for example in 3D reconstruction of the object. The error checking method is in particular performed in steps, wherein the evidence of an error of the center in one of the steps can already be sufficient to classify the center as incorrect. Thus, the error checking method is further advantageous in that a position of the object checked for errors multiple times with different approaches can be provided by the center.

Preferably, it is provided that the center is retransformed into the first image and/or into the second image and/or into a third image of the image sequence, and the error checking method is performed depending on the retransformation of the center, wherein a first error value is provided by the retransformation of the center into the first image and/or into the second image and a second error value is provided by the retransformation of the center into the third image, and the position of the object characterized by the center is incorrect if the first error value is determined as a predetermined first error limit value and/or the second error value is determined as greater than a predetermined second error limit value. For example, the retransformation can be effected by inverting the transformation of the first characteristic pixel and/or of the second characteristic pixel into the three- dimensional world coordinate system. In case only the first image and the second image are present, the center is retransformed into the first image and/or the second image and the position thereof is assessed depending on the first error limit value. Thus, it can be that the center is assessed as incorrect if a distance of the center from the first characteristic pixel in the first image and/or from the second characteristic pixel in the second image is determined as larger than the first error limit value. In case more images than the first image and the second image are present, the center can be retransformed into the third image. In the third image, preferably, a third characteristic pixel of the object is determined. The third characteristic pixel can for example also be determined by means of an optical flow method and for example be a continuation of the tracking of the first characteristic pixel and the second characteristic pixel. Thus, the center can now be assessed as incorrect if a distance from the center to the third characteristic pixel is larger than the predetermined second error limit value. The determination of the incorrectness of the center depending on the third characteristic pixel or the third image is more reliable than it can be performed based on the first image and/or the second image. This is founded in that the third characteristic pixel was not used for calculating the center. Thus, it is advantageous that a reliability assessment of the center is provided based on the first error limit value and/or the second error limit value.

Furthermore, it is preferably provided that a length of the connecting straight line is determined, and the error checking method is performed depending on the length of the connecting straight line, and the position of the object characterized by the center is assessed as incorrect if the length is larger than a predetermined length limit value. Thus, the distance of the intersection of the connecting straight line with the first ray and the intersection with the second ray is determined by the length. Thus, the length is the magnitude of the line segment of the connecting straight line. By the length, it is described how far the first ray and the second ray are apart. If this is larger than the predetermined length limit value, thus, the center is assessed as incorrect. For example, the length can be larger than the predetermined length limit value if incorrect orientation parameters of the motor vehicle have been present and have been used for the transformation. Thus, it is advantageous that statement about the reliability of the center or the position of the object in the three-dimensional world coordinate system can be made based on the length of the connecting straight line.

Furthermore, it is preferably provided that an angle between the first ray and the second ray is determined, and the error checking method is performed depending on the angle, and the position of the object characterized by the center is assessed as incorrect if the angle is larger than an angle limit value. The angle limit value can for example be determined depending on a position of the camera during the capture of the first image and a position of the camera during the capture of the second image. In particular, the difference between the positions is known by the orientation parameters of the motor vehicle. By the angle, it is preferably determined how close the first ray and the second ray are to a parallel state of the first ray and the second ray. The closer the first ray and the second ray to the parallel state, the smaller the angle between the first ray and the second ray. Thus, if the angle is larger than the angle limit value, thus, the center and therefore the position of the object are assessed as incorrect. This also means that if the angle is less than or equal to the angle limit value, thus, the center is considered as reliable. Thus, it is advantageous that the reliability of the center or of the position of the object can be determined depending on the angle. Thus, the reliability of the center can be further increased.

Furthermore, it is preferably provided that a plurality of characteristic pixels of the object are determined in a plurality of images of the image sequence, and a plurality of pixels are determined each based on two of the characteristic pixels, and the error checking method is performed depending on the plurality of the centers, and the position of the object characterized by the center is assessed as incorrect if a predetermined number of the plurality of the centers is less than a predetermined distance apart from the center. Thus, the reliability can be determined based on a local distribution of the centers. Thus, if the predetermined number of the plurality of the centers is present in the range set by the predetermined distance, thus, the center can be assessed as reliable and thus not incorrect.

In a further embodiment, it is preferably provided that the error checking method is performed depending on a plurality of distances of the plurality of the centers and/or an arithmetic mean of the number of the distances and/or a standard deviation of the plurality of the distances. Thus, the reliability or the non-present incorrectness of the center can be determined depending on the other centers. However, the other centers can in particular fast be provided since they also are in particular determined only based on two images of the image sequence. By the distribution of the other centers of the plurality of the centers, the center can be assessed as reliable based on the arithmetic mean and/or the standard deviation and/or the distances of the centers to the center. Thus, the position of the object in the three-dimensional world coordinate system can also be assessed as reliable.

Furthermore, it can be provided that a last ray and/or a next to last ray provided by two images at the end of the image sequence, thus for example the next to last ray from the next to last image of the image sequence and the last ray from the last image of the image sequence, are additionally or alternatively used for assessing the reliability of the center. The last image of the image sequence means that a further image of the image sequence is not provided after the last image. Thus, it can for example be that a next to last center of the next to last ray and a last center of the last ray are assessed as incorrect by the error checking method. This assessment can for example be effected by visualization of the center in preferably a motor vehicle coordinate system. This error checking is advantageous in case of sudden deceleration of the motor vehicle. In this case, the speed of the motor vehicle suddenly declines towards 0. This can result in the odometry and/or the visual odometry being not able to be correctly performed. Therefore, the data for the transformation also is no longer correctly present. The center thus determined in this case can now be assessed as incorrect and not be taken into account for example in a following 3D reconstruction. The invention also relates to a computer program product formed for performing a method according to the invention if the computer program product is executed on a

programmable computer device.

In addition, the invention relates to a camera system with a camera and an evaluation unit, wherein the camera system is adapted to perform a method according to the invention. The evaluation unit can for example be integrated in the camera or be present as a separate unit. The camera is preferably connected to the evaluation unit.

A motor vehicle according to the invention, in particular a passenger car, includes a camera system according to the invention or an advantageous implementation thereof.

The preferred embodiments presented with respect to the method according to the invention and the advantages thereof correspondingly apply to the computer program product according to the invention, to the camera system according to the invention as well as to the motor vehicle according to the invention.

With the statements of "top", "bottom", "horizontal", "vertical", "front", "rear" etc., the positions and orientations given with intended use and intended arrangement of the camera system, the arrangement thereof on the motor vehicle and of the motor vehicle and with an observer then standing in front of the camera system or the motor vehicle and looking in the direction of the camera system or the motor vehicle are specified.

Further features of the invention are apparent from the claims, the figures and the description of figures. The features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of figures and/or shown in the figures alone are usable not only in the respectively specified combination, but also in other combinations or alone, without departing from the scope of the invention. Thus, implementations are also to be considered as encompassed and disclosed by the invention, which are not explicitly shown in the figures and explained, but arise from and can be generated by separated feature combinations from the explained implementations. Implementations and feature combinations are also to be considered as disclosed, which thus do not have all of the features of an originally formulated independent claim.

Below, the embodiments of the invention are explained in more detail based on schematic drawings. There show:

Fig. 1 in schematic plan view an embodiment of a motor vehicle according to the invention with a camera system;

Fig. 2 a schematic illustration of a center of a connecting straight line in a three- dimensional world coordinate system;

Fig. 3 a flow diagram of a method according to the invention for determining a position of an object located in an environmental region of the motor vehicle in the three-dimensional world coordinate system;

Fig. 4 a flow diagram of an error checking method to check the center;

Fig. 5 a schematic illustration of an error checking method of the center based on a first error limit value and a second error limit value;

Fig. 6 a schematic illustration of an environmental region of the motor vehicle with a position of an object in the three-dimensional world coordinate system; and

Fig. 7 a schematic illustration of a plan view image of the motor vehicle with a position of an object in the three-dimensional world coordinate system.

In the figures, identical or functionally identical elements are provided with the same reference characters.

In Fig. 1 , a plan view of a motor vehicle 1 with a camera system 2 according to an embodiment of the invention is schematically illustrated. In the embodiment, the camera system 2 includes a camera 3 and an evaluation unit 4. The camera 3 is disposed on a rear 5 of the motor vehicle 1 in the embodiment. However, the arrangement of the camera 3 is variously possible on the motor vehicle 1 , however, preferably such that an environmental region 6 of the motor vehicle 1 can be at least partially captured. According to the embodiment of Fig. 1 , in particular, the part of the environmental region located on the rear 5 of the motor vehicle 1 is captured by the camera 3. The arrangement of the evaluation unit 4 is also variously possible on the motor vehicle 1 , however preferably such that the evaluation unit 4 can be connected to the camera 3. For example, the evaluation unit 4 can be integrated in the camera 3 or be formed as a separate unit. The camera system 2 can also include multiple cameras 3 and/or multiple evaluation units 4. An object 7 is located in the environmental region 6. The object 7 is captured by the camera 3.

Fig. 2 shows a three-dimensional world coordinate system 8. The three-dimensional world coordinate system 8 provides a three-dimensional coordinate with for example an x-value, a y-value and a z-value for each point. The relation of the motor vehicle 1 or of the camera system 2 to the three-dimensional world coordinate system 8 can be determined based on orientation parameters for example provided by odometry and/or visual odometry of the motor vehicle 1 . Furthermore, Fig. 2 shows a first position d of the camera 3 and a second position 0₂ of the camera 3. In the first position d , a first image 9 is provided by the camera 3, and in the second position 0₂, a second image 10 is provided by the camera 3. In the first image 9, a first characteristic pixel is determined. In the second image 10, a second characteristic pixel l₂ is determined. The characteristic pixels , l₂ can for example be determined by an optical flow method. An initial determination of the first characteristic pixel can for example be performed by an interest operator, for example a Harris operator. The characteristic pixels , l₂ are present in the respective image 9, 10 in a two-dimensional image coordinate system. The characteristic pixels , l₂ are

transformed from the respective image 9, 10 or the respective image coordinate system into the three-dimensional world coordinate system 8. The characteristic pixels , l₂ are thus transformed from a two-dimensional coordinate system into a three-dimensional coordinate system. The transformation can for example be effected based on orientation parameters of the motor vehicle 1 , in particular yaw information and/or pitch information and/or roll information of the motor vehicle 1 . The orientation parameters of the motor vehicle 1 can for example be determined by odometry and/or by visual odometry. The orientation parameters of the motor vehicle 1 determined by the odometry can for example be picked up on the CAN bus of the motor vehicle 1 . Those orientation parameters of the motor vehicle 1 , which are determined by the visual odometry, can for example be determined based on images of cameras of the motor vehicle 1 . The images for performing the visual odometry in particular show the environmental region 6 of the motor vehicle 1 . Furthermore, for performing the visual odometry and/or for performing the transformation from the two-dimensional coordinate system into the three-dimensional coordinate system, thus from the image coordinate system into the three-dimensional world coordinate system 8, calibration of the camera system 2 or of the camera 3 is known. In particular, the calibration includes calibration parameters, which can be present in the form of an external orientation and/or an internal orientation. By the transformation, the first characteristic pixel is present as a first ray v-i in the three-dimensional world coordinate system 8. The second characteristic pixel l₂ is present as a second ray v₂ in the three-dimensional world coordinate system 8 after the transformation. The first ray

and the second ray v₂ are in particular present as half-lines and have their origin in the first position d and in the second position 0₂, respectively. By the transformation of the characteristic pixels , l₂ from the two-dimensional coordinate system of the images 9, 1 0 into the three-dimensional world coordinate system 8, one degree of freedom cannot be determined, wherefore the characteristic pixels , l₂ in the three-dimensional world coordinate system 8 are present as a ray v_{1 ;} v₂. The first ray v-i and the second ray v₂ therefore point from the respective position d , 0₂ of the camera 3 to the object 7 in the three-dimensional world coordinate system 8.

A connecting straight line w is determined between the first ray v-i and the second ray v₂. The connecting straight line w is oriented perpendicularly to the first ray v-i and

perpendicularly to the second ray v₂. The connecting straight line w intersects the first ray v-i in a first object point Pi and the second ray v₂ in a second object point P₂. The first object point Pi and the second object point P₂ coarsely describe the position of the object 7. Furthermore, a center P is determined on the connecting straight line w. The center P is in the middle between the first object point Pi and the second object point P₂. The position of the object 7 in the three-dimensional world coordinate system 8 is described by the center P. It is assumed that the position of the object 7 is located in the middle between the first object point Pi and the second object point P₂.

The determination of the center P can be mathematically described as follows:

V w = 0 (1 ) ν ^■ w = 0 (2) w = P₂ - P₁ (3)

Auxiliary parameters t and k are introduced. Later, the auxiliary parameters t and k are resolved again. (4)

P₂ = 0₂+ kv₂ (5) v_l-(O_l-O₂+tv_l-kv₂) = 0 (6) v₂-(O_l-O₂+tv_l-kv₂) = 0 (7) v₁-(0₁-0₂) = -(tv_l-v_l-kv_l-v₂) (8) v₂ ^■ ( Ί - 0, ) = -(ft>,^■ v₂ - fev₂ ^■ v₂ ) (9)

The auxiliary variables a-_\, a₂, a₃, a₄ and a₅ are introduced and defined by known parameters.

(10a) (10b) α₃ = ν₂·ν, (10c) α₄ =ν₁·(0₁-0₂) (10d) α₅=ν₂·(0₁-0₂) (10e)

Accordingly, it applies: a₄ =—ta_x + ka₂ (11) a₅ =-ta₂ +ka₃ (12)

Because ai≠ 0 and a₂≠ 0, the auxiliary variables can be converted as follows: a₂a₄ = -ta_la₂ + ka₂a₂ (13) a_xa₅ =-ta_la₂+ka_ia_i (14)

Finally, there results for the auxiliary parameters t and k: k = (16)

The center P is now determined as follows:

P + P,

P = 2

(17)

2

Fig. 3 shows an exemplary procedure of the method according to the invention. In a step S1 , a plurality of characteristic pixels is provided. By the plurality of the characteristic pixels, the first characteristic pixel and the second characteristic pixel \₂ are

encompassed. The characteristic pixels are each determined from the object 7, but each provided by different images of an image sequence of the camera 3. The image sequence in particular includes the first image 9 and the second image 10. The characteristic pixels are transformed into the three-dimensional world coordinate system 8 by a step S3 with the aid of orientation parameters of the motor vehicle 1 from a step S2. The

transformation is in particular effected with a translation vector and at least one rotation matrix. In the step S3, now, the first ray and the second ray v₂ are present. In a step S4, ray pairs are determined. Thus, a ray pair is determined with the first ray and the second ray v₂. The ray pair is composed of two rays in the three-dimensional world coordinate system 8. In a step S5, the center P is provided based on the ray pair, thus for example the first ray and the second ray v₂. The center P is provided as already above described. The position of the object 7 in the three-dimensional world coordinate system 8 is described by the center P. After step S5, a step S6 is performed, in which the position of the object 7 in the environmental region 8 characterized by the center P is checked by an error checking method. If the center P passes the error checking method without error, thus, it is provided as a reliable position of the object 7 in a step S7. If the center P is assessed as incorrect in the step S6, thus, it is not considered as reliable and is not further taken into account in particular in determining the position of the object 7. After step S6, upon positive outcome of the error checking, a step S7 follows, and upon negative outcome of the error checking, a step S8 directly follows. The step S8 also follows after step S7. In the step S8, further positions of further objects in the

environmental region 6 of the motor vehicle 1 are provided for passing the steps S1 up to at least step S6. If all of the positions of the further objects in the environmental region 6 are processed, thus, this is determined in a step S9. If all of the positions of the further objects are processed, thus, the method is terminated in a step S10. If this is not the case, the method is continued with step S1 .

Thus, a first result Y of the error checking method or a second result N of the error checking method follows after step S6. According to the first result Y of the error checking method, the respective center P has been assessed as correct, while the respective center P has been assessed as incorrect according to the second result N.

Fig. 4 shows the detailed procedure of the error checking method according to step S6. In a step S1 1 , a loop over the centers P is generated. Thus, in particular all of the centers P are processed by the loop. In a step S12 subsequent to step S1 1 , it is checked if all of the centers P have been passed. If this is the case, thus, a step S13 follows, and the error checking method is terminated. If this is not the case, thus, a step S14 follows. In the step S14, first, a step S14a is performed. In the step S14a, a procedure described in more detail in Fig. 5 is performed. Furthermore, a step S14b is performed. In the step S14b, a length of the connecting straight line w is determined. If the length is larger than a predetermined length limit value, thus, the position of the object 7 characterized by the center P is assessed as incorrect. The length of the connecting straight line w extends from the first object point Pi to the second object point P₂. Finally, a step S14c follows. In the step S14c, an angle Θ between the first ray v-i and the second ray v₂ is determined. If the angle Θ is larger than an angle limit value, the position of the object 7 characterized by the center P is assessed as incorrect. In the step S14c, it is thus examined how close the first ray v-i and the second ray v₂ are to a parallel state PZ. The closer the rays v_{1 ;} v₂ to the parallel state PZ, the more likely the correctness of the center P. The parallelism or the state PZ close to the parallelism of the rays v_{1 ;} v₂ can be mathematically checked as follows:

PZ = a_xa₃ - a₂ ² = (v_x · v_x) * (v₂ · v₂) - (v₂ · v₂)²

« II² II II² II II² II II² 2 , m

vi|| ^v2 ^" INI ^v2 ^cos (Q>

= ||v₁||² * ||v ₂||² * sin² (#)

If the state PZ close to the parallelism is now less than a limit value, the position of the rays v_{1 ;} v₂ can be assumed as substantially parallel and the center P can be assessed as not incorrect. This in particular applies if the first image 9 and the second image 10 are captured shortly one after the other. In a step S14d, the center P is compared to a plurality of centers P. The center P is considered as correct and reliable if a predetermined number of the plurality of the centers P is less than a predetermined distance apart from the center P. In the step S14d, thus, the entire distribution of the centers P provided over the image sequence from the object 7 is used to check the reliability or correctness of the respective center P. Thus, the error checking method according to step S14d can for example be effected based on the distances of the plurality of the centers P to the respectively examined center P. Similarly, an arithmetic mean of the plurality of the distances and/or a standard deviation of the plurality of the distances can be used to perform the error checking method on the respective center P. In a step S15, it is checked if the respective center P has been assessed as correct by the error checking method according to step S14. The center P is in particular assessed as correct if the steps S14a to S14d have been passed successfully and thus within the respective limit values. If the error checking of the center P is not passed, thus, it is further continued with the step S1 1 . However, if the error checking method is successfully passed, thus, a step S16 follows, by which the center P is provided as a candidate for the position of the object 7. In a step S17, the number of the centers P for the position of the object 7 is checked. If a sufficient number of candidates of the centers P is present, thus, a step S18 follows and step S6 is terminated. If this is not the case, thus, step S1 1 follows.

Fig. 5 shows the error checking method according to step S14a. As already described, the first characteristic pixel is determined in the first image 9 and the second characteristic pixel l₂ is determined in the second image 10. The first characteristic pixel is

transformed from the two-dimensional image coordinate system into the three- dimensional world coordinate system 8. Similarly, the second characteristic pixel l₂ is transformed from the two-dimensional image coordinate system into the three- dimensional world coordinate system 8. There, the first characteristic pixel is present as the first ray Vi . The second characteristic pixel l₂ is present in the three-dimensional world coordinate system 8, as already described, as the second ray v₂. After determining the connecting straight line w and the center of the connecting straight line w by the center P, the center P is retransformed into the first image 9 and/or the second image 10. The center P retransformed into the first image 9 is present there as a first retransformed pixel l_r1. The retransformed center P is present in the second image 10 as a second

retransformed pixel l_r2. Now, depending on the distance between the first characteristic pixel and the retransformed pixel l_r1 and/or the distance between the second

characteristic pixel l₂ and the second retransformed pixel l_r2, it is determined if it is larger than a predetermined first error limit value. If the retransformed pixel l_r1 , l_r2 is farther away from the characteristic pixel , l₂ than it is provided by the predetermined first error limit value, thus, the center P is assessed as incorrect. The error checking or the error checking method based on the retransformation of the center P into the first image 9 and/or the second image 10 can be described as a self back projection error checking method since the center P has also been determined based on the first image 9 and the second image 10.

Additionally or alternatively to the self back projection checking method, a further possibility can be provided if the image sequence includes more than two images. In this case, at least a third image 1 1 is present. The third image 1 1 has a third characteristic pixel l₃ of the object 7. However, the third characteristic pixel l₃ was not taken into account in particular in determining the center P. Thus, the center P is back projected into the third image 1 1 . In the third image 1 1 , the back projected center P is present as a third retransformed pixel l_r3. In capturing the third image 1 1 , the camera is 3 in a third position 0₃. The error checking or the error checking method is in now performed depending on a distance between the third characteristic pixel l₃ and the third retransformed pixel l_r3. If the distance between the third characteristic pixel l₃ and the third retransformed pixel l_r3 is equal to or less than a predetermined second error limit value, thus, the center P is assessed as correct. However, if the distance between the third characteristic pixel l₃ and the third retransformed pixel l_r3 is larger than the second error limit value, thus, the position of the object 7 characterized by the center P is assessed as incorrect. In this case, the center P preferably is no longer taken into account for the further procedure. The distances between the characteristic pixels , l₂, l₃ and the retransformed pixels l_r1 , l_r2, l_r3 are determined in the respective image plane of the image 9, 10, 1 1 .

Fig. 6 exemplarily shows the second image 10. The second image 10 is captured by the camera 3 according to Fig. 6. The second image 10 shows the environmental region 6. Furthermore, the second image 10 shows the object 7 and further objects 12. In the second image 10 according to Fig. 6, centers P are shown, by which the position of the object 7 and/or positions of further objects in the three-dimensional world coordinate system 8 are characterized. Thus, according to Fig. 6, it is for example shown how the camera system 2 can be used for 3D reconstruction of the object 7 and/or of the further objects 12. The camera system 2 can for example also be used by a driver assistance system of the motor vehicle 1 , which is formed as a parking assistant. Thus, by the position of the object 7 in the three-dimensional world coordinate system 8, a distance from the motor vehicle 1 or from the camera 3 to the object 7 can be determined.

Similarly, a height of the object 7 above the ground of the environmental region 6 can for example be determined. Thus, it can for example also be determined if the motor vehicle 1 with its known height can pass below the object 7.

Fig. 7 shows a plan view image 13 of the environmental region 6. With a schematic picture 1 a of the motor vehicle 1 . Based on the plan view image 13, a user of a driver assistance system of the motor vehicle 1 can for example recognize if the object 7 and/or the further objects 12 have to be considered as an obstacle in reversing of the motor vehicle 1 . The centers P describing the position of the object 7 and/or of the further objects 12 in Fig. 7 are in particular present in the three-dimensional world coordinate system 8. Thus, a distance from the position of the object 7 to the motor vehicle 1 can be determined, and a collision warning can for example be output upon falling below the distance by the motor vehicle 1 .

Claims

Method for determining a position of an object (7) located in an environmental region (6) of a motor vehicle (1 ) in a three-dimensional world coordinate system (8), in which the following steps are performed:

- providing a first image (9) having the object (7) and a second image (10) having the object (7) of an image sequence by means of a camera (3) of the motor vehicle (1 ),

- determining a first characteristic pixel ( ) of the object (7) in the first image (9) and a second characteristic pixel (l₂) of the object (7) in the second image (10), characterized by

- transforming the first characteristic pixel ( ) present in a two-dimensional image coordinate system into the three-dimensional world coordinate system (8) as a first ray (v^ and the second characteristic pixel (l₂) present in a two-dimensional image coordinate system into the three-dimensional world coordinate system (8) as a second ray (v₂),

- determining a connecting straight line (w) oriented perpendicularly to the first ray (Vi ) and to the second ray (v₂),

- determining a center (P) of the connecting straight line (w) as the position of the object (7) in the environmental region (6)

wherein the position of the object (7) in the environmental region (6) characterized by the center (P) is checked by an error checking method such that the center (P) is retransformed into the first image (9) and/or into the second image (10) and/or into a third image (1 1 ) of the image sequence, and the error checking method is performed depending on the retransformation of the center (P), wherein a first error value is provided by the retransformation of the center (P) into the first image (9) and/or into the second image (10), and a second error value is provided by the retransformation of the center (P) into the third image (1 1 ), and the position of the object (7) characterized by the center (P) is incorrect if the first error value is determined as greater than a predetermined first error limit value and/or the second error value is determined as greater than a predetermined second error limit value.

2. Method according to claim 1 ,

characterized in that

the transformation from the two-dimensional image coordinate system into the three- dimensional world coordinate system (8) is performed depending on orientation parameters of the motor vehicle (1 ) determined by odometry and/or visual odometry.

3. Method according to claim 2,

characterized in that

yaw information and/or pitch information and/or roll information of the motor vehicle (1 ) are described by the orientation parameters of the motor vehicle (1 ).

4. Method according to any one of the preceding claims,

characterized in that

a ray pair is determined by the first ray (v^ and the second ray (v₂), and the center (P) of the connecting straight line (w) is only determined depending on the ray pair.

5. Method according to any one of the preceding claims,

characterized in that

the first image (9) and the second image (10) are provided as images of the image sequence immediately consecutive in time.

6. Method according to claim 1 ,

characterized in that

a length of the connecting straight line (w) is determined, and the error checking method is performed depending on the length of the connecting straight line (w), and the position of the object (7) characterized by the center (P) is assessed as incorrect if the length is larger than a predetermined length limit value.

7. Method according to claim 1 or 6,

characterized in that

an angle (Θ) between the first ray (v^ and the second ray (v₂) is determined, and the error checking method is performed depending on the angle (Θ), and the position of the object (7) characterized by the center (P) is assessed as incorrect if the angle (Θ) is larger than an angle limit value.

8. Method according to claim 1 , 6 or 7,

characterized in that

a plurality of characteristic pixels ( , l₂, of the object (7) are determined in a plurality of images (9, 10, 1 1 ) of the image sequence, and a plurality of centers (P) are determined each based on two of the characteristic pixels ( , l₂, I3) , and the error checking method is performed depending on the plurality of the centers (P), and the position of the object (7) characterized by the center (P) is assessed as incorrect if a predetermined number of the plurality of the centers (P) is less than a predetermined distance apart from the center (P).

9. Method according to claim 8,

characterized in that

the error checking method is performed depending on a plurality of distances of the plurality of the centers (P) and/or an arithmetic mean of the plurality of the distances and/or a standard deviation of the plurality of the distances.

10. Computer program product for carrying out a method according to any one of the preceding claims if the computer program product is executed on a programmable computer device.

1 1 . Camera system (2) with at least one camera (3) and an evaluation unit (4), which is adapted to perform a method according to any one of claims 1 to 9.

12. Motor vehicle (1 ) with a camera system (2) according to claim 1 1 .