WO2023088932A1

WO2023088932A1 - Title: method for determining the relative orientation of two vehicles

Info

Publication number: WO2023088932A1
Application number: PCT/EP2022/082068
Authority: WO
Inventors: Lucien Garcia
Original assignee: Continental Automotive Gmbh
Priority date: 2021-11-18
Filing date: 2022-11-16
Publication date: 2023-05-25
Also published as: CN118266005A; FR3129236B1; FR3129236A1

Abstract

A method for determining the orientation of a vehicle (VA) with respect to a reference object (VB), each comprising an image sensor (10A, 10B), comprising: - receiving an image acquired by each image sensor of the vehicle, in which image the vehicle/reference object is visible, - receiving information on the position of each image sensor with respect to the reference object/vehicle in which it is installed, - based on the received information, estimating a first position of each image sensor in the image acquired by the other image sensor, - determining the relative orientation between the two image sensors, and - deducing, based on the relative orientation between the two image sensors, the relative orientation between the vehicle and the reference object.

Description

Title: Process for determining the relative orientation of two vehicles

Technical area

This disclosure relates to a method for determining the relative orientation of two vehicles equipped with cameras. It finds advantageous applications in driving assistance functionalities such as third-party vehicle trajectory forecasting or estimation, collision detection, or the grouping of vehicles into platoons or convoys.

Prior technique

[0002] A great deal of work is now carried out in order to make vehicles autonomous, which means detecting the positions of vehicles surrounding a vehicle in question, detecting and predicting the trajectory of these vehicles, and therefore determining the position and orientation of surrounding vehicles.

[0003] For this, it is possible to use a lidar sensor, which makes it possible to obtain three-dimensional information on the environment of the vehicle. However, lidar sensors are expensive, have a high energy consumption and also have a fairly large size which makes them difficult to integrate on a small vehicle. Therefore, not all vehicles are equipped with lidar sensors.

[0004] For vehicles without Lidar sensors but equipped with a camera, determining the orientation of a surrounding vehicle is more difficult in the absence of three-dimensional information. This estimate is made from the detected position of the surrounding vehicle and an a priori on the size of the vehicle. However, the estimate obtained is not precise and may induce errors in subsequent processing.

[0005] It is also possible to estimate an orientation of a surrounding vehicle based on a lower boundary of the object, which is obtained by determining a window encompassing the entire outline of the vehicle in the image, and by identifying a low point of this window. The estimates made on this basis are not more precise, especially when the road is not horizontal.

Summary

[0006] This disclosure improves the situation.

[0007] In particular, an object of the invention is to propose a method for determining the orientation of a vehicle with respect to another object, for example a second vehicle, which is more precise than the solutions of the prior art. Another object of the invention is to be applicable to vehicles without Lidar sensors, but equipped with cameras.

[0008] In this regard, there is proposed a method for determining the orientation of a vehicle with respect to a reference object, the vehicle and the reference object each comprising an image sensor, the method being implemented by a computer and comprising:

- the reception of an image acquired by the image sensor of the vehicle, on which the reference object is visible, and of an image acquired by the image sensor of the reference object, on which the vehicle is visible,

- the reception of position information from the image sensor of the reference object with respect to the latter, and of position information from the image sensor of the vehicle with respect to the latter,

- from the information received, the estimation of a first position of the image sensor of the reference object on the image acquired by the image sensor of the vehicle, and of a first position of the sensor of images of the vehicle on the image acquired by the image sensor of the reference object,

- the determination of the relative orientation between the two image sensors so that:

* a second position of the image sensor of the reference object on the image acquired by the image sensor of the vehicle calculated from said relative orientation corresponds to the first estimated position of the image sensor of the object reference, and

* that a second position of the vehicle image sensor on the image acquired by the image sensor of the reference object calculated from said relative orientation corresponds to the estimated first position of the vehicle image sensor ,

- the deduction, from the relative orientation between the two image sensors, of the relative orientation between the vehicle and the reference object.

[0009] In embodiments, the reference object is a pedestrian, another vehicle, or an element of infrastructure.

[0010] In some embodiments, the method is implemented by a computer on board the vehicle, in the reference object, or by a remote computer.

[0011] In some embodiments, the reference object is a second vehicle, the method is implemented by a computer on board the first and/or the second vehicle, and comprises a preliminary step of establishing a communication link between the first and the second vehicle. [0012] In embodiments, the determination of the relative orientation between the two image sensors is implemented by minimizing the difference between the first estimated position and the second calculated position of the same image sensor. .

[0013] In embodiments, the determination of the relative orientation between the two image sensors is implemented by the Levenberg-Marquardt algorithm.

[0014] In embodiments, the determination of the relative orientation between the image sensors comprises the determination of the relative yaw and pitch between the image sensors, the roll being assumed to be zero.

According to another object, a computer program product is described comprising instructions for implementing the method according to the preceding description when it is executed by a processor.

According to another object, there is described a non-transitory recording medium readable by a computer on which is recorded a program for the implementation of the method according to the preceding description when this program is executed by a processor.

According to another object, a vehicle is described comprising an image sensor, a computer, and a connection interface to a telecommunications network, characterized in that the computer is configured to implement the method according to foregoing description.

The proposed method makes it possible to determine the relative orientation between a vehicle equipped with an image sensor, for example a camera, and a reference object, from an image acquired by the vehicle and the position, with respect to the reference object, of an image sensor mounted thereon. In particular, this method is applicable to the determination of the relative orientation between two vehicles both equipped with cameras. This process makes it possible to calculate the relative yaw and pitch between the two vehicles, including when the road is not flat. It can be implemented in the absence of a Lidar sensor on the vehicles.

Brief description of the drawings

[0019] Other characteristics, details and advantages will appear on reading the detailed description below, and on analyzing the appended drawings, in which:

[0020] [Fig. 1] Figure 1 schematically shows two vehicles that can implement the method.

[0021] [Fig. 2] FIG. 2 schematically represents the main steps of the method for determining the relative orientation between a vehicle and a reference object. [0022] [Fig. 3] FIG. 3 schematically represents the implementation of the method according to one embodiment.

[0023] [Fig. 4] FIG. 4 schematically represents the implementation of the method according to one embodiment.

[0024] [Fig. 5] FIG. 5 represents an example of a vehicle comprising an on-board camera and illustrates the references associated respectively with the vehicle and with the camera.

Description of embodiments

[0025] With reference to Figure 1, we will now describe a method for determining the orientation of a vehicle VA comprising an image sensor 10A, relative to a reference object also comprising an image sensor 10B and which is visible by the image sensor 10A of the vehicle. In the following, an image sensor is understood to mean a sensor suitable for detecting the position of an object, it may be a camera, or even a radar or lidar sensor.

The method will be described below by considering a particular embodiment in which the reference object is a second vehicle VB, in particular a four-wheeled vehicle of the car or truck type, traveling on the same road as the vehicle. GOES. In the following, the numerical references in accordance with this example and associated with the first and second vehicles respectively comprise an index A or B. In other embodiments, the reference object can be another type of vehicle, for example a two-wheeler. It can also be a pedestrian equipped with a camera or a mobile phone. It can also be an element of infrastructure, for example a toll or a gate.

[0027] In the case where the reference object is a second vehicle, the two vehicles can follow each other on the same lane of the road, or travel on different lanes. In embodiments, each vehicle has an image sensor 10A, 10B which is mounted on the respective vehicle with a known position and orientation of the image sensor relative to the vehicle on which it is mounted. The image sensor can be oriented from the front to the rear of the vehicle so as to acquire images on which the vehicles preceding or following the vehicle are visible. The image sensor can also be mounted on one side of the vehicle. In embodiments, one or each vehicle VA, VB, can be equipped with several cameras oriented towards the front and/or towards the rear and/or towards the sides of the vehicle.

The vehicle VA, and the reference object VB further comprise an access interface 11 A, 11 B to a telecommunications network, for example of the GSM, LTE, 3G, 4G or 5G type, or a Wi -Fi. In one embodiment illustrated schematically in Figure 2, in which the reference object is a second vehicle VB, the vehicles can communicate with each other by a vehicle-to-vehicle type communication. In this case, the vehicles also include a computer 12A, 12B configured to implement the method described below after having established communication between them, and a memory 13A, 13B storing the code instructions for the implementation of this method. , as well as information regarding the position and orientation of the camera mounted on the respective vehicle with respect to it.

In another embodiment illustrated schematically in Figure 4, the vehicle VA and the reference object are adapted to communicate with a remote server S.

With reference to Figures 2 to 4, we will now describe examples of implementation of the method. In the description below, the reference object is a second vehicle VB. In the embodiment shown in Figure 3, the method is implemented by the computer of one of the vehicles, by convention below, of the first vehicle VA. In the embodiment shown in Figure 4, the method is implemented by a remote server S.

The method comprises a first step 100 of receiving at least one image acquired by the image sensor of the first vehicle VA, on which the other vehicle or reference object is visible. In the following, we take the example where the first vehicle VA acquires an image IA where the second vehicle VB is visible. In the case where the method is implemented by the computer 12A of the first vehicle, the computer is connected to the camera and receives the images that it acquires. This step also includes the reception of an image IB acquired by the camera 11 B of the second vehicle (or of the reference object), on which the first vehicle VA is visible. In the case where the method is implemented by a computer on board the first vehicle, the latter receives the image IB of the first vehicle via a wireless network. In the case where the method is implemented by a remote server (FIG. 4), the remote server can receive the images IA, IB, acquired by the cameras of the two vehicles, via the telecommunications network.

The method then comprises the reception 200 by the computer implementing the processing, of position information from the image sensor 10B of the reference object, with respect to the latter, that is to say say in the previous example, position information denoted CB of the image sensor 10B of the second vehicle VB relative to the latter. This information can for example comprise a simplified three-dimensional representation of the reference object, as well as the position of the marker element in a marker associated with this simplified representation. This step also includes receiving position information from the image sensor 10A of the first vehicle VA by relation to this one. If the method is implemented by the computer of the first vehicle, this information is retrieved from the memory 13A.

In the embodiment in which a remote server S implements the processing and receives images acquired by the two vehicles, this step comprises the reception, by the server, of the position information (CA and CB) of each camera (10A and 10B) relative to the vehicle (VA and VB) on which it is respectively mounted.

From the information received, the computer can estimate 300: the position Pos1 B, on the image IA acquired by the first vehicle VA, of the image sensor 10B of the reference object on the image, c that is to say, for example, the position of the image sensor 10B of the second vehicle VB on the image, and the position Pos1A, on the image acquired by the reference object, of the image sensor 10A of the vehicle VA on the image, that is to say for example the position of the image sensor 10A of the first vehicle VA on the image acquired by the second vehicle VB.

To this end, for estimating a position, the computer implements an algorithm to detect the contours of the reference object or of the vehicle in the image. Many algorithms known to those skilled in the art are available to perform this function, for example the publication by J. Redmon, S. Divvala, R. Girshick and A. Farhadi "You Only Look Once: Unified, Real-Time Object Detection”, 2015. Then the computer deduces, from the relative position of the image sensor in relation to the reference object or the vehicle, a first estimated position of the image sensor on the image. If the position of the image sensor with respect to one of the first and of the second vehicle is not known, it is possible to consider that this sensor is part of the vehicle and to deduce a range of possible values for the position in X and Y of this sensor on the image.

[0036] The computer can then determine, step 400, the relative orientation between the two image sensors, denoted (RX, RY) in such a way: that a second calculated position Pos2B of the image sensor 10B of the reference object on the image IA acquired by the first vehicle VA from this relative orientation, corresponds to the first position Pos1 B estimated in the previous step, and that a second calculated position Pos2A of the image sensor 10A of the vehicle VA on the image IB acquired by the image sensor 10B of the reference object from of this relative orientation corresponds to the first estimated position Pos1A of the image sensor 10A of the vehicle VA.

[0037] The relative orientation which is calculated in step 400 includes the angles of yaw RX and pitch RY between the image sensor and the reference element, that is to say a rotation relative to an axis X represented in FIG. 5 orthogonal to the plane of the road on which the vehicle is traveling, and a rotation with respect to an axis Y also represented, which is orthogonal to the axis X and orthogonal to the direction of movement of the vehicle Z. The orientation determined does not include the roll along the Z axis. In the implementation of step 400 described below, the relative position along the X and Y axes between the image sensor and the benchmark is also determined.

This determination of the relative orientation between the two image sensors is implemented by an iterative algorithm aimed at minimizing a cost function formed by the differences between the second calculated position of each image sensor from the relative orientation between the image sensors, and the respective estimated first position.

In the mark of each image sensor, the position of this sensor is at the origin of the mark.

The computer can initialize the search for the relative orientation between the image sensors 10A, 10B by an initial relative orientation, for example equal to (co, 0, 0, 0) where the first coordinate corresponds to an angle of yaw (we take co as the initial value in the example of two vehicles each comprising an image sensor since in principle the image sensors of the two vehicles are oriented in opposite directions to take images on which the two vehicles are visible), the second coordinate corresponds to a pitch angle, the third coordinate corresponds to a translation along the X axis and the fourth coordinate corresponds to a translation along the Y axis, these translations are known to the nearest scale factor , the translation in Z is fixed at the value 1. From this initial relative orientation, the computer determines the position of the image sensor 10B of the reference object in a frame associated with the image sensor 10A of the vehicle VA , by applying a change of reference matrix determined from this initial orientation, then projects this position, from the intrinsic parameters of the image sensor 10A of the vehicle VA, to obtain the coordinates of the image sensor 10B in an image acquired by the image sensor 10A. It thus obtains the second calculated position Pos2B of the image sensor 10B of the reference object in the image IA. The same processing is implemented symmetrically to obtain the second calculated position Pos2A of the image sensor of the vehicle VA in the image IB. As indicated above, this calculation is implemented iteratively so as to find the relative orientation matrix (RX, RY, TX, TY), with TX and TY the translations respectively along the X and Y axes, minimizing the difference between the calculated position Pos2B and the estimated position Pos1 B, and between the calculated position Pos2A and the estimated position Pos1 A. This can be done using a Levenberg-Marquadt algorithm, described in the publications:

K. Levenberg, “A Method for the Solution of Certain Problems in Least Squares”, in Quart. Appl. Math. 2, 1944, p. 164-168.

D. Marquardt, “An Algorithm for Least-Squares Estimation of Nonlinear Parameters”, in SIAM J. Appl. Math. 11, 1963, p. 431-441. image sensor from the relative orientation of the image sensors.

In Figure 5, there is shown an associated mark (Xc, Yc, Zc) to an image sensor mounted on a vehicle, and a mark (Xv, Yv, Zv) associated with the vehicle. With reference to FIG. 5, once the relative orientation of the image sensors has been obtained, the relative orientation between the two vehicles can easily be deduced during a step 500 from the position and the orientation of each image sensor relative to the vehicle on which it is mounted.

Claims

-9- Claims

[Claim 1] Method for determining the orientation of a vehicle (VA) with respect to a reference object, the vehicle and the reference object each comprising an image sensor (10A, 10B), the method being implemented by a computer and comprising:

- the reception (100) of an image (IA) acquired by the image sensor (10A) of the vehicle, on which the reference object is visible, and of an image acquired (IB) by the sensor of images (10B) of the reference object, on which the vehicle (VA) is visible,

- the reception (200) of position information (CB) of the image sensor (10B) of the reference object with respect to the latter, and of position information (CA) of the image sensor ( 10A) of the vehicle (VA) in relation to it,

- from the information received, the estimation (300) of a first position (Pos1 B) of the image sensor (10B) of the reference object on the image (IA) acquired by the image sensor (10A) of the vehicle (VA), and of a first position (Pos1A) of the image sensor

(IOA) of the vehicle on the image acquired (IB) by the image sensor (10B) of the reference object,

- the determination (400) of the relative orientation (RX, RY) between the two image sensors (10A, 10B) so that:

* a second position (Pos2B) of the image sensor (10B) of the reference object on the image acquired by the image sensor (10A) of the vehicle calculated from said relative orientation corresponds to the first estimated position (Pos1 B) of the image sensor

(IOB) of the reference object, and

* that a second position (Pos2A) of the image sensor (10A) of the vehicle on the image acquired by the image sensor (10B) of the reference object calculated from said relative orientation corresponds to the first estimated position (Pos1A) of the image sensor (10A) of the vehicle.

- the deduction (500), from the relative orientation between the two image sensors (10A, 10B), of the relative orientation between the vehicle and the reference object.

[Claim 2] A method according to claim 1, wherein the reference object is a pedestrian, another vehicle, or an item of infrastructure.

[Claim 3] Method according to any one of the preceding claims, the method being implemented by a computer (12A) on board the vehicle, in the reference object, or by a remote computer (S).

[Claim 4] Method according to the preceding claim, in which the reference object is a second vehicle (VB), the method being implemented by a computer (12A) on board the first and/or the second vehicle (VA) , and comprising a preliminary step of establishing a communication link between the first (VA) and the second vehicle (VB).

[Claim 5] A method according to any preceding claim, wherein the determination (400) of the relative orientation between the two image sensors is implemented by minimizing the difference between the first estimated position and the second calculated position of the same image sensor.

[Claim 6] A method according to any preceding claim, wherein determining the relative orientation (400) between the two image sensors is implemented by the Levenberg-Marquardt algorithm.

[Claim 7] A method according to any preceding claim, wherein determining the relative orientation (400) between the image sensors includes determining relative yaw and pitch between the image sensors, roll being assumed to be zero.

[Claim 8] Computer program product comprising instructions for carrying out the method according to any one of the preceding claims when executed by a processor.

[Claim 9] Non-transitory recording medium readable by a computer on which is recorded a program for implementing the method according to one of Claims 1 to 7 when this program is executed by a processor.

[Claim 10] Vehicle (VA, VB) comprising an image sensor (10A, 10B), a computer (12A, 12B), and a connection interface (11 A, 11 B) to a telecommunications network, characterized in that the computer is configured to implement the method according to one of claims 1 to 7.