JPWO2021197649A5 - - Google Patents

Description

FIELD OF THE INVENTION The present invention relates generally to the field of vehicle assistance systems. In particular, the present invention relates to a method and system for calculating the yaw angle of a trailer coupled to a towing vehicle based on image information provided by a camera on the vehicle.

Methods are known for calculating the angle of a trailer relative to a towing vehicle based on image information provided by a camera on the vehicle.

In particular, known methods have low computational complexity but do not provide robust angular information in the case of poor image quality.

It is an object of embodiments of the invention to provide a method for calculating the yaw angle of a trailer with high robustness and reliability, which does not require obtaining the location of the towball . This problem is solved by the features of the independent claims. Preferred embodiments are described in the dependent claims. Although not explicitly described, embodiments of the invention can be freely combined with each other.

According to one aspect, the invention relates to a method of determining the yaw angle of a trailer relative to the longitudinal axis of a towing vehicle. The method includes the following methods.

First, at least first and second images of the trailer are captured using a camera. The first and second images are taken such that the orientation of the trailer relative to the vehicle is different on at least the two images.

After taking this image, at least first and second characteristics of the trailer are determined. The first and second features need to be visible on the first and second images. Additionally , this first feature is located at a different location on the trailer than the second feature. For example, the first feature may be the first obvious feature at the first location , and the second feature may be the second obvious feature at the second location .

Based on the determined first and second features, a first angle estimate is calculated. The first angle estimate characterizes a turning angle in a horizontal plane between a first feature on the first image and a first feature on the second image with respect to a fixed point of the towing vehicle. In other words, the first angle estimation is based on a first line extending between the position of the first feature on the first image and the position of this fixed point, and the position of the first feature on the second image and this fixed point. It relates to a turning angle defined between the position of a point and a second line extending between the points. This turning angle is open from the vehicle toward the trailer.

Additionally , a second angle estimate is calculated. The second angle estimation characterizes a turning angle in a horizontal plane between a second feature on the first image and a second feature on the second image with respect to a fixed point of the towing vehicle. In other words, the first angle estimation is based on a first line extending between the position of the second feature on the first image and the position of this fixed point, and the position of the second feature on the second image and this fixed point. means the angle of rotation defined between the position of a point and a second line extending between the points. This turning angle is open from the vehicle toward the trailer.

The statement "location of the first or second feature on the first image (or second image) " does not refer to a single point on this image, but rather the location of the feature on each trailer at a certain point in time when the image is taken. It is worth mentioning that , means a certain location within the vicinity of the towing vehicle.

Finally, the trailer's yaw angle is calculated based on the first and second angle estimates.

This method is advantageous because it uses two or more images and uses two or more trailer features to calculate the angle estimate, and the trailer feature detection is high. Even with noise and poor image quality, the results of yaw angle determination are very reliable and robust.

Other methods that may take into account the location of fixed points often require trigonometry on the location of the feature to generate accurate angles. As a result, these features become susceptible to noise or inaccuracies in the features being tracked. If these features are inaccurate, such methods may be mathematically unstable or produce no results at all.

According to one embodiment , in the first or second image the yaw angle of the trailer with respect to the vehicle is zero. Thereby, this image may be used as a "zero pose image", that is, as a reference for exact alignment between the longitudinal axis of the vehicle and the longitudinal axis of the trailer. However , other yaw angles may also be used as reference values. If this other yaw angle is not known, the system may calculate the change in trailer angle rather than the absolute trailer angle.

According to one embodiment , this fixed point is the camera position or the tow ball position. Using a camera as a fixed point is technically simple since the camera is used to capture images. However , it may be more precise to use the tow ball as a fixed point. In this way, the information contained in the images taken by the camera may be transformed in order to mitigate the loss of accuracy by adjusting the light beam using a fixed point, for example the location of the towball . However , if the towball is relatively close to the camera and the trailer features are relatively far away , the proposed method can determine the trailer angle that is accurate enough for an automated trailer reversing system without adjusting the towball position. You can calculate it. If the towball is close to the camera location (e.g. less than 0.3 mm in the horizontal direction) and the trailer features are separated by a distance of 2 m or more in the horizontal direction, the proposed method may improve the results. be.

According to one embodiment , calculating the first and second angle estimates comprises determining a ray between the fixed point and the first and second features in the first and second images. By ray is meant a line extending between the fixed point and the first and second features. Based on this ray, the current turning angle can be determined , for example based on a geometric method, with low computational costs.

According to one embodiment , camera calibration information is used to convert the position of the first feature and / or the second feature into a beam of light. For example, by using camera calibration information to obtain a camera position, the position of a certain feature on an image can be converted into location information depending on or correlated to the camera position.

According to one embodiment , in addition to this first and second feature, at least one further feature of the trailer is used to calculate the yaw angle. Using three or more features further improves the robustness and reliability of the yaw angle determination.

According to one embodiment , the yaw angle is calculated by setting a median value based on at least two angle estimates. This results in a very stable yaw angle determination.

According to other embodiments , the yaw angle is calculated by setting an average value of at least two angle estimates or by using statistical methods applied to this angle estimate.

According to one embodiment , the method further comprises determining an angular window. The angle window may have an upper boundary and a lower boundary around the yaw angle. Additionally , a set of features is determined. Multiple features within this set of features lead to an angle estimate that lies within this angle window. This determined set of features, preferably only features included in this set of features , are used for future yaw angle calculations. Thus, in other words, by using the information of the previous yaw angle determination, two or more features of the trailer that result in an angle estimate very close to the determined yaw angle (i.e. within the angle window) are determined and determined. Features that result in angle estimates that deviate significantly from the calculated yaw angle (i.e., outside the angle window) are not tracked. This can significantly reduce the complexity and accuracy of angle estimation calculations.

According to one embodiment , the calculated yaw angle value is increased by a certain amount or a certain percentage in order to correct the underestimation. In order to correct for an underestimation of the calculation result, for example, the calculated yaw angle may be scaled up by 5% to 15%, in particular by 10%.

According to one embodiment , the camera is a rear view camera of the vehicle. Based on a rear view camera, images of the trailer can be captured with low technical costs .

In a further aspect , a system for determining a yaw angle of a trailer relative to a longitudinal axis of a towing vehicle is disclosed. The system includes a camera that captures images of the trailer, and a processing entity that processes the captured images. This system is
- capturing at least first and second images of the trailer using a camera, where the orientation of the trailer with respect to the vehicle is different on the at least two images;
- determining at least a first feature of the trailer visible on the first and second images, the first and second features being visible on the first and second images, the first and second features being located at different positions of the trailer; determining at least a first characteristic of the trailer;
- calculating a first angle estimate, the first angle estimate characterizing a turning angle in a horizontal plane between a first feature on the first image and a first feature on the second image with respect to a fixed point of the towing vehicle; and,
- calculating a second angle estimate, the second angle estimate characterizing a turning angle in a horizontal plane between a second feature on the first image and a second feature on the second image with respect to a fixed point of the towing vehicle; and,
- calculating a yaw angle based on the first and second angle estimates.

Any of the above features described as an embodiment of the method are also applicable as system features in the system of the present disclosure.

According to yet another aspect, a vehicle is disclosed that includes a system according to any one of the above aspects.

The term "vehicle" as used in this disclosure may mean a passenger car, lorry, bus, train or any other watercraft.

As used in this disclosure, the term "yaw angle" may refer to the turning angle between the longitudinal axis of the vehicle and the longitudinal axis of the trailer.

The term "median" as used in this disclosure may refer to the value that separates the upper and lower halves of a data sample or probability distribution.

The term “essentially” or “substantially” as used in the present invention means
means a deviation of +/-10%, preferably +/-5%, from the exact value;
to mean a deviation in the form of an insignificant change in the function and/or the traffic law;
means at least one of the following .

Various aspects of the invention, including its specific features and advantages, will be readily understood from the following detailed description and accompanying drawings.

FIG. 1 shows an exemplary top view of a vehicle towing a trailer. FIG. 2 schematically shows angle estimation based on first and second features captured by camera images at different turning angles between the trailer and the towing vehicle. FIG. 3 shows a schematic block diagram illustrating the steps of a method for determining the yaw angle of a trailer with respect to the longitudinal axis of the towing vehicle .

The invention will now be described more fully with reference to the accompanying drawings, in which exemplary embodiments are shown. While the embodiments in the figures relate to preferred embodiments, all elements and features described in connection with the embodiments may be used herein, insofar as appropriate, particularly in relation to any other embodiments described above. It may also be used in combination with any other embodiments and features described. However , the invention should not be considered limited to the embodiments described herein. Throughout the following description, like reference numerals are used , where appropriate, to indicate similar elements, members, items or features.

The features of the invention disclosed by the description, the claims, the embodiment examples and / or the drawings may realize the invention in its various forms, both separately and in any combination thereof. It may be a material that

FIG. 1 shows a top view of a vehicle 1 towing a trailer 2. FIG. The vehicle 1 includes a longitudinal axis LAV passing through the center of the vehicle 1. Similarly, the trailer 2 includes a longitudinal axis LAT passing through the center of the trailer 2. The trailer 2 is connected to the vehicle 1 using a trailer hitch including a tow ball 4.

In some driving situations, the longitudinal axis LAV of the vehicle 1 and the longitudinal axis LAT of the trailer 2 may not be arranged parallel to each other and may not coincide with each other, but these axes may vary depending on the yaw angle. YA can be limited. In other words, the yaw angle YA defines the angular deviation of the longitudinal axis LAT of the trailer 2 from the longitudinal axis LAV of the vehicle 1. The yaw angle YA may be measured on a horizontal plane including the longitudinal axis LAT of the trailer 2 and the longitudinal axis LAV of the vehicle 1.

Obtaining the yaw angle YA is advantageous, inter alia, for example in trailer assistance systems.

To determine the yaw angle YA, a plurality of images of at least a part of the trailer 2 are taken using the camera 3. The camera 3 may be, for example, a rear view camera of the vehicle, and the rear view camera may also be used to take images of the surroundings of the vehicle when reversing.

FIG. 2 shows a schematic diagram showing the angular relationship of the first and second features F1, F2 of the trailer at different times when the trailer 2 has different yaw angles YA relative to the towing vehicle 1.

The camera 3 may take two or more images at different times, with different angular positions of the trailer 2 relative to the vehicle 1. For example, a series of images may be taken.

In this example, the second image may show the orientation of the trailer 2 relative to the vehicle at a yaw angle YA=0°. However , in other embodiments, yaw angle YA may be any other reference yaw angle that is known and can be used to determine the current yaw angle.

Trailer features are locally matched using feature detection and matching algorithms . For example, Harris Corner Detection Method, Scale Invariant Feature Transform (SIFT) Algorithm, Speed Up Robust Features (SURF) Algorithm, Binary Robust Invariant Scalable Keypoint (BRISK) Algorithm, Binary Robust Independent Fundamental Features (BRIEF), Directed FAST and The Rotated BRIEF (ORB) algorithm or other suitable feature detection and matching algorithm could be used .

Feature detection and matching algorithms may detect image features that are on the trailer or not on the trailer. A number of different methods may be used to separate trailer features from non- trailer features. For example, in the case of straight forward travel, trailer features may be separated from non- trailer features by looking for features that remain in the same position over time. Alternatively, the motion of background features may be modeled over time using the known motion of the vehicle. This may be extracted from CAN data regarding speed and steering. In this case, features that do not apply to the epipolar constraint of the fundamental matrix may be considered as trailer features.

On the image taken by camera 3 a number of different features may be discernible. FIG. 2 shows features F1, F2 identified in different angular positions relative to a fixed point of the vehicle 1. In this way, the upper pair of first and second features F1, F2 ( related to the solid ray connecting features F1, F2 with camera 3) is identified in the first image, The lower pair of first and second features F1, F2 (related to the dashed ray connecting to camera 3) are identified in the second image at different times. In order to determine the ray R connecting the features F1, F2 with the camera 3, the calibration information of the camera 3 may be used to convert the location of the features in image coordinates into a ray. In other words, the location of the feature on the image is combined with the location of the fixed point of the vehicle based on the calibration information of the camera 3 in order to associate the camera location and the feature location.

After determining the ray R between the fixed point and the at least two features in the first and second images, the rotation angles of the first feature F1 and the second feature F2 are determined. In FIG. 2, α1 indicates the rotation angle of the first feature F1 between two captured images, and α2 indicates the rotation angle of the second feature F2 between these images. Preferably, more than two features of the trailer are determined and tracked across multiple images. In addition , preferably more than two images are taken at different times in order to improve the results of the yaw angle estimation.

After determining the turning angles α1 and α2, the yaw angle YA can be calculated based on the turning angles α1 and α2. Yaw angle YA can be developed in various ways.

In the first embodiment, the yaw angle YA may be calculated as the median value of the developed turning angles α1 and α2. In another embodiment, the yaw angle YA may be determined by calculating the arithmetic mean of the developed turning angles α1, α2. In yet another embodiment, it may be calculated using a stochastic process. For example, the angular variance of each feature may be taken into account and the median value may be derived using only features with a small variance.

The yaw angle can be further refined with a Kalman filter or a dynamic model based on vehicle speed and steering information. Speed and steering can be derived from the CAN data, or using visual methods that process image data, for example.

One advantage of using the median is that the method is very robust. When lighting conditions are poor, the median continues to provide reliable and consistent angle estimates even if only one feature is being tracked. The median is also very robust to outliers, which can occur if feature tracking is poor or the image is particularly cluttered .

It appeared that not all features visible on the captured image were equally suitable for calculating the yaw angle YA. In order to reduce the computational complexity and robustness, those features are selected and further used in determining the yaw angle YA . This provides a turning angle that is very close to the actual yaw angle. For feature selection, only those features can be tracked in subsequent images providing the turning angle α1, α2 in a fixed window around the actual yaw angle . For example, a window may be defined by an upper boundary and a lower boundary, where the upper and lower boundaries define an angular window around this actual yaw angle. For example, the window may extend over a distance of 2° to 10°, especially 3° to 5°. In the last two or more yaw angle determination steps , all features that led to the turning angle within this window are further tracked in the next captured image.

Additionally , if the estimate of the calculated yaw angle YA is too low compared to the actual yaw angle YA, the calculated yaw angle YA may be increased by some amount or percentage to alleviate the underestimation. (scaled up) .

FIG. 3 shows a block diagram illustrating the method steps of a method for determining the yaw angle YA of the trailer 2 with respect to the longitudinal axis LAV of the towing vehicle 1.

As a first step, first and second images of the trailer are taken (S10).

After image capture, features of the trailer visible on the first and second images are determined (S11).

After determining the features, first and second angle estimates are calculated based on the determined first and second features (S12, S13).

Finally, a yaw angle is calculated based on the first and second angle estimates (S14).

It is to be noted that the specification and drawings merely illustrate the principles of the proposed invention. Those skilled in the art will be able to implement various configurations that embody the principles of the invention, even if not explicitly described or illustrated in this disclosure.

1 Vehicle 2 Trailer 3 Camera 4 Towball
α1 First angle estimation α2 Second angle estimation F1 First feature F2 Second feature LAT Trailer longitudinal axis
LAV Vehicle longitudinal axis
R Ray YA Yaw angle

Claims (13)

A method for determining a yaw angle (YA) of a trailer (2) with respect to a longitudinal axis (LAV) of a towing vehicle (1), the method comprising:
- capturing at least a first and a second image of the trailer (2) using a camera (3), in which the orientation of the trailer (2) with respect to the vehicle (1) is different on at least two images ; S10) and
- in the step (S11) of determining at least first and second features (F1, F2) of said trailer (2) visible on first and second images, said first and second features (F1, F2); a step (S11) of determining at least first and second features (F1, F2) located at different positions of said trailer (2);
- a first angle estimation (α1) is the difference between the first feature (F1) on the first image and the first feature (F1) on the second image with respect to a fixed point of the towing vehicle (1); a step (S12) of calculating a first angle estimate (α1) characterizing the turning angle in the horizontal plane between;
- a second angle estimate (α2) is determined between the second feature (F2) on the first image and the second feature (F2) on the second image with respect to the fixed point of the towing vehicle (1); a step (S13) of calculating a second angle estimate (α2) characterizing the turning angle in the horizontal plane between
- calculating the yaw angle (YA) based on the first and second angle estimates (α1, α2) (S14);
A method for determining the yaw angle (YA) of a trailer (2) with respect to the longitudinal axis (LAV) of a towing vehicle (1), comprising: 2. The yaw angle (YA) of the trailer (2) relative to the vehicle (1) on the first or second image is zero or any known yaw angle (YA) that can be used as a reference angle. The method described in 1. Method according to claim 1 or 2, wherein the fixed point is the position of the camera (3) or the position of the tow ball (4). The step of calculating first and second angle estimates (α1, α2) includes calculating a ray (R) between the fixed point and the first and second features (F1, F2) in the first and second images. 4. A method according to any one of claims 1 to 3, comprising determining. 5. The method of claim 4, wherein camera calibration information is used to convert the position of at least one of the first feature (F1) and the second feature (F2) into a light beam. Any of claims 1 to 5 , wherein in the step of calculating the yaw angle (YA), in addition to the first and second features (F1, F2), at least one further feature of the trailer (2) is used. or the method described in paragraph 1. 7. A method according to any preceding claim, wherein the yaw angle (YA) is calculated by constructing a median value based on the at least two angle estimates. 8. Any of claims 1 to 7, wherein the yaw angle (YA) is calculated by setting an average value of the at least two angle estimates or by using a statistical method applied to the angle estimates. The method described in Section 1. determining an angular window , the angular window having an upper boundary and a lower boundary around a yaw angle (YA);
determining a set of features that lead to angle estimation within the angle window ;
Use the determined set of features for future yaw angle (YA) calculations
9. A method according to any preceding claim, further comprising the step of: 10. A method according to any one of claims 1 to 9, wherein the calculated yaw angle (YA) value is increased by a certain amount or a certain percentage in order to correct the underestimation. 11. A method according to any preceding claim, wherein the camera (3) is a rear view camera of the vehicle (1). A system for determining a yaw angle (YA) of a trailer (2) with respect to a longitudinal axis (LAV) of a towing vehicle (1), the system comprising:
The system is
comprising a camera (3) that photographs an image of the trailer (2), and a processing entity that processes the photographed image;
The system is
- taking at least a first and a second image of the trailer (2) with the camera (3), in which the orientation of the trailer (2) with respect to the vehicle (1) is different on at least two images; ,
- determining at least first and second features (F1, F2) of said trailer (2) visible on first and second images, said first and second features (F1, F2) being visible on said trailer (2); (2) determining at least first and second features (F1, F2) located at different positions;
- the first angle estimation (α1) is based on the first feature (F1) on the first image and the first feature (F1) on the second image with respect to a fixed point of the towing vehicle (1); calculating a first angle estimate (α1) characterizing the turning angle in the horizontal plane between
- a second angle estimate (α2) is determined between the second feature (F2) on the first image and the second feature (F2) on the second image with respect to the fixed point of the towing vehicle (1); calculating a second angle estimate (α2) characterizing the turning angle in the horizontal plane between;
- calculating the yaw angle (YA) based on the first and second angle estimates (α1, α2);
A system for determining a yaw angle (YA) of a trailer (2) with respect to a longitudinal axis (LAV) of a towing vehicle (1), further configured to perform. A vehicle comprising a system according to claim 12.