GB2624479A

GB2624479A - Method and system for reverse parking an autonomous vehicle

Info

Publication number: GB2624479A
Application number: GB2304404.3A
Authority: GB
Inventors: Kannan Srividhya
Original assignee: Continental Autonomous Mobility Germany GmbH
Current assignee: Continental Autonomous Mobility Germany GmbH
Priority date: 2022-11-09
Filing date: 2023-03-27
Publication date: 2024-05-22
Also published as: GB202304404D0

Abstract

A parking method and system comprising, monitoring a sensor area 34 that surrounds the vehicle 10 with at least one sensor device 14 to obtain sensor data. Evaluating the sensor data using a machine learning model with a deep neural network in order to obtain object data that are indicative of the distance of an object 36 from the vehicle and/or the type of object detected. The machine learning model is trained for detecting 3D depth information and/or for object recognition; an advanced data communication unit 30 receiving the sensor data, object data and/or 3D depth information, acquiring vehicle dynamic data that are indicative of a motion of the vehicle, and generating control signals based on the sensor data, the object data and/or the 3D depth information, and the vehicle dynamic data that are configured to cause the vehicle to move into a parking spot 11. A vehicle to-everything V2X communication interface 32 receiving/sending information about the object detected, when the object is determined to come closer to the vehicle.

Description

DESCRIPTION

Method and system for reverse parking an autonomous vehicle

TECHNICAL FIELD

The invention relates to a method for reverse parking a vehicle, specifically an autonomous vehicle. The invention further relates to an associated system.

BACKGROUND

Sethakarn Prongnuch and Suchada Sitjongsataporn, "Exterior Car Parking Assistance Algorithm Based on Reconfigurable System for Future Industry", Journal of Mobile Multimedia, Vol 16 1-2, 203-220. doi: 10.13052/jmm1550-4646.161210, discloses a reconfigurable embedded system design by voice controlled parking assistance system for a prototype electric vehicle connected to a smartphone via Bluetooth. The proposed voice controlled exterior car parking assistance system is installed on a miniature electric vehicle where it is tested for both reverse parking and reverse parallel parking.

US 2019 / 0 384 304 Al discloses a path detection system for autonomous machines using deep neural networks. It provides rear-parking assistance for an autonomous vehicle using ultra sonic sensors, radar sensors to assist while parking the vehicle. Further, four fisheye cameras on the front, rear, and sides of the vehicle are used by providing a surround view of the environment around the vehicle. The machine learning model trains the collected sensor data which may include image data generated by one or more cameras of an autonomous vehicle.

DE 10 2017 100 259 Al discloses a method of parking a vehicle in a parking lot by generating steering commands for the vehicle while it is in the parking lot based on a population grid and plenoptic camera data. An autonomous park service system and method using plenoptic cameras are disclosed for reverse parking a vehicle. Cameras, ultrasound sensors, RADAR, LIDAR, and their combinations are used to detect the areas outside the vehicle. Further, the cameras are located at rear side and different sides of the vehicle. Also, the distance between the vehicle and objects around the vehicle is determined, such as curbs, pedestrians, other vehicles.

DE 11 2019 000 049 T5 discloses object detection and detection security for autonomous driving. Therein reverse parking assistance for autonomous vehicles uses a surround camera for obtaining scene information around the vehicle. Further, cameras in the front, rear, and side views are used to provide an all-round view around the vehicle. The objects are trained to determine their orientation that are present around the vehicle. The data is collected from the sensors and used to perform a safe parking maneuver. The environment around the vehicle can be provided to the user in the 3D view.

SUMMARY OF THE INVENTION

It is the object of the invention to improve reverse parking of vehicles.

The invention provides a method for reverse parking a vehicle, the method comprising: a) with at least one sensor device monitoring a sensor area that surrounds the vehicle to obtain sensor data; b) with a control unit: evaluating the sensor data using a machine learning model with a deep neural network in order to obtain object data that are indicative of the distance of an object from the vehicle and/or of an object type of the object, wherein the machine learning model is trained for detecting 3D depth information and/or for object recognition of the object; c) with an advanced data communication unit: receiving the sensor data, object data and/or 3D depth information, acquiring vehicle dynamic data that are indicative of a motion of the vehicle, and generating control signals based on the sensor data, the object data and/or the 3D depth information, and the vehicle dynamic data that are configured to cause the vehicle to move into a parking spot; and d) with a vehicle-to-everything (V2X) communication interface: receiving/sending information about the object detected in step b), when the object is determined to come closer to the vehicle.

Preferably, step a) comprises at least two sensor devices of a different sensor type, and each sensor device monitors the sensor area and generates sensor data.

Preferably, each sensor device is selected from a group consisting of a front camera, a rear camera, a side camera, two side cameras on opposite sides of the vehicle, an ultrasonic sensor device, a surround radar device, a 4D radar device, and an ultrasonic sensor array.

Preferably, in step b) the object type is determined by determining a height of the object above a base line, such as a ground surface or a road surface.

Preferably, in step b) the object type is determined by determining an azimuthal angular extent of the object.

Preferably, in step b) the control unit causes a warning signal to the driver based on the object data.

Preferably, in step c) the ADCU generates a driving status feedback signal that informs the driver about the current driving status.

Preferably, in step d) the object is determined to come closer by determining a change in distance of the object using the sensor data of two different points in time.

Preferably, in step d) the object is determined to come closer due to the object sending information that indicates the object is coming closer and receiving the information with the V2X communication interface.

The invention provides a parking assistance system for a vehicle, preferably an autonomous vehicle, the system comprising - at least one sensor device configured for monitoring a sensor area that surrounds the vehicle to obtain sensor data; - a control unit configured for evaluating the sensor data using a machine learning model with a deep neural network in order to obtain object data that are indicative of the distance of an object from the autonomous vehicle and/or of an object type of the object, wherein the machine learning model is trained for detecting 3D depth information and/or for object recognition of the object; - an advanced data communication unit configured for receiving the sensor data and acquiring vehicle dynamic data that are indicative of a motion of the vehicle, and generating control signals based on the sensor data and vehicle dynamic data that are configured to cause the vehicle to move into a parking spot; and - a vehicle-to-everything (V2X) communication interface configured for sending/receiving information about the object detected in step b), when the object is determined to come closer to the autonomous vehicle, wherein the parking assistance system is configured to perform a preferred method.

Preferably, the system comprises at least two sensor devices of a different sensor type, and each sensor device is configured for monitoring the sensor area and for generating sensor data.

Preferably, the control unit, preferably the machine learning model, is configured to determine the object type by determining a height of the object above a base line, such as a ground surface or a road surface. Preferably, the control unit, preferably the machine learning model, is configured to determine the object type by determining an azimuthal angular extent of the object. Preferably, the control unit is configured for causing a warning signal to the driver based on the object data.

Preferably, the ADCU is configured for generating a driving status feedback signal that informs the driver about the current driving status.

Preferably, the control unit, preferably the machine learning model, or the ADCU is configured to determine the object coming closer by determining a change in distance of the object using the sensor data of two different points in time. Preferably, the control unit, preferably the machine learning model, or the ADCU is configured to determine the object coming closer due to the object sending information that indicates the object is coming closer and receiving the information with the V2X communication interface.

The invention provides vehicle having a preferred parking assistance system.

A car accident while parking the car may be caused by the car driver, who may have blind spots around the car. There are limited solutions for parking assistance when the driver has limited information about the surroundings. The drivers may sense the reverse parking as a stressful situation, in particular in a narrow parking space. Additional, car parking is a major issue especially in crowded and cramped urban settings. Even for experienced drivers, the time to reverse park a car in a cramped space is typically large.

A combination of camera, radar and ultrasonic sensor can be used for smart reverse parking. The aim of the parking-assist system is to enhance parking safety with efficiency. Exterior car parking assistance is a kind of crash protection technology that causes a change in reducing car accidents and keeping drivers safe. Sensors can be, for example, surround view camera, stereo camera, surround radar, and 4D radar. 4D radar can be useful to provide 3D depth information of the scene.

Typically, the car has to navigate back and forth multiple times to park. Here the 3D view of the surrounding can help in easy parking of the vehicle. The steering movement can also be previewed to match the path of the reverse parking for fast and easy parking.

A surround radar like the SRR600 using 76-81GHz can be used for supporting backward and forward looking applications (incl. NCAP support) at an affordable cost.

4D is called 4D due to high-resolution long-range radar sensor capability that not only detects the distance, relative speed, and azimuth (angular measurement in a spherical coordinate system) of objects, but also their height above road level. Time is considered the fourth dimension (4D). These radars are not really mapping time, but rather utilize time to understand the 3D environment with regards to elevation. This can help a car decide, whether a stationary object up ahead is a person or a tree branch.

Unlike traditional radar solutions that are based around two to three transmitting antennas and three to four receiving antennas, 4D imaging radar may leverage a multiple-input, multiple-output (MIMO) 48-antenna array for high-resolution mapping of surroundings. The point cloud data output -a dataset that represents objects -combined with a wide azimuth-elevation field of view (FOV) allows detection and tracking with greater accuracy for situations such as a traffic jam under a bridge.

Surround view cameras and rear view cameras like RVS3xx / 5VC216 -195° FoV and RVS217 /218 -195° FoV can detect additional information for fusion with other parking systems and supports also trailer functions.

A surround view can be offered in different automotive segments. The basic version offers a pure 360-degree 3D image to assist in parking and maneuvering at low speeds. In contrast to existing systems, it may employ only two cameras: one in the front mounted in the grill and another in the rear, e.g., mounted near the license plate. This entry-level system is especially suited for use in the compact-car segment due to its attractive cost structure. Still other levels can be available, all the way up to an intelligent, active surround view system.

The system may have four cameras -typically in the front, in the rear and on the outside rear view mirrors -that monitor the sensing area all around the vehicle. The system may also recognize pedestrians, warn the driver or even stop the vehicle in critical situations. The system is particularly suited to city driving because, for example, it is able to recognize nearby pedestrians earlier. The system is also capable of capturing other vehicles crossing the path of the car, holding a lane and even recognizing curbing.

The ultrasonic sensor for parking is a sensor dedicated to be used in low speed maneuvering situations (speed < 10 km/h). The ultrasonic sensor can measure the time of flight to an object (echo-localization principle). The sensor may send an ultrasound echo (wave) and receives a signal (wave reflection by an object) after a certain period of time. The time of flight is converted to a measured distance.

A machine learning model having a deep neural network can be trained for 3D depth information and further detection on the scene to warn the driver of obstacles or closer objects in the surrounding.

The advanced data communication unit (ADCU) can receive data from multiple sensors, e.g., cameras, radars, ultrasonic, vehicle-to-everything (V2X) communication interfaces, and/or may acquire vehicle dynamic data from a virtual computing unit (VCU) so that it can support decision-making and safe parking. The outputs of the ADCU can be used for the driving status feedback, vehicle control, and various autonomous driving features. V2X can be a nearby vehicle or pedestrian which can send information if it comes closer.

The user can visualize the rear information in a 3D world and navigation path controlled by an app (in the car or smartphone). The 3D scene information can help in easier reverse parking without much effort and allows to avoid accidentally hitting an object in the surrounding.

In contrast to the known approaches, the invention uses different types of complementary data. In particular a combination of non-imaging sensors, imaging sensors, such as cameras, stereo cameras, surround view and/or 4D image radar data is used for reverse parking. With this it is possible to estimate the depth information of the objects around the vehicle. The depth information is preferably estimated using a deep neural network. The deep neural network can also be used for further detecting objects and warning the driver of obstacles or objects close to or getting closer within the surrounding. Furthermore, improvement can be achieved by sending/receiving information about a nearby object, such as a vehicle or pedestrian, when the object comes closer to the vehicle. The ADCU allows support of decision-making and safe parking. The ADCU may also output driving status feedback and various autonomous driving signals. The ADCU can also visualize the information in a 3D world. The ADCU may also show a navigation path controlled by the application.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are described in more detail with reference to the accompanying schematic drawings.

Fig. 1 depicts an embodiment of reverse parking Fig. 2 depicts an embodiment of reverse parallel parking; Fig. 3 depicts a vehicle interior during parking; and Fig. 4 depicts an embodiment of a control unit.

DETAILED DESCRIPTION OF EMBODIMENT

Referring to Fig. 1, an autonomous vehicle 10 is shown during reverse parking into a parking spot 11. The autonomous vehicle 10 includes a parking assistance system 12 that supports the reverse parking.

The parking assistance system 12 comprises a plurality of sensor devices 14. A first sensor device may be a front camera 16. A second sensor device may be a rear camera 18. A third and fourth sensor device may each be a side camera 20, that is preferably arranged on a sideview mirror. Furthermore, further sensor devices may be ultrasonic sensor devices 22. It is also possible that a sixth sensor device is a surround radar device 24. A seventh sensor device may be a 4D radar device 26. An eighth sensor device may be a vehicle-to-everything (V2X) communication interface 32 (to be described in more detail). The sensor devices 14 generate sensor data.

The parking assistance system 12 comprises a control unit 28. The control unit 28 includes a machine learning model having a deep neural network. The machine learning model is trained for object recognition and/or extraction of 3D depth information from the sensor data.

The parking assistance system 12 comprises an advanced data communication unit (ADCU) 30. The ADCU 30 is preferably adapted to receive the sensor data gathered by the sensor devices 14. The ADCU 30 may be configured to acquire vehicle dynamic data that are indicative of a motion of the vehicle. Vehicle dynamic data include, but are not limited to, vehicle speed, vehicle acceleration, and steering angle.

The ADCU 30 can generate a control signal based on the sensor data and the vehicle dynamic data. The control signal is such that the vehicle 10 is guided from its current position into the parking spot 11.

The parking assistance system 12 comprises a vehicle-to-everything (V2X) communication interface 32. The V2X communication interface 32 can connect to objects, such as to a pedestrian or rather their smartphone or to other vehicles.

In the following an embodiment of a reverse parking process is described in more detail.

The vehicle 10 determines its current location and gathers sensor data within a sensor area 34 that is defined to surround the vehicle 10.

The sensor data can be received by the control unit 28 and processed by the machine learning model to obtain object data. As indicated in Fig. 1, the machine learning model may recognize various objects 36 in the vicinity of the vehicle 10. The objects 36 are recognized, for example, as other vehicles 38, an obstacle 40 and a curb 42.

The sensor data can be received by the ADCU 30. The ADCU 30 may acquire vehicle dynamic data. Based upon the sensor data and/or the vehicle dynamic data, the ADCU 30 generates control signals that cause the vehicle 10 to follow a path towards the parking spot. The ADCU 30 may also display information to the driver, such that they are able to monitor the parking process and may intervene if necessary.

If during the parking process one of the objects 36 that also has a V2X. such as another vehicle 38, comes closer towards the vehicle 10, the object 36 uses its V2X to send information indicating the motion to the V2X communication interface 32.

In response to receiving the information with the V2X communication interface, the ADCU 30 may modify the control signal for the vehicle 10 and/or may display a warning to the driver.

The reverse parking process may be interrupted or prioritized, and subsequently finished appropriately.

Referring to Fig. 2, another embodiment of reverse parking in the form of reverse parallel parking is shown. For sake of brevity, this embodiment is only described insofar as it is different from the previous embodiment. The vehicle 10 is reverse parallel parking into the parking spot 11 similarly to the first embodiment.

As indicated in Fig. 2, the machine learning model may recognize various objects 36 in the vicinity of the vehicle 10. The objects 36 are recognized, for example, as other vehicles 38 and a curb 42.

Again, if during the parking process one of the objects 36 that also has a V2X, such as another vehicle 38, comes closer towards the vehicle 10, the object 36 uses its V2X to send information indicating the motion to the V2X communication interface 32.

The reverse parallel parking process may be interrupted or prioritized, and subsequently finished appropriately.

Referring to Fig. 3, an interior view of the vehicle 10 is shown. A display 44 may show a 3D image 46 of the surroundings of the vehicle 10. The 3D image 46 is generated by the ADCU 30 based on the sensor data, the vehicle dynamic data and/or the object data that were determined by the machine learning model. The display 44 may also be used to display warnings generated by the ADCU 30.

Referring to Fig. 4, a schematic view of the control unit 28. Sensor data in the form of image data 47, gathered by the cameras 16, 18, 20, for example, and 4D radar depth image data 48, gathered by the 4D radar device 26, for example, are fed to the machine learning model 50. The machine learning model 50 includes a deep neural network. The deep neural network has a feature extraction portion 52, a cost volume portion 54, and a cost aggregation portion 56, as is well known in the art.

The deep neural network is trained to perform object recognition and/or distance determination based on the input data. Preferably, the deep neural network can also be trained to generate 3D image data 58 that can be visualized using the ADCU 30. The ADCU 30 may display the 3D image data as the 3D image 46.

With the method and system described herein, a driver can be supported in reverse (parallel) parking of the vehicle 10.

REFERENCE SIGNS

autonomous vehicle 11 parking spot 12 parking assistance system 14 sensor device 16 front camera 18 rear camera side camera 22 ultrasonic sensor device 24 surround radar device 26 4D radar device 28 control unit advanced data communication unit (ADCU) 32 vehicle-to-everything (V2X) communication interface 34 sensor area 36 object 38 other vehicle obstacle 42 curb 44 display 46 3D image 47 image data 48 4D radar depth image data 50 machine learning model 52 feature extraction portion 54 cost volume portion 56 cost aggregation portion 58 3D image data

Claims

CLAIMS1. A method for reverse parking a vehicle (10), the method comprising: a) with at least one sensor device (14): monitoring a sensor area (34) that surrounds the vehicle (10) to obtain sensor data; b) with a control unit (28): evaluating the sensor data using a machine learning model (50) with a deep neural network in order to obtain object data that are indicative of the distance of an object (36) from the vehicle (10) and/or of an object type of the object (36), wherein the machine learning model (50) is trained for detecting 3D depth information and/or for object recognition of the object (36); c) with an advanced data communication unit (30): receiving the sensor data, object data and/or 3D depth information, acquiring vehicle dynamic data that are indicative of a motion of the vehicle (10), and generating control signals based on the sensor data, the object data and/or the 3D depth information, and the vehicle dynamic data that are configured to cause the vehicle (10) to move into a parking spot (11); and d) with a vehicle-to-everything (V2X) communication interface (32): receiving/sending information about the object (36) detected in step b), when the object is determined to come closer to the vehicle (10).
2. The method according to claim 1, wherein step a) comprises at least two sensor devices (14) of a different sensor type, and each sensor device (14) monitors the sensor area (34) and generates sensor data.
3. The method according to any of the preceding claims, wherein each sensor device (14) is selected from a group consisting of a front camera (16), a rear camera (18), a side camera (20), two side cameras (20) on opposite sides of the vehicle (10), an ultrasonic sensor device (22), a surround radar device (24), a 4D radar device (26), and an ultrasonic sensor array.
4. The method according to any of the preceding claims, wherein in step b) the object type is determined by determining a height of the object (36) above a base line, such as a ground surface or a road surface; and/or wherein in step b) the object type is determined by determining an azimuthal angular extent of the object (36).
5. The method according to any of the preceding claims, wherein in step b) the control unit (28) causes a warning signal to the driver based on the object data.
6. The method according to any of the preceding claims, wherein in step c) the ADCU (30) generates a driving status feedback signal that informs the driver about the current driving status.
7. The method according to any of the preceding claims, wherein in step d) the object (36) is determined to come closer by determining a change in distance of the object (36) using the sensor data of two different points in time.
8. The method according to any of the preceding claims, wherein in step d) the object (36) is determined to come closer due to the object (36) sending information that indicates the object (36) is coming closer and receiving the information with the V2X communication interface (32).
9. A parking assistance system (12) for a vehicle (10), the system comprising - at least one sensor device (14) configured for monitoring a sensor area (34) that surrounds the vehicle (10) to obtain sensor data; - a control unit (28) configured for evaluating the sensor data using a machine learning model (50) with a deep neural network in order to obtain object data that are indicative of the distance of an object (36) from the autonomous vehicle (10) and/or of an object type of the object (36), wherein the machine learning model (50) is trained for detecting 3D depth information and/or for object recognition of the object; - an advanced data communication unit (30) configured for receiving the sensor data and acquiring vehicle dynamic data that are indicative of a motion of the vehicle (10), and generating control signals based on the sensor data and vehicle dynamic data that are configured to cause the vehicle (10) to move into a parking spot (11); and - a vehicle-to-everything (V2X) communication interface (32) configured for sending/receiving information about the object (36) detected in step b), when the object (36) is determined to come closer to the autonomous vehicle (10), wherein the parking assistance system (12) is configured to perform a method according to any of the preceding claims.
10. The system (12) according to claim 9, wherein the system comprises at least two sensor devices (14) of a different sensor type, and each sensor device (14) is configured for monitoring the sensor area (34) and for generating sensor data.
11. The system (12) according to claim 9 or 10, wherein each sensor device (14) is selected from a group consisting of a front camera (16), a rear camera (18), a side camera (20), two side cameras (20) on opposite sides of the vehicle (10), an ultrasonic sensor device (22), a surround radar device (24), a 4D radar device (26), and an ultrasonic sensor array.
12. The system (12) according to any of the claims 9 to 11, wherein the control unit (28), preferably the machine learning model (50), is configured to determine the object type by determining a height of the object (36) above a base line, such as a ground surface or a road surface; and/or the object type by determining an azimuthal angular extent of the object (36); and/or wherein the control unit (28) is configured for causing a warning signal to the driver based on the object data.
13. The system (12) according to any of the claims 9 to 12, wherein the ADCU (30) is configured for generating a driving status feedback signal that informs the driver about the current driving status.
14. The system (12) according to any of the claims 9 to 13, wherein the control unit (28), preferably the machine learning model (50), or the ADCU (30) is configured to determine the object (36) coming closer by determining a change in distance of the object (36) using the sensor data of two different points in time; and/or wherein the control unit (28), preferably the machine learning model (50), or the ADCU (30) is configured to determine the object (36) coming closer due to the object (36) sending information that indicates the object (36) is coming closer and receiving the information with the V2X communication interface (32).
15. A vehicle (10) having a parking assistance system (12) according to any of the claims 9 to 14.