DE102020123808A1 - Method and system for acquiring environmental data to ensure environmental recognition - Google Patents

Abstract

The present disclosure relates to a method for capturing environmental data to ensure environmental recognition. The method includes providing a first vehicle having a first sensor assembly and a second vehicle having a second sensor assembly; a detection, by the first sensor arrangement of the first vehicle, of first surroundings data; a detection, by the second sensor arrangement of the second vehicle, of second environment data, the first environment data and the second environment data being synchronized in time; and a merging of the first environment data and the second environment data by a central unit for safeguarding an environment detection.

Description

The present disclosure relates to a method for capturing environmental data to validate environmental recognition, a storage medium for executing the method, and a system for capturing environmental data to validate environmental recognition. The present disclosure relates in particular to a dynamic sensor network for expanding the field of vision for safeguarding the environment detection.

State of the art

The development of driver assistance functions, for example for (partially) autonomous driving, is becoming increasingly important. In order to enable driver assistance systems to perceive the environment as precisely as possible, sensor measurements from various sensor types such as cameras, LiDAR and radars can be used. Each sensor type is subject to different physical measurement principles. In reality, the vehicle environment consists of many different objects/subjects such as different vehicle types, people and animals, static objects such as garbage cans, traffic control systems, natural objects and terrain. Since the perception of all these objects/subjects also depends on the measuring principle used, it is difficult to ensure a precise perception of the surroundings.

In order to secure the recognition of the environment in the vehicle, a mapping (ground truth) of the vehicle environment that is as accurate and reliable as possible is required for the comparison. One approach to generating ground truth is based on offline processing of recorded sensor data. It was observed that the detection quality is still largely determined by the sensor resolution (e.g. lidar), which decreases with distance. For a further improvement in the ground truth, there would also have to be a higher density of sensor detections from more distant objects. One possible approach for this is the use of high-resolution sensors. However, these are expensive and cannot solve the problem related to occluded objects. In addition, retrofitting a roof structure for each vehicle to be protected is not desirable.

Disclosure of Invention

It is an object of the present disclosure to specify a method for acquiring environment data to validate environment recognition, a storage medium for executing the method, and a system for acquiring environment data to validate environment recognition, which improves validation of environment recognition, in particular for automated drivers vehicles.

This object is solved by the subject matter of the independent claims. Advantageous configurations are specified in the dependent claims.

According to an independent aspect of the present disclosure, a method for capturing environmental data for safeguarding environmental recognition is specified. The method includes providing a first vehicle having a first sensor assembly and a second vehicle having a second sensor assembly; a detection, by the first sensor arrangement of the first vehicle, of first surroundings data; a detection, by the second sensor arrangement of the second vehicle, of second environment data, the first environment data and the second environment data being synchronized in time; and a merging of the first environment data and the second environment data by a central unit to generate a reliable reference for safeguarding an environment detection.

According to the invention, the surroundings data of a vehicle are supplemented by surroundings data from another vehicle, so that an improved perception of the surroundings can take place, e.g. during test drives with test vehicles. For example, an object may be obscured from one vehicle but visible or detectable to the other vehicle. A more comprehensive perception of a traffic situation is thus made possible by the use of additional vehicles to extend the field of vision of an individual vehicle. The recorded data is e.g. merged offline without online communication at raw data level and made available for the digital mapping of the environment (e.g. object data, lane information, etc.).

In particular, due to the extended sensor field of view, a more accurate and reliable reconstruction of the environment can be carried out. The “ground truth” obtained in this way allows weak points in online functions to be identified more easily by means of a comparison, which means that the environment recognition in (partially) autonomous vehicles can be more reliably safeguarded.

The term vehicle includes cars, trucks, buses, mobile homes, motorcycles, etc., which are used to transport people, goods, etc. In particular, the term includes motor vehicles for passenger transport. The first vehicle and the second vehicle (and the at least one additional Vehicle) can in particular be test vehicles that are used in the development of driver assistance systems for automated driving.

Preferably, the first vehicle and the second vehicle are dedicated reference vehicles. In particular, dedicated reference vehicles, each equipped with improved reference sensors, can be made available to validate different vehicle derivatives. This offers a high degree of flexibility for validation, as the field of view of vehicles in which only one series sensor setup is installed can be expanded directly using a dedicated reference vehicle. In order to provide a comprehensive data base, several reference vehicles in the vicinity of the vehicle to be secured can also be integrated, which achieve greater coverage.

The first environment data and the second environment data are preferably recorded while the first vehicle and the second vehicle are driving in a convoy. For example, the first vehicle can be a reference vehicle that undertakes a test drive to collect environment data that is used to validate the environment detection. The second vehicle can be a following vehicle that follows the reference vehicle and expands its environment recognition or its field of vision. Alternatively, the second vehicle can be a vehicle driving ahead, which drives in front of the reference vehicle and expands its environment recognition or its field of vision.

However, the present disclosure is not limited to the first vehicle and the second vehicle, and more than two vehicles with corresponding sensor arrangements can be involved in the method according to the invention. This makes the process scalable. The more vehicles involved, the more accurate the environment detection becomes. For example, a reference vehicle can be provided which is followed and/or preceded by one or more following vehicles in order to extend the field of view of the reference vehicle.

In particular, at least one additional vehicle, on which an additional sensor arrangement is present, can be provided. The further sensor arrangement can record further surroundings data, which are merged with the first surroundings data of the first vehicle and the second surroundings data of the second vehicle. The descriptions in this document apply equally to the use of a single additional vehicle or multiple additional vehicles to extend the field of vision.

The method preferably also includes determining a relative position between the first vehicle and the second vehicle, the merging of the first environment data and the second environment data including aligning the first environment data and the second environment data with one another based on the relative position. In particular, the relative positions of the additional vehicles to the vehicle under consideration should be known in order to be able to align the raw data from the sensors of the individual vehicles with one another offline. Various methods can be used to reliably determine these positions, which are explained below. The methods can be used individually or in combination.

In a first exemplary method, the relative position between the first vehicle and the second vehicle is determined based on an absolute localization of the first vehicle and the second vehicle, in particular using odometry and/or DGPS (“Differential Global Positioning System”).

In a second exemplary method, the relative position between the first vehicle and the second vehicle is determined based on a mutual localization of the first vehicle and the second vehicle using an environment sensor system of the respective vehicles.

In a third exemplary method, the relative position between the first vehicle and the second vehicle is determined based on a registration of raw environmental data that originate from the vehicles, based on a static environment of the vehicles.

In a fourth exemplary method, the relative position between the first vehicle and the second vehicle is determined based on a localization of the vehicles, in each case by registering raw data of the surroundings against a map of the surroundings.

The environmental data are preferably mapped onto a body-fixed coordinate system of the first vehicle. In particular, the raw sensor data of all vehicles can be mapped to the physical coordinate system of an individual vehicle to be secured. The subsequent processing generates the ground truth specifically for one vehicle.

The environmental data are preferably mapped onto a non-rotating global coordinate system. This allows for the generation of a ground truth that is less vulnerable to its own path estimation of an individual vehicle and at the same time is available to all vehicles involved. As a result, several vehicles can be protected at the same time, which in turn can increase the mileage to be protected while the processing effort remains the same.

Preferably, the first vehicle and the second vehicle each include an environment sensor system (i.e. the first sensor arrangement and the second sensor arrangement), which is set up to record the environment data. The environmental sensor system preferably includes at least one LiDAR system and/or at least one radar system and/or at least one camera and/or at least one ultrasound system and/or at least one laser scanner. The environment sensor system can provide the environment data (also referred to as “environment data”), which depict an area surrounding the vehicle.

The first sensor arrangement is preferably arranged or mounted on a roof of the first vehicle. In addition or as an alternative, the second sensor arrangement is arranged or mounted on a roof of the second vehicle.

According to another independent aspect of the present disclosure, a software (SW) program is provided. The SW program can be set up to be executed on one or more processors, and thereby to execute the method described in this document for capturing environment data for safeguarding environment recognition.

According to another independent aspect of the present disclosure, a storage medium is provided. The storage medium can include a SW program which is set up to be executed on one or more processors and to thereby execute the method described in this document for capturing environmental data in order to safeguard environmental recognition.

According to a further independent aspect of the present disclosure, software with program code for carrying out the method for capturing data of the surroundings for safeguarding a recognition of the surroundings is to be executed when the software runs on one or more software-controlled devices.

According to a further independent aspect of the present disclosure, a system for capturing environmental data for safeguarding environmental recognition is specified. The system comprises at least one processor unit that is set up to carry out the method for acquiring data about the surroundings in order to ensure recognition of the surroundings.

The central unit that receives the first environment data and the second environment data in order to perform the raw data fusion can be a server and/or backend of a vehicle manufacturer, for example.

The at least one processor unit or the processor is a programmable arithmetic unit, ie a machine or an electronic circuit that controls other elements according to the commands that have been passed and thereby advances an algorithm (process).

The first vehicle and the second vehicle (and the at least one other vehicle) can in particular be test vehicles that are used in the development of driver assistance systems for automated driving.

In the context of the document, the term "automated driving" can be understood as driving with automated longitudinal or lateral guidance or autonomous driving with automated longitudinal and lateral guidance. Automated driving can be, for example, driving on the freeway for a longer period of time or driving for a limited time as part of parking or manoeuvring. The term "automated driving" includes automated driving with any degree of automation. Exemplary degrees of automation are assisted, partially automated, highly automated or fully automated driving. These degrees of automation were defined by the Federal Highway Research Institute (BASt) (see BASt publication "Research compact", issue 11/2012).

With assisted driving, the driver constantly performs longitudinal or lateral guidance, while the system takes over the other function within certain limits. With semi-automated driving (TAF), the system takes over longitudinal and lateral guidance for a certain period of time and/or in specific situations, whereby the driver has to constantly monitor the system, as with assisted driving. With highly automated driving (HAD), the system takes over longitudinal and lateral guidance for a certain period of time without the driver having to constantly monitor the system; however, the driver must be able to take control of the vehicle within a certain period of time. With fully automated driving (VAF), the system can automatically handle driving in all situations for a specific application; a driver is no longer required for this application.

The four levels of automation mentioned above correspond to SAE levels 1 to 4 of the SAE J3016 standard (SAE - Society of Automotive Engineering). For example, highly automated driving (HAF) corresponds to level 3 of the SAE J3016 standard. Furthermore, SAE J3016 also provides SAE Level 5 as the highest degree of automation, which is not included in the BASt definition. SAE Level 5 corresponds to driverless driving, in which the system can automatically handle all situations like a human driver throughout the journey; a driver is generally no longer required.

An automated vehicle steers and/or brakes and/or accelerates independently based on a driving strategy. The driving strategy can be determined based on environmental data from the environmental sensors, road conditions, traffic conditions, weather conditions, etc. The specific driving strategy is implemented by the drive, the steering and/or the brakes.

character list

• 1 ein Flussdiagram eines Verfahrens zum Erfassen von Umgebungsdaten zur Absicherung einer Umfelderkennung gemäß Ausführungsformen der vorliegenden Offenbarung,
• 2A und 2B eine Erweiterung eines Sichtfelds eines Fahrzeugs durch wenigstens ein anderes Fahrzeug gemäß Ausführungsformen der vorliegenden Offenbarung,
• 3A bis C Methoden zur Lokalisierung von Fahrzeugen gemäß Ausführungsformen der vorliegenden Offenbarung,
• 4A ein Abbilden der Rohsensordaten aller Fahrzeuge auf das körperfeste Koordinatensystem eines einzelnen abzusichernden Fahrzeugs,
• 4B ein Abbilden der Rohsensordaten aller Fahrzeuge auf ein nichtdrehendes globales Koordinatensystem, und
• 5 einen Ablauf eines Verfahrens zum Erfassen von Umgebungsdaten zur Absicherung einer Umfelderkennung gemäß weiteren Ausführungsformen der vorliegenden Offenbarung.

Exemplary embodiments of the disclosure are shown in the figures and are described in more detail below. Show it:

• 1 a flowchart of a method for capturing environment data to ensure environment detection according to embodiments of the present disclosure,
• 2A and 2 B an extension of a field of view of a vehicle by at least one other vehicle according to embodiments of the present disclosure,
• 3A until C Methods for locating vehicles according to embodiments of the present disclosure,
• 4A a mapping of the raw sensor data of all vehicles to the body-fixed coordinate system of an individual vehicle to be secured,
• 4B mapping all vehicles' raw sensor data to a non-rotating global coordinate system, and
• 5 a sequence of a method for acquiring environmental data for safeguarding an environment detection according to further embodiments of the present disclosure.

Embodiments of the Disclosure

Unless otherwise noted, the same reference symbols are used below for the same elements and those with the same effect.

There is currently no generally applicable procedure for securing environment detection for autonomous driving. A potential approach lies in the comparison of the vehicle's own environment detection with an offline-processed environment detection. The advantage of offline processing lies in the consideration of future information for the state estimation. A roof structure with reference sensors can be used to further decouple the ground truth from the vehicle's own environment detection and to obtain better sensor data. The limitation here is that in order to protect different derivatives, the roof structure has to be converted to individual vehicles, which involves conversion work.

To generate the ground truth, only the sensor data of a single vehicle is currently evaluated. The intended use of several vehicles to provide the sensor data falls within the scope of Car2X approaches, which enable a vehicle to communicate online with other road users. Information relating to an environment can be exchanged between vehicles. However, the content is limited to high-level information such as object data. A transmission of raw sensor data is not possible due to the high amount of data and the limited transmission bandwidth. However, a data fusion based on raw data enables a much better utilization of the sensor data for environment recognition.

According to the invention, the surroundings data of a vehicle are supplemented by surroundings data from another vehicle, so that an improved perception of the surroundings can take place, e.g. during test drives with test vehicles. For example, an object may be obscured from one vehicle but visible or detectable to the other vehicle. A more comprehensive perception of a traffic situation is thus made possible by the use of additional vehicles to extend the field of vision of an individual vehicle. The recorded data is e.g. merged offline without online communication at raw data level and made available for the digital mapping of the environment (e.g. object data, lane information, etc.).

1 FIG. 12 schematically shows a flowchart of an exemplary method 100 for acquiring environmental data to validate environmental recognition according to embodiments of the present disclosure.

The method 100 can be implemented by appropriate software that is provided by one or more processors (e.g. a CPU) can be executed.

In block 110, the method 100 includes providing a first vehicle on which a first sensor arrangement is present and a second vehicle on which a second sensor arrangement is present; in block 120, first surroundings data are detected by the first sensor arrangement of the first vehicle; in block 130, the second sensor arrangement of the second vehicle detects second surroundings data, the first surroundings data and the second surroundings data being synchronized in time; and in block 140 a merging of the first surroundings data and the second surroundings data by a central unit for safeguarding a recognition of the surroundings.

This field of view expansion is in the 2A and 2 B represented graphically. The first vehicle 10 serves as a reference vehicle whose field of vision is to be expanded by at least one additional vehicle 20 . In this example it is shown that the first vehicle 10 detects an object 30 in its surrounding area (represented by the detection cone A). The object 30 is partially covered. The second vehicle 20 drives in front of the first vehicle and can also detect the object 30 in its surrounding area (represented by the detection cone B). The merging of the data surrounding the first vehicle 10 and the second vehicle 20 can thus expand a field of view of the first vehicle 10 by the second vehicle 20 .

In order to enable the first environment data and the second environment data to be merged by the central unit to ensure environment detection, the sensor data from the additional vehicles that expand the field of vision, for example of the first vehicle 10, are synchronized in terms of time with the vehicle under consideration. This is done, for example, using a GPS time stamp that each vehicle records using the built-in DGPS measurement sensors. The synchronization of the different sensor data within a vehicle can be carried out using the arrival time stamp in the respective data logger.

In order to be able to align the raw data from the sensors of the individual vehicles with one another offline, the relative positions of the additional vehicles to the vehicle under consideration should be known. Different methods can be used to reliably determine these positions, as described in 3A , 3B and 3C are shown. The methods can be used individually or in combination.

In a first exemplary method ( 3A) the relative position between the first vehicle and the second vehicle is determined based on an absolute localization of the first vehicle and the second vehicle, in particular using odometry and/or DGPS. The relative position between the first vehicle and the second vehicle is preferably determined independently of a direct visual condition.

In a second exemplary method ( 3B) the relative position between the first vehicle and the second vehicle is determined based on a mutual localization of the first vehicle and the second vehicle using an environment sensor system of the respective vehicles. For example, the vehicles involved are mutually localized using their lidar sensors and stored CAD models by registering point clouds (regardless of the quality of an existing DGPS signal).

In a third exemplary method ( 3C ) the relative position between the first vehicle and the second vehicle is determined based on a registration of raw environmental data originating from the vehicles, based on a static environment of the vehicles. For example, the lidar point clouds originating from different vehicles are registered with one another using the static environment.

In a fourth exemplary method ( 3C ) the relative position between the first vehicle and the second vehicle is determined based on a localization of the vehicles, in each case by registering raw data from the surroundings against a map of the surroundings. For example, the vehicles are localized by registering the lidar point clouds against a highly accurate offline map of the area.

In some embodiments, the raw sensor data of all vehicles can be mapped onto the body-fixed coordinate system of an individual vehicle to be protected, as is shown in the example in FIG 4A is shown. The subsequent processing generates the ground truth specifically for one vehicle.

The data can also be mapped to a non-rotating global coordinate system as shown in 4B is shown. This allows the generation of a ground truth that is less sensitive to the self-motion estimation of a single vehicle while contributing to all equivalent vehicles is available. As a result, several vehicles can be protected at the same time and the mileage to be protected can be increased with the same processing effort.

Dedicated reference vehicles, each equipped with improved reference sensors, can be made available to validate different vehicle derivatives. This offers a high degree of flexibility for validation, as the field of view of vehicles in which only one series sensor setup is installed can be expanded directly using a dedicated reference vehicle. In order to provide a comprehensive data base, several reference vehicles in the vicinity of the vehicle to be secured can also be integrated, which achieve greater coverage.

5 FIG. 1 shows a sequence of a method for capturing environment data for safeguarding an environment detection according to further embodiments of the present disclosure.

According to the invention, the surroundings data of a vehicle are supplemented by surroundings data from another vehicle, so that an improved perception of the surroundings can take place, e.g. during test drives with test vehicles. For example, an object may be obscured from one vehicle but visible or detectable to the other vehicle. A more comprehensive perception of a traffic situation is thus made possible by the use of additional vehicles to extend the field of vision of an individual vehicle. The recorded data is e.g. merged offline without online communication at raw data level and made available for the digital mapping of the environment (e.g. object data, lane information, etc.).

Although the invention has been illustrated and explained in more detail by means of preferred exemplary embodiments, the invention is not restricted by the disclosed examples and other variations can be derived therefrom by a person skilled in the art without departing from the protective scope of the invention. It is therefore clear that a large number of possible variations exist. It is also understood that the embodiments given as examples are really only examples and should not be construed as limiting in any way the scope, applications or configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to concretely implement the exemplary embodiments, whereby the person skilled in the art, knowing the disclosed inventive concept, can make a variety of changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without having to Leaving the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

Claims (10)

Method (100) for capturing environment data to ensure environment detection, comprising: Providing (110) a first vehicle on which a first sensor arrangement is present and a second vehicle on which a second sensor arrangement is present, Acquisition (120), by the first sensor arrangement of the first vehicle, of first environment data, Detection (130), by the second sensor arrangement of the second vehicle, of second environment data, the first environment data and the second environment data being synchronized in time; and Merging (140) the first environment data and the second environment data by a central unit to secure an environment detection. The method (100) after claim 1 , wherein the first environment data and the second environment data are recorded while the first vehicle and the second vehicle are driving in a convoy. The method (100) after claim 1 or 2 , further comprising: determining a relative position between the first vehicle and the second vehicle, wherein the merging (140) of the first environment data and the second environment data comprises aligning the first environment data and the second environment data to each other based on the relative position. The method (100) after claim 3 , wherein the determination of a relative position between the first vehicle and the second vehicle is based on at least one of the following aspects: - an absolute localization of the first vehicle and the second vehicle, in particular using odometry and/or DGPS; and/or mutual localization of the first vehicle and the second vehicle using an environment sensor system of the respective vehicles; and/or - a registration of raw environmental data originating from the vehicles, based on a static environment of the vehicles; and/or - a localization of the vehicles via a registration of raw environmental data against an environmental map. The method (100) after claim 4 , wherein the environmental data are mapped onto a body-fixed coordinate system of the first vehicle. The method (100) after claim 4 , where the environment data is mapped to a non-rotating global coordinate system. The method (100) according to one of Claims 1 until 6 , wherein the first sensor arrangement is arranged on a roof of the first vehicle, and/or wherein the second sensor arrangement is arranged on a roof of the second vehicle. The method (100) according to one of Claims 1 until 7 , wherein the first vehicle and the second vehicle are dedicated reference vehicles. A storage medium comprising a software program arranged to be executed on one or more processors, and thereby performing the method (100) according to any one of Claims 1 until 8th to execute. System for capturing environmental data to secure an environment detection, comprising at least one processor unit that is set up to the method (100) according to one of Claims 1 until 8th to execute.

Images