EP3822924B1

EP3822924B1 - System and method for recording field data in a vehicle

Info

Publication number: EP3822924B1
Application number: EP19209071.0A
Authority: EP
Inventors: Dirk Gehrke
Original assignee: Rohde and Schwarz GmbH and Co KG
Current assignee: Rohde and Schwarz GmbH and Co KG
Priority date: 2019-11-14
Filing date: 2019-11-14
Publication date: 2024-04-03
Anticipated expiration: 2039-11-14
Also published as: EP3822924A1

Description

The present invention relates to a testing system for autonomous vehicles, more specifically to a system for recording field data in a vehicle for later use in "field to lab" tests. In other words, the present invention relates to a data recorder for transportation scenarios, which allows for holistic data logging. The present invention in particular provides a system for obtaining radio frequency (RF) data and sensor data in a synchronized manner.
Simulations for autonomous vehicles focus on scenarios under various conditions. Worldwide tests are required to test autonomous vehicles in real-world scenarios. Long development and multiple testing cycles are required, wherein regulations and the growing complexity of autonomous vehicles require more and more complex tests. That is, automotive vendors continuously need to optimize evaluation- and development-processes of vehicles and need to meet regulatory laws. This is, for example, done by transitioning pure field tests to "field to lab" tests. During such tests, field data, which is typically obtained during a real life test scenario, is provided in a laboratory environment where a test vehicle can be exposed to the obtained real life field data. This can be done in a physical setup, or in a simulation.
Conventional solutions for example allow for identifying characteristics (i.e. field data) of a dynamic object in a physical environment of a vehicle, wherein the characteristics can then be used in a simulation of the behavior of the dynamic object and the vehicle in a simulated environment.
However, due to increased automation of autonomous vehicles, test scenarios are required for which the field data, which is obtained by conventional solutions, is not sufficient. Regulatory rules get more stringent due to the risk of autonomous driving and public safety aspects.
US 2018/088584 A1 discloses a sensor synchronization system for an autonomous vehicle.
US 2019/156600 A1 discloses a locomotive sensor system for monitoring engine and lubricant health.
US 2008/036617 A1 discloses an energy harvesting, wireless structural health monitoring system.
EP 3 090 354 A1 discloses methods and systems for synchronizing data received from multiple sensors of a device.
That is, there is lack of a system, which can obtain field data, which is sufficient for "field to lab" tests for autonomous vehicles.
Therefore, the object of the invention is to provide a system for recording field data in vehicles, wherein the field data includes RF data and sensor data, which are synchronized based on timestamps in the RF data and the sensor data. Moreover, the object is to provide an according method, computer program product and computer-readable storage medium.
The object is solved by the features of the first independent claim for the system and by the features of the second independent claim for the method. Further, it is solved by the features of an associated computer program product and by a computer-readable storage medium. The dependent claims contain further developments.
The present invention is defined by the independent claims. Additional features of the invention are provided in the dependent claims. In the following, parts of the description and drawings referring to embodiments which are not covered by the claims are not presented as embodiments of the invention, but as examples useful for understanding the invention.
This is beneficial, as a holistic simulation tool for autonomous vehicles can be provided. Optionally, such a tool can e.g. be equipped with visualization, management, and documentation functionality. In particular, this simulation tool can bring RF data and sensor data for real-world scenario simulation to a laboratory environment. In addition to only doing tests and simulations, which are only based on sensor data, wireless interference and anomaly simulations can be performed based on the synchronized RF data and sensor data, to increase the variety of test conditions and scenarios.
In addition to typical as well as unusual traffic situations, simulations can also cover safety threats like wireless spectrum interferences and anomalies, EMC interferences, jamming (flooding a vehicle with RF noise), spoofing (attacking a vehicle with false RF information).
In addition to standard test scenarios, unlimited individual tests can be covered with artificial intelligence (AI) based scenario creation. AI for example can inject and/or replace objects in a simulated environment, generate new environments, create wireless interference, or change a weather condition.
The present invention can also be used for data logging in autonomous vehicles to trace accident data for insurances and improvements.
The present invention is not only limited to be used in the automotive industry, but can also be applied in other transportation segments such as a train, a ship, urban air mobility, a plane, a drone, a motorbike, a bicycle, or autonomous driving.
Again in other words, the present invention allows brining real world RF scenarios to a laboratory environment, enabling a user to run the real life scenario in a virtual environment and if required add disturbing or additionally wanted signals to it.
The holistic data logging solution provided by the present invention is in particular beneficial as it covers all possible aspects by recording sensor data in real time, and brings them to the lab in addition to the RF data. Moreover, RF data and sensor data can be provided to the laboratory in a synchronized manner.
As vehicles do have a huge amount of sensors (e.g. Radar, Camera, Lidar, Pressure, Temperature, Humidity, Acceleration, Positioning, Air quality, Driver health condition), the present invention can be used for prototype development as well as for regulatory compliance tests in simulation tools.
In order to provide holistic scenario, recording wireless data and sensor data logging is combined in a single setup. This means that RF data (e.g. comprising I/Q data), all sensor data (radar, camera, lidar, pressure, temperature, humidity, acceleration, gyro, positioning (GPS or GNSS), air quality, driver health condition) as well as a processing data set (which is based on the above RF data and sensor data, e.g. in a synchronized manner) can be recorded in real time.
In particular, the field data, which is recorded in the vehicle, is external field data to which the vehicle is exposed. In particular, external field data is field data, which originates from a source not being part of the vehicle, and/or from a source outside the vehicle.
In particular, the vehicle is a car, a motorbike, a bicycle, a truck, a train, a helicopter, a drone, a plane, or a ship.
In particular, the RF data acquisition unit is configured to add a timestamp to the RF data. In particular, the digital data acquisition unit is configured to add a timestamp to the digital data.
Advantageously, the RF data acquisition unit is further configured to convert a received RF signal into an in-phase quadrature, IQ, signal, to obtain the RF data. This ensures that the RF data can be analyzed and processed in a more efficient and effective manner.
In particular, the RF data comprises the IQ signal. Advantageously and preferably, the system further comprises an antenna configured to receive the RF signal.
This ensures that the system can receive RF signals in a reliable manner.
Advantageously, the RF signal represents an electromagnetic environment of the vehicle.
This ensures that the simulation, which is based on the synchronized RF data and sensor data, is correctly supplied with a real world scenario recorded by the system.
In particular, the RF signal, which represents an electromagnetic environment of the vehicle, is an RF signal, which is of origin external to the vehicle. That is, the RF signal is not produced or generated by the vehicle.
Advantageously and preferably, the data collection unit is further configured to generate a synchronized dataset based on the synchronized RF data and digital data.
This ensures that the field data which is recorded by the system can be stored in the synchronized dataset, which can be read out later, e.g. in a "field to lab" test.
Advantageously and preferably, the system further comprises a data processing unit.
Advantageously and preferably, the data processing unit is further configured to receive, process and replay the synchronized dataset.
This ensures that the system effectively can be used in a "field to lab" test.
In particular, replaying the synchronized dataset includes outputting RF data and digital data based on the synchronized dataset.
Advantageously and preferably, the sensor data includes at least one of: radar data, image data, lidar data, pressure data, temperature data, air quality data, pulse frequency data, ultrasonic data, position data, time data.
This allows for various kinds of sensor data to be recorded by the system.
In particular, lidar data can also be called light detection and ranging data. In particular, lidar is a method for optical distance and velocity measurement and remote measurement of atmospheric parameters based on laser technology.
Advantageously and preferably, the radar data, the image data, the lidar data, the pressure data, the temperature data, the air quality data, the pulse frequency data, the ultrasonic data, the position data, or the time data comprises raw data and/or pre-processed data.
This allows for even more various kinds of sensor data to be recorded by the system.
In particular, pre-processed data includes data obtained by means of image detection and/or sensor fusion. For example, image detection can be performed on the image data, before it is provided to the digital data acquisition unit. For example, the above sensor data may include fused information, which is obtained from at least two different sensors.
Advantageously and preferably, the data collection unit comprises an RF to digital conversion unit configured to convert the received RF data into a digital signal.
This allows for optimized processing of the RF signal in the data collection unit.
In particular, the RF signal is converted to a digital signal when it is received by the data collection unit before other processing of the RF signal in the data collection unit. In particular, the further processing of the RF signal in the data collection unit is based on the converted digital signal of the RF signal.
Advantageously and preferably, the RF signal and/or the converted digital signal comprises at least one of: UKW data, MW data, GSM data, LTE data, 5G data, GPS data, V2X data.
This ensures that many kinds of RF signals can be represented in a digital manner for further processing in the system.
In particular, UKW data may also be called very high frequency, VHF, data. In particular, MW data may also be called medium frequency, MF, data.
Advantageously and preferably, the system is further configured to obtain at least one of: GPS information, real time clock, RTC, information, or GSM information; and the RF data acquisition unit is configured to add the timestamp to the received RF data based on the GPS information, RTC information and/or GSM information; and/or the digital data acquisition unit is configured to add the timestamp to the received digital data based on the GPS information, RTC information and/or GSM information.
This ensures that the timestamps can be provided based on reliable time sources.
A method according to the invention for recording field data in a vehicle comprises the steps defined by independent claim 11.
Advantageously, the method further comprises, by the RF data acquisition unit, converting a received RF signal into an in-phase quadrature, IQ, signal, to obtain the RF data.
Advantageously, the RF signal represents an electromagnetic environment of the vehicle.
The inventive method comprises the same advantages as the inventive system.
An inventive computer program product comprises instructions, which when the program is executed by a computer, cause the computer to carry out the steps of the above described inventive method or any of its advantageous implementation forms.
The inventive computer program product comprises the same advantages as the inventive device.
An inventive computer-readable storage medium comprises instructions which, when executed by a computer, cause the computer to carry out the steps of the above described inventive method or any of its advantageous implementation forms.
The inventive computer-readable storage medium comprises the same advantages as the inventive device.
An exemplary embodiment is now further explained with respect to the drawings by way of examples only, in which

FIG. 1: shows a schematic view of a system according to an embodiment;

FIG. 2: shows a schematic view of a system according to an embodiment in more detail;
FIG. 3: shows a schematic view of an operating scenario according to an embodiment;

FIG. 4: shows a schematic view of a method according to an embodiment not corresponding to the invention.

In the following, the structure and function of an embodiment of the system is described based on FIG. 1 in general, and based on FIG. 2 in more detail. In FIG. 3, an operating scenario is described. In view of FIG. 4, the function of an embodiment method is described.
FIG. 1 shows a system 100 for recording field data in a vehicle. As shown in FIG. 1, the system 100 comprises an RF data acquisition unit 101, a digital data acquisition unit 103 and a data collection unit 105. The RF data acquisition unit 101 is configured to obtain RF data 102. This is e.g. done by receiving an RF signal (e.g. by means of an antenna) and obtaining the RF data 102 based on the received RF signal. Optionally, this may also include generating a digital signal from the received RF signal, to obtain the RF data 102. The digital data acquisition unit 103 is configured to obtain digital data 104. The digital data 104 can, for example, be obtained from sensors, which are connected to the system 100 (which are not part of the system 100, however).
The data collection unit 105 is configured to receive the RF data 102 from the RF data acquisition unit 101 and is configured to receive the digital data 104 from the digital data acquisition unit 103. As it is shown in FIG. 1, the RF data 102 comprises a timestamp 106 and the digital data 104 comprises a timestamp 107. The timestamp 106, for example, is added to the RF data 102 in the RF data acquisition unit 101. The timestamp 107, for example, is added to the digital data 104 in the digital data acquisition unit 103. Once the data collection unit 105 receives the RF data 102 and the digital data 104, the data collection unit 105 synchronizes the RF data 102 and the digital data 104 based on the timestamp 106 and based on the timestamp 107.
The RF data 102 processed by the system 100 comprises data representing an in-phase and/or a quadrature component of an RF signal. As already mentioned, the digital data 104 processed by the system 100 represents sensor data.
Thereby, all kinds of available sensor data, which are generated in a vehicle, can be collected by the system 100. The sensor data may, for example, be obtained from a LIN/CAN bus of a vehicle. In parallel, wireless data and/or a wireless spectrum to which a vehicle (which comprises the system 100) is exposed, can be recorded. In turn, the recorded wireless data and the recorded sensor data can be combined to enable holistic real world scenario test simulations.
The data, which is recorded by the system 100, can for example be used in simulation software for testing autonomous vehicles in a laboratory environment with real world data. Such a simulation software may for example include scenario visualization with a 3D engine (e.g. unity). Optionally, AI generated test scenarios which are based on the real world data may enable unlimited real world scenario constellations. In addition, this allows for simulating and testing wireless interference and anomalies. The data, which is obtained by the system 100, can for example be provided by means of a cloud-based approach to facilitate scalability of parallel simulations.
FIG. 2 shows a schematic view of the system 100 in more detail. The system 100 as shown in FIG. 2 comprises all features and function of the system 100 as shown in FIG. 1. To this end, same reference signs as in FIG. 1 are used for same features in FIG. 2. Moreover, the system 100 as shown in FIG. 2 includes the following features:
The system 100 is configured to receive a RF signal 201 and to convert the received RF signal 201 into an in-phase quadrature (IQ) signal to obtain the RF data 102. The system 100 may optionally comprise and antenna 202, which is configured to receive the RF signal 201. The RF signal 201 which is received by the system 100 (e.g. by means of the antenna 202) represents an electromagnetic environment of the vehicle in which the system 100 is applied. That is, the RF signal 201 represents electromagnetic information to which the vehicle is exposed, however, which is not originated by the vehicle (or by parts of the vehicle) itself.
Optionally, the data collection unit 105 is configured to generate a synchronized data set 203 as it is shown in FIG. 2. The generation of the synchronized data set 203 is based on the RF data 102 and the digital data 104 which are synchronized by the data collection unit 105 based on the time stamp 106 and the time stamp 107.
As illustrated in FIG. 2, the system 100 optionally comprises a data processing unit 204, which is configured to receive the synchronized data set 203. Although this is not shown in FIG. 2, the synchronized data set 203 can also be stored on a storage medium and provided to other entities except the system 100 for further processing.
By means of the data processing unit 204 the synchronized data set 203 can be received, processed and replayed in the system 100 itself. That is, the system 100 provides a solution which is capable of recording synchronized wireless data and sensor data in a vehicle, and which is capable of replaying said data (which represents the real world scenario) in a laboratory environment (that is the synchronized dataset 203 is replayed).
As it is illustrated in FIG. 2, the data collection unit 105 optionally comprises a RF to digital conversion unit 205. The RF to digital conversion unit 205 is configured to convert the received RF data into a digital signal. Thereby, the RF data 102 can be further processed in a digital manner in the data collection unit 105.
Optionally, the system 100 is further configured to obtain at least one of: GPS information, real time clock (RTC) information, or GSM information. Based on this information, the RF data acquisition unit 101 can add the timestamp 106 to the received RF data 102, based on at least one of the GPS information, the RTC information, and/or the GSM information. Further optionally, the digital data acquisition unit 103 can add the timestamp 107 to the received digital data 104 based on at least one of the GPS information, the RTC information, and/or the GSM information.
FIG. 3 shows a schematic view of an operating scenario according to an embodiment. FIG. 3 is, however, just for illustrating the operational principle and therefore does not show all features which are e.g. described in view of FIG. 1 and FIG. 2.
As shown in FIG. 3, sensor data 104 of several sensors is obtained. That is, data from a radar sensor 301, a camera 302, a LiDAR sensor 303 and optionally from several other sensors 304 is obtained. The obtained data is fused, that is, the data from the several sensors is combined to one kind of digital data 104. FIG. 3 also shows a wireless spectrum which is recorded to obtain the RF data 102. The RF data 102 and the digital data 104 are synchronized based on timestamps in the data, which forms the real world data shown in FIG. 3. The real world data (that is the synchronized RF data 102 and digital data 104) can be output as a synchronized data set 203 by the system 100. The synchronized data set 203 may then be used for simulation in a laboratory environment. The synchronized data set 203 may be enriched with additional AI generated scenarios based on the real world data and e.g. additional interferences, before it is used for simulation.
FIG. 4 shows a method 400 for recording field data in a vehicle. As illustrated in FIG. 4, the method comprises the step of obtaining 401, RF data 102, by a RF data acquisition unit 101. The method 400 further comprises the step of obtaining 402, digital data 104, by a digital data acquisition unit 103. The method 400 further comprises the step of receiving 403, by a data collection 105, RF data 102 from the RF data acquisition unit 101, and receiving 404, by the data collection unit 105, the digital data from the digital data acquisition unit 103. Further, the method 400 comprises the step of synchronizing 405, by the data collection unit 105, the RF data 102 and the digital data 104 based on a time stamp 106 in the RF data 102 and a time stamp 107 in the digital data 104. The digital data 104 represents sensor data and the RF data 102 comprises data representing an in-phase and/or a quadrature component of an RF signal.
It is important to note that the inventive system and method very closely correspond. Therefore, all of the above said regarding the system is also applicable to the method and vice versa.

Claims

A system (100) for recording field data in a vehicle, wherein the system (100) comprises:
- a radio frequency, RF, data acquisition unit (101), configured to obtain RF data (102);

- a digital data acquisition unit (103), configured to obtain digital data (104);

- a data collection unit (105), configured to receive the RF data (102) from the RF data acquisition unit (101) and to receive the digital data (104) from the digital data acquisition unit (103);

wherein the data collection unit (105) is configured to synchronize the RF data (102) and the digital data (104) based on a timestamp (106) in the RF data (102) and a timestamp (107) in the digital data (104);

wherein the RF data (102) comprises data representing an in-phase and/or a quadrature component of an RF signal,

and the digital data (104) represents sensor data, wherein the RF data acquisition unit (101) is further configured to convert a received RF signal (201) into an in-phase quadrature IQ signal, to obtain the RF data (102),

wherein the RF signal (201) represents an electromagnetic environment of the vehicle.
The system (100) according to claim 1, wherein the system (100) further comprises an antenna (202) configured to receive the RF signal (201).
The system (100) according to any one of the preceding claims, wherein the data collection unit (105) is further configured to generate a synchronized dataset (203) based on the synchronized RF data (102) and digital data (104).
The system (100) according to any one of the preceding claims, wherein the system (100) further comprises a data processing unit (204).
The system (100) according to claim 4 when depending on claim 3, wherein the data processing unit (204) is further configured to receive, process and replay the synchronized dataset (203).
The system (100) according to any one of the preceding claims, wherein the sensor data (104) includes at least one of: radar data, image data, lidar data, pressure data, temperature data, air quality data, pulse frequency data, ultrasonic data, position data, time data.
The system (100) according to claim 6, wherein the radar data, the image data, the lidar data, the pressure data, the temperature data, the air quality data, the pulse frequency data, the ultrasonic data, the position data, or the time data comprises raw data and/or pre-processed data.
The system (100) according to any one of the preceding claims, wherein the data collection unit (105) comprises an RF to digital conversion unit (205) configured to convert the received RF data (102) into a digital signal.
The system (100) according to claim 8, wherein the RF signal and/or the converted digital signal comprises at least one of: UKW data, MW data, GSM data, LTE data, 5G data, GPS data, V2X data.
The system (100) according to any one of the preceding claims, further configured to obtain at least one of: GPS information, real time clock, RTC, information, or GSM information; and
wherein the RF data acquisition unit (101) is configured to add the timestamp (106) to the received RF data (102) based on the GPS information, RTC information and/or GSM information; and/or

wherein the digital data acquisition unit (103) is configured to add the timestamp (107) to the received digital data (104) based on the GPS information, RTC information and/or GSM information.
A method (400) for recording field data in a vehicle, wherein the method (400) comprises the steps of:
- obtaining (401), radio frequency, RF, data (102), by a RF data acquisition unit (101);

- obtaining (402), digital data (104), by a digital data acquisition unit (103);

- receiving (403), by a data collection unit (105), the RF data (102) from the RF data acquisition unit (101);

- receiving (404), by the data collection unit (105), the digital data (104) from the digital data acquisition unit (103);

- synchronizing (405), by the data collection unit (105), the RF data (102) and the digital data (104) based on a timestamp (106) in the RF data (102) and a timestamp (107) in the digital data (104);

wherein the digital data (104) represents sensor data and the RF data (102) comprises data representing an in-phase and/or a quadrature component of an RF signal,

wherein the method further comprises converting, by the RF data acquisition unit (101), a received RF signal (201) into an in-phase quadrature IQ signal, to obtain the RF data (102), wherein the RF signal (201) represents an electromagnetic environment of the vehicle.
A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method (400) of claim 11.
A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method (400) of claim 11.