DE102020210510A1 - Method and device for generating a data index based on raw data from a driver assistance system of a motor vehicle - Google Patents

Abstract

Method for generating a data index based on raw data from a driver assistance system of a motor vehicle, having the steps: detecting (1) at least one manual annotation of raw data by means of a user interface (9); Recording (2) at least one automatic annotation of raw data by means of an electronic data interface (10); Transmitting (4) the at least one manual annotation and the at least one automatic annotation to a database which is designed to store and/or merge (5) the at least one transmitted manual annotation and the at least one transmitted automatic annotation in this way, see above that a data index is created, and a corresponding device for creating such a data index.

Description

The invention relates to a method, a computer program product and a device for generating a data index based on raw data from a driver assistance system of a motor vehicle.

The development and validation of driver assistance systems in vehicles are becoming increasingly complex and expensive, especially with new functions. The amount of data generated reaches a size that can no longer be surveyed or efficiently evaluated using traditional methods.

Currently, individual annotations/labels of data, such as test data, are made and stored separately from each other. For example, GPS paths/tracks are recorded with manual annotations and saved locally, or special events such as automatic emergency braking (AEB) are automatically transferred to cloud storage. Therefore, there are several different, independent systems that have to be maintained in parallel and do not allow efficient indexing of the data or an overview of all data.

For example, from the document DE 10 2018 000 101 A1 a control system and a control method are known in which, after a predetermined time interval has elapsed, a data set with environment data, position data, speed data and time information is repeatedly created and transmitted to a database which can store and provide the transmitted data sets.

The object of the invention is to functionally improve a method and computer program product mentioned at the outset. In addition, the invention is based on the object of improving the structure and/or functionality of a device mentioned at the outset.

The task is solved with a method having the features of claim 1. The task is also solved with a computer program product with the features of claim 11. The task is also solved with a device with the features of claim 12. Advantageous embodiments and/or developments are the subject of the dependent claims.

A method for generating a data index based on raw data from a driver assistance system of a motor vehicle can include the step of detecting at least one manual annotation of raw data using a user interface. The data index can be or have an indexing, for example of data such as information. The raw data can be recorded and/or made available by the driver assistance system, for example using sensors. The raw data can include environment data and/or vehicle data and/or position data and/or speed data and/or time information. The sensors can be camera sensors, radar sensors, lidar sensors or the like. An annotation can have annotation data or be formed from it. Annotation can also be referred to as annotation data. An annotation can be a label. An annotation can be a data set, for example an annotation data set. A manual annotation can be made by a user, for example by manually selecting and/or manually entering an annotation. The user interface can be designed to transmit and/or provide an annotation, for example one that has been entered and/or selected manually. The user interface can be a human-machine interface. The user interface may include an input device such as a keyboard or the like. The user interface can be designed for entering and/or selecting data and/or information and/or at least one annotation. The user interface may include a display device. The display device can be a screen, such as a display. The display device can be a touch-sensitive screen, such as a touch display, for example. The display device can be designed to display information and/or operate an application, for example to record at least one manual annotation. The user interface and/or display device can be embodied as or implemented in a mobile device such as a mobile phone, smartphone or tablet. The user interface and/or display device may be part of a computer or computer system.

The method can also include the step of acquiring at least one automatic annotation of raw data using an electronic data interface. The automatic annotation can be an annotation that is generated automatically, for example based on the raw data. The automatic annotation can be an ad hoc annotation or an annotation generated beforehand, for example based on raw data. The automatic annotation can be generated in the motor vehicle or outside of the motor vehicle. The automatic annotation can take place or be generated by an electronic system, such as an arithmetic unit or control unit, for example a computer or processor, or the like. The electronic data interface can be in the electron ic system and / or implemented in the processing unit. The electronic data interface can provide at least one annotation, for example one that is generated automatically. The electronic data interface can be designed to transmit and/or provide an annotation, for example an annotation that is generated automatically.

The at least one manual annotation and/or the at least one automatic annotation can be recorded separately and/or in addition to the raw data. The raw data can be stored and/or treated and/or processed separately from the annotations. The raw data can be transmitted to local and/or external storage, such as cloud storage.

The method may further include the step of: submitting the at least one manual annotation and the at least one automatic annotation to a database. The database can be a database for a data index or can be designed to store and/or provide the data index. The database can be designed to store and/or merge the at least one transmitted manual annotation and the at least one transmitted automatic annotation in such a way that a data index is created. The data index may be a merged index/data index. The method can include the step: storing and/or merging the at least one, in particular transmitted, manual annotation and the at least one, in particular transmitted, automatic annotation in the database. The data index can be stored and/or implemented in the database. The data index may include annotations, such as manual and/or automatic annotations. The database may be configured to store the annotations and/or the data index and/or provide stored annotations. The database can be designed to merge the annotations. An electronic system, such as an arithmetic unit or control unit, for example a computer or processor, or the like can be used to transmit the at least one manual annotation and/or the at least one automatic annotation to the database and/or store it therein and/or to merge into it. The database can have at least one physical data store and/or a database logic unit. The database logic unit can be designed to store and/or store and/or merge the annotations or annotation data records in a structured manner in the database and/or retrieve them from the database.

In the method, at least one, for example additional, annotation of raw data can be retrieved from a local memory and/or from a cloud memory. A communication module, for example a receiving module and/or transmitting module, can serve or be provided for this purpose. In addition to the at least one manual annotation and the at least one automatic annotation, this annotation can be retrieved and/or recorded and/or transmitted to the database for storage and/or merging. The method can include the step: storing and/or merging the at least one, in particular transmitted, manual annotation, the at least one, in particular transmitted, automatic annotation and the additionally retrieved annotation in the database. A data interface can be provided for detecting the additional annotation. The annotation retrieved from local storage and/or cloud storage may be manual and/or automatic annotation. The motor vehicle and/or a control system of the motor vehicle can have the local memory. The cloud storage can be part of an external data center and/or database server. Annotations or annotation datasets can be stored on the local storage and/or the cloud storage. The local storage and/or the cloud storage can provide annotations or annotation datasets, for example via a data interface. The data interface can have a transmission module, for example for wireless transmission. The annotation retrieved from the local storage and/or from the cloud storage can be transmitted to the database for storage and/or merging of the retrieved annotation.

The database can be implemented inside or outside the motor vehicle, for example in a local database server present in the motor vehicle or in a database server implemented outside of the motor vehicle. The at least one manual annotation and the at least one automatic annotation can be transmitted to local storage, external storage or cloud storage, for example for storage in the database. The database can be implemented in an external memory, a cloud or a cloud memory. The local storage, external storage or cloud storage can be network storage.

The transmission to the database can be done through a wireless connection. A communication module or transmission module can serve or be provided for this purpose. The wireless connection can be a cellular connection or a satellite communication connection. The wireless connection can be established at least temporarily and/or for a defined period of time be. The wireless connection can be initiated by a manual or automatic command.

The method can be carried out while the motor vehicle is driving, for example a test drive. Selected steps of the method can also be carried out before and/or after a trip. For example, a motor vehicle can be connected to or connected to an external system, for example a cloud/cloud memory and/or a computing system and/or a control system, by means of a communication module.

In the method, an index of the data, such as a data index, in particular of annotations, can be created while driving, for example a test drive, of the motor vehicle from the fusion of various individual annotations/labels. These can be stored on an on-premise server or in a cloud, such as cloud storage. The various individual annotations can have at least one manual annotation and/or at least one automatic annotation and/or at least one annotation retrieved from an external storage device such as cloud storage and/or internet storage. The method can provide for the simultaneous use of data, such as annotations, from manual input, automatically generated data/annotations in the vehicle or externally and from the cloud and/or internet storage.

The respective annotation can include one or more, for example two or three, of the following information: reference sensor and/or reference sensor type (for example: GNSS, IMU, reference camera, reference radar, reference lidar, rain sensor or the like), reference and/or file name for the raw data (these can be stored and/or treated separately, for example), vehicle signal and/or vehicle sensor, continuous signal or event-based signal (e.g. triggered by automatic emergency braking, AEB, camera glare, undershooting or overshooting of a threshold), status information (for example S/W version, VIN, etc.), signal statistics, EDR data, manual annotation and/or manual labeling (e.g. by users, such as driver, co-driver, passenger, observer, developer, etc.), user information, local data or external and/or online data (e.g. local map or from a navigation system), online service ce or online service, cloud data or cloud storage data, online weather data (e.g. temperature, rain intensity, humidity, cloud cover, sunny sand, etc.), online map data (e.g. HD maps, etc.), offline maps.

The respective annotation and/or information can be recorded or recorded and/or stored and/or merged separately and/or in addition to the raw data. The information can be used as input. At least two or more of the aforementioned items of information can be stored and/or merged to create the indexing, for example indexing of the data. In the method, at least one annotation suggestion can be proposed and/or displayed to a user (for example annotator, labeller, driver, co-driver, passenger, observer, developer, etc.) via the user interface for selection. The annotation suggestion can contain at least one of the above-mentioned pieces of information. If a signal or raw data has a low level of confidence, at least one corresponding annotation suggestion can be proposed and/or displayed to the user for selection. For example, if a speed limit is recognized but the value cannot be interpreted correctly.

When storing and/or merging the annotations/label data, both classic methods and artificial intelligence (AI) methods can be used. In the method, the at least one manual annotation and the at least one automatic annotation can be stored and/or merged using artificial intelligence, for example an artificial neural network. In the method, the at least one manual annotation, the at least one automatic annotation and the at least one annotation additionally retrieved from a memory can be stored and/or merged using artificial intelligence, for example an artificial neural network.

One method of artificial intelligence can be machine learning, for example supervised learning or unsupervised learning. In the method, the storage and/or merging and/or indexing can take place using machine learning, such as supervised learning or unsupervised learning.

In supervised learning, the artificial neural network can be given an input pattern. The output that the neural network produces in its current state can be compared to the value that it is actually supposed to output. By comparing the target and actual output, conclusions can be drawn about the changes to be made to the network configuration. In supervised learning, the user can be given a suggestion, such as an annotation suggestion or label suggestion, be made or displayed. This can be done, for example, when the confidence of the detection or of the signal or raw data is low. For example, if a street sign was seen but the content could not be assigned exactly (eg 70 km/h or 90 km/h), or whether the city was left or not. The user can select an alternative to the suggestion, or confirm or correct the suggestion. The system or the artificial neural network can thus learn. Additionally or alternatively, the raw data, such as vehicle signals, and/or the information mentioned above, such as a reference sensor, online/offline map or online service/cloud, can also be used for this purpose. This can improve future predictions when annotating/labeling.

Unsupervised learning can be done by entering the patterns to be learned. The artificial neural network can change itself according to the input patterns. The data, such as annotations or annotation data, can be projected into an n-dimensional state space in unsupervised learning (e.g. location, time, season, lighting conditions, weather, road type, speed, ESP slip, object constellations and their movements, etc.). By analyzing the state space, n-dimensional clusters and/or outliers can be found (e.g. anomaly detection). The clusters and/or the outliers/anomalies can be reported and/or displayed to the user. The user can either react to this immediately or at least carry out a targeted check during data evaluation and/or manual annotation/labeling (hand labeling). Unsupervised learning can be performed online or offline, for example in batch mode.

For example, a trained system or artificial neural network that is notified of a zebra crossing on a freeway can perceive or identify this as an outlier in the state space and both inform the user and mark the data specifically. This means that the data can later be evaluated and checked for plausibility. In this way, the user or driver could drive to the same place several times and thus see under what circumstances the (probable) incorrect detection occurs. In addition, the user or driver/passenger is able to record much more data directly in order to be able to recognize what the actual underlying truth (ground truth) is immediately and not only later during a post-analysis.

The boundary conditions of the anomaly can be analyzed. At least one suggestion may be made or displayed to the user to accommodate a scenario with similar states. Other motor vehicles in a test and/or validation fleet can be informed, for example by means of a wireless connection and/or online, that scenes with similar states are required. The other motor vehicles in the test and/or validation fleet can be sent to corresponding locations.

If a motor vehicle happens to enter a condition that another motor vehicle should enter (e.g. a daytime roadworks on a country road at night and in the rain), this can be communicated to the other motor vehicle and/or included in its planning.

The coverage of the state space can be observed, in particular in parallel and/or continuously. Suggestions can be issued and/or displayed as to which states are still to be validated in order to cover the state space in a meaningful way. For example, it can be suggested to select a parallel street in order to record anomalies that are currently occurring there, such as a demo on a federal highway.

The method can be used not only for development and/or validation, but also after the so-called "Start of Operation" (SOP), i.e. when a motor vehicle is used by a customer. In such a motor vehicle, for example, which is wirelessly connected to an external system such as a cloud or cloud storage or data center/control center, the annotations and/or information mentioned above and/or below can be used to build a data index. This allows different applications to be covered. Each application can run individually or in combination with one or more other applications.

Scenes of interest can be determined by means of the data index, e.g. activation of an automatic safety brake (AEB) in the dark and in the rain in France, and/or corresponding more detailed data, e.g. associated images in the event data recorder (event data recorder) together with internal data of the control units are recorded and/or sent to the cloud or cloud storage for evaluation. This can be done automatically or, for example afterwards, at a user request, in each case for individual motor vehicles, partial fleets of motor vehicles or the entire fleet. For example, a request can be a request for detailed data on all AEB events in the last month in the dark and rain in Germany.

The driver can provide feedback as to whether a malfunction has occurred or whether a function has worked differently than desired. This feedback can be included along with the data indices. The feedback can be evaluated automatically or manually. In the case of predetermined feedback or feedback under certain conditions, correspondingly more in-depth data, for example associated images in the event data recorder (Event Data Recorder) together with internal data from the control units, can be recorded and/or sent to the cloud/cloud memory for evaluation. This can be done automatically or, for example afterwards, at a user request, in each case for individual motor vehicles, partial fleets of motor vehicles or the entire fleet. For example, a request can be a request for detailed data on all AEB events in the last month in the dark and rain in Germany.

The steps or points mentioned above (after the "Start of Operation") can each be combined individually or together with the methods of artificial intelligence described above.

In the case of unsupervised learning, the recorded or created data index can be projected and/or clustered into an n-dimensional state space. For example, outliers can be identified and/or detailed data for the examination of the respective motor vehicles can be requested and/or transmitted, e.g. directly automatically or on request.

With supervised learning, the driver can give feedback, for example about the (un)desired behavior. Using the collected data and an artificial intelligence, such as an artificial neural network, the function can be predicted and/or optimized up to the desired behavior.

Driver assistance functions, such as AD/ADAS functions that are already implemented and/or running in motor vehicles of customers, can be evaluated with the aforementioned applications and/or steps for data acquisition/data collection or data indexing. The evaluation can be used, among other things, to optimize the function. For example, new parameters can be loaded either offline or online into a control system of a motor vehicle or fleet of motor vehicles. Furthermore, the data can be used to improve future functions.

The method can be stored at least partially as a computer program on a computer, microcomputer, in an electronic control and/or computing unit or on a storage medium and/or can be implemented there. In terms of software, the computer program can be distributed to one or more storage media, control and/or computing units or computers, etc.

A computer program product can cause a device, such as a controller, a control and/or computing unit/device, a control system, a processor or a computer, to execute the method for generating a data index described above and/or below. For this purpose, the computer program product can have corresponding data sets and/or the computer program.

A device may include a processor, memory, user interface, and/or electronic data interface. The device can have a communication module and/or transmission module. The device can be set up and intended to carry out the method described above and/or below for generating a data index. The computer program product may be stored in the memory of the device.

With the invention, everyone involved in the development and/or validation of driver assistance functions, such as AD/ADAS functions, can quickly and efficiently obtain an overview of all data. The merged index, like data index, is several orders of magnitude smaller than the raw data and thus can be computed/created and hosted/stored on a server on-premise or in a cloud storage in a data-size-efficient manner. This allows subsequent automatic filing/storage of the raw data on servers/storage based on information from the merged index. A fast and efficient finding of data by filters and by all participants is made possible. Statistics of the merged annotations/labels can be created quickly and efficiently without having to evaluate the raw data. This can also be done before a broad/high or fast data connection has been established with the motor vehicles in which the data is recorded. Data can be analyzed quickly and efficiently using classic methods or methods of artificial intelligence without having to evaluate the raw data.

• 1 ein Flussdiagramm eines Verfahrens zum Erzeugen eines Datenindex basierend auf Rohdaten eines Fahrerassistenzsystems eines Kraftfahrzeugs; und
• 2 eine Vorrichtung zum Erzeugen eines Datenindex basierend auf Rohdaten eines Fahrerassistenzsystems eines Kraftfahrzeugs.

Exemplary embodiments of the invention are described in more detail below with reference to figures described, showing schematically and by way of example:

• 1 a flow chart of a method for generating a data index based on raw data of a driver assistance system of a motor vehicle; and
• 2 a device for generating a data index based on raw data from a driver assistance system of a motor vehicle.

1 shows a flowchart of a method for generating a data index based on raw data from a driver assistance system of a motor vehicle.

In a step 1, at least one manual annotation of raw data is recorded using a user interface. The manual annotation was entered manually by a user using the user interface.

In a step 2, at least one automatically generated annotation of raw data is recorded using an electronic data interface. The automatic annotation was automatically generated based on the raw data.

In a step 3, at least one annotation of raw data retrieved from an external cloud memory is recorded. The retrieved annotation may be an annotation previously entered manually by a user, or an annotation generated ad-hoc or previously automatically based on raw data.

In a step 4, the at least one manual annotation, the at least one automatically generated annotation and the annotation retrieved from the external cloud storage device are transmitted to a database.

In a step 5, the at least one manual annotation, the at least one automatically generated annotation and the annotation retrieved from the external cloud memory are stored and merged in the database such that a merged data index is created.

The above with reference to 1 The method described can be part of a computer program product that causes a device to execute the method for generating a data index based on raw data from a driver assistance system of a motor vehicle.

2 shows a device 6 for generating a data index based on raw data of a driver assistance system of a motor vehicle.

The device 6 has a processor 7 , a memory 8 , a user interface 9 and an electronic data interface 10 . The user interface 9 is set up to capture a manual annotation entered manually by a user. The electronic data interface 10 is set up to record an automatically generated annotation.

The device 6 also has a communication module 11 for wirelessly receiving and sending data, such as annotations and/or raw data. The communication module 11 is connected to an external cloud memory 12 and can receive data from it and transmit it to it. A database for the data index is implemented in cloud storage 12 .

The device 6 has a computer program product with data sets, which causes the device 6 to execute the method described above and/or below for generating a data index. The computer program product is stored in memory 8 of device 6 .

Incidentally, in addition, in particular 1 and the associated description.

In particular, “may” denotes optional features of the invention. Accordingly, there are also developments and/or exemplary embodiments of the invention which additionally or alternatively have the respective feature or features.

If necessary, isolated features can also be selected from the combinations of features disclosed here and used in combination with other features to delimit the subject matter of the claim, eliminating any structural and/or functional relationship that may exist between the features. The order and/or number of all steps of the method can be varied.

reference list

1

Step to capture a manual annotation

2

Step to capture an automatic annotation

3

Step of capturing an annotation retrieved from cloud storage

4

Step to submit the annotations

5

Step for storing and merging the annotations

6

Device for generating a data index

7

processor

8th

Storage

9

user interface

10

electronic data interface

11

communication module

12

Cloud storage

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102018000101 A1 [0004]

Claims (10)

Method for generating a data index based on raw data from a driver assistance system of a motor vehicle, with the steps: - detecting (1) at least one manual annotation of raw data by means of a user interface (9); - detecting (2) at least one automatic annotation of raw data by means of an electronic data interface (10); - Transmitting (4) the at least one manual annotation and the at least one automatic annotation to a database which is designed to store and/or merge (5) the at least one transmitted manual annotation and the at least one transmitted automatic annotation in this way, so that a data index is created. procedure after claim 1 , in which an annotation of raw data is retrieved (3) from local storage and/or cloud storage (12) and transmitted (4) to the database for storage and/or merging (5) of the retrieved annotation. Method according to at least one of the preceding claims, wherein the database is implemented inside or outside the motor vehicle, for example in a local database server present in the motor vehicle or in a database server (12) implemented outside of the motor vehicle. Method according to at least one of the preceding claims, in which the transmission (4) to the database takes place via a wireless connection, and/or in which the method is carried out while the motor vehicle is driving, for example a test drive. Method according to at least one of the preceding claims, wherein the at least one manual annotation and/or the at least one automatic annotation is recorded (1, 2) separately and/or in addition to the raw data. Method according to at least one of the preceding claims, wherein the respective annotation comprises one or more of the following information: - reference sensor and/or reference sensor type, - reference and/or file name to the raw data, - vehicle signal and/or vehicle sensor, - continuous or event-based signal, - status information, - signal statistics, - EDR data, - manual annotation and/or manual label, - user information, - local data or external and/or online data, - Online Service and/or Online Service - Cloud data or cloud storage data - online weather data, - Online or offline map data. procedure after claim 6 , wherein at least two pieces of information are merged (5) to generate the indexing. Method according to at least one of the preceding claims, wherein at least one annotation suggestion is suggested and/or displayed to a user for selection by means of the user interface (9). Method according to at least one of the preceding claims, wherein the storage and/or merging (5) of the at least one manual annotation and the at least one automatic annotation takes place using artificial intelligence, such as an artificial neural network. Device (6) with a processor (7), a memory (8), a user interface (9) and an electronic data interface (10), wherein the device (6) is set up and intended to use a method for generating a data index according to at least carry out one of the preceding claims.

Images