DE102020213862A1 - Method and environment detection system for providing environment data - Google Patents

Abstract

The invention relates to a method for providing environmental data describing the environment of a vehicle (1), sensor data describing the environment being recorded with at least one sensor (12) of the vehicle (1) and the at least one sensor (12) storing the sensor data with a defined resolution, a defined rate and in a defined coordinate system, and environmental data based on the sensor data are provided by transmission to other road users (6) and/or to an infrastructure facility (100) as a receiver, characterized in that the environmental data are provided via a Conversion of the sensor data is obtained, in which at least one or more conversions are made, which are selected from i) increasing or decreasing the resolution, ii) increasing or decreasing the rate, and ii) performing a deconstruction of the sensor data, which abstracts the sensor data and thereby anonymized. Width Re aspects of the invention relate to an environment detection system (10) for a vehicle (1) and a computer program, which are set up to carry out the method.

Description

The invention relates to a method for providing environment data which describes the environment of a vehicle, with sensor data describing the environment being recorded with at least one sensor of the vehicle and with the sensor supplying the sensor data with a defined resolution and a defined rate and based on the sensor data Area data are provided by transmission to other road users and / or to an infrastructure facility as a recipient. Further aspects of the invention relate to an environment detection system for a vehicle and a computer program, which are set up to carry out the method.

State of the art

Modern vehicles increasingly have assistance systems that support the driver of the vehicle in performing various driving maneuvers. Vehicles are also known which have automated driving functions which, at least under certain conditions, allow autonomous driving without the intervention of a driver.

These assistance systems and autonomous driving functions rely on precise data about the vehicle's surroundings. For this purpose, the vehicles each have sensors that record sensor data. These sensors are used, for example, to locate objects around the autonomous vehicle. Based on this, an autonomous driving function can be used to plan a trajectory and control vehicle actuators within an autonomous vehicle. The lateral guidance, ie the steering of the vehicle, and/or the longitudinal guidance, ie the acceleration or braking of the vehicle, are then influenced by the vehicle actuators.

The sensor data describing the environment is processed, for example, with the aid of intelligent object recognition algorithms, which can be implemented, for example, as neural networks or artificial intelligence. The object recognition algorithms usually use sensor data with a defined resolution and data rate. The processing algorithms within an autonomous vehicle are set up to deal with the resolutions of the different sensors arranged on the vehicle, or the evaluation algorithms are parameterized for the corresponding resolutions and data rates of the sensor data of the different sensors of the respective vehicle or, in the case of neural networks, are trained.

Furthermore, it is known that vehicles can exchange data with one another (car-to-car communication) or with an infrastructure device (car-to-infrastructure communication). This exchanged data can in particular include environmental data that describe the environment of the vehicle.

DE 10 2017 200 654 A1 describes a method for providing sensor-based vehicle functions. In the method, the sensors are coupled via an integration component and are designed to be independent of other vehicle functions. The integration component reads sensor data and converts it into environmental data or generally into status data. In addition, external devices can be connected, for example via Car2X communication or vehicle-to-vehicle communication. For this purpose, it can be provided that the integration component transmits at least some of its status data to at least one outsourced vehicle function executed outside the vehicle. In-vehicle vehicle functions can call up status data just like sensor data via the integration component, so that the vehicle functions can make no difference whether they come from an in-vehicle sensor or not.

DE 10 2017 223 634 A1 describes a method for determining a road condition by evaluating sensor data from a road vehicle. One embodiment provides for a sensor unit of the vehicle to send sensor data to a central database. Standardized raw data is transmitted in the process.

When sensor data from different sensors are exchanged between different vehicles or with an infrastructure device, the problem arises that the evaluation algorithms present in a vehicle are set up for specific resolutions and/or data rates. This makes it more difficult or even impossible to use the exchanged sensor data. Furthermore, data protection concerns can arise, especially when image data is exchanged, especially if people or number plates can be identified in the exchanged images. Such data protection concerns can mean that this sensor data cannot be exchanged and is therefore not available for evaluation by other vehicles or by an infrastructure device.

Disclosure of Invention

1. i) dem Erhöhen oder Verringern der Auflösung,
2. ii) dem Erhöhen oder Verringern der Rate, und
3. iii) dem Durchführen einer Dekonstruktion der Sensordaten, welche die Sensordaten abstrahiert und dadurch anonymisiert.

A method is proposed for providing environment data which describes the environment of a vehicle, with sensor data describing the environment being recorded with at least one sensor of the vehicle and with the at least one sensor transmitting the sensor data with a defined resolution, a defined rate and in a defined Provides coordinate system and based on the sensor data environment data are provided by transmission to other road users and / or to an infrastructure facility as a recipient. Provision is also made for the environmental data to be obtained via a conversion of the sensor data, in which at least one or more conversions are carried out, which are selected from

1. i) increasing or decreasing the resolution,
2. ii) increasing or decreasing the rate, and
3. iii) performing a deconstruction of the sensor data, which abstracts the sensor data and thereby renders it anonymous.

In addition, the sensor data is preferably converted into the vehicle coordinate system of the receiver or into a global coordinate system (world coordinate system).

The at least one sensor can be, for example, a video camera, a microphone, a camera, an ultrasonic sensor, a LiDAR sensor or a radar sensor. In particular, the vehicle can have more than one sensor, it being possible for the vehicle to have a plurality of sensors of the same type and/or to have sensors of different types.

According to the type of at least one sensor, the sensor data can be different types of data, for example image data representing images, videos (in the form of a sequence of images or image data), waveforms which can represent a sound recording, three-dimensional point clouds and the like . These data types each have a resolution that indicates a precision of the data. Furthermore, the data types have a rate which can correspond in particular to a repetition rate of the measurement carried out by the respective sensor.

For example, the resolution is an image resolution or color depth of image data, a bit depth of samples of a waveform, a numerical precision of a discrete sensor value, the numerical precision of a 3D model or a point cloud.

The rate is, for example, a frame rate of a stream of image data, a sampling rate of a waveform, a repetition rate of discrete sensor values, the update rate of a 3D model or a point cloud.

Algorithms and/or AI methods are preferably used to evaluate the sensor data, which, for example, extract specific features from the sensor data in order to identify objects in the area surrounding the vehicle in particular. As a rule, these algorithms and/or AI methods expect the sensor data to be in a specific input format, i.e. in particular in a specific resolution and/or rate. In order to make the sensor data processable for processing by other road users and/or an infrastructure facility, the invention provides for the sensor data to be converted into a specified target format and the environmental data obtained in this way made available for use by others.

The other road users are, in particular, other vehicles that are in the vicinity. By transmitting the environmental data based on their own sensor data, these other vehicles can gain an insight into the current traffic situation from a different perspective. This makes it possible, for example, to obtain information about areas of the environment that cannot be viewed by the vehicle's own sensors, since these are covered by objects, for example. Each road user can process the environmental data obtained using their own evaluation algorithms and/or AI methods, so that the functionality of all of the vehicle's own assistance functions and autonomous driving functions is fully available, regardless of whether their own sensor data or environmental data obtained are used.

The infrastructure device can be a central server, for example, in particular a cloud server. The infrastructure device can make the environmental data obtained available to other road users and/or carry out its own evaluations of the environmental data. For example, it is conceivable for the infrastructure device to determine a traffic density or identify free parking spaces using the environmental data obtained using an algorithm or an AI method. This derived information can then in turn be made available to road users.

The target format of the conversion can take part in an exchange of environment data for all taking road users and / or infrastructure facilities are specified identically. Alternatively, the target format can be specified individually by a recipient. It can be provided in particular that the recipient requests environmental data from the vehicle in the specified target format and thus transmits information about the desired target format with the request.

A wireless communication method is preferably used for the transmission of the environment data. In this case, for example, a direct connection can be established between two road users or a connection to a communication network such as the Internet can be established. Suitable communication methods include, in particular, mobile radio technologies such as GSM, UMTS, LTE, 5G and methods for wireless networking such as Bluetooth or WiFi.

One or more algorithms, machine learning methods or artificial intelligence methods and combinations thereof are preferably used to carry out the conversion, with different algorithms, machine learning methods or artificial intelligence methods being used for different types of sensor data and/or for different conversions, or an algorithm Machine learning method and / or an artificial intelligence method performs multiple conversions for one or more types of sensor data.

When using different algorithms for converting the sensor data, for example, one algorithm can convert sensor data into a uniform resolution, while a downstream algorithm adjusts the rate and another downstream algorithm calculates a deconstruction of the sensor data.

However, it can also be provided that only certain parameters of the sensor data are changed during the conversion. For example, it can be provided that only the rate or only the resolution is adjusted during the conversion.

The rate is preferably increased using extrapolation algorithms or interpolation algorithms from two adjacent data points of the sensor data, with intermediate points being calculated. To reduce the rate, the data points of the sensor data to be transmitted are preferably selected according to the rate specified as the target and/or intermediate points are calculated from two adjacent data points of the sensor data using extrapolation algorithms or interpolation algorithms.

If the sensor data is image data, a neural network is preferably used to change the image resolution of the image data and/or to deconstruct the image data.

A changed resolution or image resolution of image data can be calculated using a neural network. If the resolution of the image data is reduced, for example specific pixels of the image data can be omitted and the resulting image can be smoothed. If the resolution is increased, the neural network is preferably used to extrapolate missing pixels from the existing pixels of the image data.

Using neural networks to change the resolution of image data is known to those skilled in the art. An example of this is described in the publication Mehdi SM Sajjadi, Bernhard Schölkopf, Michael Hirsch "EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis", arXiv:1612.07919v2.

A changed rate or, in the case of a video, the number of images over time is preferably also calculated using a neural network. For example, sensor data is only transmitted to other road users or to an infrastructure facility offset by one cycle. One cycle corresponds to the rate at which the environment sensor supplies data. In the case of image data as sensor data, when the rate (frame rate) is increased by a factor of two, for example, an image is extrapolated from two consecutive images using a neural network. For example, the neural network is used to determine the differences between the various pixels between the two images and, based on this, the missing image between the two images is calculated using an extrapolation algorithm. All 3 images are then transferred. In this way, image data can be generated from the existing image data at twice the rate in this example and then transmitted to other road users or an infrastructure facility. The factor of two can only be seen as an example. The more intelligent the algorithm, the more image data can be generated step by step from the two adjacent images, depending on the required frame rate on the receiver side. A reduction in the frame rate using the neural network is also conceivable. In this case, for example, only every second image is transmitted to a recipient.

For example, if the frame rate is to be reduced by a third, two or three adjacent frames are used and based on that rend uses an extrapolation algorithm within the neural network to calculate the pixel content of each pixel in the image for a third of the frame rate.

In sensor data deconstruction, input sensor data is processed by a neural network. The neural network converts, for example, image data into a machine format that can only be understood by a machine. A downstream neural network, for example, can then extract certain features from this machine format. The features extracted in this way are suitable, for example, for describing the positions of objects and/or for classifying the type of object, so that a distinction can be made between vehicles, trees and pedestrians, for example. However, the machine format obtained through the deconstruction or the extracted features do not allow people to be identified or license plates to be read. It is also not possible to reconstruct the original image data using the deconstructed data or the extracted features. In this way it is possible, for example, to anonymize image data and transmit it as environmental data without any data protection concerns.

The sensor data of the vehicle are usually first recorded with reference to a local vehicle coordinate system as a defined coordinate system, which is related, for example, to a sensor position or a reference point of the vehicle. The sensor data of the vehicle are preferably transformed into a vehicle coordinate system of the receiver or into a global coordinate system before transmission. For this purpose, for example, a corresponding conversion can be carried out on the basis of a received position and orientation of another road user using a known position of the vehicle. Furthermore, it can be provided that the defined coordinate system of the sensor data is converted into a global coordinate system. Such a global coordinate system is preferably unique worldwide and can be processed directly by an infrastructure facility, for example. Other road users can, if necessary, transform the environmental data obtained into their own local coordinate system via a further transformation.

As an alternative to this, it can be provided that the position and orientation of the vehicle and, if necessary, the position and orientation of the sensor are transmitted to the receiver together with the environmental data, so that the latter can carry out a corresponding transformation of the coordinate system.

The position and, if necessary, the alignment of the vehicle can be determined, for example, via receivers for signals from navigation satellites. If necessary, signals from sensors of the vehicle, such as an odometry sensor or a compass, can be taken into account to support localization.

The environmental data provided are preferably cryptographically signed so that their integrity and origin can be traced.

For this purpose, the environmental data can be provided with a uniform and unique hash value before transmission to a recipient. In this way, the environmental data can be assigned to a specific road user at any time when stored on the other road user or the infrastructure facility. The hash value of the environment data is calculated using a unique key that fits exactly one vehicle (private key). The hash value of the environmental data outside the vehicle is then checked using a public key.

Another aspect of the invention is providing an environment sensing system for a vehicle. The environment detection system comprises at least one sensor with which sensor data describing the environment can be detected, as well as a communication device for transmitting environment data to other road users or an infrastructure device and a control device.

The environment detection system is set up to carry out one of the methods described herein. Therefore, features described within the scope of one of the methods apply accordingly to the environment detection system and vice versa, features described within the scope of one of the methods apply accordingly to the environment detection system.

The at least one sensor is preferably a video camera, a microphone, a stereo camera, an ultrasonic sensor, a LiDAR sensor, a radar sensor or a combination of several of these sensors. In this case, in particular, several identical sensors and/or combinations of several different types of sensors can be used.

The communication device is preferably set up for wireless communication, for example using mobile radio technology such as GSM, UMTS, LTE, 5G and/or methods for the wireless networking such as Bluetooth or WiFi can be implemented.

The control device can in particular comprise a microcontroller or a processor which is programmed accordingly for the execution of the method and is set up in particular for carrying out the conversion of the sensor data.

According to the invention, a computer program is also proposed, according to which the steps of the method described herein to be executed by the vehicle are carried out when the computer program is executed on a programmable computer device. The computer program can be, for example, a module for implementing a driver assistance system or a subsystem thereof in a vehicle, or an application for driver assistance functions, which can be executed on a smartphone or tablet, for example. The computer program may be stored on a machine-readable storage medium, such as a permanent or rewritable storage medium, or associated with a computing device, or on a removable CD-ROM, DVD, Blu-ray Disc, or USB stick. Additionally or alternatively, the computer program may be made available on a computing device, such as a server, for download, e.g., over a data network such as the Internet or a communications link such as a telephone line or wireless connection.

Advantages of the Invention

In the method according to the invention, sensor data from a sensor for detecting the surroundings of a vehicle are converted into a predefined format for provision as surroundings data before they are transmitted to other road users or an infrastructure facility. This allows the environment data to be evaluated or further processed directly on the receiver side without further conversion having to take place on the receiver side. In this way, the evaluation of the environment data is accelerated on the receiver side. In advantageous variants of the method, the recipient can specify the target format for the conversion in order to receive the environmental data in exactly the format that is required for further processing.

During the conversion, sensor data, in particular image data, can advantageously be deconstructed. This ensures that the privacy of pedestrians in the image data, for example, is guaranteed during subsequent processing by the recipient. Even with strict data protection regulations, this makes it possible to transmit image data as environment data to other road users or to an infrastructure facility without data protection concerns, so that the recipients have more data about the environment at their disposal. The safety of road users is significantly increased by this additional environment data, since a more precise and complete image of the environment is accordingly available.

In the case of the recipients of the environmental data, the algorithms or AI methods used for processing can be designed in a particularly simple manner, since they always receive the environmental data in the expected format.

Brief description of the drawings

Embodiments of the invention are explained in more detail with reference to the drawings and the following description.

The only figure shows a crossroads where three vehicles meet, two of which have an environment detection system according to the invention.

The figure represents the subject matter of the invention only schematically.

Embodiments of the invention

• the 1 shows a road junction 2 schematically. A vehicle 1 and another road user 6 are each equipped with an environment detection system 10 . The environment detection system 10 has in the simplified example of FIG 1 each have a sensor 12, a control unit 14 and a communication device 16. Of course, the environment detection system 10 can also have multiple sensors 12 . It is conceivable here for the control device 14 and, if appropriate, the communication device 16 to be configured together with the sensor 12 as an integrated unit.

The sensors 12 each have a field of vision 13, 13' within which they can detect objects. The sensor 12 of the vehicle 1 can detect an oncoming vehicle 8 since this is located in the field of view 13 of the sensor 12 of the vehicle 1 . The sensor 12 of the other road user 6 cannot detect the oncoming vehicle 8 because in 1 In the situation shown, a house 4 limits the field of view 13' of the sensor 12 of the other road user 6.

Thus, assistance systems or autonomous driving functions of the other road user 6 initially have no knowledge that the oncoming vehicle 8 is also approaching the intersection.

However, the vehicle 1 can provide the other road user 6 via the environment detection system 10 with environment data based on sensor data from the sensor 12 of the vehicle 1 . Provision is made for this purpose to convert sensor data from sensor 12 of vehicle 1 into standardized environmental data, with a rate and a resolution being changed by conversion, for example. This standardized environment data is then transmitted via the communication device 16 to the environment detection system 10 of the other road user 6 .

The sensor data of the vehicle 1 are preferably transformed into the vehicle coordinate system of the road user 6 before transmission. For this purpose, based on a received position and orientation of the other road user 6 , a corresponding conversion can be carried out by the control unit 14 using a known position of the vehicle 1 . As an alternative to this, it can be provided that the position and orientation of the vehicle 1 and, if applicable, the position and orientation of the sensor 12 are transmitted together with the environmental data to the other road user 6, so that a corresponding transformation can be carried out by the control unit 14 of the other road user 6. To this end, the vehicle 1 and the other road user 6 can determine their respective position in relation to a global coordinate system, for example via signals from navigation satellites.

Assistance systems or an autonomous driving function of the other road user 6 can now process the environmental data obtained as if they had been recorded by a sensor 12 of the other road user 6 itself. The other road user 6 can thus already take into account the oncoming vehicle 8 when planning a trajectory before it enters the field of view 13 ′ of the sensor 12 of the other road user 6 . Driving safety is thus increased.

In addition, the environment detection systems 10 of the vehicle 1 and/or the other road user 6 can provide environment data to an infrastructure device 100 .

The infrastructure device 100 can be, for example, a central server that is in 1 example has a communication unit 102 and a computing unit 104 . The infrastructure device 100 can carry out its own evaluations of the environmental data received. For example, the infrastructure device 100 can determine a traffic density or identify free parking spaces using an algorithm or an AI method via the environmental data obtained. This derived information can then in turn be made available to the vehicle 1 and the other road user 6 .

The invention is not limited to the exemplary embodiments described here and the aspects highlighted therein. Rather, within the range specified by the claims, a large number of modifications are possible, which are within the scope of expert action.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102017200654 A1 [0006]
• DE 102017223634 A1 [0007]

Claims (11)

Method for providing environmental data which describes the environment of a vehicle (1), the sensor data describing the environment being recorded with at least one sensor (12) of the vehicle (1) and the at least one sensor (12) storing the sensor data with a defined resolution , a defined rate and in a defined coordinate system and environmental data based on the sensor data are provided by transmission to other road users (6) and/or to an infrastructure facility (100) as a receiver, characterized in that the environmental data are obtained by converting the sensor data in which at least one or more conversions are performed, which are selected from i) increasing or decreasing the resolution, ii) increasing or decreasing the rate, and iii) performing a deconstruction of the sensor data, which abstracts the sensor data and thereby anonymized. procedure after claim 1 , characterized in that the sensor data is additionally converted into the vehicle coordinate system of the receiver or into a global coordinate system. procedure after claim 1 or 2 , characterized in that the provided environmental data are cryptographically signed, so that their integrity and origin can be traced. Procedure according to one of Claims 1 until 3 , characterized in that a target format of the conversion is specified identically for all road users (6) and/or infrastructure facilities (100) participating in an exchange of environmental data, or that the target format is specified individually by a recipient, the recipient being informed by the vehicle ( 1) Requests environment data in the specified target format. Procedure according to one of Claims 1 until 4 , characterized in that the resolution is an image resolution or color depth of image data, the bit depth of a waveform, the numerical precision of a discrete sensor value, the numerical precision of a 3D model and/or that the rate is a frame rate of a stream of image data, a sampling rate of a waveform, a repetition rate of discrete sensor values, which is the update rate of a 3d model. Procedure according to one of Claims 1 until 5 , characterized in that one or more algorithms, machine learning methods or artificial intelligences and combinations thereof are used to carry out the conversion, with different algorithms, machine learning methods or artificial intelligences being used in each case for different types of sensor data and/or for different conversions, or one algorithm , a machine learning method and/or an artificial intelligence performs multiple conversions for one or more types of sensor data. procedure after claim 6 , wherein, to increase the rate, using extrapolation algorithms or interpolation algorithms, intermediate points are calculated from two adjacent data points of the sensor data and, to reduce the rate, the data points of the sensor data to be transmitted are selected according to the rate specified as the target and/or using extrapolation algorithms or interpolation algorithms Intermediate points are calculated from two adjacent data points of the sensor data. procedure after claim 6 or 7 , wherein a neural network is used to change the resolution of image data and/or for a deconstruction of image data. Environment detection system (10) for a vehicle (1), wherein the environment detection system (10) comprises at least one sensor (12) with which sensor data describing the environment can be detected, and a communication device (16) for transmitting environment data to other road users (6 ) or an infrastructure device (100) and a control unit (14), characterized in that the environment detection system (10) for executing one of the methods according to one of Claims 1 until 8th is set up. Environment detection system (10) according to claim 9 , characterized in that the at least one sensor (12) is a video camera, a microphone, a stereo camera, an ultrasonic sensor, a LiDAR sensor, a radar sensor or a combination of several of these sensors. Computer program by the vehicle (1) to be executed steps of the method according to one of Claims 1 until 8th executes when running on a computer.

Images