DE102019119929B4 - Method for detecting a vehicle environment - Google Patents

Abstract

Method (100), in particular for detecting a vehicle environment (1), in particular an object (1) of a vehicle environment, of a vehicle (10), wherein- in a first step at least one detection module (1) is put into operation for detecting the vehicle environment, wherein the detection module (1) at least partially detects the vehicle environment and the image data (B1) then recorded by this detection are initially- in a further step forwarded to a correction unit (2), which at least partially eliminates image errors (F1) contained in the image data (B1), so that the correction unit (2) generates corrected image data (BF1) which are at least partially freed from the image error (F1), wherein- in a further step these then corrected image data (BF1) are fed to a control unit (ECU), which on the basis of these corrected image data (BF1) outputs at least one control signal, by means of which at least one actuator (5) of the vehicle (10) is controlled and/or regulated,characterized in that a data transmission of the image data (B1) from the detection module (1) to the control unit (ECU), preferably exclusively, via the correction unit (2), preferably so that no image data (B1) passes directly from the detection module (1) to the control unit (ECU), wherein within the correction unit (2) the image data (B1) fed into it are compared with image data (BDat1) stored in a database of the correction unit (2) and if a deviation of the image data (BDat1) from the fed-in image data (B1) is detected, the image data (B1) is supplemented by further image data by means of a correction process and/or the image data (B1) is subjected to an interpolation and/or extrapolation process by the correction unit (2), so that the corrected image data (BF1) is then generated by the correction unit (2), wherein the detection module (1) comprises at least one sensor (1.1), by means of which the image data (B1) is at least partially detected, wherein the deviation of the image data (B1) is caused by a foreign object on a detection surface of the sensor (1.1) and/or by a malfunction of the sensor (1.1) itself.

Description

The present invention relates to a method, in particular for detecting a vehicle environment of a vehicle, and a device, in particular for detecting a vehicle environment of a vehicle, and a device network, in particular for detecting a vehicle environment of a vehicle according to the respective preambles of patent claims 1, 7 and 8.

Such methods are already well known from the prior art. In particular, a vehicle environment can be any items and/or objects arranged outside the vehicle that are arranged externally around the vehicle and/or move relative to the vehicle.

As a rule, a control unit based on detection modules of the device generates an at least partially virtual image of the, for example, current, surroundings of the vehicle.

However, the inventors have recognized that in the event of a sensor failure and/or a sensor malfunction of one or more sensors of the detection module and/or, for example, due to at least partial covering of a sensor surface of a sensor, the control unit creates such a virtual image of the environment, which has corresponding reproduction errors. This is because the at least partially covered image of the environment transmitted by the malfunctioning sensor and/or by the sensor can lead to the environment not being able to be reproduced in a level of detail that corresponds to reality and is necessary for further processing.

Since the control unit usually actuates corresponding actuators of the vehicle, the situation can arise that, based on these faulty image data, which are processed by the control device, actuators are actuated incorrectly or not at all in order to be able to react to a changing vehicle environment of the vehicle.

It is conceivable that a vehicle parked at the side of the road is only partially detected or not detected at all by an at least partially covered sensor surface of the detection module, so that the control unit does not count this vehicle as part of the environment, which means that the vehicle is not detected and is therefore not part of the environment model. Corresponding actuators, which normally lead to a sudden safety brake or a diversion of the vehicle in order to avoid a collision with this vehicle, are therefore not activated at all in the worst case scenario, resulting in an accident. This must be avoided at all costs.

EN 10 2015 205 678 A1 shows an on-board system consisting of a camera, a driver assistance device and a single or multiple on-board devices.

EN 10 2013 220 005 A1 relates to a device for displaying the surroundings of a vehicle and to a driver assistance system with such a device with at least one sensor for generating sensor data of the surroundings of a vehicle.

EN 10 2010 030 616 A1 shows a method for detecting a destroyed object in at least one camera image of a camera image sequence.

EN 10 2018 121 697 A1 relates to a computing device programmed to form a convoy with a second vehicle upon determining an error in sensor data in a first vehicle.

EN 10 2007 004 349 A1 shows a camera system for a vehicle that can capture the vehicle's surroundings even at night. The infrared lighting emits radiation that is reflected by objects in the surroundings.

EN 11 2015 003 722 T5 discloses a dead pixel correction method.

It is therefore an object of the present invention to eliminate the above-mentioned problems and thus to offer a method and a corresponding device which, in a particularly simple manner, makes it possible to record a vehicle's surroundings as accurately as possible in order to minimize the risk of an accident.

This problem is solved, inter alia, by the subject matter of method claim 1.

According to the invention, the image data is transmitted from the acquisition module to the control unit, preferably exclusively, via the correction unit, preferably so that no image data passes directly from the acquisition module to the control unit.

The correction unit according to the invention described here is interposed in terms of method and device between a detection step of the detection module and a processing step in the control unit. In this Concept, correction of the image data by the control unit can therefore be dispensed with.

In particular, according to the invention, in a first step at least one detection module is put into operation for detecting the vehicle environment, wherein the detection module at least partially detects the vehicle environment and the image data then recorded by this detection are first forwarded in a further step to the correction unit, which at least partially eliminates image errors contained in the image data, so that the correction unit generates corrected image data which are at least partially freed of the image error, wherein in a next step these then corrected image data are fed to the control unit, which, on the basis of these corrected image data, outputs at least one control signal by means of which at least one actuator of the vehicle is controlled and/or regulated.

In contrast to the previously known state of the art, the image data captured by the capture module are therefore modified before they are fed into the control unit in such a way that errors contained in the image data do not lead to stationary or moving objects in the vicinity of the vehicle being captured at least partially, not at all, or incorrectly.

Therefore, in the sense of the invention, image errors can be those errors which are generated due to a malfunctioning sensor of the detection module, a defect and/or malfunction of the detection module itself and/or sensor capabilities limited by external objects.

Image errors can also include image noise, which is generated by the capture module itself. Image noise is the deterioration of a digital or electronically recorded image due to interference that has no connection to the actual image content, the image signal. The disturbing pixels differ in color and brightness from those of the actual image. The signal-to-noise ratio is a measure of the noise component.

If a sensor of the detection module is at least partially malfunctioning and/or a sensor surface is at least partially covered by an external object, it is conceivable that this sensor does not generate a sensor image or generates an incorrect sensor image. In the sense of the application, this sensor then generates a "dead pixel" or this sensor represents a dead pixel. Such a dead pixel can therefore represent an image error. For example, such a dead pixel is shown as black on a screen. Such a dead pixel therefore represents a disturbance in the image, whereby the present invention at least partially eliminates this disturbance by the correction unit.

In the sense of the invention, a “unit” can be defined as haptic structural components (computer elements, chips, etc.).

Such image errors are therefore at least partially corrected retrospectively by the correction unit according to the invention described here. This can be done, among other things, by replacing at least one faulty image pixel (=dead pixel) with the virtual pixels surrounding the dead pixel based on a mathematical algorithm. The image of a dead pixel that has reached the correction unit can therefore be at least partially replaced by a functioning image pixel. Such a functioning image pixel can represent a pixel immediately adjacent to the dead pixel.

Although this corrected pixel does not reproduce the corresponding environment 1:1, an attempt can be made to create the most accurate image of a functioning pixel. For example, a corrected pixel then reproduces the corresponding environment with an accuracy level of at least 90%, preferably at least 95%. In other words, the correction unit can extrapolate and/or interpolate as accurately as possible from the immediately adjacent functioning pixels to a virtual image of the dead pixel. The dead pixel is then virtually replaced by a virtual pixel approximated accordingly by the correction unit.

Furthermore, in the sense of the invention, a pixel is the image of the environment generated by one or more sensors, which can be displayed on a screen.

According to at least one embodiment, the method according to the invention described here for detecting a vehicle environment of a vehicle comprises a first step, within which at least one detection module is put into operation for detecting the vehicle environment, wherein the detection module at least partially detects the vehicle environment and the image data then recorded by the detection are first forwarded in a further step to a correction unit, which at least partially eliminates image errors contained in the image data, so that the correction unit generates corrected image data which are at least partially freed of the image error, wherein in a further step these then corrected image data are fed to a control unit, which on the basis of these corrected image data at least one control signal from by means of which at least one current vehicle is controlled and/or regulated.

According to the invention, data transmission of the image data from the acquisition module to the control unit preferably takes place exclusively via the correction unit, preferably so that no image data passes directly from the acquisition module to the control unit.

Within the correction unit, the image data fed into it are compared with image data stored in a database of the correction unit and/or another data quality criterion and if a deviation of the image data from the fed-in image data is detected by means of a correction process, the image data are supplemented with further image data and/or the image data are subjected to an interpolation and/or extrapolation process or other processes by the correction unit, so that the then corrected image data are generated by the correction unit.

For example, the image data or other data quality criteria stored in the database can correspond to data that corresponds to optical sharpness or completeness of the captured image data. Of course, not all image data that corresponds to the surroundings of the vehicle can be stored in the database, as this is known to change constantly. However, the image data stored in the database can be data that can be used to decide whether the captured image data is faulty. For example, the database can contain image data that can be attributed to an image that is free of a dead pixel, so that the image data fed in can be compared with such a base image of an image free of a dead pixel. In such an example, it can therefore only be about the recording and detection of a dead pixel.

According to at least one embodiment, the correction unit classifies the image data and/or the corrected image data such that a comparison of these classified data with the stored image data or other data quality criteria results in at least one correction value on the basis of which the corrected image data are generated.

The idea of generating and handling the data classes described here, i.e. the classified data, offers a particularly elegant and cost-effective way to guarantee more energy-efficient operation of the entire process.

In object-oriented programming, a data class (also called an object type) is an abstract model or blueprint for a series of other objects.

The class is considered a blueprint for mapping real objects (i.e. the environment of the vehicle) into software objects (virtual image of the environment) and describes attributes (properties) and methods (behaviors) of the objects.

In particular, a real object can correspond to one, preferably exactly one, specific data class.

According to at least one embodiment, the data classes of the image data and/or the data classes of the corrected image data, which can be managed by the correction unit or are stored in the correction unit, correspond to a data type of a real object recorded by the detection module or stored in the database, such as a vehicle license plate, a vehicle height, a vehicle width, a noise level of a vehicle, a vehicle color, vehicle type, vehicle shape, a vehicle class, person size, spatial person extent, object size, spatial person extent or other objects.

According to at least one embodiment, a classification of the image data is carried out on the basis of motion vectors of an object to be detected and/or a detected object in the vehicle environment in such a way that a motion profile is first created by the correction unit from the temporal movement of the object, wherein the motion profile includes the object's motion vectors. The motion vectors can be generated by or at least on the basis of the image data recorded by the sensor or sensors. The image data can therefore be converted to motion vectors in the detection module and/or processed into such.

Detection and classification of objects from the optical information can therefore also be based on motion vectors, which are either directly available from the detection element or are generated by it.

In particular, the acquisition module and/or the correction unit can combine the data acquired into so-called data blops or data clusters, which can then be classified in terms of their form and/or extent.

In the sense of the invention, a data cluster is a group of data objects with similar properties. The set of clusters found in this data set is called a cluster ring, and a method for calculating such a grouping is called cluster analysis. Data objects that do not belong to a cluster are called outliers, outliers or noise. For example, such outliers can be the errors in the image data described above, for example the dead pixels described above.

According to at least one embodiment, it can therefore be provided that the correction unit combines the image data and/or the corrected image data into so-called data blops or data clusters, which are classified in their form and/or extent by the correction unit.

According to at least one embodiment, the image data includes data generated by an optical detection, a radar detection, a lidar detection and/or a GPS detection of the detection module. The detection module can in particular be supplied with optical data with lidar and/or GPS data generated by external sensors. In particular, data from different image data sources can be combined with one another in order to be able to represent the real or imaginary object.

According to at least one embodiment, the detection module comprises at least one sensor by means of which the image data are at least partially detected.

The sensor can be an optical sensor, a sound sensor, a lidar sensor, a radar sensor and/or a GPS or other sensor that also detects in the infrared and/or UV range, for example. It is also conceivable that the image data comes at least partially from other sensor sources and/or other processing units and/or other connections. This includes data acquisition based on V2X, HDmaps, HMI, IMU, etc.

V2X (Vehicle-to-everything; V2X) is the electronic communication between traffic participants in the forms of vehicle-to-vehicle (V2V), vehicle-to-road (V2R), vehicle-to-infrastructure (V2I), vehicle-to-network (V2N) and vehicle-to-person (V2P).

HDmaps refers to a mapping of the vehicle's surroundings. HMI (Human Machine Interface) (HMI) is also called "Man Machine Interface" (MMI) and allows the operator to observe the system status and intervene in the process in addition to operating the machine.

IMU can include information that comes, for example, from an acceleration sensor or other information domains.

The deviation in the image data (= dead pixel) is caused by a foreign object on a detection surface of a sensor and/or by a malfunction of the sensor itself. A deviation in the image data can therefore be reflected in the faulty image data that must then be corrected. As already described above, the detected image and/or the virtual image (= imaginary image) that can be generated by the control unit can therefore have errors in the sense of the invention, which are then at least partially corrected by the correction unit.

Furthermore, the present invention relates to a device for detecting a vehicle environment. In this respect, all features and embodiments as disclosed in connection with the method described above apply to the device described here and vice versa.

According to at least one embodiment, the device described here comprises at least one detection module for detecting the vehicle environment, wherein the detection module at least partially detects the vehicle environment and the image data recorded by this detection can be forwarded to a correction unit which at least partially eliminates image errors contained in the image data, so that the correction unit generates corrected image data which are at least partially freed from the image error, which can then be fed with corrected image data to a control unit which, on the basis of this corrected image data, outputs at least one control signal by means of which at least one current vehicle can be controlled and/or regulated.

According to the invention, the image data is transmitted from the acquisition module to the control unit, preferably exclusively, via the correction unit, preferably so that no image data passes directly from the acquisition module to the control unit.

Furthermore, the present invention relates to a device network for detecting a vehicle environment of a vehicle.

The device network described here comprises a plurality of devices according to the embodiment described above, which are in data communication with each other via a control network, wherein the detection, processing and/or The transmission of the image data of each acquisition module is controlled by at least one of the control units. In particular, a master-slave architecture can be implemented by such a device network.

A device as described here can therefore take on the role of a master, with the other devices, which can be installed within the same vehicle and/or other vehicles, taking on the role of a respective slave.

According to at least one embodiment, a so-called control network can be divided into at least two network segments (V-LAN) that are only logically separated from one another by means of at least one V-LAN switch, wherein each of the detection modules can be controlled by each of the network segments depending on the control by a V-LAN switch and/or by the control device.

In the following, the method described here as well as the device described here and the device network described here are described in more detail using figures in an exemplary embodiment.

The 1 shows a schematic of an information flow starting from the detection of an object to the actuator control.

In the embodiments and the figures, identical or identically functioning components are provided with the same reference symbols. The elements shown here are not to be regarded as being to scale. Rather, individual elements may be shown exaggeratedly large for better understanding.

In the 1 A schematic view shows an information flow and a device 1000 from which a method 100 for detecting a vehicle environment of a vehicle 10 can be seen, wherein in a first step a vehicle environment is detected by means of a detection module 1. The detection module comprises sensors 1.1 to 1.4, wherein the sensors can comprise a camera sensor 1.1, a radar sensor 1.2, a lidar sensor 1.3, a sound sensor 1.4 and a GPS sensor 1.5.

The image data B1 then acquired by the acquisition module 1 are forwarded in a further step to a correction unit 2, within which image errors B1, F1 are at least partially eliminated and/or corrected, so that the correction unit 2 generates corrected image data BF1 which are at least partially freed from the image errors F1.

In a further step, it can be seen that the correction unit 2 then forwards the corrected image data BF1 to a control unit ECU. The control unit ECU comprises an ECU-1 module in which the corrected image data BF1 can initially be divided into corresponding object classes.

In addition, the control unit ECU includes an ECU-2 module, which creates a virtual vehicle environment based on the corrected image data BF1. This vehicle environment is one that represents a purely virtual reproduction of the real vehicle environment as accurately as possible.

Within an ECU-3 module, a decision is made as to whether an actuator 5 of the vehicle 10 should be controlled on the basis of the “virtual environment module” of the ECU-2 unit.

Before this happens, however, a trajectory and a control process are carried out within an ECU-4 module in order to actually check the decision made by the decision unit ECU-3 again and, if necessary, adapt it to the corresponding target specifications.

After the ECU-4 unit has released this signal, the control unit ECU controls the respective actuators 5 of the vehicle 10. The actuator 5 of the vehicle 10 can be a steering wheel control, a valve control, and/or a brake control.

In one embodiment, it can also be provided that the fed-in image data B1 are compared with stored image data BDat1 in order to carry out a corresponding correction process of the image data B1 and to supplement the image data B1 with further image data.

In the 2 a corresponding device network 200 for detecting a vehicle environment of a vehicle 10 is shown.

A plurality of devices 1000 can be seen, which form a control network 4 and/or are in data communication with one another via a control network 4, wherein the capture, processing and/or reproduction of the image data B1 of each capture module 1 is controlled by at least one of the control units ECU.

The invention is not limited by the description of the embodiment. Rather, the invention covers any new feature and any combination of features, which in particular includes any combination of features in the patent claims, even if this feature or combination itself is not explicitly stated in the patent claims or the embodiment.

List of reference symbols

1

Recording module

1.1

Camera sensor

1.2

Radar sensor

1.3

Lidar sensor

1.4

Sound sensor

1.5

GPS sensor

2

Correction unit

4

Control network

5

Actuator

10

vehicle

100

Method for detecting a vehicle environment

200

Device network

1000

contraption

B1

Image data

BDat1

stored image data

BF1

Image errors

ECU

Control unit

ECU-1

module

ECU-2

module

ECU-3

module

ECU-4

module

F1

Image errors

Claims (8)

Method (100), in particular for detecting a vehicle environment (1), in particular an object (1) of a vehicle environment, of a vehicle (10), wherein - in a first step, at least one detection module (1) is put into operation for detecting the vehicle environment, wherein the detection module (1) at least partially detects the vehicle environment and the image data (B1) then recorded by this detection are initially - in a further step forwarded to a correction unit (2), which at least partially eliminates image errors (F1) contained in the image data (B1), so that the correction unit (2) generates corrected image data (BF1) which are at least partially freed from the image error (F1), wherein - in a further step, these then corrected image data (BF1) are fed to a control unit (ECU), which, on the basis of these corrected image data (BF1), outputs at least one control signal by means of which at least one actuator (5) of the vehicle (10) is controlled and/or regulated, characterized in that a data transmission of the image data (B1) from the detection module (1) to the control unit (ECU), preferably exclusively, via the correction unit (2), preferably so that no image data (B1) passes directly from the detection module (1) to the control unit (ECU), wherein within the correction unit (2) the image data (B1) fed into it are compared with image data (BDat1) stored in a database of the correction unit (2) and if a deviation of the image data (BDat1) from the fed-in image data (B1) is detected, the image data (B1) is supplemented by further image data by means of a correction process and/or the image data (B1) is subjected to an interpolation and/or extrapolation process by the correction unit (2), so that the corrected image data (BF1) is then generated by the correction unit (2), wherein the detection module (1) comprises at least one sensor (1.1), by means of which the image data (B1) is at least partially detected, wherein the deviation of the image data (B1) is caused by a foreign object on a detection surface of the sensor (1.1) and/or by a malfunction of the sensor (1.1) itself. Procedure (100) according to Claim 1 , characterized in that within the correction unit (2) the image data (B1) fed into it are additionally compared with another data quality criterion, in particular algorithm-based. Procedure (100) according to Claim 1 or 2 , characterized in that the correction unit (2) classifies the image data (B1) and/or the corrected image data (BF1), so that a comparison of these then classified data (B1, BF1) with the stored image data (BDat1) results in at least one correction value on the basis of which the corrected image data (BDat1) are generated. Method (100) according to at least one of the preceding claims, characterized in that a classification of the image data (B1) is carried out on the basis of movement vectors of the object (1) to be detected in the vehicle environment in such a way that a movement profile is first created by the correction unit (2) from the temporal movement of the object (1), wherein the movement movement profile of the object (1) includes motion vectors thereof. Method (100) according to at least one of the preceding claims, characterized in that the correction unit (2) combines the image data (B1) and/or the corrected image data (BF1) into so-called data blops or data clusters, which are then classified in their form and/or extent by the correction unit (2). Method (100) according to at least one of the preceding claims, characterized in that the image data (B1) comprise data which are generated by an optical detection, a radar detection, a LIDAR detection and/or by a GPS detection of the detection module (1) or others. Device (1000), in particular for detecting a vehicle environment, of a vehicle (10), comprising - at least one detection module (1) for detecting the vehicle environment, wherein the detection module (1) at least partially detects the vehicle environment and the image data (B1) then recorded by this detection - can be forwarded to a correction unit (2) which at least partially eliminates image errors (F1) contained in the image data (B1), so that the correction unit (2) generates corrected image data (BF1) which are at least partially freed from the image error (F1), wherein - these then corrected image data (BF1) can be fed to a control unit (ECU), which, on the basis of these corrected image data (BF1), outputs at least one control signal by means of which at least one actuator (5) of the vehicle (10) can be controlled and/or regulated, characterized in that a data transmission of the image data (B1) from the detection module (1) to the control unit (ECU), preferably exclusively, only via the correction unit (2), preferably so that no image data (B1) reaches the control unit (ECU) directly from the detection module (1), wherein within the correction unit (2) the image data (B1) fed into it are compared with image data (BDat1) stored in a database of the correction unit (2), and if a deviation of the image data (BDat1) from the fed-in image data (B1) is detected, the image data (B1) are supplemented by further image data by means of a correction process and/or the image data (B1) are subjected to an interpolation and/or extrapolation process by the correction unit (2), so that the corrected image data (BF1) are then generated by the correction unit (2), wherein the detection module (1) comprises at least one sensor (1.1), by means of which the image data (B1) are at least partially detected, wherein the deviation of the image data (B1) is generated by a foreign object on a detection surface of the sensor (1.1) and/or by a malfunction of the sensor (1.1) itself. Device network (200) for detecting a vehicle environment of a vehicle (10), comprising - a plurality of devices (1000) according to Claim 7 which are in data communication with one another via a control network (4), wherein the acquisition, processing and/or forwarding of the image data (B1) of each acquisition module (1) is controlled by at least one of the control units (ECU).

Images