DE102021131054A1 - Method, system and computer program product for evaluating a driving situation for the predictive control of an automated driving function - Google Patents

Abstract

The invention relates to a method for evaluating a driving situation for the predictive control of an automated driving function of a vehicle (10), comprising the following method steps: - recording sensor and image data (20) from a camera and sensor device (12) of the vehicle (10), the image and sensor data (20) depicting the environment and objects in the environment of the vehicle (10) in order to determine a current driving situation at a first point in time (t1);- forwarding the determined sensor and image data (20) and/or further historical data from at least one database (40) to a data analysis device (30), - determining at least one future driving situation at a second point in time (t2) in the data analysis device (40) by means of an analysis system (300); - classifying the at least a future driving situation with regard to the probability of occurrence and/or other features, states or safety categories;- evaluating the classified future driving situation and assigning an evaluation index (70);- controlling (S70) at least one driving function (85) of the driver assistance module (80) with the Rating index (70).

Description

The invention relates to a method, a system and a computer program product for evaluating a driving situation for the predictive control of an automated driving function.

When a motor vehicle is driven by a person, the driver of the vehicle observes very precisely and intuitively whether there are any possible sources of danger on the roadway. In particular, vehicles ahead are closely observed to obtain various information, such as the speed of the vehicle ahead or whether an overtaking maneuver is planned. In addition, the road surface and the road layout are observed in order to be able to adjust driving behavior accordingly. A trained human driver makes these observations intuitively while driving and is often even aware of how he processes the information and assigns it to a possible dangerous situation.

Driver assistance systems and highly automated driving functions are increasingly being used in vehicles to relieve the driver or to enable autonomous driving. In semi-autonomous and autonomous vehicles, camera systems and sensors are used to obtain information about the vehicle's surroundings. The development of highly automated driving (HAD) is therefore accompanied by an increase in the requirements for vehicle sensor systems to record suitable sensor data such as image data in particular. In addition, the recorded sensor data must be carefully interpreted in order to draw the right conclusions regarding a possible dangerous situation.

An example of a driver assistance system with highly automated driving functions is adaptive cruise control in a motor vehicle, which uses the regulation of the distance to a vehicle driving ahead as a control variable for controlling the speed of the motor vehicle. Such a regulation is also referred to as adaptive cruise control (ACC). Adaptive cruise control uses a sensor to determine the position and speed of the vehicle in front and adjusts the engine and brake control accordingly.

However, in the case of the known driver assistance systems and driving functions, the course of a lane of a motor vehicle due to the topography of the terrain, such as inclines and curves, and the actual traffic situation, such as vehicles cutting in, have only been insufficiently taken into account. Cutting-in vehicles are either ignored or a reactive reaction is made to the possible cutting-in vehicle. Current driver assistance systems therefore only have two states that either plan for a possible event such as a vehicle cutting in or ignore it. An intermediate state with only a slight adaptation of the driving function, such as a lower gas supply and thus only a slight reduction in the speed of the motor vehicle due to a drag torque, has not previously been provided.

The DE 10 2017 206 862 A1 discloses a driving system for automated driving of a motor vehicle. The driving system is set up to detect an actual traffic situation in the vicinity of the motor vehicle with at least one other road user and to use at least two alternative options for action based on the actual traffic situation for a first time step. At least one possible future traffic situation is determined for each of these options for action, with the possible future traffic situation being evaluated and one of the options for action being selected based thereon.

The DE 10 2020 201 016 A1 discloses a method for providing at least one trajectory for an automated vehicle, future behavior being estimated based on data surrounding the vehicle using a prediction device, and the estimated future behavior being assigned to one of at least two evaluation categories.

The DE 10 2017 114 876 A1 discloses a method for collision avoidance of a potential collision between a vehicle and another road user. The future trajectories for the vehicle and the other road users are predicted and at least two probability areas with different time intervals are determined along the predicted trajectories, with all probability areas with the same time intervals being compared with one another and escalation measures being initiated if an overlap is identified.

The object on which the invention is based is now to create a method, a system and a computer program product for evaluating a driving situation for the predictive control of an automated driving function for a vehicle, which is characterized by high reliability, safety and accuracy and is simple implement.

According to the present invention, a method, a system and a computer program product are proposed, by means of which a driving situation is reliably detected and evaluated, so that predictive control of an automated driving function is made possible, which ensures a high level of safety.

This object is achieved according to the invention with regard to a method by the features of patent claim 1, with regard to a system by the features of patent claim 11, and with regard to a computer program product by the features of claim 15. The further claims relate to preferred embodiments of the invention.

• - Aufnehmen von Sensor- und Bilddaten von einer Kamera- und Sensoreinrichtung des Fahrzeugs, wobei die Bild- und Sensordaten die Umgebung und Objekte in der Umgebung des Fahrzeugs abbilden zur Ermittlung einer aktuellen Fahrsituation zu einem ersten Zeitpunkt t1;
• - Weitergeben der ermittelten Sensor- und Bilddaten und/oder weiterer historischer Daten aus zumindest einer Datenbank an eine Datenanalyseeinrichtung, wobei die Datenanalyseeinrichtung ein Analysesystem umfasst, welches Algorithmen aus dem Bereich der künstlichen Intelligenz (KI) und der maschinellen Bildanalyse verwendet;
• - Ermitteln zumindest einer zukünftigen Fahrsituation zu einem zweiten Zeitpunkt t2 in der Datenanalyseeinrichtung mittels des Analysesystems;
• - Klassifizieren der zumindest einen zukünftigen Fahrsituation hinsichtlich der Wahrscheinlichkeit des Eintreffens und/oder weiterer Merkmale, Zustände oder Sicherheitskategorien;
• - Bewerten der klassifizierten zukünftigen Fahrsituation und Zuordnen eines Bewertungsindex;
• - Zuführen des Bewertungsindex an ein Fahrzeugassistenzmodul;
• - Steuern zumindest einer automatisieren Fahrfunktion des Fahrassistenzmoduls mit dem Bewertungsindex.

According to a first aspect, the invention provides a method for evaluating a driving situation for the predictive control of an automated driving function of a vehicle. The procedure comprises the following procedural steps:

• - Recording of sensor and image data from a camera and sensor device of the vehicle, the image and sensor data depicting the surroundings and objects in the surroundings of the vehicle for determining a current driving situation at a first point in time t1;
• - Passing on the determined sensor and image data and/or further historical data from at least one database to a data analysis device, the data analysis device comprising an analysis system which uses algorithms from the field of artificial intelligence (AI) and machine image analysis;
• - determining at least one future driving situation at a second point in time t2 in the data analysis device by means of the analysis system;
• - Classifying the at least one future driving situation with regard to the probability of occurrence and/or other features, states or safety categories;
• - Assess the classified future driving situation and assign an assessment index;
• - Supplying the evaluation index to a vehicle assistance module;
• - Controlling at least one automated driving function of the driving assistance module with the evaluation index.

A further development provides for the evaluation index to have at least three control states, which can be represented as “1—no reaction”, “2—active reaction” and “3—adaptive reaction”.

In an advantageous embodiment, the analysis system comprises a detection module for detecting the current driving situation at a first point in time t1, a classification module for classifying the current driving situation and/or at least one future driving situation, a prediction module for calculating at least one future driving situation at a second point in time t2 and at Estimation of a probability value for the occurrence of the future driving situation and an evaluation module for evaluating the future driving situation using evaluation criteria.

In particular, the recognition module has at least one encoder module for extracting image features from the sensor and image data and at least one decoder module for evaluating the extracted image features.

In a further embodiment it is provided that the camera and sensor device has at least one camera and/or at least one such as a LIDAR (Light Detection and Ranging) sensor for optical distance and speed measurement and/or an ultrasonic sensor and/or a radar sensor (Radio Detection and Ranging) and is designed for a fast recording frequency.

In an advantageous embodiment, it is provided that the algorithms from the field of artificial intelligence (AI) and machine image analysis are algorithms from the field of machine learning, preferably deep learning with deep neural networks and/or convolutional neural networks ) and/or learning reinforcement agents (LV) (Reinforcement Learning Agent) for the analysis and evaluation of a driving situation by means of the recorded image and sensor data and/or the further historical data.

In particular, the data analysis device is connected to at least one database and/or a cloud computing infrastructure by means of a communication link.

The communication connection is advantageously designed as a mobile radio connection and/or a near-field communication connection such as ^Bluetooth® , Ethernet, NFC (near field communication) or Wi-Fi®.

In a further development it is provided that the data analysis device and/or the camera and sensor device is/are equipped with radio modules of the 5G standard.

In an advantageous embodiment, it is provided that at least one driving function is designed as a lane keeping assistance or brake support or autonomous driving function.

According to a second aspect, the invention provides a system for evaluating a driving situation for the predictive control of an automated driving function of a vehicle. The system includes a camera and sensor device that is designed to record sensor and image data, the image and sensor data depicting the surroundings and objects in the surroundings of the vehicle for determining a current driving situation at a first point in time t1; a data analysis device that includes an analysis system that uses algorithms from the field of artificial intelligence (AI) and machine image analysis and is designed to determine at least one future driving situation at a second point in time t2 from the image and sensor data and other historical data, to classify the at least one future driving situation with regard to the probability of occurrence and/or other features, states or safety categories, to evaluate the classified future driving situation and to assign an evaluation index to it; and a vehicle assistance module that is designed to control at least one automated driving function with the evaluation index.

In a further development it is provided that the evaluation index has at least three control states, which can be represented as “1—no reaction”, “2—active reaction” and “3—adaptive reaction”.

In an advantageous embodiment, the analysis system comprises a detection module for detecting the current driving situation at a first point in time t1, a classification module for classifying the current driving situation and/or at least one future driving situation, a prediction module for calculating at least one future driving situation at a second point in time t2 and at Estimation of a probability value for the occurrence of the future driving situation and an evaluation module for evaluating the future driving situation using evaluation criteria.

In a further development, algorithms from the field of artificial intelligence (AI) and machine image analysis are used as algorithms from the field of machine learning, preferably deep learning with deep neural networks and/or convolutional neural networks and/or Learning reinforcement agents (LV) (Reinforcement Learning Agent) used to analyze and evaluate a driving situation using the captured image and sensor data and / or other data.

According to a third aspect, the invention provides a computer program product comprising executable program code configured, when executed, to carry out the method according to the first aspect.

The invention is explained in more detail below with reference to exemplary embodiments illustrated in the drawing.

• 1 eine schematische Darstellung eines ersten Ausführungsbeispiels des erfindungsgemäßen Systems;
• 2 eine schematische Darstellung eines erfindungsgemäßen Analysesystems;
• 3 ein Flussdiagramm zur Erläuterung der einzelnen Verfahrensschritte des erfindungsgemäßen Verfahrens;
• 4 ein Blockdiagramm eines Computerprogrammprodukt gemäß einer Ausführungsform des dritten Aspekts der Erfindung.

It shows:

• 1 a schematic representation of a first embodiment of the system according to the invention;
• 2 a schematic representation of an analysis system according to the invention;
• 3 a flowchart to explain the individual steps of the method according to the invention;
• 4 a block diagram of a computer program product according to an embodiment of the third aspect of the invention.

Additional features, aspects and advantages of the invention or embodiments thereof will become apparent from the detailed description coupled with the claims.

1 shows a first embodiment of a system 100 for evaluating a driving situation for controlling an automated driving function for a vehicle 10. The vehicle 10 can be, for example, a motor vehicle, an autonomously driving vehicle, an agricultural vehicle such as a combine harvester, a robot in the production or in service and maintenance facilities, or a watercraft or a flying object such as a drone.

The vehicle 10 has a camera and sensor device 12 which records the environment and objects in the environment of the vehicle 10 and generates sensor and image data 20 . The camera and sensor device 12 records the sensor and image data 20 in a recording area that records the surroundings of the vehicle 10 as comprehensively as possible. In particular, image and sensor data 20 are recorded from a lane ahead, but the side area and the rear area of the vehicle 10 are also important in order to be able to monitor overtaking maneuvers by other road users on multi-lane roads, for example. Other moving objects such as vehicles, pedestrians, cyclists, motorcyclists, etc. or immovable objects such as houses, street signs, fences, trees etc. The camera and sensor device 12 forwards the recorded sensor and image data 20 to a data analysis device 30 for further processing.

The camera and sensor device 12 includes in particular at least one RGB camera in the visible range with the primary colors blue, green and red. However, there can also be at least one UV camera in the ultraviolet range and/or at least one IR camera in the infrared range be provided. The cameras, which differ in terms of their recording spectrum, can thus display different lighting conditions in the recording area.

The recording frequency of the camera and sensor device 12 is designed for high speeds of the vehicle 10 and can record sensor and image data 20 with a high image recording frequency. Furthermore, the camera and sensor device 12 can be equipped with a microphone for the detection of acoustic signals. As a result, rolling noises from the tires or engine noises can be recorded, which allow conclusions to be drawn about the road surface and the road conditions.

Provision can also be made for the camera and sensor device 12 to automatically start the image recording process when there is a significant change in the recording area of the camera and sensor device 12, for example when clear changes in the road surface are recognizable. This enables a selective data acquisition process and only relevant sensor and image data 20 are processed by the data analysis device 30 . This allows computing capacities to be used more efficiently.

In particular, it is provided to use a weatherproof action camera as at least one camera, which can be arranged, in particular, outside of the vehicle 10 . Action cameras have wide-angle fisheye lenses, which makes it possible to achieve a visible radius of around 180°. In this way, the road ahead 20 can be mapped comprehensively. Action cameras can usually record videos in Full HD (1,920 x 1,080 pixels), but action cameras in Ultra HD or 4K (at least 3,840 x 2,160 pixels) can also be used, resulting in a significant increase in image quality. The frame rate is typically 60 frames per second in 4K and up to 240 frames per second in Full HD. An integrated image stabilizer can also be provided. In addition, action cameras are often equipped with an integrated microphone. In addition, methods of differential signal processing can be used to selectively suppress background noise.

Furthermore, the camera and sensor device 12 can have sensors such as at least one LIDAR sensor (Light Detection and Ranging) for optical distance and speed measurement and/or at least one ultrasonic sensor and/or at least one radar sensor (Radio Detection and Ranging ) contain.

The sensor and image data 20 recorded by the camera and sensor device 12 are forwarded to the data analysis device 30 by means of communication connections such as a CAN bus system (Controller Area Network). However, wireless connections can also be provided. A wireless communication connection is designed in particular as a mobile radio connection and/or a near-field communication connection such as ^Bluetooth® , Ethernet, NFC (near field communication) or Wi-Fi®.

Furthermore, a GPS connection 14 is advantageously provided in order to determine the geographic location of the vehicle 10 and to assign this to the recorded sensor and image data 20 .

In addition, the data analysis device 30 can access one or more other databases 40 . For example, classification parameters for analyzing the recorded image and sensor data 20 or other historical data and images and/or parameters can be stored in the database 40 . Furthermore, target variables and target values that define a safety standard can be stored in the database 40 . Cartographic and topographical data can also be stored. Furthermore, a user interface 50 can be provided for entering further data and for displaying the calculation results created by the data analysis device 30 . In particular, the user interface 50 is designed as a display with a touch screen.

On the basis of the geographic location determined, further data on the topographical structure of the roadway on which vehicle 10 is located can also be called up by data analysis device 30 from database 40 . The topographical structure can be, for example, the crest of a hill, so that the road leading down behind the crest cannot be seen by the driver and he therefore cannot see oncoming traffic either. Or it is a sharp turn that only should be driven at a reduced speed.

In particular, the route can be entered into a navigation device so that the data on this route can be retrieved from a database 40 .

Weather data for the current location of vehicle 10 and for a planned route can be retrieved as additional data from database 40 or another data device, since these, such as rain, fog or snow in particular, can have an impact on driving behavior. Other historical data may represent traffic volume data.

The data analysis device 30 preferably has a memory module 32 and a processor 34 which processes the sensor and image data 20 using an analysis system 300 . The processor 34 or another processor is also designed to control the camera and sensor device 12 . The data analysis device 30 can be integrated in the vehicle 10 or be embodied as a cloud-based solution. In particular, the data analysis device 30 can be connected to a cloud computing infrastructure 45 or another computing unit via a communication link. The database 40 can also be integrated in the cloud computing infrastructure 45 .

In the context of the invention, a “processor” can be understood to mean, for example, a machine or an electronic circuit. A processor can in particular be a central processing unit (CPU), a microprocessor or a microcontroller, for example an application-specific integrated circuit or a digital signal processor, possibly in combination with a memory unit for storing program instructions, etc .

A processor can also be understood to mean a virtualized processor, a virtual machine or a soft CPU. It can also be a programmable processor, for example, which is equipped with configuration steps for executing the mentioned method according to the invention or is configured with configuration steps in such a way that the programmable processor has the inventive features of the method, the component, the modules, or other aspects and/or or implemented partial aspects of the invention. In particular, the processor 34 can contain highly parallel computing units and powerful graphics modules. Processor 34 may also advantageously use AI hardware acceleration such as a Coral Dev Board to enable real-time processing. This is a microcomputer with a tensor processing unit (TPU) that allows a pre-trained software application to evaluate up to 70 frames per second.

In the context of the invention, a “memory unit” or “memory module” and the like can mean, for example, a volatile memory in the form of a random-access memory (RAM) or a permanent memory such as a hard disk or a data carrier or e.g. B. be understood as a removable memory module. However, the storage module can also be a cloud-based storage solution.

In connection with the invention, a “module” can be understood to mean, for example, a processor and/or a memory unit for storing program instructions. For example, the processor is specially set up to execute the program instructions in such a way that the processor and/or the control unit executes functions in order to implement or realize the method according to the invention or a step of the method according to the invention.

In connection with the invention, “data” means both raw data and data that has already been processed from the measurement results of the camera and sensor device 12 and/or data stored in the database 40 and/or the cloud computing infrastructure 45 .

"Database" means both a storage algorithm and the hardware in the form of a storage unit. In particular, the database is designed as a cloud computing infrastructure 45 .

The communication connection between the data analysis device 30 and the database 40 or cloud computing infrastructure 45 is designed in particular as a mobile radio connection and/or as a near field communication connection such as ^Bluetooth® , Ethernet, NFC (near field communication) or Wi-Fi®.

In particular, the data analysis device 30 and/or the camera and sensor device 12 have mobile radio modules of the 5G standard. 5G is the fifth-generation mobile communications standard and, compared to the 4G mobile communications standard, is characterized by higher data rates of up to 10 Gbit/sec, the use of higher frequency ranges such as 2100, 2600 or 3600 megahertz, increased frequency capacity and thus increased data throughput and real-time data data transmission, as up to one million devices per square kilometer can be addressed simultaneously. The latency times are a few milliseconds to less than 1 ms, so that real-time transmission of data and calculation results is possible. The sensor and image data 20 recorded by the camera and sensor device 12 can be sent in real time to the cloud computing infrastructure 45, where the analysis of the sensor and image data 20 is carried out. The analysis and calculation results can be sent back to the data processing module 30 or another control module in the vehicle 10 .

This data transmission speed is required if cloud-based solutions are to be used for processing the sensor and image data 20 . Cloud-based solutions offer the advantage of high and therefore fast computing power. In order to protect the connection to the cloud computing infrastructure 45 by means of a mobile radio connection, cryptographic encryption methods are provided in particular. However, cryptographic encryption methods can also be provided for the connection between the camera and sensor device 12 and the data analysis device 30 .

The data analysis device 30 can be a separate unit with regard to the hardware configuration and is arranged in particular in the vehicle 10, but it can also use other hardware and software components in the vehicle 10 or in the cloud computing infrastructure 45 to carry out the function.

The analysis system 300 includes artificial intelligence and machine image analysis algorithms to select and classify the sensor and image data 20 . The analysis system 300 advantageously uses algorithms from the field of machine learning, preferably deep learning with, for example, folded neural networks (convolutional neural networks) and/or learning reinforcement agents (LV) (Reinforcement Learning Agent) to analyze and evaluate a future driving situation by means of of the captured image and sensor data 20.

A neural network consists of neurons that are arranged in several layers and connected to each other in different ways. A neuron is able to receive information from outside or from another neuron at its input, to evaluate the information in a certain way and to pass it on to another neuron in a changed form at the neuron output or to output it as the end result. Hidden neurons are located between the input neurons and output neurons. Depending on the network type, there can be several layers of hidden neurons. They ensure the forwarding and processing of the information. Finally, output neurons deliver a result and output it to the outside world. Different types of neural networks such as feedforward networks, recurrent networks or convolutional neural networks are created by the arrangement and linking of the neurons. The networks can be trained through unsupervised or supervised learning

The convolutional neural network is a special form of an artificial neural network. It has multiple layers of convolution and is well suited for machine learning and artificial intelligence (AI) applications in the field of image and speech recognition. The functionality of a convolutional neural network is to a certain extent modeled on biological processes and the structure is comparable to the visual cortex of the brain. The training of a convolutional neural network usually takes place under supervision. Conventional neural networks consist of fully or partially meshed neurons in several levels. However, these structures reach their limits when processing images, since there would have to be a number of inputs corresponding to the number of pixels. The convolutional neural network is made up of different layers and is based on the basic principle of a partly locally meshed neural feedforward network. The individual layers of the CNN are the convolutional layer, the pooling layer and the fully meshed layer. The convolutional layer is the actual convolution layer and is able to recognize and extract individual features in the input data. In image processing, these can be features such as lines, edges or specific shapes. The input data is processed in the form of tensors such as a matrix or vectors. The pooling layer, also known as the subsampling layer, compresses and reduces the resolution of the recognized features using appropriate filter functions. The reduced data volume increases the calculation speed.

The convolutional neural network (CNN) therefore offers numerous advantages over conventional non-convoluted neural networks. It is suitable for machine learning and artificial intelligence applications with large amounts of input data, such as in image recognition. The network works reliably and is not sensitive to distortions or other optical changes. CNN can process images captured under different lighting conditions and from different perspectives. It recognizes him nor the typical features of an image. Since the CNN is divided into several local, partially meshed layers, it requires significantly less storage space than fully meshed neural networks. The convolutional layers drastically reduce storage requirements. The training time of the convolutional neural network is also greatly reduced. With the use of modern graphics processors, CNNs can be trained very efficiently.

A reinforcement learning agent (LV) selects a specific state _si ∈ S from a set of available states for at least one action a _i ∈ A from a set of available actions. For the selected action a _i the agent receives a reward which can be positive, neutral or negative. The agent receives the states s _i ∈ S from a state module, which processes data from various sensors and data sources and assigns states s _i ∈ S to these processed data, which the LV agent can access.

2 shows the analysis system 300 according to the invention, which comprises a recognition module 310, a classification module 320, a prediction module 330 and an evaluation module 340.

The detected sensor and image data 20 are processed in the detection module 310 in order to detect the surroundings and the objects in the surroundings and to determine a current driving situation at a first point in time t1. In particular, the sensor and image data 20 are processed using a convolutional neural network. A driving situation at a first point in time t1 thus represents a representation of different objects at different locations in a three-dimensional space in relation to the current location of the vehicle 10. However, the objects and the vehicle 10 are moving at different speeds, so that a future Driving situation changes at a second point in time t2 compared to the current driving situation.

The recognition module 310 can in turn have at least one encoder module for the extraction of image features. The extracted image features are forwarded to at least one decoder module for evaluation. As a result, the demands on the computing capacity for processing the sensor and image data with regard to specific characteristics such as object recognition and depth estimation for precise position determination can be reduced.

In the classification module 320, the detected objects are classified in connection with further data, in particular from the database 40, according to various characteristics, states and/or security categories. A route can be classified as non-hazardous or dangerous based on the topography and assigned a security level from low to high. Or the driving style of another vehicle can be classified as harmless due to the speed or regarded as a safety risk. These features, states, and/or security categories have preferably been predefined, communicated to the classification module 320, and implemented during a neural network training phase.

The recognized and classified driving situation is processed in the prediction module 330 with regard to the prediction of a future driving situation at a point in time t2. The prediction module 330 calculates at least one future driving situation at a second point in time t2 on the basis of a prediction about the change in the spatial position of the various objects in the area surrounding the vehicle 10 . The shorter the time interval between t1 and t2, the more accurate the prediction of the future driving situation. The future driving situation at a second point in time t2 is therefore weighted with a probability of the actual event occurring, ie the future driving situation. In particular, the prediction module 330 can calculate a number of future driving situations at a second point in time t2, which are each evaluated with a probability of actual occurrence. However, prediction module 330 can also calculate at least one future driving situation at a plurality of points in time ti, for example in a time sequence, each of which is evaluated with a probability of actual occurrence Classification module 320 are classified according to characteristics, states and / or security categories. A first future driving situation can thus have a high probability of occurrence and be evaluated with a low safety level. A second future driving situation, on the other hand, may have a low probability of occurrence, but be rated with a high safety level.

The at least one future driving situation calculated by prediction module 330 is evaluated by evaluation module 340 . The evaluation criteria weigh the probability of arrival and the other characteristics, conditions and security categories against each other. Since there are a number of ethical issues in the field of automated driving in order to ensure the safety of all road users, ethical criteria can also be included in the evaluation criteria, which, for example, have been developed by an ethics committee for automated driving.

The evaluation module 340 outputs an evaluation index 70 based on the evaluation of the at least one future driving situation at a point in time t2 and/or at a further point in time ti. Evaluation index 70 corresponds to a control state for a driver assistance module 80, with three control states being provided in particular, which can be labeled “1—no reaction”, “2—active reaction” and “3—adaptive reaction”.

In the first control state "1 - no reaction" there is no change in driving behavior since the probability of a critical future event occurring is low and/or the resulting danger and/or reduced comfort when driving can be accepted.

In the second control state "2 - active reaction" there is an active change in driving behavior, since the probability of a critical future event occurring is high and/or the probability of a critical future event occurring is low, but the risk potential of the event is very high is, and therefore the resulting danger and / or less comfort when driving must be counteracted in order to ensure safe and / or comfortable driving when the future driving situation occurs.

In the third control state "3 - adaptive reaction" there is a moderate adaptation of the driving behavior, such as no gas supply and a slight reduction in speed, since the probability of a critical future event occurring is in the medium range. The risk can be reduced and/or the driving comfort improved by moderately adapting the driving behavior. In the event that the event does not occur, this third control state is more optimal than the reactive behavior in the first control state.

In a further development, however, it can also be provided that, in addition to the three control states described, a continuum of several control states is provided between the first control state "1 - no reaction" and the second control state "2 - active reaction", each of which has a predictive individual reaction to enable a certain future situation.

The evaluation index 70 is thus forwarded to the driver assistance module 80 and integrated into at least one automated driving function 85 and/or transmitted to the user interface 50 . A driving function 85 is, for example, lane keeping assistance, braking assistance or an autonomous driving function. In the case of an automated driving function 85, for example, the driving speed can be automatically reduced or automatic braking can be carried out by issuing corresponding control commands to control devices 14 of vehicle 10 such as the engine and/or the brakes. In addition, the seat belts can be tightened automatically.

In addition, recommendations for action or warnings can be displayed to the driver of the vehicle 10 via the user interface 50 . For example, a warning tone or a visual indication depending on the evaluation index 70 can be output via the user interface 50 in order to draw the driver's attention to a particular traffic situation that requires increased attention.

Since the evaluation index 70 must be calculated in real time for the driving function 85 to be controlled promptly, the calculation speed is decisive for the large number of calculation operations in the various modules of the analysis system 300 . In order to enable the arithmetic operations to be carried out in real time, the use of the cloud computing infrastructure 45 is therefore advantageous since this can ensure rapid calculation. A 5G mobile phone connection is advantageous for the communication between the data analysis device 30 and the cloud computing infrastructure 45 since data can be transmitted in real time in this way.

In a further development it can also be provided that at least some of the objects in the vicinity of the vehicle 1 also send their recorded data to the cloud computing infrastructure 45, so that the cloud computing infrastructure 45 uses this data on the objects and the Data of the vehicle 10 can simulate the current driving situation and future driving situations.

Safe, comfortable and safe driving is made possible by taking into account future events that are to be expected, estimated or known, such as gradients or curves due to the topography or vehicles cutting into the lane. Future driving situations are calculated using prediction algorithms and weighted with a probability of their occurrence. From this, an evaluation index for the future driving situation is calculated, which enables predictive control of driving functions of vehicle 10 . In particular three control states are proposed, which enable a better adaptation of the driving functions to a future driving situation. The driving behavior in terms of safety, comfort and performance can thus be fine-tuned, which means that the trust in automatic driving functions and thus the driving experience can be significantly improved.

In 3 describes the procedural steps for evaluating a driving situation for the predictive control of an automated driving function.

In a step S10 , sensor and image data 20 are recorded by a camera and sensor device 12 of a vehicle 10 , the image and sensor data 20 depicting the surroundings and objects in the surroundings of the vehicle 10 .

In a step S20, the sensor and image data 20 and/or other historical data from a database 40 are passed on to a data analysis device 30, with the data analysis device 30 comprising an analysis system 300 which uses algorithms from the field of artificial intelligence (AI) and machine image analysis used.

In a step S30, at least one future driving situation is determined from the sensor and image data 20 in the data analysis device 40 using the analysis system 300.

In a step S40, the at least one future driving situation is classified with regard to the probability of its occurrence and/or other features, states or safety categories.

In a step S50, the classified future driving situation is evaluated and an evaluation index 70 is assigned to it.

In a step S60 the evaluation index 70 is supplied to a vehicle assistance module 80 .

In a step S70, an automated driving function is controlled using the evaluation index 70 of the future driving situation.

4 Figure 12 schematically illustrates a computer program product 400 comprising executable program code 450 configured to perform the method according to the first aspect of the present invention when executed.

With the method and the system 100 according to the present invention, the probability of the occurrence of a critical future driving situation can thus be predicted reliably and in real time. Through a finely tuned assessment of the future driving situation, the control parameters of automated driving functions can in turn be adjusted more closely, so that overall safety and comfort when driving are increased.

Reference List

10

vehicle

12

Camera and sensor setup

14

control devices

20

sensor and image data

30

data analysis facility

32

memory module

34

processor

40

Database

45

cloud computing infrastructure

50

user interface

70

rating index

80

driver assistance module

85

driving function

100

system

300

analysis system

310

detection engine

320

classification module

330

prediction module

340

rating module

400

computer program product

450

program code

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102017206862 A1 [0006]
• DE 102020201016 A1 [0007]
• DE 102017114876 A1 [0008]

Claims (15)

Method for evaluating a driving situation for the predictive control of an automated driving function of a vehicle (10), comprising the following method steps: - Recording (S10) of sensor and image data (20) from a camera and sensor device (12) of the vehicle (10), the image and sensor data (20) depicting the environment and objects in the environment of the vehicle (10). for determining a current driving situation at a first point in time (t1); - Forwarding (S20) the determined sensor and image data (20) and/or other historical data from at least one database (40) to a data analysis device (30), the data analysis device (30) comprising an analysis system (300) which algorithms used in the field of artificial intelligence (AI) and machine image analysis; - determining (S30) at least one future driving situation at a second point in time (t2) in the data analysis device (40) by means of the analysis system (300); - Classifying (S40) the at least one future driving situation with regard to the probability of occurrence and/or other features, states or safety categories; - Evaluating (S50) the classified future driving situation and assigning an evaluation index (70); - Supplying (S60) of the evaluation index (70) to a vehicle assistance module (80); - Controlling (S70) at least one automated driving function (85) of the driver assistance module (80) with the evaluation index (70). procedure after claim 1 , wherein the evaluation index (70) has at least three control states that can be represented as "1 - no reaction", "2 - active reaction" and "3 - adaptive reaction". procedure after claim 1 or 2 , The analysis system (300) has a detection module (310) for detecting the current driving situation at a first point in time (t1), a classification module (320) for classifying the current driving situation and/or at least one future driving situation, a prediction module (330) for calculating at least one future driving situation at a second point in time (t2) and for estimating a probability value for the occurrence of the future driving situation, and an evaluation module (340) for evaluating the future driving situation using evaluation criteria. procedure after claim 1 , wherein the detection module (310) has at least one encoder module for extracting image features from the sensor and image data (20) and at least one decoder module for evaluating the extracted image features. Procedure according to one of Claims 1 until 3 , wherein the camera and sensor device (12) has at least one camera and/or at least one such as a LIDAR (Light Detection and Ranging) sensor for optical distance and speed measurement and/or an ultrasonic sensor and/or a radar sensor ( Radio Detection and Ranging) and is designed for a fast recording frequency. Procedure according to one of Claims 1 until 4 , whereby algorithms from the field of artificial intelligence (AI) and machine image analysis are algorithms from the field of machine learning, preferably deep learning with deep neural networks and/or convolutional neural networks and/or learning reinforcement Agents (LV) (Reinforcement Learning Agent) are used to analyze and evaluate a driving situation using the recorded image and sensor data (20) and/or other historical data. Procedure according to one of Claims 1 until 5 , The data analysis device (30) being connected to at least one database (40) and/or a cloud computing infrastructure (45) by means of a communication link. procedure after claim 5 , wherein the communication connection is designed as a mobile radio connection and/or a near-field communication connection such as ^Bluetooth® , Ethernet, NFC (near field communication) or Wi-Fi®. Procedure according to one of Claims 1 until 7 , wherein the data analysis device (30) and/or the camera and sensor device (12) is/are equipped with radio modules of the 5G standard. Procedure according to one of Claims 1 until 9 , wherein a driving function (85) is designed as lane keeping assistance or brake support or autonomous driving function. System (100) for evaluating a driving situation for the predictive control of an automated driving function of a vehicle (10), comprising a camera and sensor device (12) which is designed to record sensor and image data (20), the image and sensor data (20) map the surroundings and objects in the surroundings of the vehicle (10) to determine a current driving situation at a first point in time (t1); a data analysis device (30) having an analysis system system (300), which uses and is designed with algorithms from the field of artificial intelligence (AI) and machine image analysis, at least one future driving situation at a second point in time (t2) from the image and sensor data (20) and other historical data to determine, to classify the at least one future driving situation with regard to the probability of occurrence and/or other features, states or safety categories, to evaluate the classified future driving situation and to assign an evaluation index (70; and a vehicle assistance module (80) to it, which is designed to control at least one automated driving function (85) with the evaluation index (70). system (100) after claim 11 , wherein the evaluation index (70) has at least three control states that can be represented as "1 - no reaction", "2 - active reaction" and "3 - adaptive reaction". system (100) after claim 11 or claim 12 wherein the analysis system (300) has a recognition module (310) for recognizing the current driving situation at a first point in time (t1), a classification module (320) for classifying the current driving situation and/or at least one future driving situation, a prediction module (330) for calculating at least a future driving situation at a second point in time (t2) and for estimating a probability value for the occurrence of the future driving situation, and an evaluation module (340) for evaluating the future driving situation using evaluation criteria. System (100) according to one of Claims 11 until 13 , whereby algorithms from the field of artificial intelligence (AI) and machine image analysis are algorithms from the field of machine learning, preferably deep learning with deep neural networks and/or convolutional neural networks and/or learning reinforcement Agents (LV) (Reinforcement Learning Agent) are used to analyze and evaluate a driving situation using the recorded image and sensor data (20) and/or other data. Computer program product (400), comprising an executable program code (450), which is configured so that when it is executed it performs the method according to any one of Claims 1 until 10 executes

Images