DE102022110761A1 - Driving assistance system and driving assistance method for a vehicle - Google Patents

Abstract

The present disclosure relates to a driving assistance system for a vehicle, in particular a motor vehicle. The driving assistance system includes an environmental sensor system that is set up to detect a vehicle environment and provide corresponding environmental data; an evaluation module that is set up to recognize vegetation in the vehicle surroundings based on the environmental data and to determine at least one biological status parameter for at least one vegetation object in the recognized vegetation; and at least one control module that is set up to control at least one vehicle function based on the at least one biological state parameter.

Description

The present disclosure relates to a driving assistance system for a vehicle, a vehicle with such a driving assistance system, a driving assistance method for a vehicle and a storage medium for executing the driving assistance method. The present disclosure particularly relates to camera-based threat detection based on semantic segmentation.

State of the art

Modern vehicles have built-in environmental sensors that can detect the vehicle's surroundings. The environmental sensor system can include, for example, one or more cameras, radars and/or LiDAR. In reality, the vehicle environment contains many different dynamic objects/subjects such as vehicles, people and animals, static objects such as garbage cans, traffic control systems, natural objects and terrain. When controlling vehicle functions, such as a driver assistance function for automated driving, generally only those objects/subjects that are relevant to driving are taken into account.

However, there are other factors in the vehicle environment that can pose a potential danger to the vehicle, regardless of the driving mode. For example, the vehicle can be damaged by pollen, sugar dew from flowering trees, tree sap, falling fruit, broken branches, falling trees, etc. However, vehicle occupants often cannot or only with difficulty assess such dangers due to a lack of expert knowledge. For example, extensive knowledge of botany would be necessary to recognize that the lime tree will begin to blossom in the next few days or that the pine tree will lose large amounts of resin.

Disclosure of the invention

It is an object of the present disclosure to provide a driving assistance system for a vehicle, a vehicle with such a driving assistance system, a driving assistance method for a vehicle, and a storage medium for executing the driving assistance method, which can prevent damage to the vehicle. In addition, it is an object of the present disclosure to enable reliable hazard detection and hazard avoidance by a driving assistance system.

This task is solved by the subject matter of the independent claims. Advantageous refinements are specified in the subclaims.

According to an independent aspect of the present disclosure, a driving assistance system for a vehicle, in particular a motor vehicle, is specified. The driving assistance system includes an environmental sensor system that is set up to detect a vehicle environment and provide corresponding environmental data; an evaluation module that is set up to recognize vegetation in the vehicle surroundings based on the environmental data and to determine at least one biological status parameter for at least one vegetation object in the recognized vegetation; and at least one control module that is set up to control at least one vehicle function based on the at least one biological state parameter.

The evaluation module and the control module can be implemented in a common software and/or hardware module. Alternatively, the evaluation module and the control module can each be implemented in separate software and/or hardware modules.

According to the invention, as part of an environment recognition, a condition of a vegetation is recognized and used for a hazard assessment and/or hazard avoidance. For example, the vehicle can avoid a parking lot where there is a risk that the vehicle will be contaminated by tree sap from a tree standing there. In another example, the vehicle can automatically change a navigation route if it detects the danger that a branch from a tree is about to fall onto the road. By detecting danger in this way and reacting appropriately to it, damage and/or contamination of the vehicle can be avoided.

The term “vegetation” as used herein refers to plant formations in the vehicle environment. In particular, the vegetation includes the at least one vegetation object, such as a tree, a bush or components thereof. The tree can be, for example, a linden tree, a cherry tree, a lemon tree, a chestnut tree, a spruce, a pine or a fir. The shrub can be, for example, a lilac or a willow. The component can be, for example, a branch, a trunk or a treetop. However, the present disclosure is not limited to this and the at least one vegetation object can be any other plant or component thereof in the vehicle environment that is detected by the environmental sensor system and can pose a potential danger to the vehicle.

The environmental sensor system preferably comprises at least one LiDAR system and/or at least one radar system and/or at least one camera and/or at least one ultrasound system and/or at least one laser scanner. The environmental sensor system can provide the environmental data (also referred to as “environmental data”) that depicts the surrounding area of the vehicle.

The evaluation module is preferably set up to carry out a classification of the at least one vegetation object based on the environmental data. In the context of environment recognition, classification generally means labeling objects and/or subjects recognized in the environment data (e.g. “vehicle”, “pedestrian”, “tree”, etc.). It should be understood that the at least one vegetation object can be a whole plant (e.g. tree or bush) or a component of the whole plant (e.g. branch, treetop and/or fruits) and can be classified accordingly.

Preferably, the classification of the at least one vegetation object (e.g. the tree and/or its components) takes place using semantic segmentation. The term “semantic segmentation” is generally understood to mean a classification of image segments or pixels of an input image into a (fixed) number of classes. In particular, with semantic segmentation, each pixel of an image is assigned a semantic meaning, for example to distinguish between vehicles, pedestrians, bicycles, streets, curbs, buildings, poles, signs, etc. Semantic segmentation can assign a meaning to each individual pixel of the input image assign, both for non-countable “stuff” classes (e.g. roads) and for countable “thing” classes (e.g. vehicles or people).

Preferably, the classification of the at least one vegetation object, in particular the semantic segmentation, uses a neural network.

Preferably, the evaluation module is set up to create a three-dimensional (3D) model of the at least one vegetation object, for example using a depth map derived from the environmental data. In particular, each pixel of an input image can be assigned a depth value.

Preferably, the semantic segmentation is based on camera data, and the depth map can be created from LiDAR data. The camera data and the LiDAR data can therefore be used or merged in combination to create the three-dimensional model of the at least one vegetation object. However, the present disclosure is not limited to camera data and LiDAR data, and other types of sensor types and/or environmental data may be used to enable or perform classification (particularly semantic segmentation) and depth map creation.

Preferably, the LiDAR data is only used for training an algorithm, such as a neural network, for depth estimation. During real operation of the vehicle, the depth can then be determined based on the camera data.

In some embodiments, creating the depth map may be done using a neural network.

Preferably, the evaluation module is further set up to determine at least one geometric parameter of the at least one vegetation object based on the three-dimensional model of the at least one vegetation object. The at least one geometric parameter of the at least one vegetation object can include or be, for example, a height and/or a width and/or a diameter of the at least one vegetation object. For example, the at least one geometric parameter can include or be a breast height diameter and/or a height and/or a crown diameter of a tree.

Preferably, the at least one biological state parameter is selected from the group that includes, or consists of, a healthy state, a diseased state, a dead state, a damaged state, a release state of biological material and a resting state without release of biological material. The at least one biological state parameter can in particular enable a classification of the biological state of the at least one vegetation object. For example, living and dead tree tops (e.g. based on a color: green and/or brown) can be recognized and classified accordingly. Such a recognition or classification, for example of a health status of the at least one vegetation object, can optionally take place taking into account further circumstances, such as a current season and/or a species (e.g. tree species) and/or species-specific appearance of the at least one vegetation object.

In some embodiments, determining the at least one biological condition parameter and/or the health status of the at least one vegetation object can be done using a neural network.

Preferably, the evaluation module is further set up to determine and/or define a danger area around the at least one vegetation object based on the at least one biological status parameter. The danger area can be determined, for example, based on an actual vehicle position, a potential vehicle position (e.g. in a parking lot), and/or a travel route. For example, the danger zone for a parking lot can be estimated or determined based on a height and/or a crown diameter of a flowering or resin-secreting or dead tree that is adjacent to a parking lot.

The driving assistance system is preferably set up to receive at least one additional parameter, in particular to receive it externally. The evaluation module can be set up to further determine the at least one biological status parameter based on the at least one additional parameter. Additionally or alternatively, the control module can be set up to further control the at least one vehicle function based on the at least one additional parameter. In particular, the information obtained from the environmental data can be fused with additional information (or not derived from the environmental data) in order to identify potential dangers to the vehicle.

Preferably, the at least one additional parameter is selected from the group that includes a weather condition (e.g. current weather report, wind, precipitation, etc.), a time (e.g. date, time, time of day, season, etc.), a type of the at least one vegetation object ( e.g. tree species), map information (e.g. digital map data), biological background knowledge (e.g. botanical expert knowledge) and external hazard information (e.g. local backend information on previously recognized trees including species, crown structure, health status, etc.) includes, or consists of.

Preferably, the evaluation module is further set up to determine the danger area around the at least one vegetation object based on the at least one biological status parameter and the at least one additional parameter. For example, the evaluation module can be set up to classify possible dangerous situations, for example based on statistical methods using a type of the at least one vegetation object, a 3D model of the at least one vegetation object, biological background knowledge, a time, a date, a season, Map information, weather information and/or feedback from fleet vehicles regarding hazards.

Preferably, the at least one control module is set up to control the at least one vehicle function based on the at least one biological state parameter in such a way that an acoustic and/or visual and/or haptic warning is issued to a vehicle occupant, in particular a driver. For example, the driver can be informed that a branch is in danger of falling onto the road in front of the vehicle, or that a parking space is at risk of contamination (e.g. from pollen, sugar dew, tree sap, etc.) and/or damage (e.g. from falling objects branches, falling trees and/or fruit).

The acoustic warning can be issued, for example, through a loudspeaker. The visual warning may be displayed on a vehicle display device, such as a head-up display. The haptic warning can be issued via a vibration of a steering wheel.

Preferably, the at least one control module is set up to control the at least one vehicle function based on the at least one biological state parameter in such a way that a travel route, in particular a navigation route, is changed. For example, on a road that the vehicle is traveling on, it can be recognized that there is a risk of falling branches. In this case, a route can be changed, for example by bypassing the section of road that has been identified as dangerous. The change in the route can take place both in manual driving mode and in automated, in particular autonomous, driving mode. In both cases, the change in route can be suggested to a vehicle occupant, in particular a driver.

In some embodiments, the route may be changed based solely on the vehicle receiving an appropriate indication of the danger situation externally (e.g. from a backend or another vehicle). In other words, the route can be changed independently of the ego vehicle's environmental detection.

Preferably, the at least one control module is set up to control the at least one vehicle function based on the at least one biological state parameter in such a way that a parking space in an area surrounding the at least one vegetation object is blocked for parking or indicated as unsuitable. For example, the vehicle can classify the parking space as unsuitable for itself and/or can transmit corresponding information to a backend and/or other vehicles. This information can be used for a parking guidance system, for example.

Preferably, the at least one control module is set up to control the at least one vehicle function based on the at least one biological state parameter in such a way that a functional scope and/or at least one driving parameter of a driving assistance function for automated driving are adapted. In some embodiments, the driving assistance function can be downgraded to automated driving based on the detected dangerous situation. For example, an automated vehicle can avoid a parking space and/or reduce its speed if a corresponding dangerous situation is determined based on the vegetation detected in the vehicle's surroundings.

Preferably, the at least one control module is set up to control the at least one vehicle function based on the at least one biological state parameter in such a way that information relating to the recognized vegetation is sent to at least one external unit. The at least one external unit can be, for example, a backend and/or at least one third-party vehicle.

However, the present disclosure is not limited to the above-mentioned examples of actions on the identified dangerous situation(s) and other actions can be carried out by the driving assistance system in addition or as an alternative. Examples of this include creating a danger scale and deriving a danger heat map based on map information and/or a position of the at least one vegetation object, in particular relative to the vehicle.

According to a further independent aspect of the present disclosure, a vehicle, in particular a motor vehicle, is specified. The vehicle includes the driving assistance system according to the embodiments of the present disclosure.

The term vehicle includes cars, trucks, buses, mobile homes, motorcycles, etc. that are used to transport people, goods, etc. In particular, the term includes motor vehicles for passenger transport.

The driving assistance system is preferably set up for automated driving.

In the context of the document, the term “automated driving” can be understood to mean driving with automated longitudinal or lateral guidance or autonomous driving with automated longitudinal and lateral guidance. Automated driving can, for example, involve driving for a longer period of time on the motorway or driving for a limited period of time as part of parking or maneuvering. The term “automated driving” includes automated driving with any level of automation. Examples of levels of automation include assisted, partially automated, highly automated or fully automated driving. These levels of automation were defined by the Federal Highway Research Institute (BASt) (see BASt publication “Research Compact”, edition 11/2012).

During assisted driving, the driver permanently performs longitudinal or lateral guidance while the system takes over the other function within certain limits. In partially automated driving (TAF), the system takes over longitudinal and lateral guidance for a certain period of time and/or in specific situations, whereby the driver must continuously monitor the system, as with assisted driving. In highly automated driving (HAF), the system takes over longitudinal and lateral guidance for a certain period of time without the driver having to permanently monitor the system; However, the driver must be able to take over control of the vehicle within a certain period of time. With fully automated driving (VAF), the system can automatically handle driving in all situations for a specific application; A driver is no longer required for this application.

The four levels of automation mentioned above correspond to SAE levels 1 to 4 of the SAE J3016 standard (SAE - Society of Automotive Engineering). Furthermore, SAE J3016 specifies SAE level 5 as the highest level of automation, which is not included in the BASt definition. SAE Level 5 corresponds to driverless driving, in which the system can automatically handle all situations throughout the journey like a human driver; a driver is generally no longer required.

According to a further independent aspect of the present disclosure, a driving assistance method for a vehicle, in particular a motor vehicle, is specified. The driving assistance method includes detecting a vehicle environment using an environmental sensor system and providing corresponding environmental data; detecting vegetation in the vehicle surroundings based on the environmental data; determining at least one biological state parameter for at least one vegetation object in the recognized vegetation; and controlling at least one vehicle function based on the at least one biological state parameter.

The driving assistance method can implement the aspects of the driving assistance system described in this document.

According to another independent aspect of the present disclosure, a software (SW) program is provided. The SW program can be set up to run on one or more processors and thereby carry out the driving assistance method described in this document.

According to another independent aspect of the present disclosure, a storage medium is provided. The storage medium may comprise a SW program which is set up to be executed on one or more processors and thereby to carry out the driving assistance method described in this document.

According to a further independent aspect of the present disclosure, software with program code for carrying out the driving assistance method described in this document is to be executed when the software runs on one or more software-controlled devices.

According to another independent aspect of the present disclosure, a driver monitoring system is provided. The system includes one or more processors; and at least one memory connected to the one or more processors and containing instructions executable by the one or more processors to carry out the driver assistance method described in this document.

A processor or a processor module is a programmable arithmetic unit, i.e. a machine or an electronic circuit that controls other elements according to instructions given and thereby drives an algorithm (process).

Brief description of the drawings

• 1 schematisch ein Fahrzeug mit einem Fahrassistenzsystem gemäß Ausführungsformen der vorliegenden Offenbarung,
• 2 schematisch ein Fahrzeug und ein Vegetationsobjekt gemäß Ausführungsformen der vorliegenden Offenbarung,
• 3 schematisch ein Fahrzeug mit einem Fahrassistenzsystem zum automatisierten Fahren gemäß Ausführungsformen der vorliegenden Offenbarung, und
• 4 ein Flussdiagram eines Fahrassistenzverfahrens gemäß Ausführungsformen der vorliegenden Offenbarung.

Embodiments of the disclosure are shown in the figures and are described in more detail below. Show it:

• 1 schematically a vehicle with a driving assistance system according to embodiments of the present disclosure,
• 2 schematically a vehicle and a vegetation object according to embodiments of the present disclosure,
• 3 schematically a vehicle with a driver assistance system for automated driving according to embodiments of the present disclosure, and
• 4 a flowchart of a driving assistance method according to embodiments of the present disclosure.

Embodiments of the disclosure

In the following, unless otherwise noted, the same reference numerals are used for elements that are the same and have the same effect.

1 schematically shows a vehicle 10 with a driving assistance system 100 according to embodiments of the present disclosure.

The driving assistance system 100 includes an environmental sensor system 110, an evaluation module 120 and a control module 130.

In some embodiments, the environmental sensor system 110 may include at least one LiDAR system and/or at least one radar system and/or at least one camera and/or at least one ultrasound system and/or at least one laser scanner. The environmental sensor system 110 can provide environmental data (also referred to as “environmental data”) that depicts the surrounding area of the vehicle 10.

The evaluation module 120 is set up to detect vegetation in the vehicle environment based on the environmental data provided by the environmental sensor system 110 and to determine at least one biological status parameter for at least one vegetation object in the detected vegetation.

The control module 130 is set up to at least one vehicle function based on that determined by the evaluation module 120 to control at least one biological status parameter.

The driving assistance system 100 according to the invention can thus recognize a state of vegetation and carry out a risk assessment and/or risk avoidance for the vehicle 10. For example, the vehicle 10 can avoid a parking lot where there is a risk that the vehicle 10 will be contaminated by tree sap from a tree standing there. In another example, the vehicle 10 may automatically change a navigation route when it detects a threat that a branch from a tree is about to fall onto the roadway. By detecting danger in this way and reacting appropriately to it, damage and/or contamination of the vehicle 10 can be avoided.

2 schematically shows a vehicle 10 and a vegetation object 20 according to embodiments of the present disclosure.

Im in the 2 In the application shown as an example, the vehicle 10, which includes a camera, can be traveling on a street, such as an avenue. The camera of the vehicle 10 can capture the vehicle environment and classify it using, for example, semantic segmentation. For example, a deciduous tree 20 adjacent to the road can be recognized, classified and reconstructed in a three-dimensional model.

In an exemplary case, it can be recognized that some parts of the crown of the deciduous tree 20 have died because it is recognized that the branches of the deciduous tree 20 are devoid of leaves in summer. This information can be stored, for example, in the vehicle 10 and/or in a backend (e.g. danger of branches breaking during a weather event). If a snowstorm occurs in winter, the vehicle 10 or another vehicle can drive around an area around the deciduous tree 20 in order to avoid potentially dangerous situations. Additionally or alternatively, for example, a road maintenance department can be informed, which can then repair the danger zone.

In another application example, the vehicle approaches a parking lot. The vehicle's camera can capture the vehicle's surroundings and classify them using, for example, semantic segmentation. For example, a deciduous tree adjacent to the parking lot (e.g. a linden tree) can be recognized, classified and reconstructed in a three-dimensional model.

Based on a current date (e.g. June) and the weather conditions (e.g. sunny and warm), a high probability of linden blossom can be identified, which would cause severe dirt on the vehicle's paint. The parking space under the deciduous tree can then be classified as potentially hazardous (e.g. damaging to paintwork) for the current time and optionally the following days. This information can be stored in the vehicle and/or a backend. In particular, the parking space can be indexed and/or reported as currently unusable.

3 schematically shows a vehicle 10 with a driving assistance system 300 for automated driving according to embodiments of the present disclosure.

The driving assistance system 300 can include or be the driving assistance system according to the invention, and can detect a state of vegetation in the vehicle surroundings and carry out a risk assessment and/or danger avoidance for the vehicle 10. For example, a functional scope and/or at least one driving parameter of the driving assistance system 300 can be adjusted based on the risk assessment carried out. In particular, a degradation of the automated driving can take place based on the dangerous situation recognized by the driving assistance system 300. For example, the vehicle 10 can automatically avoid a parking space and/or reduce its speed if a corresponding dangerous situation is detected due to the vegetation.

Regarding the 3 The longitudinal and/or lateral guidance of the vehicle 10 occurs automatically during automated driving. The driving assistance system 300 therefore takes over driving the vehicle. For this purpose, the driving assistance system 300 controls the drive 30, the transmission 32, the hydraulic service brake 34 and the steering 36 via intermediate units, not shown.

To plan and carry out automated driving, environmental information from the environmental sensors that monitor the vehicle's surroundings is received by the driver assistance system 300. In particular, the vehicle 10 can include at least one environmental sensor 12, which is set up to record the environmental data that indicates the vehicle surroundings. The at least one environmental sensor 12 can include, for example, a LiDAR system, one or more radar systems and/or one or more cameras and/or one or more laser scanners.

4 schematically shows a flowchart of a driving assistance method 400 according to embodiments of the present disclosure. The driving assistance method 400 can be implemented by appropriate software that can be executed by one or more processors (eg a CPU).

The driving assistance method 400 includes, in block 410, detecting a vehicle environment using an environmental sensor system and providing corresponding environmental data; in block 420, detecting vegetation in the vehicle surroundings based on the environmental data; in block 430, determining at least one biological condition parameter for at least one vegetation object in the detected vegetation; and in block 440, controlling at least one vehicle function based on the at least one biological state parameter.

According to the invention, as part of an environment recognition, a condition of a vegetation is recognized and used for a hazard assessment and/or hazard avoidance. For example, the vehicle can avoid a parking lot where there is a risk that the vehicle will be contaminated by tree sap from a tree standing there. In another example, the vehicle can automatically change a navigation route if it detects the danger that a branch from a tree is about to fall onto the road. By detecting danger in this way and reacting appropriately to it, damage and/or contamination of the vehicle can be avoided.

Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the examples disclosed and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a large number of possible variations exist. It is also to be understood that exemplary embodiments are truly examples only and should not be construed in any way as limiting the scope, application, or configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to concretely implement the exemplary embodiments, whereby the person skilled in the art can make a variety of changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, with knowledge of the disclosed inventive concept, without To leave the scope of protection, which is defined by the claims and their legal equivalents, such as further explanations in the description.

Claims (10)

Driving assistance system (100) for a vehicle (10), comprising: an environmental sensor system (110) that is set up to detect a vehicle environment and provide corresponding environmental data; an evaluation module (120) that is set up to: to detect vegetation in the vehicle surroundings based on the environmental data, and to determine at least one biological status parameter for at least one vegetation object (20) in the recognized vegetation; and at least one control module (130) that is set up to control at least one vehicle function based on the at least one biological state parameter. The driving assistance system (100) according to Claim 1 , wherein the evaluation module (120) is set up to carry out a classification of the at least one vegetation object (20), in particular using semantic segmentation. The driving assistance system (100) according to Claim 1 or 2 , wherein the evaluation module (120) is set up to create a three-dimensional model of the at least one vegetation object (20), in particular using a depth map derived from the environmental data. The driving assistance system (100) according to one of the Claims 1 until 3 , wherein the at least one biological state parameter is selected from the group consisting of a healthy state, a diseased state, a dead state, a damaged state, a release state of biological material and a resting state without release of biological material. The driving assistance system (100) according to one of the Claims 1 until 4 , wherein the evaluation module (120) is set up to determine a danger area around the at least one vegetation object (20) based on the at least one biological status parameter. The driving assistance system (100) according to one of the Claims 1 until 5 , wherein the driving assistance system (100) is set up to receive at least one additional parameter, and wherein: the evaluation module (120) is set up to further obtain the at least one biological state parameter based on the at least one addition to determine parameters, and / or the control module (130) is set up to further control the at least one vehicle function based on the at least one additional parameter, in particular wherein the at least one additional parameter is selected from the group consisting of a weather condition, a time, a Type of at least one vegetation object, map information, biological background knowledge and external hazard information. The driving assistance system (100) according to one of the Claims 1 until 6 , wherein the at least one control module (130) is set up to control the at least one vehicle function based on the at least one biological state parameter in such a way that: an acoustic and / or visual and / or haptic warning is issued to a vehicle occupant; and/or a travel route, in particular a navigation route, is changed; and/or a parking space in an area surrounding the at least one vegetation object (20) is blocked for parking or indicated as unsuitable; and/or a range of functions and/or at least one driving parameter of a driving assistance function can be adapted for automated driving; and/or information relating to the detected vegetation is sent to at least one external unit. Vehicle (10), in particular motor vehicle, comprising the driving assistance system (100) according to one of Claims 1 until 7 . Driving assistance method (400) for a vehicle, comprising: Detecting (410) a vehicle environment using an environmental sensor system and providing corresponding environmental data; Detecting (420) vegetation in the vehicle surroundings based on the environmental data; Determining (430) at least one biological state parameter for at least one vegetation object in the recognized vegetation; and Controlling (440) at least one vehicle function based on the at least one biological state parameter. Storage medium comprising a software program that is set up to be executed on one or more processors and thereby the driving assistance method (440) according to Claim 9 to carry out.

Images