DE102018104270A1 - Method for predicting the behavior of at least one pedestrian - Google Patents

Abstract

The invention provides a method for predicting the behavior of at least one pedestrian in a traffic scene for controlling a motor vehicle having a driving support system, the method comprising the steps of:
Providing one or more images of at least one pedestrian outside the motor vehicle, which are received by at least one camera of the vehicle,
Detecting the at least one pedestrian within the one or more images using a computer vision object detection algorithm,
Determining the position of the at least one pedestrian relative to the motor vehicle,
Estimating the head, torso and / or leg posture of the at least one pedestrian using the computer vision object detection algorithm,
Determining the trajectory and the state of motion of the at least one pedestrian,
- estimating the intention of the at least one pedestrian to predict the behavior of the at least one pedestrian, and
Making a decision on maneuvers of the motor vehicle based on the predicted behavior of the at least one pedestrian.

Description

The invention relates to a method for predicting the behavior of at least one pedestrian in a traffic scene, in particular for controlling a motor vehicle having a driving support system.

Driving assistance systems, such as driver assistance systems, are systems that have been developed to automate, adapt, and improve vehicle systems in terms of safety and improved handling. Security features are designed to prevent collisions and accidents by providing technologies that warn the driver of potential problems or avoid collisions by implementing security measures and taking control of the vehicle. In autonomous vehicles, the driving assistance systems provide inputs for performing control of the vehicle. Adaptive features can automate the lighting, provide adaptive cruise control, automate the braking process, integrate traffic alerts, communicate with smartphones, e.g. Warn the driver about other cars or various dangers, keep the vehicle in the correct lane, or indicate what is in blind spots. Driving assistance systems incorporating the aforementioned driver assistance systems are often based on inputs from multiple data sources, such as on-vehicle imaging devices, image processing, radar sensors, LiDAR, ultrasonic sensors, and other sources. Neural networks have recently been involved in processing such inputs of data in driver assistance systems, or generally in driving support systems.

Recently, driving assistance systems for automobiles incorporating driver assistance systems having all-round vision systems have become more and more popular. In addition, there has been a surge in research into convolutional neural networks (CNNs). Their design has been supported by increasing computing power in computer architectures.

Convolutional neural networks (CNNs) are highly successful in classifying and categorizing tasks, but much of the research is on standard RGB photometric images and does not focus on embedded vehicle devices. Vehicle hardware must have a low power consumption and thus low computing power.

In machine learning, a convolutional neural network is a class of deep, feedforward artificial neural networks that has been successfully used to analyze visual images. CNNs use a variation of multilayer perceptrons that are designed to require minimal preprocessing. Convolutional networks have been inspired by biological processes in which the connectivity pattern between neurons is inspired by the organization of the animal visual cortex. Individual cortical neurons respond to stimuli only in a restricted area of the visual field known as the receptive field. The receptive fields of different neurons partially overlap to cover the entire visual field.

CNNs use relatively little preprocessing compared to other image classification algorithms. This means that the network learns the filters that have been manually developed in conventional algorithms. This independence of prior knowledge and human effort in feature design is a great advantage. CNNs are used in image and video recognition, recommendation systems and natural language processing.

In this regard, the US 9381916 B1 a method and system for autonomously controlling a vehicle that predicts the behavior of objects by detecting position and tracking over time. The method comprises detecting a plurality of mobile objects using one or more sensors of the vehicle, selecting a first one of the plurality of mobile objects corresponding to a second vehicle by one or more processors to determine a likely behavior of the first object, and determining a position and moving the plurality of mobile objects from the perspective of the first object through the one or more processors, determining a likely future behavior of the first object by the one or more processors by determining how the vehicle would behave when the vehicle is on the position of the first object would be arranged if the position and movement of the plurality of mobile objects are predetermined from the perspective of the first object, and autonomously controlling the vehicle with respect to the first object based on the determined probable future behavior d It first object through the one or more processors.

Conventional methods known in the art capture a pedestrian only in a frame by frame with some tracking over time. However, these methods can not handle static pedestrians.

Driving assistance systems, such as driver assistance systems or sophisticated driver assistance systems (ADAS), are one of the fastest growing segments in automotive electronics, and there is a need for improved methods and systems for predicting the behavior of objects in traffic.

It is an object of the present invention to provide a method for predicting the behavior of objects in a traffic scene that may also predict the likely future behavior of static objects.

The object is achieved by the subject matter of the independent claims. Preferred embodiments are described in the subclaims.

• - Bereitstellen eines oder mehrerer Bilder mindestens eines Fußgängers außerhalb des Kraftfahrzeugs, die durch mindestens eine Kamera des Fahrzeugs aufgenommen werden,
• - Erfassen des mindestens einen Fußgängers innerhalb des einen oder der mehreren Bilder unter Verwendung eines Computer-Vision-Objekterfassungsalgorithmus,
• - Bestimmen der Position des mindestens einen Fußgängers relativ zu dem Kraftfahrzeug,
• - Schätzen der Kopf-, Oberkörper- und/oder Beinhaltung des mindestens einen Fußgängers unter Verwendung des Computer-Vision-Objekterfassungsalgorithmus,
• - Bestimmen der Trajektorie und des Bewegungszustands des mindestens einen Fußgängers, und
• - Abschätzen der Absicht des mindestens einen Fußgängers, um das Verhalten des mindestens einen Fußgängers vorherzusagen.

The invention provides a method for predicting the behavior of at least one pedestrian in a traffic scene, in particular for controlling a motor vehicle having a driving support system, the method comprising the following steps:

• Providing one or more images of at least one pedestrian outside the motor vehicle, which are received by at least one camera of the vehicle,
• Detecting the at least one pedestrian within the one or more images using a computer vision object detection algorithm,
• Determining the position of the at least one pedestrian relative to the motor vehicle,
• Estimating the head, torso and / or leg posture of the at least one pedestrian using the computer vision object detection algorithm,
• Determining the trajectory and the state of motion of the at least one pedestrian, and
• Estimate the intention of the at least one pedestrian to predict the behavior of the at least one pedestrian.

The standard algorithm for pedestrian detection known in the art carries out detection via a bounding box, and the footing on the ground is tracked and readjusted to obtain the motion trajectory. In contrast, the inventive method also predicts the intention of the at least one pedestrian to anticipate and predict the trajectory. The method according to the invention is preferably designed for driving scenarios in the inner city area where there is a high probability of interactions with pedestrians. This is particularly advantageous for static pedestrians where tracing information can not be used to predict the trajectory.

Driving assistance systems include driver assistance systems that are already known and used in conventional vehicles. The developed driving assistance systems are provided to automate, adapt and improve vehicle systems in terms of safety and better handling. Security features are designed to prevent collisions and accidents by providing technologies that warn the driver of potential problems or avoid collisions by implementing security measures and taking control of the vehicle. In autonomous vehicles, the driving assistance systems provide inputs for performing control of the vehicle. Adaptive features can automate the lighting, provide adaptive cruise control, automate the braking process, integrate traffic alerts, communicate with smartphones, e.g. Warn the driver about other cars or various dangers, keep the vehicle in the correct lane, or indicate what is in blind spots. Driving assistance systems incorporating the aforementioned driver assistance systems are often based on inputs from multiple data sources, such as on-vehicle imaging devices, image processing, radar sensors, LiDAR, ultrasonic sensors, and other sources.

The method according to the invention can be applied to the following applications: detection of at least one static pedestrian near the roadside and in some specific cases of pedestrians looking in the wrong direction of a dynamic pedestrian moving towards the roadside, detecting a Group behavior of multiple pedestrians, the behavior of a distracted pedestrian, for example: people using their mobile phones or other mobile devices, children, detecting obstacles on the sidewalk and predicting whether the at least one pedestrian will bypass an obstacle and is likely to appear on the street, at least a pedestrian carrying a large crate causing disability, at least one pedestrian looking for an opportunity, across the road at an unmarked junction, ie not at a pedestrian crossing running, and managing a long-term trajectory behavior of at least one pedestrian.

The main input of the system may consist of one or more images supplied by at least one, preferably at least two, more preferably at least three and most preferably at least four cameras, preferably all-round vision cameras such as fish eye lens cameras. The at least four cameras can provide a combined surround view of the surrounding area of the vehicle. The one or more provided images are used to capture at least one pedestrian within the one or more images. Determining the position of the at least one pedestrian relative to the motor vehicle may also include using odometry signals, GPS (Global Positioning System) signals, mechanical signals representing the ego movement of the vehicle, and detecting road context features.

The process according to the invention can be divided into two stages. The first stage includes capturing the one or more images and capturing the at least one pedestrian and locating the pedestrian using the odometry signals representing the ego movement of the vehicle. The term "locate" refers to determining the relative position of the at least one pedestrian with respect to the motor vehicle. The second stage of the method of the invention may include providing limited areas of interest (ROIs) containing the at least one pedestrian and its position as input to a computer vision algorithm that is used to make an estimation of the head, upper body and / or the leg posture of the at least one pedestrian is used. The pedestrian trajectory may be derived using the head, torso and / or stance information and position information to estimate the intent of the at least one pedestrian. In addition, the head, upper body and / or containment information and the position information of the at least one pedestrian may also include information about a possible distraction of the at least one pedestrian, e.g. when a pedestrian is on the phone.

The one or more images captured by at least one camera are then used by a pedestrian sensing system to detect at least one pedestrian within the one or more images. The pedestrian sensing system may be a computer vision object detection algorithm that may include a neural network. Preferably, at least one pedestrian is navigated using a fully convolutional neural network (CNN), e.g. of a single-shot detector (SSD). The pedestrian acquisition output may be a bounding box. The method according to the invention can also detect a plurality of pedestrians or a large number of pedestrians, who can also interact with one another.

Subsequently, the position of the at least one pedestrian is determined relative to the motor vehicle. At least one pedestrian or one or more pedestrians may be located based on a base point estimate and / or a dense structure-from-motion process. This may give an idea of the danger associated with the detected one or more detected pedestrians, with pedestrians located near the vehicle being more vulnerable.

Tracking and trajectory estimation may be performed based on fusion of optical flow information and also by tracking the foot of the detected at least one pedestrian, which may be represented by a state motion block.

After determining the position of the at least one pedestrian relative to the motor vehicle, an estimation of the head posture, the upper body posture and / or the leg posture of the detected at least one pedestrian is carried out. The estimation of upper body posture and leg posture can be combined as an estimation of posture. For reasons of efficiency, the computer vision object detection algorithm used in detecting the at least one pedestrian may include, for example, a neural network, such as a convolutional neural network (CNN), in estimating head posture, upper body posture, and / or leg posture at least one pedestrian can be reused, with additional layers added to determine posture. The neural network, preferably a convolutional neural network (CNN), may have separate posture modules for separate body areas or body parts of the at least one pedestrian, eg, a head posture module, a torso posture module, and / or a leg retention module. Detecting the posture of the legs of the at least one pedestrian may be used to anticipate and grasp the gesture of leg movement preparing to walk. In addition, the constraints of head and body stiffness and their alignment with each other can be modeled because they are in a geometric relationship with each other. A separate determination for the above-mentioned body parts or body parts, For example, head, torso and / or legs, can also give disability robustness.

In addition, determining the position of the at least one pedestrian relative to the motor vehicle may also include detecting road context features such as road, lanes, parking area characteristics, walkway and their obstacles to obtain a global context of the scene and better the intent of the at least one pedestrian to understand.

The method according to the invention can be applied to various combinations of pedestrian and vehicle scenarios. In particular, scenarios may be classified in scenarios where there is less danger to the at least one pedestrian and scenarios where the at least one pedestrian is at greater risk of where a braking or deceleration process may be required. The method of the invention can incorporate all detected objects and pedestrian attributes into a probabilistic graphical model to learn to make decisions in braking and steering.

According to a preferred embodiment of the invention, detecting at least one pedestrian and making an estimate of the head, torso, and / or leg position of the pedestrian using a computer vision object detection algorithm comprises using a neural network. Appreciating the upper body and leg posture can also be combined as an estimate of posture. Preferably, the neural network is a convolutional neural network (CNN). Such a CNN may e.g. be a single-shot detector (SSD).

Preferably, determining the position of the at least one pedestrian relative to the motor vehicle comprises using odometry signals, GPS (Global Positioning System) signals, mechanical signals representing the ego movement of the vehicle, and detecting road context features.

The detection of road context features may include detecting road, lane, walkway, and / or parking area characteristics, wherein the parking area characteristics may include road markings, pedestrian walks, pedestrian crossings, parking area signs, pedestrian signposts, and parking lot occupancy states.

In a preferred embodiment of the invention, the output of the detection of the at least one pedestrian is represented by a bounding box using a computer vision object detection algorithm. The input for making an estimate of the head, torso, and / or leg position of the at least one pedestrian using the computer vision object detection algorithm may be one or more limited areas of interest (ROIs) and their locations indicative of the at least one pedestrian ,

The method preferably has an additional step for making a decision about maneuvers of the motor vehicle based on the predicted behavior of the at least one pedestrian, in particular decisions about braking and / or steering maneuvers of the vehicle. Therefore, the control of the vehicle may be e.g. be executed to avoid contact with the pedestrians, either by stopping the vehicle or continuing a movement of the vehicle, while the pedestrian is dodged.

In a preferred embodiment of the invention, the method predicts the behavior of a plurality of pedestrians. Several pedestrians can interact with each other and even be distracted. Therefore, the behavior of multiple pedestrians may provide other unique characteristics for predicting the behavior of at least one pedestrian.

• - Bereitstellen eines oder mehrerer Bilder mindestens eines Fußgänger außerhalb des Kraftfahrzeugs, die durch mindestens eine Kamera des Fahrzeugs aufgenommen werden,
• - Erfassen des mindestens einen Fußgängers innerhalb des einen oder der mehreren Bilder unter Verwendung eines Computer-Vision-Objekterfassungsalgorithmus,
• - Bestimmen der allgemeinen Richtung des mindestens einen Fußgängers,
• - Schätzen der beabsichtigten Route des mindestens einen Fußgängers,
• - Erfassen von Parkbereichseigenschaften unter Verwendung einer oder mehrerer Kameras,
• - Kombinieren der Ergebnisse der Bestimmung der allgemeinen Richtung und der Schätzung der beabsichtigten Route des mindestens einen Fußgängers mit den erfassten Parkbereichseigenschaften und
• - Vorhersagen der Route, der der mindestens eine Fußgänger folgen wird, basierend auf den kombinierten Ergebnissen.

The invention further provides a method for predicting the behavior of at least one pedestrian on a parking area for controlling a motor vehicle having a driving support system, the method comprising the steps of:

• Providing one or more images of at least one pedestrian outside the motor vehicle taken by at least one camera of the vehicle,
• Detecting the at least one pedestrian within the one or more images using a computer vision object detection algorithm,
• Determining the general direction of the at least one pedestrian,
• Estimating the intended route of the at least one pedestrian,
• Detecting parking area characteristics using one or more cameras,
• Combining the results of determining the general direction and estimating the intended route of the at least one Pedestrian with the detected parking area features and
• Predict the route the at least one pedestrian will follow, based on the combined results.

In general, the description of the above method for predicting the behavior of at least one pedestrian in a traffic scene is also applicable to the method of predicting the behavior of at least one pedestrian on a parking area.

According to a preferred embodiment of this method of the invention, detecting at least one pedestrian within the one or more images using a computer vision object detection algorithm comprises using a neural network. Preferably, the neural network is a convolutional neural network (CNN). Such a CNN may e.g. be a single-shot detector (SSD).

This embodiment of the invention allows a more accurate prediction of pedestrian condition and allows for a more comprehensive early warning system. The model for the pedestrian is improved and supplemented with topological information, for example, a pedestrian's course, a pedestrian route determination algorithm, road markings, signage, and parking area occupancy state. This meta-information allows the drive assist system of the motor vehicle to predict a more accurate position of the at least one pedestrian and a probable path of the at least one pedestrian further into the future, allowing the vehicle propulsion system to take action earlier or alert the driver earlier.

This embodiment of the invention offers the advantage of allowing more accurate modeling of the pedestrian, while conventional models can only predict whether the pedestrian will continue his or her course. By observing some characteristics of pedestrian behavior and properties of the parking area, this embodiment of the invention can more accurately locate the at least one pedestrian and enable longer term predictions for use in driving assistance systems having early warning systems.

Pedestrian characteristics that are helpful and that can be observed according to this embodiment of the invention are: the general direction of the at least one pedestrian, i. their course, and the pedestrian route determination algorithm, such as Manhattan or Shortest Path. By providing the above-mentioned two information items, the vehicle can generally know where the at least one pedestrian is going and how to navigate generally. The method may predict probable routes that the at least one pedestrian will take by combining the above information with the parking area characteristics.

Parking area characteristics which can be observed by this embodiment of the invention are e.g. Road markings, pedestrian walkways, pedestrian crossings, parking area signage, pedestrian signposts indicating the direction to the entrance of a shopping mall and parking area occupancy. By combining this information with the pedestrian characteristics, the method can predict whether a particular pedestrian or a group of pedestrians will continue on a path when he or she can cross the lane, or e.g. whether the pedestrians go to their car or to the entrance of a shopping mall.

The following applications demonstrate the advantages of this embodiment of the invention.

A pedestrian is walking along a row of cars. Ahead is a turnoff. Methods known in the art assume that the pedestrian will continue straight ahead until he ceases to do so and the condition is updated with a new position. In contrast, in this embodiment of the invention, a driving support system configured to carry out the method described above observes the pedestrian walking on a marked path. The pedestrian typically follows a Manhattan path and has a course across the park area. There are no vehicles parked directly on the other side of the vehicle. The method according to the invention predicts that the pedestrian is likely to cross the pedestrian crossing. The driving support system alerts the driver accordingly.

A pedestrian appears from a row of cars. Methods known in the art assume that the pedestrian will continue straight ahead until he ceases to do so, whereupon the condition will be updated with the new position. In contrast, in this embodiment of the invention, a drive assisting system configured to execute the above-described method observes that the pedestrian is heading in the opposite direction to the shopping center signage, indicating that he is returning to his vehicle could. The pedestrian appears from a row of vehicles indicating that he or she might be on a shortest path route. The drive assisting system configured to execute the above-described method knows that vehicles are parked on the opposite side of the vehicle. Accordingly, the vehicle drive assist system predicts that the pedestrian might attempt to cross the lane.

A pedestrian appears from a row of cars. Methods known in the art assume that the pedestrian will continue straight ahead until he ceases to do so, whereupon the condition will be updated with the new position. In contrast, in this embodiment of the invention, a driving assistance system configured to carry out the above-described method observes that the pedestrian steers course in a direction aligned with the signage for a mall entrance. The pedestrian appears from a row of vehicles indicating that he or she might be on a shortest path route. The driving support system configured to carry out the method described above knows that the entrance to a shopping center is on the opposite side. Accordingly, the vehicle driving assistance system may predict that the pedestrian crosses the traffic lane.

A pedestrian walks along a row of vehicles. Ahead is a turnoff. Methods known in the art assume that the pedestrian will continue to walk straight until he does not, then the state will be updated with the new position. In contrast, in the present embodiment of the invention, a driving support system configured to execute the method described above observes that the pedestrian has crossed the lane earlier on a Manhattan route. Ahead is a turnoff. The method according to the present embodiment of the invention then predicts that the pedestrian continues to follow his route and turns at the intersection instead of crossing it.

The invention also specifies the use of the methods described herein in a driving assistance system of a motor vehicle. In particular, the invention provides the use of the method for predicting the behavior of at least one pedestrian in a traffic scene for controlling a motor vehicle in a driving assistance system of a motor vehicle, as described above. The invention further provides the use of the method for predicting the behavior of at least one pedestrian on a parking area for controlling a motor vehicle in a driving support system of a motor vehicle, as described above.

The invention further provides a vehicle ride assist system comprising a camera system having at least one camera, the ride assist system configured to perform the methods described herein.

The camera system of the motor vehicle preferably has at least two, preferably at least three, more preferably at least four cameras, the cameras preferably providing an all-round view of the vehicle. The cameras used in the camera system of the motor vehicle can e.g. Be cameras with fish-eye lenses. Several cameras, for example four cameras, can provide a combined all-round view.

Preferably, the driving assistance system further comprises a GPS receiver and / or detectors for detecting mechanical signals indicative of the ego movement of the vehicle, and for detecting road context features and / or parking area context features. The detection of mechanical signals indicative of the vehicle's own motion by odometry may be performed by detectors and / or sensors that measure steering and / or wheel rotation. Wheel rotation detectors or sensors can detect the full rotational movement of a wheel as well as multiple partial rotations of a wheel with certain intermediate positions.

According to a preferred embodiment of the invention, the driving support system comprises a computer vision object detection algorithm, preferably a neural network, more preferably a convolutional neural network (CNN). The neural network, preferably the convolutional neural network (CNN), may have separate posture modules having separate posture modules for the head, torso, and / or legs.

The invention also provides a non-transitory computer-readable medium having instructions stored thereon that, when executed on a processor, cause a driving support system to perform the methods described herein.

• einer Datenverarbeitungseinrichtung,
• einem nichtflüchtigen computerlesbaren Medium mit darauf gespeicherten Anweisungen, die, wenn sie auf einem Prozessor ausgeführt werden, ein Fahrunterstützungssystem veranlassen, die hierin beschriebenen Verfahren auszuführen, und
• ein Fahrunterstützungssystem für ein Kraftfahrzeug mit einem Kamerasystem, das mindestens eine Kamera aufweist, wobei das Fahrunterstützungssystem dafür konfiguriert ist, die hierin beschriebenen Verfahren auszuführen.

The invention also provides a motor vehicle, comprising:

• a data processing device,
• a non-transitory computer readable medium having instructions stored thereon that, when executed on a processor, cause a driving support system to perform the methods described herein, and
• a drive assist system for a motor vehicle having a camera system having at least one camera, the drive assist system configured to perform the methods described herein.

These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments and examples described below. Individual features set forth in the embodiments, alone or in combination, constitute one aspect of the present invention. Features of the various embodiments may be transferred from one embodiment to another embodiment. Embodiments of the present invention will be further described in the following examples, which are given by way of illustration only and are not intended to limit the invention in any way.

• 1 schematisch einen statischen Fußgänger, der nach einer Gelegenheit zum Überqueren der Straße sucht, als eine beispielhafte Verkehrsszene, in der das erfindungsgemäße Verfahren das Verhalten des Fußgängers vorhersagen kann;
• 2 schematisch zwei statische Fußgänger, die nach einer Gelegenheit zum Überqueren der Straße suchen, die Touristen sind, die fälschlicherweise in die falsche Richtung schauen, als eine beispielhafte Verkehrsszene, in der das erfindungsgemäße Verfahren das Verhalten der Fußgänger vorhersagen kann;
• 3 schematisch Fußgänger, die die Straße überqueren, während sie durch ihre Telefone abgelenkt sind, als eine beispielhafte Verkehrsszene, in der das erfindungsgemäße Verfahren das Verhalten der Fußgänger vorhersagen kann;
• 4 schematisch Fußgänger, die in einem nicht gekennzeichneten Bereich herumlaufen, um die Straße zu überqueren, als eine beispielhafte Verkehrsszene, in der das erfindungsgemäße Verfahren das Verhalten der Fußgänger vorhersagen kann;
• 5 schematisch ein Blockdiagramm einer Ausführungsform des erfindungsgemäßen Verfahrens, das die Schritte aufweist: Bereitstellen mehrerer Bilder mindestens eines Fußgängers, die durch ein Fischaugen-Multikamerasystem mit mehreren Kameras 3 des Fahrzeugs 1 aufgenommen werden, Erfassen des Fußgängers innerhalb des einen oder der mehreren Bilder unter Verwendung eines Computer-Vision-Objekterfassungsalgorithmus (Fußgängererfassung), Bestimmen der Position des mindestens einen Fußgängers relativ zum Kraftfahrzeug 1 (Fußgänger-Auto-Lokalisierung), Abschätzen der Kopf-, Oberkörper- und/oder Beinhaltung des Fußgängers unter Verwendung des Computer-Vision-Objekterfassungsalgorithmus (Fußgängerhaltungserfassung), Bestimmen der Trajektorie und des Bewegungszustandes des Fußgängers (Zustandsbewegung), Abschätzen der Absicht des mindestens einen Fußgängers (Fußgängerabsichtsschätzung), um das Verhalten des mindestens einen Fußgängers (Fußgängerabsicht und -trajektorie) vorherzusagen, worauf basierend eine Entscheidung über Manöver des Kraftfahrzeugs 1 getroffen werden kann;
• 6 schematisch ein detaillierteres Blockdiagramm des ersten Teils (Stufe 1) einer Ausführungsform des erfindungsgemäßen Verfahrens, das die Schritte aufweist: Bereitstellen mehrerer Bilder mindestens eines Fußgängers, die durch ein Fischaugen-Multikamerasystem mit mehreren Kameras 3 des Fahrzeugs 1 aufgenommen werden, Erfassen des Fußgängers in dem einen oder in den mehreren Bildern unter Verwendung eines Computer-Vision-Objekterfassungsalgorithmus (Fußgängererfassung), Bestimmen der Position des mindestens einen Fußgängers relativ zum Kraftfahrzeug 1 (Fußgängerlokalisierung, Auto-Fußgänger-Referenz), was die Verwendung von Odometriesignalen, GPS-Signalen, mechanischen Signalen, die die Ego-Bewegung des Fahrzeugs anzeigen (GPS + Ego-Bewegung) und das Erfassen von Straßenkontextmerkmalen (Fahrspurerfassung) beinhaltet, und Bereitstellen eines Begrenzungsrahmens des Fußgängers (Fußgängereingrenzung) und der Position des Fußgängers und des Fahrzeugs (Fußgänger- und Auto-Lokalisierung);
• 7 schematisch ein detaillierteres Blockdiagramm des zweiten Teils (Stufe 2) einer Ausführungsform des erfindungsgemäßen Verfahrens, das die Schritte aufweist: Bereitstellen einer Eingrenzung des Fußgängers (Fußgängereingrenzung), Schätzen der Kopfhaltung und der Körperhaltung (die die Oberkörper- und/oder Beinhaltung beinhaltet) des mindestens einen Fußgängers unter Verwendung des Computer-Vision-Objekterfassungsalgorithmus, Bestimmen der Trajektorie und des Bewegungszustands des mindestens einen Fußgängers (Körpertrajektorienschätzung, Fußgängerbewegungschätzung), Abschätzen der Absicht des mindestens einen Fußgängers (Fußgängeraufmerksamkeitsschätzung) basierend auf der Fußgängertrajektorie und dem Bewegungszustand, um das Verhalten des mindestens einen Fußgängers (Fußgängerabsicht) vorherzusagen wird, worauf basierend eine Entscheidung über Manöver des Kraftfahrzeugs 1 getroffen werden kann;
• 8 schematisch ein Kraftfahrzeug 1, das einen Fußgänger erfasst, wobei ein Fußgänger durch einen Begrenzungsrahmen für die Körperhaltung, die den Oberkörper und die Beine beinhaltet, und einen Begrenzungsrahmen für die Kopfhaltung dargestellt ist, und wobei eine Ausgabe des Verfahrens die Begrenzungsrahmen, Haltungen und Abstände aufweist;
• 9 schematisch verschiedene Interaktionsmodelle des Kraftfahrzeugs 1, das mit einem Fußgänger interagiert, wobei der Fußgänger beabsichtigen kann, die Straße zu überqueren (oben), oder wobei der Fußgänger beabsichtigen kann, weiterhin auf dem Gehweg zu gehen (unten);
• 10 schematisch verschiedene Interaktionsmodelle des Kraftfahrzeuges 1, das mit Fußgängern interagiert, wobei zwei Fußgänger die Absicht haben können, die Straße zu überqueren, basierend darauf, dass ihre Kopfausrichtung vom Kraftfahrzeug 1 weg zeigt, während ihre Trajektorie zum Pfad der Trajektorie des Fahrzeugs 1 hin zeigt (oben), und ein weiterer Fußgänger beabsichtigen kann, die Straße an einer Abzweigung zu kreuzen, wobei seine Trajektorie zum Pfad der Trajektorie des Fahrzeugs 1 hin zeigt, das dabei ist, rechts abzubiegen (unten); und
• 11 schematisch ein Kraftfahrzeug 1, das ein Rundumsichtkamerasystem mit vier Kameras 3 aufweist, d.h. mit einer Frontkamera 31, einer Rückkamera 32, einer rechten Spiegelkamera 33 und einer linken Spiegelkamera 34, wobei das Rundumsichtkamerasystem einen Fußgänger an einem Zebrastreifen erfasst, gemäß einer Ausführungsform der Erfindung.

Show it:

• 1 schematically a static pedestrian looking for an opportunity to cross the road, as an exemplary traffic scene in which the method of the invention can predict the behavior of the pedestrian;
• 2 schematically, two static pedestrians looking for an opportunity to cross the road, who are tourists who are falsely looking in the wrong direction, as an exemplary traffic scene in which the method of the invention can predict the behavior of pedestrians;
• 3 schematically, pedestrians crossing the street while distracted by their phones as an exemplary traffic scene in which the method of the invention can predict pedestrian behavior;
• 4 schematically, pedestrians walking around an unmarked area to cross the road as an exemplary traffic scene in which the method of the invention can predict pedestrian behavior;
• 5 schematically a block diagram of an embodiment of the inventive method, comprising the steps of providing a plurality of images of at least one pedestrian by a fisheye multi-camera system with multiple cameras 3 of the vehicle 1 Detecting the pedestrian within the one or more images using a computer vision object detection algorithm (pedestrian detection), determining the position of the at least one pedestrian relative to the motor vehicle 1 (Pedestrian car localization), estimating the head, torso and / or leg position of the pedestrian using the computer vision object detection algorithm (pedestrian attitude detection), determining the trajectory and the state of motion of the pedestrian (state movement), estimating the intention of the at least a pedestrian (pedestrian intention estimation) to predict the behavior of the at least one pedestrian (pedestrian intention and trajectory), based thereon a decision on maneuvers of the motor vehicle 1 can be taken;
• 6 schematically a more detailed block diagram of the first part (step 1 ) of an embodiment of the method according to the invention, comprising the steps of: providing a plurality of images of at least one pedestrian by a multi-camera fisheye multi-camera system 3 of the vehicle 1 Detecting the pedestrian in the one or more images using a computer vision object detection algorithm (pedestrian detection), determining the position of the at least one pedestrian relative to the motor vehicle 1 (Pedestrian localization, car pedestrian reference), which includes the use of odometry signals, GPS signals, mechanical signals indicating the ego movement of the vehicle (GPS + ego movement) and the detection of road context features (lane detection), and providing a pedestrian's limitation frame (pedestrian confinement) and the position of the pedestrian and the vehicle (pedestrian and car localization);
• 7 schematically a more detailed block diagram of the second part (stage 2 ) an embodiment of the method of the invention comprising the steps of: providing pedestrian confinement, estimating head posture and posture (including upper body and / or leg posture) of the at least one pedestrian using the computer vision object detection algorithm Determining the trajectory and state of motion of the at least one pedestrian (body trajectory estimation, pedestrian motion estimation), estimating the intention of the trainer at least one pedestrian (pedestrian attention estimation) based on the pedestrian trajectory and the state of motion to predict the behavior of the at least one pedestrian (pedestrian intention), based on which a decision on maneuvers of the motor vehicle 1 can be taken;
• 8th schematically a motor vehicle 1 visualizing a pedestrian, wherein a pedestrian is represented by a posture limiter frame including the torso and the legs, and a head restraint frame, and wherein an output of the method comprises the bounding frames, postures, and distances;
• 9 schematically different interaction models of the motor vehicle 1 interacting with a pedestrian, wherein the pedestrian may intend to cross the street (above), or the pedestrian may intend to continue on the sidewalk (below);
• 10 schematically different interaction models of the motor vehicle 1 interacting with pedestrians, whereby two pedestrians may intend to cross the street based on their head alignment from the motor vehicle 1 shows off while her trajectory to the path of the trajectory of the vehicle 1 indicates (top), and another pedestrian may intend to cross the road at a junction, with its trajectory to the path of the trajectory of the vehicle 1 pointing to the right turn (below); and
• 11 schematically a motor vehicle 1 , which is a four-camera all-round vision camera system 3 has, ie with a front camera 31 , a rear camera 32 , a right mirror camera 33 and a left mirror camera 34 wherein the all-round vision camera system detects a pedestrian at a crosswalk, according to an embodiment of the invention.

example 1

According to this exemplary embodiment of the method for predicting the behavior of at least one pedestrian in a traffic scene for controlling a motor vehicle having a driving support system, the method may include controlling the behavior of one or more pedestrians in the traffic scenes 1 to 4 Predict traffic scenes presented by way of example.

1 schematically shows a static pedestrian, looking for a way to cross the road. 2 schematically shows two static pedestrians looking for a way to cross the road. They are tourists who falsely look in the wrong direction. 3 schematically shows pedestrians crossing the street while distracted by their phones. 4 schematically shows pedestrians walking around in an unmarked area to cross the road.

• - Bereitstellen eines oder mehrerer Bilder mindestens eines Fußgängers außerhalb des Kraftfahrzeugs 1, die durch mindestens eine Kamera 3 des Fahrzeugs aufgenommen werden,
• - Erfassen des mindestens einen Fußgängers innerhalb des einen oder der mehreren Bilder unter Verwendung eines Computer-Vision-Objekterfassungsalgorithmus,
• - Bestimmen der Position des mindestens einen Fußgängers relativ zum Kraftfahrzeug 1,
• - Abschätzen der Kopf-, Oberkörper- und/oder Beinhaltung des mindestens einen Fußgängers unter Verwendung des Computer-Vision-Objekterfassungsalgorithmus,
• - Bestimmen der Trajektorie und des Bewegungszustands des mindestens einen Fußgängers, und
• - Schätzen der Absicht des mindestens einen Fußgängers, um das Verhalten des mindestens einen Fußgängers vorherzusagen.

According to this example, a motor vehicle leads 1 approaching the above exemplary traffic scenes, a method for predicting the behavior of at least one pedestrian in a traffic scene for controlling a motor vehicle 1 that's a driving support system 2 comprising, the method comprising the steps of:

• - Providing one or more pictures of at least one pedestrian outside the motor vehicle 1 passing through at least one camera 3 be absorbed by the vehicle,
• Detecting the at least one pedestrian within the one or more images using a computer vision object detection algorithm,
• - Determining the position of the at least one pedestrian relative to the motor vehicle 1 .
• Estimating the head, torso and / or leg posture of the at least one pedestrian using the computer vision object detection algorithm,
• Determining the trajectory and the state of motion of the at least one pedestrian, and
• Estimating the intention of the at least one pedestrian to predict the behavior of the at least one pedestrian.

Subsequently, a decision on maneuvers of the motor vehicle 1 based on the predicted behavior of the at least one pedestrian.

5 1 schematically shows a block diagram of the above exemplary embodiment of the method according to the invention, comprising the steps of: providing a plurality of images of at least one pedestrian passing through a multi-camera fisheye multi-camera system 3 of the vehicle 1 Detecting the pedestrian within the one or more images using a computer vision object detection algorithm (pedestrian detection), determining the position of the at least one pedestrian relative to the motor vehicle 1 (Pedestrian car localization), estimation of head, torso and / or leg posture of the pedestrian using the computer vision object detection algorithm, determining the trajectory and state of motion of the pedestrian (state movement), estimating the intention of the at least one pedestrian (pedestrian intention estimation) to determine the behavior of the at least one pedestrian (pedestrian intention and trajectory) predict what based a decision on maneuvers of the motor vehicle 1 can be taken.

The standard algorithm for pedestrian detection known in the art carries out detection over a bounding box, tracking and adjusting the root on the ground to obtain the motion trajectory. In contrast, the method according to this exemplary embodiment also predicts the intention of the at least one pedestrian to anticipate and predict the trajectory. The method was developed for urban driving scenarios where pedestrian interactions are highly likely. This is particularly beneficial for static pedestrians, as in the in the 1 and 2 shown traffic scenes, where no tracking information can be used to predict the trajectory.

Driving assistance systems 2 in the motor vehicle 1 can be used include driver assistance systems that are already known and used in conventional vehicles. The developed driving assistance systems are provided to automate, adapt and improve vehicle systems in terms of safety and improved handling. Security features are designed to prevent collisions and accidents by providing technologies that warn the driver of potential problems or avoid collisions by implementing security measures and taking control of the vehicle. In autonomous vehicles provide the driving support systems 2 Inputs for executing a control of the vehicle 1 ready. Adaptive features can automate the lighting, provide adaptive cruise control, automate braking, integrate traffic alerts, communicate with smartphones, alerting the driver to other cars or various dangers, keeping the vehicle in the correct lane, or displaying what is in blind spots located.

Driving assistance systems incorporating the aforementioned driver assistance systems are often based on inputs from multiple data sources, such as on-vehicle imaging devices, image processing, radar sensors, LiDAR, ultrasonic sensors, and other sources.

The method according to this example can be applied to the following applications: tracking and intent detection of at least one static pedestrian near the roadside, as in 1 and pedestrians who are looking in the wrong direction, as in 2 For example, tourists who falsely look in the wrong direction.

Other examples of pedestrian behavior detected in this example are a dynamic pedestrian going to the side of the road, detecting a group behavior of several pedestrians, behavior of a distracted pedestrian, eg: people using their mobile phone, children, such as in 3 detecting obstacles on the sidewalk and predicting to see if the at least one pedestrian is likely to appear on the road to bypass the obstacle, at least one pedestrian carrying a large crate, causing disability, at least a pedestrian looking for a way to cross the road at an unmarked junction, ie where there is no crosswalk, such as in 4 and managing the long-term trajectory behavior of at least one pedestrian.

The main input for the driver assistance system 2 The process that performs this procedure is one or more images taken by multiple cameras 3 be supplied, for example, all-round vision cameras, such as cameras with fish-eye lenses, a combined panoramic view of the surrounding area of the motor vehicle 1 provide. 11 schematically shows an exemplary embodiment of a motor vehicle 1 with a panoramic camera system that has four cameras 3 has, ie a front camera 31 , a back camera 32 , a right mirror camera 33 and a left mirror camera 34 in which the panoramic camera system detects a pedestrian at a crosswalk.

The one or more provided images are used to capture at least one pedestrian within the one or more images. Determining the position of the at least one pedestrian relative to the motor vehicle 1 It also includes the use of odometry signals, Global Positioning System (GPS) signals, mechanical signals representing the vehicle's ego movement, and the detection of road context features.

The method according to this exemplary embodiment may be divided into two stages. The first stage, as on the left in the 5 and 6 includes capturing one or more images by at least one camera 3 and detecting the at least one pedestrian and locating the at least one pedestrian using the odometry signals representative of the ego movement of the motor vehicle 1 represent. The term "locate" refers to determining the relative position of the at least one pedestrian with respect to the motor vehicle 1 ,

In particular shows 5 schematically a more detailed block diagram of the first stage (stage 1 ) an embodiment of the method according to the invention, comprising the steps of providing a plurality of images of at least one pedestrian, which are taken by a fisheye multi-camera system, the plurality of cameras 3 of the vehicle 1 comprising detecting the pedestrian within the one or more images using a computer vision object detection algorithm (pedestrian detection), determining the position of the at least one pedestrian relative to the vehicle 1 (Pedestrian localization, car pedestrian reference), which include the use of odometry signals, GPS (Global Positioning System) signals, mechanical signals representing the ego movement of the vehicle (GPS + Ego motion), and the detection of road context features ( Lane detection), and providing pedestrian confinement (pedestrian confinement) and locating the pedestrian and the vehicle (pedestrian and vehicle location).

The second stage of the process according to this embodiment (step 2 ), as on the right side of 5 and 7 Fig. 12 illustrates providing limited areas of interest (ROIs) containing the at least one pedestrian and its position as input to a computer vision algorithm for making an estimate of head, torso, and / or leg posture the at least one pedestrian is used. The pedestrian trajectory is derived using the head, torso and / or stance information and position information to estimate the intent of the at least one pedestrian. In addition, the head, upper body and / or containment information and the position information of the at least one pedestrian also provide information about a possible distraction of the at least one pedestrian, for example, whether a pedestrian is on the phone, as in 3 is shown.

In particular shows 7 2 schematically illustrates a more detailed block diagram of the second stage of an embodiment of the method of the invention, including the steps of providing pedestrian confinement, estimation of head posture and posture (including upper body and / or leg posture) of the at least one pedestrian using the Computer vision object detection algorithm, determining the trajectory and state of motion of the at least one pedestrian (body trajectory estimation, pedestrian motion estimation), and estimating the intention of the at least one pedestrian (pedestrian attention estimation) based on the pedestrian trajectory and motion state to determine the behavior of the at least one pedestrian ( Pedestrian intention) to propose what based a decision on maneuvers of the motor vehicle 1 can be taken.

The one or more pictures taken by at least one camera 3 are used by a pedestrian detection system for detecting at least one pedestrian in the one or more images. The pedestrian detection system is a computer vision object detection algorithm that includes a neural network. In this exemplary embodiment, the at least one pedestrian is detected using a fully convolutional neural network (CNN), eg, a single-shot detector (SSD). The pedestrian detection issue is a bounding box. The method according to this exemplary embodiment may also detect a plurality of pedestrians or a plurality of pedestrians, who may also interact with each other and may even be distracted.

Subsequently, the position of the at least one pedestrian relative to the motor vehicle 1 certainly. At least one pedestrian or one or more pedestrians are located based on a base point estimate and / or a dense structure-from-motion process. This may provide an idea of the danger associated with the detected one or more pedestrians, with pedestrians near the motor vehicle 1 may be more vulnerable.

The tracking and the trajectory estimation are performed based on a fusion of optical flow information and also by tracking the foot of the detected at least one pedestrian, represented by a state motion block.

After determining the position of the at least one pedestrian relative to the motor vehicle 1 An estimation of the head posture is made Upper body posture and / or the leg position of the detected at least one pedestrian. For the sake of efficiency, the computer vision object detection algorithm, in this example a convolutional neural network (CNN), used in detecting the at least one pedestrian is used again to estimate the head posture, upper body posture and / or containment of the detected at least one pedestrian, with additional layers added to determine the posture. The convolutional neural network (CNN) has separate posture modules for separate body areas or body parts of the at least one pedestrian, eg, a head posture module, a torso posture module, and a leg posture module. Detecting the posture of at least one pedestrian's legs is used to anticipate and recognize the gestures of a first leg movement that is ready for walking. In addition, the boundary conditions for the stiffness of the head and body and their orientation to each other are modeled, since they are in a geometric relationship to each other. The separate determination for the aforementioned body parts or body areas, eg head, torso and legs, also brings robustness to disabilities.

In addition, determining the position of the at least one pedestrian relative to the motor vehicle includes 1 also detecting road context features, such as road, lanes, parking area properties, walkway and its obstacles, to maintain a global context of the scene and to better understand the intention of the at least one pedestrian.

The method according to this exemplary embodiment may be applied to various combinations of pedestrian and automobile scenarios. In particular, scenarios may be classified into scenarios where there is less danger to the at least one pedestrian and in scenarios where there is a high risk to the at least one pedestrian where a deceleration or deceleration process might be necessary. The method of this example incorporates all captured object and pedestrian attributes into a probabilistic graphical model to learn to make decisions for braking and steering maneuvers.

According to this example, detecting at least one pedestrian and estimating the head, torso, and / or leg position of the pedestrian using a computer vision object detection algorithm includes using a convolutional neural network (CNN) as a neural network. Estimating the upper body posture and leg posture can also be combined to obtain a bounding frame as a posture for posture, as in 8th is shown. Such a CNN may be, for example, a single-shot detector (SSD).

Determining the position of the at least one pedestrian relative to the motor vehicle 1 involves using odometry signals, GPS (Global Positioning System) signals, mechanical signals representing the ego movement of the vehicle, and detecting road context features.

Detecting road context features involves detecting road, lane, walkway and / or parking area characteristics. Such parking area features include road markings, pedestrian walks, pedestrian crossings, parking area signage, pedestrian signage, and parking area slip.

The output of the detection of the at least one pedestrian using a computer vision object detection algorithm is represented by a bounding box, as in FIGS 8th . 9 and 10 is shown. The input for estimating the head, torso and / or leg position of the at least one pedestrian using the convolutional neural network (CNN) are one or more limited areas of interest (ROIs) and their locations indicative of the at least one pedestrian.

In particular shows 8th schematically a motor vehicle 1 capturing a pedestrian. The pedestrian is represented by a posture limiter frame comprising the torso and legs, and a head restraint frame. The output of the method includes the bounding boxes, stalls, and spacings.

9 schematically shows various interaction models of the motor vehicle 1 interacting with a pedestrian, where the pedestrian may intend to cross the street (above), or the pedestrian may intend to continue on the sidewalk (below).

10 schematically shows various interaction models of the motor vehicle 1 that interacts with pedestrians. Two pedestrians intend to cross the road. Your head alignment points from the motor vehicle 1 away, while their trajectory towards the path of the trajectory of the vehicle 1 points (above). Another pedestrian may intend to cross the road at a turnoff, with its trajectory toward the trajectory of the vehicle 1 shows that is just turning right (below).

The decision on maneuvers of the motor vehicle 1 based on the predicted behavior of the at least one pedestrian involves decisions about braking and / or steering maneuvers. The method according to this exemplary embodiment may also predict the behavior of multiple pedestrians. Several pedestrians can interact with each other and even be distracted.

Example 2

• - Bereitstellen eines oder mehrerer Bilder mindestens eines Fußgängers außerhalb des Kraftfahrzeugs 1, die durch mindestens eine Kamera 3 des Fahrzeugs aufgenommen werden,
• - Erfassen des mindestens einen Fußgängers innerhalb des einen oder der mehreren Bilder unter Verwendung eines Computer-Vision-Objekterfassungsalgorithmus,
• - Bestimmen der allgemeinen Richtung des mindestens einen Fußgängers,
• - Schätzen der beabsichtigten Route des mindestens einen Fußgängers,
• - Erfassen von Parkbereichseigenschaften unter Verwendung einer oder mehrerer Kameras 3,
• - Kombinieren der Ergebnisse der Bestimmung der allgemeinen Richtung und der Schätzung der beabsichtigten Route des mindestens einen Fußgängers mit den erfassten Parkbereichseigenschaften, und
• - Vorhersagen der Route, der der mindestens eine Fußgänger folgen wird, basierend auf den kombinierten Ergebnissen.

Another exemplary embodiment is a method for predicting the behavior of at least one pedestrian on a parking area for controlling a motor vehicle 1 , which is a driving support system 2 comprising, the method comprising the steps of:

• - Providing one or more pictures of at least one pedestrian outside the motor vehicle 1 passing through at least one camera 3 be absorbed by the vehicle,
• Detecting the at least one pedestrian within the one or more images using a computer vision object detection algorithm,
• Determining the general direction of the at least one pedestrian,
• Estimating the intended route of the at least one pedestrian,
• - Detecting parking area properties using one or more cameras 3 .
• Combining the results of determining the general direction and the estimate of the intended route of the at least one pedestrian with the detected parking area characteristics, and
• Predict the route the at least one pedestrian will follow, based on the combined results.

This embodiment of the invention allows a more accurate prediction of pedestrian condition and allows for a more comprehensive early warning system. The model for the pedestrian is improved and supplemented with topological information, such as the pedestrian course, a pedestrian route determination algorithm, road markings, signage and parking area occupancy. This meta-information enables the driving support system 2 of the motor vehicle 1 to further predict a more accurate position of the at least one pedestrian and a likely path of the at least one pedestrian into the future to the driving support system 2 allow the vehicle to take action earlier or to warn the driver earlier.

This embodiment of the invention offers the advantage of allowing more accurate modeling of the pedestrian, while conventional models can only predict whether the pedestrian will continue his or her course. By observing some characteristics of pedestrian behavior and parking area characteristics, this embodiment of the invention can more accurately locate the at least one pedestrian and allows longer term prediction for use in driving support systems having early warning systems.

Pedestrian characteristics that may be helpful in this exemplary embodiment and can be observed are: the general direction of the at least one pedestrian, ie, its course, and the pedestrian route determination algorithm, such as Manhattan or the Shortest Path. Using the above-mentioned two information elements, the motor vehicle 1 general knowledge of where the at least one pedestrian is going and how he generally navigates. The method may predict likely routes that the at least one pedestrian will take by combining the aforementioned information elements with the parking area characteristics.

Parking area characteristics observed by this exemplary embodiment are e.g. Road markings, pedestrian walkways, pedestrian crossings, parking area signage, information signs directing the pedestrian to the entrance of a mall, and parking area occupancy. By combining this information with the pedestrian characteristics, the method can predict whether a particular pedestrian or a group of pedestrians will continue on a path, whether he or she can cross the lane, or whether e.g. the pedestrians go to their vehicle or to the entrance of a shopping center.

In this example, the following applications demonstrate the advantages of the method according to this embodiment:

A pedestrian walks along a row of cars. Ahead is a turnoff. Methods known in the art assume that the pedestrian will continue to go straight until he ceases to do so, whereupon the condition will be updated with the new position. In contrast, the driving support system is watching 2 in this example configured to perform the method described above, the pedestrian running on a marked path. The pedestrian follows a typical Manhattan path and heads for the park area. There are no vehicles parked directly on the other side of the vehicle. The method according to this exemplary embodiment predicts that the pedestrian is likely to cross the pedestrian crossing. The driving support system 2 warns the driver of the vehicle 1 corresponding.

A pedestrian appears from a row of cars. Methods known in the art assume that the pedestrian continues to walk straight until he ceases to do so, whereupon the condition is updated with the new position. In contrast, in this example, the driving support system is observed 2 configured to execute the method described above, that the pedestrian follows a course in the direction opposite the signage for the shopping center, indicating that he could return to his vehicle. The pedestrian appears from a row of vehicles, indicating that he or she could follow a shortest path route. The driving support system 2 configured to execute the method described above, knows that vehicles are parked on the opposite side of the vehicle. Accordingly, the driving support system says 2 of the motor vehicle 1 before that the pedestrian might try to cross the lane.

A pedestrian appears from a row of cars. Methods known in the art assume that the pedestrian continues to walk straight until he ceases to do so, whereupon the condition is updated with the new position. In contrast, in this example, the driving support system is observed 2 configured to execute the above-described method that the pedestrian follows a course that is aligned with the signage for a mall entrance. The pedestrian appears from a row of vehicles indicating that he or she might be on a shortest path route. The driving support system 2 , which is configured to carry out the method described above, knows that the entrance of a shopping center is on its opposite side. Accordingly, the driving support system says 2 of the motor vehicle 1 before that the pedestrian might try to cross the lane.

The pedestrian walks along a row of vehicles. Ahead is a turnoff. Methods known in the art assume that the pedestrian continues to walk straight until he ceases to do so, whereupon the condition is updated with the new position. In contrast, in this example, the driving support system is observed 2 configured to execute the above-described method that the pedestrian has previously crossed the lane on a Manhattan route. Ahead is a turnoff. The method according to this exemplary embodiment then predicts that the pedestrian will continue his route and turn at the intersection instead of crossing it.

LIST OF REFERENCE NUMBERS

1

motor vehicle

2

Driving assistance system

3

camera

31

front camera

32

reverse camera

33

right mirror camera

34

left mirror camera

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 9381916 B1 [0007]

Claims (15)

Method for predicting the behavior of at least one pedestrian in a traffic scene, in particular for controlling a motor vehicle (1) having a driving support system (2), the method comprising the steps: Providing one or more images of at least one pedestrian outside the motor vehicle (1), which are received by at least one camera (3) of the vehicle, Detecting the at least one pedestrian within the one or more images using a computer vision object detection algorithm, Determining the position of the at least one pedestrian relative to the motor vehicle (1), Estimating the head, torso and / or leg posture of the at least one pedestrian using the computer vision object detection algorithm, Determining the trajectory and the state of motion of the at least one pedestrian, and Estimate the intention of the at least one pedestrian to predict the behavior of the at least one pedestrian. Method according to Claim 1 wherein detecting the at least one pedestrian and estimating the head, torso and / or leg posture of the pedestrian using a computer vision object detection algorithm comprises using a neural network, preferably using a convolutional neural network (CNN). Method according to Claim 1 or 2 wherein determining the position of the at least one pedestrian relative to the motor vehicle (1) comprises using odometry signals, GPS (Global Positioning System) signals, mechanical signals representing the ego movement of the vehicle, and detecting road context features. Method according to Claim 3 wherein detecting road context features comprises detecting road, lane, walkway, and / or parking area characteristics, the parking area characteristics including road markings, pedestrian walks, pedestrian crossings, parking area signs, pedestrian signage, and parking area occupancy. Method according to one of Claims 1 to 4 wherein the output of the detection of the at least one pedestrian is represented by a bounding frame using a computer vision object detection algorithm, and preferably wherein the input is for estimating the head, torso and / or leg posture of the at least one pedestrian using the computer Vision object detection algorithm is one or more limited areas of interest (ROIs) and their locations indicating the at least one pedestrian. Method according to one of Claims 1 to 5 with an additional step for making a decision on maneuvers of the motor vehicle (1) based on the predicted behavior of the at least one pedestrian, in particular decisions about braking and / or steering maneuvers of the vehicle. Method according to one of Claims 1 to 6 The method predicts the behavior of several pedestrians. A method for predicting the behavior of at least one pedestrian on a parking area for controlling a motor vehicle (1) having a driving support system (2), the method comprising the steps of: Providing one or more images of at least one pedestrian outside the motor vehicle (1), which are received by at least one camera (3) of the vehicle, Detecting the at least one pedestrian within the one or more images using a computer vision object detection algorithm, Determining the general direction of the at least one pedestrian, Estimating the intended route of the at least one pedestrian, Detecting parking area characteristics using one or more cameras (3), Combining the results of the determination of the general direction and the estimate of the intended route of the at least one pedestrian with the detected parking area characteristics and Predict the route the at least one pedestrian will follow, based on the combined results. Use of the method according to one of Claims 1 to 8th in a driving support system (2) of a motor vehicle (1). A driving support system (2) for a motor vehicle (1) comprising a camera system having at least one camera (3), the driving support system (2) being configured to perform the method of any one of Claims 1 to 9 perform. Driving support system (2) after Claim 10 , wherein the camera system has at least two, preferably at least three, more preferably at least four cameras (3), wherein the cameras (3) preferably provide an all-round view of the vehicle (1). Driving support system after Claim 10 or 11 wherein the driving assistance system (2) further comprises a GPS receiver and / or detectors for detecting mechanical signals representing the ego movement of the vehicle and for detecting road context features and / or parking area context features. Driving assistance system according to one of Claims 10 to 12 wherein the driving support system (2) comprises a computer vision object detection algorithm, preferably a neural network, more preferably a convolutional neural network (CNN), the convolutional neural network (CNN) preferably having separate attitude modules comprising separate head positioning modules Upper body and / or legs have. A non-transitory computer readable medium having instructions stored therein that, when executed by a processor, cause a driving support system (2) to perform the method of any one of Claims 1 to 9 perform. A motor vehicle (1) comprising: a data processing device, a non-transitory computer-readable medium Claim 14 and the driving support system (2) according to any one of Claims 10 to 13 ,

Images