DE102012009703A1 - Method for actively protecting e.g. cyclist in environment of motor car, involves determining endangered object by data fusion unit, and providing actuator in car controller when endangerment measure is larger than predetermined threshold - Google Patents

Abstract

The method involves detecting image data of an environment of a motor car using an image-receiving sensor system (1). A fully or partially detected person objects in the environment of the motor car is determined by a person recognition unit. Completely visible face objects are determined by a face detection unit, and an endangered object is determined by a data fusion unit of the detected face objects with the fully or partially detected person objects. A control actuator (4) is provided in a motor car controller when an endangerment measure is larger than a predetermined threshold. An independent claim is also included for a device for actively protecting a person in an environment of a motor car.

Description

The invention relates to a method for active protection of persons moving in the environment of a motor vehicle, such as pedestrians or cyclists, in a motor vehicle according to the preamble of claim 1 and a corresponding device according to the preamble of claim 7.

Assistance systems in motor vehicles to avoid or at least reduce the consequences of an accident of a motor vehicle with an object are known. For example, emergency braking systems attempt to avoid an imminent collision with an object in the driving lane by corresponding actions or to reduce the consequences of the collision if the driver does not take any measures within a given time window. Therefore, if a point in time is reached, from which a collision is practically unavoidable taking into account the reaction time of the driver, the assistance system takes over the control of the vehicle and initiates appropriate actions, such as an evasive maneuver or full braking.

On the one hand, such emergency brake systems can be implemented as independent systems in motor vehicles or as part of complex assistance systems, as may be the case, for example, with an ACC system with start-stop function or with a parking assistant. Now such assistance systems with emergency braking function are usually designed to respond to other located in the vicinity of the vehicle or motor vehicles standing objects, so on relatively large objects.

Problems are provided by forward-looking protection systems or protective methods which are intended to avoid collisions with persons in the vicinity of the motor vehicle or to reduce the consequences of accidents, since accidents between a motor vehicle and a person are generally at the expense of the person involved in the accident. People are understood here to be essentially pedestrians or cyclists, but they may also be the driver of a moped. A person should therefore be understood as an unprotected human road user in this context.

Such predictive protection systems for pedestrians or cyclists evaluate on the basis of sensor data, such as a monocamera or a radar, the danger of a traffic situation with a person participating in the traffic. Depending on the determined dangerousness, which is realized by a suitable measure, in the case of a potential collision between a motor vehicle and a person participating in the traffic, i. H. Pedestrians or cyclists, initiated actions to avoid a collision or to reduce the collision consequences, for example by emergency braking.

From the publication US 7,961,906 B2 For example, an apparatus and method for detecting people and their discrimination from the background or other obstacles is known, wherein an infrared camera used in conjunction with at least one other optical sensor, such as a color camera, stereo camera or a LIDAR system, to determine the identical physical situation are sought, in the respective images for suitable characteristics, from which the existence and position of a human person can be derived. The system can also be used in motor vehicles.

Furthermore, from the document EP 2 288 138 A1 a device of a motor vehicle for the detection of objects in advance of the motor vehicle is known, wherein a pedestrian is detected in a camera image by the detection of characteristic edges in a plurality of sub-areas by means of a gray scale gradient method and their evaluation by means of suitable thresholds. Thus, a type of contour recognition of a person is performed, wherein the size of the contour of a possible person is a function of the measured distance of the object found to the vehicle. The disadvantage here is that the found contour can only be assigned with certainty to a human person when the contour is substantially complete, in other words, the contour of the person is completely visible in the camera image and is not partially obscured.

In addition to the methods for the recognition of persons, which are based essentially on the contour recognition, especially in the field of computer vision method of face recognition in development, as for example in the publication P. Viola et. al .: "Robust Real-Time Face Detection", International Journal of Computer Vision 57 (2), 137-154, 2004 , is described.

The invention has for its object to provide a method and an apparatus for the active protection of moving in the environment of a motor vehicle persons with improved recognition ability of the person.

This object is achieved by a method for the active protection of persons moving in the vicinity of a motor vehicle with the features of claim 1 and by a system for active protection of persons moving in the vicinity of a motor vehicle with the features of Claim 7 solved. Preferred embodiments of the invention are the subject of the dependent claims.

• – Erfassen von Bilddaten des Umfeldes des Kraftfahrzeugs mittels eines bildaufnehmenden Sensorsystems,
• – Durchführen einer Bildverarbeitung der erfassten Bilddaten mit – einer Personenerkennung zur Bestimmung von vollständig oder teilweise erkannten Personenobjekten in der Kraftfahrzeugumgebung, – einer Gesichtserkennung zur Bestimmung vollständig sichtbarer Gesichtsobjekte, und – eine Datenfusion erkannter Gesichtsobjekte mit vollständig oder teilweise erkannten Personenobjekten zur Bestimmung potentiell gefährdeter Objekte,
• – Beurteilung des Maßes der Gefährdung der potentiell gefährdeten Objekte, und
• – Durchführen eines Eingriffs in die Kraftfahrzeugsteuerung, falls das Gefährdungsmaß größer als eine vorgegebene Schwelle ist.

The method according to the invention for the active protection of persons in the vicinity of a motor vehicle, in particular pedestrians and cyclists, comprises the following steps:

• Acquiring image data of the environment of the motor vehicle by means of an image-capturing sensor system,
• Performing an image processing of the acquired image data with a person recognition for the determination of completely or partially recognized person objects in the motor vehicle environment, a face recognition for the determination of completely visible face objects, and a data fusion of recognized face objects with completely or partially recognized person objects for the determination of potentially endangered objects,
• - assessment of the degree of endangerment of potentially endangered objects, and
• - Performing an intervention in the motor vehicle control, if the risk measure is greater than a predetermined threshold.

By combining person recognition and face recognition, different characteristics of a person, i. H. a pedestrian or a cyclist, are linked together, which leads to improved detection reliability and thus to a reduced false triggering rate. For person recognition, pattern recognition methods are preferably used which search, for example, for pedestrian-like contours in the captured images. Likewise, the face recognition is preferably carried out by means of pattern recognition methods, whereby the detected images are searched in a targeted manner for faces and facial features. Faces are distinguished in frontal views, side views and rotated views.

By linking partially recognized persons with the detection of full faces, there is the possibility that the method described here reacts to pedestrians or cyclists vulnerable to vehicle and personal movement, which are not completely visible, but whose face is unobstructedly visible to an image-receiving sensor.

Preferably, the person recognition for the recognition of personal objects and / or the face recognition for the recognition of facial objects over a predetermined number of time sequentially recorded images of the image data is performed. In this way, a so-called tracking can be performed so that temporal changes, ie movements of people and faces are recognizable and comprehensible. In this way, even short-term partial coverage of detected persons can be taken into account.

More preferably, the data fusion comprises a plausibility check of the data to be merged and the merged data. A plausibility check can be carried out by linking position and distance data of the persons and faces or generally by means of position data. The plausibility check can be done for the detected or partly recognized persons and completely recognized faces, it can also be a plausibility check of the merged data.

Preferably, upon complete or partial detection of personal objects and fully visible facial objects, a quality of recognition of the fused data is determined which is used to trigger the intervention in the motor vehicle control. Usually, the recognition quality of fused data of completely recognized persons and faces is higher than the recognition quality of the respective individual unfused data. The same applies to the merged data of partially recognized persons and fully recognized faces.

Further preferably, in the case of recognized facial objects, to which no person object can be assigned, a triggering of an intervention in the motor vehicle control takes place on the basis of a plausibility check of the recognition quality of the recognized facial object. In this way, the safety for pedestrians and cyclists is increased, since triggering an emergency measure, such as braking the vehicle, even with a fully recognized face can be done, although no person matching the face has been currently recognized. Such a situation may be present, for example, if the person is covered so far that a person contour is not recognizable in the picture.

More preferably, a plausibility check and / or fusion of recognized facial objects with data from non-image-capturing environmental sensors takes place. Such non-image-taking environment sensors can be, for example, radar sensors, which determine the distance of an obstacle that can be used to check the plausibility of a completely recognized face.

• – ein bildaufnehmendes Sensorsystem zur Erfassung von Bilddaten der Umgebung des Kraftfahrzeugs,
• – eine Einrichtung zur Verarbeitung der Bilddaten des bildaufnehmenden Sensorsystems mit – einer Personenerkennungseinrichtung zur Bestimmung von vollständig oder teilweise erkannten Personenobjekten in der Kraftfahrzeugumgebung, – einer Gesichtserkennungseinrichtung zur Bestimmung vollständig sichtbarer Gesichtsobjekte, und – eine Datenfusionseinrichtung zur Datenfusion erkannter Gesichtsobjekte mit vollständig oder teilweise erkannten Personenobjekte zur Bestimmung gefährdeter Objekte,
• – eine Bewertungs- und Entscheidungseinrichtung zur Bewertung der gefährdeten Objekte, und
• – eine Einrichtung zur Ansteuerung von Aktoren basierend auf den Ergebnissen der Bewertung- und Entscheidungseinrichtung.

The device according to the invention for the active protection of persons moving in the environment of a motor vehicle, in particular pedestrians and cyclists, for carrying out the method described above comprises:

• An image-capturing sensor system for capturing image data of the environment of the motor vehicle,
• A device for processing the image data of the image-capturing sensor system with a person recognition device for determining fully or partially recognized personal objects in the motor vehicle environment, a face recognition device for determining completely visible facial objects, and a data fusion device for data fusion of recognized facial objects with completely or partially recognized humanoid objects Determination of endangered objects,
• - an evaluation and decision-making facility for the assessment of vulnerable objects, and
• A device for controlling actuators based on the results of the evaluation and decision device.

The device preferably has a tracking device for tracking human objects and facial objects in successive time-sequential image data. With the tracking device, a time tracking of potentially endangered objects is possible, whereby the probability of a false triggering an emergency intervention in the vehicle control is reduced.

More preferably, the person recognition device and the face recognition device each have a pattern recognition device. In other words, on the one hand, preferably person contours such as pedestrians or cyclist contours in the images and, on the other hand, preferably facial contours and facial contents are sought and analyzed on the basis of suitable pattern recognition algorithms.

The data fusion device preferably has a plausibility device for checking the plausibility of the data to be merged or the merged data. The plausibility check improves the security of the assignment of the data to be merged, which increases the detection of potentially endangered objects.

With further preference, the data fusion device has a device for determining the recognition quality of the fused data. This measure further increases the detection of potentially endangered objects and the reliability of triggering a measure.

Furthermore, the device can have a non-image-capturing environment sensor system, for example a radar sensor or a lidar sensor for distance measurement. By means of the additional environment sensor system, it is possible in particular to cover the case of an action triggering in which only complete facial objects have been recognized, but to which no person object can be assigned.

A preferred embodiment of the invention will be explained below with reference to the drawings. It shows

1 the processing chain of the process in a schematic representation, and

2 the sensor data processing in detail.

1 shows the sequence of the method according to the invention and also corresponds to the actual implementation in hardware. Shown is an image sensor 1 or sensor system for creating images of the environment of a motor vehicle, not shown, at predetermined time intervals, ie with predetermined timing. The image pickup sensor 1 can be a camera. Below the image pickup sensor 1 is a sensor data processing 2 arranged, which processes the image data. The results of sensor data processing 2 be in an evaluation and decision-making unit 3 assessed and the decision is taken to decide whether to take measures such as a braking maneuver, an evasive maneuver and / or a driver warning. If the decision for a measure is made, then the control unit is activated 4 Control signals for controlling one or more actuators generated. In the corresponding actuator 5 , such as brake or steering, the control signal is converted and performed the appropriate measure name.

2 shows the sensor data processing 2 in detail. The time-sequential image data is processed in an image data preprocessing 7 prepared, for example by a suitable filtering. After the image data are preprocessed, they are incorporated in a person recognition 8th examined for the presence of persons in the picture. In parallel, they are in a face recognition 9 searched for the presence of faces. The data of found face objects and person objects are in a fusion device 10 fused, whereby a plausibility check can be made here. The merged data then becomes the valuation and decision unit 3 of the 1 fed.

• – eine Sensordatenverarbeitung 2, umfassend:
• – eine Bildvorverarbeitung 7 mit vorbereitenden Verarbeitungsschritten, wie beispielsweise einer Filterung;
• – eine Fußgänger- und/oder Radfahrererkennung, d. h. eine Personenerkennung 8, von vollständig oder teilweise im aufgenommenen Bild erkennbaren Fußgängern oder Radfahrern basierend auf einem geeigneten Mustererkennungsverfahren;
• – eine Gesichtserkennung 9, basierend auf einem geeigneten Mustererkennungsverfahren, welche die Fähigkeit aufweist vollständig sichtbare Gesichter in verschiedenen Ansichten, bis hin zur Seitenansicht, bei ausreichend genauer Abbildung zu erkennen;
• – wobei sowohl Fußgänger-/Radfahrererkennung 8 als auch Gesichtserkennung 9 durch ein Trackingverfahren, nämlich eine zeitliche Verfolgung eines Objekts über mehrere zeitsequenziell aufgenommene und verarbeitete Bilder mit möglicher Plausibilisierung, ergänzt werden können;
• – einen Sensordatenfusionsalgorithmus 10, welcher die Einheit von auf verschiedenen Wegen, also mittels verschiedener Algorithmen und/oder verschiedener Sensoren, erkannten Objekten ermittelt und deren Eigenschaften je nach Erkennungsgüte verknüpft bzw. kombiniert;
• – einen Bewertungs- und Entscheidungsalgorithmus 3, welcher anhand der erkannten Objektdaten ein Maß der Gefährdung für erkannte Fußgänger oder Radfahrer ermittelt und darauf basierend einen oder mehrere Aktoren über die jeweilige Ansteuerungseinheit ansteuert (Eingriff).

In summary, the sensor data processing lead 2 and the valuation unit 3 doing the following different algorithms:

• - a sensor data processing 2 , full:
• - an image preprocessing 7 with preparatory processing steps, such as filtering;
• A pedestrian and / or cyclist recognition, ie a person recognition 8th , pedestrians or cyclists recognizable wholly or partially in the captured image, based on a suitable pattern recognition process;
• - a face recognition 9 based on a suitable pattern recognition method, which has the ability to recognize fully visible faces in different views, right through to the side view, with sufficiently accurate imaging;
• - where both pedestrian / cyclist recognition 8th as well as face recognition 9 by a tracking method, namely a temporal tracking of an object over several time sequentially recorded and processed images with a possible plausibility, can be supplemented;
• A sensor data fusion algorithm 10 which determines the unit of objects detected in different ways, ie by means of different algorithms and / or different sensors, and combines or combines their properties depending on the quality of detection;
• - an evaluation and decision algorithm 3 which, on the basis of the recognized object data, determines a measure of the risk to recognized pedestrians or cyclists and, based on this, actuates one or more actuators via the respective actuation unit (intervention).

• – Bei vollständig erkannten Fußgängern oder Radfahrern mit gleichzeitiger Erkennung der zugehörigen Gesichter wird durch Sensordatenfusion eine Erkennungsgüte der fusionierten Daten ermittelt, welche dabei höher ausfallen kann als die jeweils einzelnen Erkennungsgütewerte. Dabei ist eine verringerte Falschauslöserate zu erwarten. Die Sensordatenfusion kann beispielsweise über eine Verknüpfung der Positions- und geschätzten Entfernungsdaten oder Plausibilisierung der Position erfolgen.
• – Bei teilerkannten und damit weniger sicher erkannten Fußgängern oder Radfahrern mit gleichzeitiger vollständiger Erkennung der zugehörigen Gesichter wird durch Sensordatenfusion eine Erkennungsgüte der fusionierten Daten ermittelt, welche dabei höher ausfallen kann als die jeweils einzelnen Erkennungsgüten. Eine Teilerkennung kann beispielsweise in einer Verdeckung oder unvollständigen Abbildung durch die Beschränkungen des bildaufnehmenden Sensors, wie beispielsweise aufgrund des Öffnungswinkels, begründet sein. Die Sensordatenfusion kann auch hier beispielsweise über eine Verknüpfung der Positions- und geschätzten Entfernungsdaten oder Plausibilisierung der Position erfolgen. Insgesamt kann das System auch hier häufiger eingreifen als bei bekannten Systemen, die keinem Eingriff wegen Teilerkennung und damit zu geringer Erkennungsgüte zulassen.
• – Bei mit hoher Sicherheit erkannten Gesichtern, was beispielsweise bei vielen erkannten Gesichtmerkmalen und/oder mittels eines Trackings über mehrere Bilder möglich ist, denen aber kein durch die Fußgänger/Radfahrererkennung erkannter Fußgänger oder Radfahrer zugeordnet werden kann, kann nach einer geeigneten Plausibilisierung auf dieser Basis ein Eingriff erfolgen. Dabei ist eine verringerte Falschauslöserate zu erwarten.
• – Objekteigenschaften eines erkannten Gesichtes können auch mit Daten von nichtbildaufnehmenden Umfeldsensoren fusioniert/plausibilisiert werden. So kann eine Fusion oder Plausibilisierung mit Daten eines oder mehrerer weiterer Umfeldsensoren erfolgen, auf deren Basis ebenfalls mit signalverarbeitenden Verfahren ein Fußgänger erkannt wurde, welcher dem Gesicht zugeordnet werden kann. Dadurch kann eine höhere Erkennungssicherheit erzielt werden als bei der getrennten Erkennung mit Hilfe der beteiligten Sensoren. Auf Basis dieses fusionierten/plausibilisierten Fußgängers oder Radfahrers kann ein Eingriff erfolgen und damit das Wirkfeld vergrößert werden. Gegenüber Eingriffen auf Basis nur eines Sensors ist eine verringerte Falschauslöserate zu erwarten.

To increase the effectiveness of measures to reduce accidents or avoid accidents as well as to reduce false trigger rates, the following links are made:

• - In the case of completely recognized pedestrians or cyclists with simultaneous recognition of the associated faces, sensor data fusion will determine a quality of detection of the fused data, which may be higher than the respective individual recognition quality values. In this case, a reduced false triggering rate is to be expected. The sensor data fusion can be done, for example, by linking the position and estimated distance data or plausibility of the position.
• - For partially recognized and therefore less secure pedestrians or cyclists with simultaneous full recognition of their faces is detected by sensor data fusion a recognition quality of the fused data, which may turn out to be higher than the respective individual Erkennungsgüten. Partial detection may be due, for example, to occlusion or incomplete imaging due to the limitations of the image sensing sensor, such as due to the aperture angle. The sensor data fusion can also be done here, for example, by linking the position and estimated distance data or plausibility of the position. Overall, the system can intervene here more often than in known systems that do not allow intervention because of partial recognition and thus to low recognition quality.
• - For detected with high security faces, which is possible for example with many recognized facial features and / or by means of a tracking over several images, but which can not be assigned by the pedestrian / cyclist recognition recognized pedestrian or cyclist, can after a suitable plausibility check on this basis an intervention take place. In this case, a reduced false triggering rate is to be expected.
• Object properties of a detected face can also be fused / plausibilized with data from non-image-capturing environmental sensors. Thus, a merger or plausibility check with data of one or more other environment sensors can be carried out, on the basis of which a pedestrian has also been identified with signal processing methods, which can be assigned to the face. As a result, a higher detection reliability can be achieved than in the separate detection with the help of the sensors involved. On the basis of this fused / plausibilized pedestrian or cyclist, an intervention can be made and thus the effective field can be increased. Compared to interventions based on only one sensor, a reduced false release rate is to be expected.

LIST OF REFERENCE NUMBERS

1

Image capture Senor

2

Sensor data processing

3

Evaluation and decision

4

Control Actuator

5

actuator

6

additional sensors

7

Bilddatenvorverarbeitung

8th

people detection

9

face recognition

10

data Fusion

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 7961906 B2 [0006]
• EP 2288138 A1 [0007]

Cited non-patent literature

• P. Viola et. al .: "Robust Real-Time Face Detection", International Journal of Computer Vision 57 (2), 137-154, 2004 [0008]

Claims (12)

Method for the active protection of persons in the vicinity of a motor vehicle, in particular pedestrians and cyclists, with the following steps: acquisition of image data of the environment of the motor vehicle by means of an image-receiving sensor system ( 1 ), Performing image processing of the acquired image data with a person recognition ( 8th ) for determining fully or partially recognized personal objects in the motor vehicle environment, a face recognition ( 9 ) for the determination of completely visible facial objects, and a data fusion ( 10 ) recognized face objects with completely or partially recognized person objects for the determination of endangered objects, assessment ( 3 ) of the degree of endangerment of an object, and performing an intervention ( 4 ) in the vehicle control, if the risk measure is greater than a predetermined threshold. Method according to claim 1, characterized in that the person recognition ( 8th ) for recognition of personal objects and / or face recognition ( 9 ) is performed for recognizing facial objects over a predetermined number of time-sequentially recorded images of the image data. Method according to one of the preceding claims, characterized in that the data fusion ( 10 ) includes a plausibility check of the data to be merged and the merged data. Method according to one of the preceding claims, characterized in that upon complete or partial recognition of person objects and completely visible face objects, a quality of recognition of the fused data is determined, which is used to trigger the intervention ( 4 ) is used in the vehicle control. Method according to one of claims 2 or 3, characterized in that when detected face objects to which no person object can be assigned, triggering an intervention ( 4 ) takes place in the motor vehicle control due to a plausibility of the recognition quality of the recognized face object. Method according to one of the preceding claims, characterized in that a plausibility check and / or fusion of recognized face objects with data from non-image-capturing environment sensors. Device for the active protection of persons moving in the vicinity of a motor vehicle, in particular pedestrians and cyclists, for carrying out the method according to one of the preceding claims, comprising: an image-receiving sensor system ( 1 ) for capturing image data of the environment of the motor vehicle, a device ( 2 ) for processing the image data of the image-receiving sensor system with a person recognition device ( 8th ) for determining fully or partially recognized personal objects in the motor vehicle environment, a face recognition device ( 9 ) for determining fully visible facial objects, and a data fusion device ( 10 ) for data fusion of recognized facial objects with fully or partially recognized personal objects for the determination of endangered objects, an evaluation and decision-making device ( 3 ), an institution ( 4 ) for controlling actuators based on the results of the evaluation and decision device ( 2 ). Apparatus according to claim 7, characterized in that the device comprises a tracking device for tracking of human objects and face objects in successive time-sequential image data. Apparatus according to claim 7 or 8, characterized in that the person recognition device ( 7 ) and the face recognition device ( 8th ) each have a pattern recognition device. Device according to one of claims 7 to 9, characterized in that the data fusion device ( 10 ) has a plausibility device for checking the plausibility of the data to be merged or merged. Apparatus according to claim 10, characterized in that the data fusion device ( 10 ) has means for determining the quality of recognition of the fused data. Device according to one of claims 7 to 11, characterized in that the device comprises a non-image-receiving environment sensor ( 6 ) having.

Images