DE112016003241T5 - Method, control unit and system for detecting and tracking vulnerable road users - Google Patents

Abstract

Method (400) and control unit (310) for recognizing and tracking a VRU (200). The method (400) comprises detecting (401) an object (200) by a camera (110) of the vehicle (100); classifying (402) the detected (402) object (200) as a VRU (200) detecting (403) the object (200) by a sensor (120) of the vehicle (100); associating (404) the ranked (402) VRU (200) with the (403) object (200) identified with the sensor (120); and tracking (405) the VRU (200) by the sensor (120).

Description

Technical field

This document relates to a method, a control unit and a system in a vehicle. In particular, a method, a control unit and a system for detecting and tracking a vulnerable road user (VRU) are described.

background

Non-motorized road users, such as pedestrians and cyclists, as well as motorcyclists and persons with disabilities and / or limited mobility and orientation, are sometimes referred to as Vulnerable Road Users (VRUs). This heterogeneous group is disproportionately represented in statistics on injuries and victims of traffic accidents.

A particularly dangerous scenario is when VRUs are in the blind spot of the driver of a vehicle when the vehicle is turning at low speed.

An advanced warning system for VRUs in the blind spot of a vehicle is not yet known. In today's market, there are simple systems based on ultrasonic sensors that detect the presence of "anything" beside the vehicle as it turns or uses direction indicators. However, such systems, for example, are unable to distinguish between a lamppost and a stationary pedestrian.

US 201200358466 relates to a method for an external environment detection apparatus for vehicles and a vehicle system using the method. A rangefinder sensor, such as a radar, and an image sensor, such as a camera, are used to detect objects.

Environmental sensors such as RADAR, LIDAR or cameras all have several advantages and disadvantages. For example, a camera system may be affected by dirt, snow, fog and low light conditions in a manner that does not generally distinguish between road users and still objects unless the detected object has been observed to be moving. Therefore, in a RADAR- or LIDAR-based system, a stunned pedestrian waiting at a pedestrian crossing generally can not be distinguished from other immobile objects such as a lamppost.

It would therefore be desirable to discover a method of detecting VRUs by a vehicle and distinguishing a VRU from a stationary object, which could be used, for example, in a VRU warning system.

Summary

It is therefore an object of this invention to solve at least some of the above problems and to improve traffic safety.

According to a first aspect of the invention, this object is achieved by a method in a vehicle for recognizing and tracking a vulnerable road user (VRU). The method includes detecting an object by a camera of the vehicle; classifying the detected object as a VRU and making an estimate of a VRU motion prediction reliability, wherein unaccompanied animals and persons smaller than an adjustable threshold size are ranked to have limited reliability of motion prediction. The method further comprises detecting the object by a sensor of the vehicle; associating the ranked VRU with the object detected by the sensor; and tracking the VRU by the sensor.

According to a second aspect of the invention, this object is achieved by a control unit in a vehicle. The control unit aims to detect and track a VRU. The control unit is configured to recognize an object by a camera of the vehicle in accordance with the above. According to a third aspect of the invention, this object is achieved by a computer program comprising program code for carrying out a method according to the first aspect, when the computer program is executed in a control unit according to the second aspect.

According to a fourth aspect, this object is achieved by a system for recognizing and tracking a VRU. The system comprises a control unit according to the second aspect. Furthermore, the system also includes a camera of the vehicle for recognizing an object. The system additionally comprises at least one sensor of the vehicle for recognizing the object.

Thanks to the described aspects, VRUs can be effectively identified and tracked by combining the corresponding advantages of the camera and the sensor. Accurate VRU tracking is essential, for example, to providing a reliable VRU warning system that warns / intervenes when a collision occurs VRU is really likely, ie, if the predicted path of the vehicle and a predicted path for the VRU overlap. Such a system will gain a high level of acceptance and trust as superfluous warnings are eliminated or at least reduced, and this in turn is expected to reduce deaths from turnaround accidents. Therefore, increased traffic safety is achieved.

Therefore, the traffic safety can be increased since the driver's attention could be increased in situations where the VRU is particularly unpredictable. It also eliminates unnecessary warnings (adult, reliable VRUs). This will give the system a high level of driver acceptance, causing the driver to turn to alarm signals.

Other advantages and additional novel features will become apparent from the following detailed description.

list of figures

• 1 ein Fahrzeug gemäß einer Ausführungsform der Erfindung zeigt;
• 2A ein Beispiel eines Verkehrsszenarios und eine Ausführungsform der Erfindung zeigt;
• 2B ein Beispiel eines Verkehrsszenarios und eine Ausführungsform der Erfindung zeigt;
• 3 ein Beispiel eines Fahrzeuginnenraums gemäß einer Ausführungsform zeigt;
• 4 ein Ablaufdiagramm ist, das eine Ausführungsform des Verfahrens veranschaulicht; und
• 5 eine Darstellung ist, die ein System gemäß einer Ausführungsform zeigt.

Embodiments of the invention will now be described in detail with reference to the accompanying drawings, in which:

• 1 shows a vehicle according to an embodiment of the invention;
• 2A shows an example of a traffic scenario and an embodiment of the invention;
• 2 B shows an example of a traffic scenario and an embodiment of the invention;
• 3 shows an example of a vehicle interior according to an embodiment;
• 4 Fig. 10 is a flow chart illustrating an embodiment of the method; and
• 5 Fig. 10 is a diagram showing a system according to an embodiment.

Detailed description

Embodiments of the invention described herein are defined as a method, a control unit, and a system that may be implemented in the embodiments described below. However, these embodiments may be illustrated and practiced in many different forms, and are not limited to the examples set forth herein; rather, these illustrative examples of embodiments are provided so that this disclosure will be thorough and complete.

Nevertheless, other objects and features may be apparent from the following detailed description taken in conjunction with the accompanying drawings. It should be understood, however, that the drawings are designed for the purpose of illustration only, and not as a definition of the limits of the embodiments disclosed herein, for which reference should be made to the appended claims. Furthermore, the drawings are not necessarily to scale, and are merely illustrative of the structures and acts described herein, unless otherwise specified.

1 shows a scenario with a vehicle 100 , The vehicle 100 drives on a road in one direction of travel 105 ,

The vehicle 100 For example, it may include a truck, a bus or a car, or any similar vehicle or other means of transportation.

Furthermore, the vehicle described herein 100 controlled by a driver or driverless, in some embodiments an autonomous controlled vehicle 100 , However, for the sake of clarity, they are described below as including a driver.

The vehicle 100 includes a camera 110 and a sensor 120 , In the embodiment shown, which is merely an arbitrary example, the camera may 110 for example, at a front of the vehicle 100 be arranged behind the windshield of the vehicle 100 , An advantage of placing the camera 110 behind the windshield is that the camera 110 protected from dirt, snow, rain and to a certain extent from damage, vandalism and / or theft.

The camera 110 can go to the front of the vehicle 100 be directed in the direction of travel 105 , This allows the camera 110 a VRU in the direction of travel 105 in front of the vehicle 100 detect. In various embodiments, the camera may include, for example, a camera, a stereo camera, an infrared camera, a video camera, or a time-of-flight camera.

Attaching the camera 110 behind the windshield (looking forward) has some advantages compared to an externally mounted camera system. These advantages include the ability to use the wipers for cleaning and to use the light of the headlamps to illuminate objects in the field of view of the camera. Such a multifunctional camera 110 can also be used for a variety of other tasks.

The sensor 120 can be on one side of the vehicle 100 arranged and intended to be objects on the side of the vehicle 100 to recognize. The sensor 120 For example, in various embodiments, it may include a radar, a lidar, an ultrasound device, a time-of-flight camera, or the like.

In some embodiments, the sensor 120 For example, it may include a motion detector and / or it may be based on a passive infrared (PIR) sensor that is sensitive to a person's skin temperature due to emitted blackbody radiation in the mid-infrared wavelength range, as opposed to background objects at room temperature; or by emitting a continuous wave of microwave radiation and detecting motion by the principle of a Doppler radar; or by emitting an ultrasonic wave and detecting and analyzing the reflections; or by a tomographic motion detection system based on detection of radio wave interference, to name a few possible implementations.

By using at least one camera 110 and at least one sensor 120 the advantages of the corresponding types of devices can be combined. The advantage of the camera 110 is that it is able to distinguish, for example, between a VRU and another object, even if the VRU is unmoved. The advantages of the transmitter 120 are the recognition range, the price, the robustness and the ability to work in all weather conditions. As a result, recognitions and orders of high reliability can be achieved. Thanks to the combination of the camera 110 which can also recognize the VRU when it is stationary and the sensor 120 taking any of the camera 110 detected VRU, a powerful function of a VRU warning / intervention system is achieved, possibly without adding a side-facing camera to the vehicle 100 , This can eliminate a need for dedicated side-looking VRU detection sensors.

By having a side-facing sensor 120 and the camera 110 overlapping fields of view are present, motionless VRUs can first with the camera 110 be detected when they are driven past them, and then they can with the sensor 120 out of sight of the camera 110 To be "tracked". This allows for VRU alert / VRU intervention with respect to still objects outside the field of view of the camera 110 What is needed for a VRU warning in the blind spot of the driver.

However, the side-facing sensor must be 120 and the camera 110 do not necessarily have overlapping fields of view; they can also have side-by-side fields of view or fields of view with a distance between them. To assign one from the camera 110 recognized object and the same object, that of the side-facing sensor 120 In the latter case, in some embodiments, a calculation may be performed.

2A schematically illustrates a scenario similar to that described above in 1 shown scenario, but where the vehicle 100 from a perspective from above and being a VRU 200 is shown.

If the vehicle 100 in the direction of travel 105 drives, recognizes the camera 110 the VRU 200 , An image recognition program can be the VRU 200 recognize as a VRU and possibly also classify him as a pedestrian, child, cyclist, animal, etc.

If the vehicle 100 in the direction of travel 105 drives forward and joins the VRU 200 approaching, is the VRU 200 for a moment in an area where he is both through the camera 110 as well as through the sensor 120 is recognized. It will be for the sensor 120 possible, the VRU 200 to recognize as a VRU, even if the VRU 200 is unmoved.

If the vehicle 100 continue in the direction of travel 105 moved, the VRU device 200 from the perspective of the camera 110 while still within range of the sensor 120 is located as in 2 B is shown. The VRU 200 can then through the sensor 120 be tracked as long as it is within the detection range of the sensor 120 located.

An accurate recognition and tracking of each VRU 200 near the vehicle 100 are the backbone for creating a reliable VRU warning system that warns / intervenes only when a collision with a VRU is indeed likely and forthcoming. Such systems will gain greater acceptance and confidence, which is expected to reduce turn-down fatality deaths.

However, the disclosed method of detecting a VRU is not limited to VRU warning systems, but may be used for various other purposes.

3 shows an example of a vehicle interior of the vehicle 100 and represents how the previous scenario in 1 and / or in 2A of the Driver of the vehicle 100 can be perceived.

The vehicle 100 includes a control unit 310 , The control unit 310 is capable of doing so, the VRU 200 based on one or more of the camera 110 provided images as a VRU recognize. The control unit 310 is further adapted to detect signals from the sensor 120 to receive and the detected VRU from the sensor 120 Assign received detection signals. In addition, the control unit 310 continue to set up the VRU 200 over the sensor 120 to pursue as long as the VRU 200 within the range of the sensor 120 located.

As shown, the vehicle can 100 in some embodiments, a sensor 120 - 1 on the right side of the vehicle 101 sensor 120 - 2 on the left side. However, the vehicle can 100 in other embodiments, only one sensor 120 on the right side of the vehicle 100 include, wherein the number of sensors 120 in the vehicle 100 is reduced. However, the vehicle can 100 in other embodiments, multiple sensors 120 on each side of the vehicle 100 include. The sensors 120 may be of the same type or different types, such as radar, lidar, ultrasound, time-of-flight camera, etc.

In the example shown, the vehicle includes 100 a camera 110 in the front of the vehicle 100 is arranged behind the windshield. In other embodiments, the vehicle may 100 however, include a camera attached to a rear of the vehicle 100 is arranged, directed in one direction, the normal direction of travel 105 directed against. That way, VRUs can 200 be recognized while the vehicle 100 is reset. The camera 110 may be arranged within the rear disc in such a case to be protected from dirt, snow, etc.

The control unit 310 can with the camera 110 and the sensor 120 communicate, for example via a communication bus of the vehicle 100 or via a wired or wireless connection.

4 shows an example of a method 400 according to one embodiment. The flowchart in 4 shows the procedure 400 for use in a vehicle 100 , The procedure 400 aims to be a VRU 200 to recognize and pursue.

The vehicle 100 may be, for example, a truck, a bus, a car, a motorcycle or the like.

In order to be able to the VRU 200 To correctly recognize and pursue the procedure 400 a number of steps 401 - 405 include. Some of these steps 401 - 405 however, can be done in several alternative ways. In addition, the steps described 401 - 405 are performed to some extent deviating chronological sequence as the numbering suggests. The procedure 400 can the successive steps include:

step 401 involves the recognition of an object by a camera 110 of the vehicle 100 ,

step 402 includes classifying the detected 401 object 200 as a VRU 200 ,

Classifying the detected 401 object 200 can be performed based on image recognition.

The ranking may further include estimating a reliability of a VRU motion prediction 200 wherein unaccompanied animals and persons smaller than an adjustable threshold size are classified as having limited reliability of motion prediction.

The ranking may also be, according to another addition, estimating a reliability of VRU motion prediction 200 wherein motorcyclists are ranked to have increased reliability of motion prediction.

step 403 includes a recognition of the object 200 through a sensor 120 of the vehicle 100 ,

step 404 includes a mapping of the ranked 402 VRU 200 to the through the sensor 120 detected object 200 ,

steps 405 includes tracking the VRU 200 through the sensor 120 ,

5 shows an embodiment of a system 500 to recognize and track a VRU 200 , The system 500 can be at least one of the steps described above 401 - 405 according to the above described and in 4 illustrated method 400 carry out.

The system 500 includes a control unit 310 in the vehicle 100 , The control unit 310 is set up the VRU 200 to recognize and pursue. The control unit 310 is designed to be an object 200 through a camera 110 of the vehicle 100 to recognize. Further, the control unit 310 set up the detected object 200 as a VRU 200 classify. The control unit 310 is also set up the object 200 through a sensor 120 of the vehicle 100 to recognize. In addition, the control unit 310 also set up the ruled VRU 200 to the through the sensor 120 recognized object 200 assigned. The control unit 310 is set to the VRU 200 through the sensor 120 to pursue.

The control unit 310 In some embodiments, it may also be configured to recognize the detected object 200 based on image capture. Further, the control unit may be configured to estimate a reliability of a motion prediction of the VRU 200 where unaccompanied animals and persons smaller than an adjustable threshold size are classified as having limited reliability of motion prediction. In some embodiments, the control unit may 310 be configured to estimate a reliability of VRU motion prediction 200 to make, whereby motorcyclists are classified so that they have an increased reliability of a movement prediction.

The control unit 310 includes a receiver circle 510 which is to receive a signal from the camera 110 and the sensor 120 is set up.

Furthermore, the control unit comprises 310 According to some embodiments, a processor 520 which is set up, at least one step of the procedure 400 perform.

Such a processor 520 may include one or more instances of a processing circuit, ie, a central processing unit (CPU), a processing unit, a processing circuit, an application specific integrated circuit (ASIC), a microprocessor, or other processing logic that can interpret and execute instructions. The term "processor" as used herein may therefore represent processing circuitry that includes multiple processing circuits, such as any, some, or all of those enumerated above.

Furthermore, the control unit 310 in some embodiments, a memory 525 include. The optional memory 520 may include a physical device used to temporarily or permanently store data or programs, ie, sequence of instructions. According to some embodiments, the memory 520 include integrated circuits comprising silicon-based transistors. The memory 525 For example, in some embodiments, it may include a memory card, a flash memory, a USB memory, a hard disk, or other similar volatile or nonvolatile storage unit for storing data such as ROM (Read Only Memory), PROM (Programmable Read Only). Only Memory), EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), etc.

In addition, the control unit 310 in some embodiments, a signal transmitter 530 include. The signal transmitter 530 may be configured to transmit a signal, for example, to a display device or to a VRU warning system or to a warning device.

In addition, the system includes 500 also a camera 110 of the vehicle 100 to recognize an object 200 , The camera 110 For example, it may include a camera, a stereo camera, an infrared camera, a video camera, or the like. In addition, the camera can 110 behind the windshield of the vehicle 100 be arranged, directed forward in the direction of travel 105 of the vehicle 100 ,

Furthermore, the system can 500 at least one sensor 120 of the vehicle 100 to recognize the object 200 include. The at least one sensor 120 can be on one side of the vehicle 100 and, in some embodiments, may include any of radar, lidar, ultrasonic sensor, time-of-flight camera, or thermal camera.

The camera 110 and the at least one sensor 120 In some embodiments, they may have overlapping fields of view.

The in the vehicle 100 to be performed above steps 401 - 405 can be through the one processor or the multiple processors 520 within the control unit 310 be implemented, along with a computer program product, to perform at least some of the functions of the steps 401 - 405 , In this way, a computer program product may be provided instructions for performing the steps 401 - 405 in the control unit 310 includes, the procedure 400 perform at least some of the steps 401 - 405 to recognize and track the VRU 200 includes when the computer program in the one processor or in the multiple processors 520 the control unit 310 is loaded.

Further, some embodiments of the invention may be a vehicle 100 comprising the control unit 310 that has become one Predicting a path of a vehicle 100 according to at least some of the steps 401 - 405 is set up.

The above-mentioned computer program product may be provided, for example, in the form of a data carrier, the computer program code for performing at least some of the steps 401 - 405 according to some embodiments, when incorporated into the one or more processors 520 the control unit 310 is loaded. The data carrier may be, for example, a hard disk, a CD-ROM disc, a memory stick, an optical storage device, a magnetic storage device, or any other suitable medium, such as a disk or tape, carrying machine-readable data in a non-transitory manner. The computer program product may also be provided as computer program code on a server and to the control unit 310 be downloaded remotely, for example, via an Internet or an Internet connection.

The terminology used in the description of the embodiments illustrated in the accompanying drawings is not intended to facilitate the described method 400 , the control unit 310 , the computer program, the system 500 and / or restrict the vehicle. Various changes, substitutions and / or alterations may be made without departing from embodiments of the invention which are defined by the appended claims.

Where the term "and / or" is used herein, it includes any and all combinations of one or more of the correspondingly listed items. Where the term "or" is used herein, it should be interpreted as a mathematical OR, i. H. as an inclusive distinction; but not as a mathematically exclusive OR (XOR), unless otherwise stated. In addition, the singular forms "a" / "an" and "the" / "the" / "the" as "at least one" / "at least one" / "at least one" are to be interpreted so that they may comprise several units of the same kind, unless otherwise indicated. It should be further understood that the terms "include," "include," "including," and / or "comprising" specify but are not limited to the presence of specified features, acts, integers, steps, operations, elements, and / or components Preclude the presence or addition of one or more other features, acts, integers, steps, operations, elements, components, and / or groupings thereof. A single unit, such as a processor, may perform the function of several parts enumerated in the claims. The mere fact that certain measures are listed in mutually different dependent claims does not indicate that a combination of these measures can not be used to advantage. A computer program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium co-provided with or as part of other hardware, but may be in other forms, such as the Internet or others wired or wireless communication systems are distributed.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 201200358466 [0005]

Claims (8)

A method (400) in a vehicle (100) for detecting and tracking a vulnerable road user, VRU (200), the method (400) comprising: Detecting (401) an object (200) by a camera (110) of the vehicle (100); Classifying (402) the recognized (402) object (200) as a VRU (200) and making an estimate of a VRU motion prediction reliability (200), wherein unaccompanied animals and persons smaller than an adjustable threshold size are be classified as having limited reliability of motion prediction; Detecting (403) the object (200) by a sensor (120) of the vehicle (100); Associating (404) the ranked (402) VRU (200) with the (403) object (200) recognized by the sensor (120); and Following (405) the VRU (200) by the sensor (120). Method (400) after Claim 1 wherein the ranking (402) of the recognized (401) object (200) is based on image recognition. Method (400) according to one of Claims 1 or 2 wherein the ranking (402) further comprises estimating a reliability of a motion prediction of the VRU (200), wherein motorcyclists are ranked to have increased reliability of motion prediction. A control unit (310) in a vehicle (100) for detecting and tracking a vulnerable road user, VRU (200), wherein the control unit (310) is arranged to: Detecting (401) an object (200) by a camera (110) of the vehicle (100); Classifying (402) the recognized (402) object (200) as a VRU (200) and making an estimate of a VRU motion prediction reliability (200), wherein unaccompanied animals and persons smaller than an adjustable threshold size are be classified as having limited reliability of motion prediction; Detecting (403) the object (200) by a sensor (120) of the vehicle (100); Associating (404) the ranked (402) VRU (200) with the (403) object (200) recognized by the sensor (120); and Following (405) the VRU (200) by the sensor (120). Computer program comprising program code for performing a method (400) according to one of Claims 1 - 3 when the computer program is in a processor in a control unit (310) Claim 4 is performed. System (500) for detecting and tracking a vulnerable road user, VRU (200), the system (500) comprising: a control unit (310) according to Claim 4 ; a camera (110) of the vehicle (100) for detecting an object (200); and at least one sensor (120) of the vehicle (100) for detecting the object (200). System (500) after Claim 6 wherein the camera (110) is disposed behind the windshield of the vehicle (100) and facing forward in the direction of travel (105) of the vehicle, and wherein the at least one sensor (120) is disposed on a side of the vehicle (100) and any one of Includes radar, lidar, ultrasonic sensor, time-of-flight camera or thermal camera. System (500) according to one of Claims 6 or 7 wherein the camera (110) and the at least one sensor (120) have overlapping fields of view.

Images