DE102019202634B3 - Method, control device for an automated road vehicle, computer program product for recognizing objects in road traffic and automated road vehicle for mobility services - Google Patents

Abstract

Method for recognizing objects (1) in road traffic for automated road vehicles (2) comprising the steps: obtaining audio signals of the objects (1) by means of microphones (3) which can be arranged on the road vehicle (2), receiving image signals of an environment of the road vehicle ( 2) by means of at least one imaging sensor (4) which can be arranged on the road vehicle (2), locating and classifying the objects (1) as a function of the audio signals, assigning these localizations and classifications to the objects (1) in the vicinity of the road vehicle (2) in Dependency of a comparison with the image signals, determination of a degree of concealment for at least one of the objects (1) depending on the assignment and output of the degree of concealment. The invention further relates to a control device (50), a computer program product and an automated vehicle vehicle (2) for mobility services.

Description

The invention relates to a method, a control device for an automated road vehicle, a computer program product for recognizing objects in road traffic and an automated road vehicle for mobility services.

In the field of automated driving, environments are perceived using lidar, radar, cameras and similar 2D or 3D environment detection sensors. These sensors can be objects that are covered by other objects, for example a person who is between two vehicles parked on a roadside and is covered by at least one of these vehicles, or a person who is covered by a tree or a building, not notice.

The DE 10 2017 111 468 A1 discloses a vehicle system and method for determining whether a vehicle occupant has perceived an object external to the vehicle.

The DE 10 2017 103 374 A1 discloses an audio processing module that identifies features that may correspond to a vehicle.

Further prior art is disclosed in DE 10 2016 120 218 A1 , DE 10 2010 025 351 A1 and US 1 014 0417 B1 .

This is where the invention comes in. The invention has for its object to enable the detection of hidden people.

The method according to the invention for recognizing objects in road traffic is designed for road vehicles that can be operated automatically and achieves the task by receiving audio signals of the objects by means of microphones that can be arranged on the road vehicle and image signals of an environment of the road vehicle by means of at least one imaging sensor that can be arranged on the road vehicle. Depending on the audio signals, the objects are localized and classified. These localizations and classifications are then assigned to the objects in the vicinity of the road vehicle depending on a comparison with the image signals. In a further step, a degree of concealment for at least one of the objects is determined as a function of the assignment. The degree of concealment is output.

Road vehicles are vehicles suitable for road traffic with internal combustion engines and petrol, diesel or synthetic fuels, electric drives, hybrid electric drives or fuel cell drives.

Operable automatically means that the road vehicle includes technical equipment, in particular control units and actuators, which enables automated control and / or regulation of the longitudinal and / or transverse guidance up to a fully automated or autonomous control and / or regulation without a human fallback level. Longitudinal guidance is regulated, for example, via drive torque control, for example via electronic engine power control, and / or braking torque control. Lateral guidance regulates the lateral dynamics of the vehicle, for example lane and / or directional loyalty, steering maneuvers and / or yaw rate.

Objects in road traffic are in particular pedestrians, children, cyclists, runners, wheelchairs, wheelchair users, animals and other road vehicles.

An audio signal is an electrical signal that transports acoustic information, in particular sounds of objects in traffic, for example children's sounds or pedal sounds. Microphone signals are audio signals.

Image signals are electrical signals from an imaging sensor. An imaging sensor is contained, for example, in a camera, a lidar or radar sensor. In this sense, the camera, lidar or radar sensor are examples of imaging sensors.

Environment is the area that can affect the road vehicle, for example via objects in the area. The detection or perception area or field of view (FOV) of the microphones and the imaging sensor cover at least a part of the environment.

Localization means that at least one position of the objects is determined relative to the road vehicle. Classifying means that at least the type of object, for example human, machine or animal, is determined. Localization and classification takes place, for example, by means of an amplitude, frequency and / or phase analysis of the audio signals.

The degree of masking indicates how likely it is that an object is masked by another object.

By comparing the acoustic localizations and classifications of the objects with the image signals, the acoustically detected objects are assigned to the corresponding objects in image recordings of the imaging sensor. In order to the audio signals and the image signals are fused. The merger in particular provides coherent information that cannot be provided individually from audio signals or image signals. It can be used to determine whether an object that has been acoustically detected is covered by another object. The process has advantages, particularly in an urban environment. In the urban environment, the density of buildings and the density of vehicles parked on the side of the road are high, so that people are relatively often concealed in this environment. For example, a truck parked on the side of the road obscures the area behind the truck relative to an approaching road vehicle. The area behind the truck is not visible to imaging sensors in the optical, infrared or radar range of the electromagnetic spectrum. This area is therefore not in the detection or perception area or FOV of the imaging sensors. If there is a person behind the truck, this is detected acoustically according to the invention, for example by footsteps or speech noises. The information from the imaging sensor about the hidden area together with the acoustic information indicates that a person is moving in a hidden area or that a person is likely to be moving in a hidden area. Such a confirmation by means of the imaging sensor increases the probability of recognizing hidden objects. The process is based on human behavior. For example, if a human driver hears noise from a child while driving a road vehicle, but does not see the child on the road, the driver concludes that the child is likely to be covered by another road vehicle or tree. As a result, the driver will drive slower or with more attention. The invention transfers this human behavior to self-driving road vehicles in order to recognize hidden objects and to output information about the hidden objects, for example to a passenger or to an automated control unit of the road vehicle.

According to one aspect of the invention, a control signal is determined for a control device in order to automatically control the road vehicle in order to reduce a collision with one of the hidden objects. For example, the road vehicle is braked when it detects hidden objects.

In the sense of the invention, a control device, in English electronic control unit, abbreviated ECU, processes input signals, processes them by means of an electronic circuit and provides logic and / or power levels as control and / or control signals.

The road vehicle can thus be controlled automatically in order to reduce or prevent collisions, for example by throttling the vehicle speed.

According to the invention, the type of objects, number of objects, speed of the objects, position of the objects, size of the objects and / or direction of movement of the objects relative to the road vehicle are determined as a function of the audio signals. This information is preferably determined using a first trained artificial neural network, preferably a convolution network, also called a convolutional network. The first artificial neural network receives the audio signals and the speed of the road vehicle in an input layer. This information is preferably output in the form of a list of likely hidden objects. According to the invention, a second trained artificial neural network is provided, which receives the specific information about the objects in an input layer and outputs a list of likely hidden objects validated by means of fusion of audio signals and image signals in an output layer. Alternatively, an artificial neural network is provided which fulfills the functions of the first and the second artificial neural network. The type of object indicates whether it is, for example, a human or an animal, for example a dog. Size of objects includes mass of objects. The mass of a person can be estimated, for example, on the basis of their step volume. The position of the objects can be determined, for example, by triangulating the audio signals. The speed and direction of movement of the objects relative to the road vehicle can be determined via the Doppler effect. The speed of the road vehicle, which can be tapped, for example, via a CAN bus, is preferably taken into account in this determination. By taking into account the distance and the relative speed of the concealed object, it is judged according to the invention whether a reduction in the vehicle speed may be necessary in order to avoid accidents if the concealed object would appear in front of the road vehicle.

According to a further aspect of the invention, the method is computer-implemented. This means that the process is carried out by a computer. A computer is, for example, a computing system of a control device. The computer is designed to process the audio signals and image signals, for example by means of pattern recognition.

According to a further aspect of the invention, an artificial neural network receives the audio signals and the image signals in an input layer. The artificial neural network outputs the objects determined to be hidden in an output layer. The artificial neural network comprises further layers between the input layer and the output layer. The number and connections of the further layers to one another were adapted to the recording of audio signals and image signals in a learning phase of the artificial neural network. In the learning phase, the audio signals and / or image signals are prepared for monitored learning by means of filters, amplifiers and / or label labels. The artificial neural network learns to determine the hidden objects by changing weighting factors for individual connections in and / or between the layers.

The artificial neural network learns to react appropriately to new information. This shortens the computing time, increases the accuracy and improves the security of the method provided.

In order to be able to react appropriately to new information, it is necessary that the artificial neural network first learns the meaning of predetermined information. For this purpose, the artificial neural network is trained with training data. In addition to the actual data, training data in particular also contain information about the meaning of the respective data. This means that the training data on which the artificial neural network learns are labeled. This training phase is inspired by the learning process of a brain.

Training with training data is called machine learning. A subset of machine learning is deep learning, which uses a series of hierarchical layers of neurons, so-called hidden layers, to carry out the machine learning process. The artificial neural network is preferably a deep neural network.

Neurons are the functional units of an artificial neural network. An output of a neuron generally results as the value of an activation function evaluated via a sum of the inputs weighted with weighting factors plus a systematic error, the so-called bias. An artificial neural network with several hidden layers is a deep neural network.

The artificial neural network is, for example, a fully connected network, referred to as a fully connected network. In a fully connected network, each neuron in a layer is connected to all neurons in the previous layer. Each connection has its own weighting factor. The artificial neural network is preferably a fully convolutional network. In a convolutional neural network, a filter is applied to a layer of neurons with the same weighting factors regardless of the position. The convolutional neural network comprises several pooling layers between the convolutional layers. Pooling layers change the dimension of a two-dimensional layer in width and height. Pooling layers are also used for higher dimensional layers.

In the so-called learning phase, connections between neurons are evaluated using weighting factors. Forward feeding means that information is fed into the input layer of the artificial neural network, goes through the following layers and is output in the output layer. Backward feeding, known as backward propagation, means that information passes through the layers backwards, i.e. from the output layer towards the input layer. The deviations of the respective layers are obtained by successively feeding back a received deviation of target and actual data from the output layer into the previous layer up to the input layer. The deviations are a function of the weighting factors. The difference between the actual output and the target output is evaluated by a cost function. When feeding backwards, the gradient of the error is fed backwards according to the individual weights. So you know whether and how much the difference between the actual and target output is minimized if you increase or decrease the respective weight. By minimizing the deviation in the training phase, for example using the least squares method, the cross-entropy known from information theory or the gradient descent method, the weighting factors are changed. As a result, when the input is fed in again, the desired output is approximated. For example, reverse feeding is described in detail in Michael A. Nielsen, Neural Networks and Deep Learning, Determination Press, 2015.

The control device according to the invention is designed for an automated operable road vehicle for regulating and / or automatically controlling the longitudinal and / or transverse guidance of the road vehicle depending on objects determined to be hidden. The control device solves the task mentioned at the beginning by means of at least one first interface with a high data transmission rate in order to receive audio signals and image signals. Furthermore, the control device comprises at least one computing system for processing the audio signals and image signals by means of at least one artificial intelligence. For this purpose, the computing system comprises at least one processor comprising a microarchitecture for parallel processing and execution of computing operations in order to carry out the at least one artificial intelligence. In addition, the computing system comprises at least one memory with a high data transfer rate for data exchange with the processor. The computing system is designed to execute the inventive method in real time in order to determine a degree of concealment of the objects. In addition, the control device comprises a second interface in order to provide control signals for actuators for the longitudinal and / or transverse guidance of the road vehicle depending on the objects determined to be hidden. Furthermore, the control device includes a protective housing with integrated cooling in order to protect the computing system against vibrations, moisture, heat and / or cold and to dissipate waste heat from the computing system.

An interface is a component between at least two functional units on which an exchange of logical quantities, for example data, or physical quantities, for example electrical signals, takes place, either only unidirectionally or bidirectionally. The exchange can be analog or digital. The exchange can also be wired or wireless.

The first interface is in particular a deserializer. Deserializers extract data structures from series of bits. Imaging sensors transmit raw data according to a specific serializer protocol. Serialization translates data structures into storable formats, for example using a series of bits. The states of a bit are symbolized as 0 or 1 written down. The states can simply be transmitted as a voltage signal. The state 0 is, for example, low voltage level. The state 1 is, for example, high voltage level.

The sensors preferably comprise an FPD link serializer or a GMSL serializer. With a GMSL serializer, for example, data transfer rates of 2.5 Gb per second can be achieved. The data are preferably transmitted according to the so-called low voltage differential signaling, abbreviated LVDS, standard. LVDS is a standard for high-speed data transmission. The deserializer is a corresponding logic module with a correspondingly high data transmission rate in order to extract the data structures from series of bits. The first interface is, for example, a radio interface, in particular a WLAN interface.

The computing system is a programmable electronic circuit. The computing system is preferably designed as a system-on-chip. This means that all or a large part of the functions of the computing system are integrated on one chip. In particular, the computing system merges the audio signals and the image signals.

The processor is preferably a multi-core processor. In a multi-core processor, multiple cores are arranged on a single chip. Multi-core processors achieve higher computing power and are less expensive to implement in a chip compared to multi-processor systems, in which each individual core is arranged in a processor socket and the individual processor sockets are arranged on a motherboard.

The multi-core processor preferably comprises central processors and / or graphics processors. Central processors are so-called central processing units, abbreviated CPU. The central processors, for example, have a 64-bit architecture. The graphics processor, in the English graphic processing unit, abbreviated GPU, preferably comprises at least one process unit for executing tensor and / or matrix multiplications. Tensor and / or matrix multiplications are the central arithmetic operations for deep learning, a form of artificial intelligence.

The computing system preferably also includes hardware / software accelerators for artificial intelligence, in particular so-called deep learning accelerators.

The computing system executes in particular a first and a second artificial neural network as described above or an artificial neural network according to the method according to the invention.

The memory is connected, for example, via a bus system to the computer system for signal / data exchange. In order to be able to achieve a high data transfer rate, the memory transfers data, for example, using a double data rate, quadruple data rate or octal data rate method. For example, the memory is a double data rate synchronous dynamic RAM, abbreviated DDR SDRAM, memory. The memory is preferably a low power DDR SDRAM memory.

The protective housing enables the control unit to be used in the automotive sector. The protective housing comprises, for example, a plastic, a metal or an alloy. The cooling dissipates waste heat from the computing system. The cooling is implemented, for example, by a water cooling circuit provided specifically for the control device. Alternatively, the cooling is provided by an engine cooling system already present in the vehicle, heat exchange elements being integrated in the protective housing in order to exchange heat via the vehicle cooling circuit.

Such a control unit can process very large amounts of data in real time and thus enables fully automated to driverless driving. This computing power of the control unit also enables cryptographic calculations, anomaly detection and plausibility checks, in particular with regard to information security, particularly with regard to cyber security, and operational security, especially with regard to functional security. The computing system is dimensioned in such a way that it can be integrated in a control unit for a vehicle. This means that the large computing power does not require supercomputers that fill cabinets distributed over the premises of a stationary data center.

The second interface is, for example, an interface to actuators for engine control, a braking system and / or steering system. The second interface is in particular an interface to an internal vehicle communication system in which different control devices, sensor units and multimedia units communicate with one another. For example, the second interface is an interface to a CAN bus system. The second interface is wired, for example.

The computer program product according to the invention recognizes objects in road traffic and comprises commands which cause a control device according to the invention to execute a method according to the invention when the computer program runs on the control device.

The commands of the computer program product, for example implemented in software code sections, represent a sequence of commands by means of which the control device, when the computer program is loaded, is prompted to determine hidden objects and to output a signal as a function of this determination, in particular for controlling safety-relevant vehicle functions. The computer program product thus has a technical effect.

The automated road vehicle for mobility services according to the invention comprises at least two microphones arranged on an outside of the road vehicle. The road vehicle further comprises at least one imaging sensor arranged on an outside of the road vehicle. The road vehicle further comprises a control device according to the invention in order to control the longitudinal and / or transverse guidance of the road vehicle as a function of objects determined to be hidden. The control unit is connected to the microphones and the imaging sensor for signaling purposes.

Mobility services, in English as mobility-as-a-service, are transportation services related to the transportation of goods and / or people. Ride sharing and e-hailing services are examples of mobility services. The road vehicle is in particular a minibus, which can accommodate up to 15 people, with a base area of the minibus no larger than that of a large sedan. This makes the minibus agile. The minibus includes an electric axle drive for emission-free travel. This means the minibus can provide mobility services in an urban environment.

The road vehicle preferably comprises an arrangement of a plurality of microphones, for example one microphone each in the front and rear right and left regions of the road vehicle. The road vehicle preferably comprises an arrangement of four microphones arranged side by side in the front and rear right and left areas. Such arrangements are surprisingly very well suited to absorbing noise and are relatively easy to obtain. In particular, this enables a 360 ° detection of ambient noise.

The data collected by the road vehicle, in particular, can be used to optimize an urban infrastructure in the planning and utilization of parking spaces at road edges with regard to improved safety for pedestrians.

• 1 ein Ausführungsbeispiel eines verdeckten Objekts,
• 2 ein Ausführungsbeispiel eines erfindungsgemäßen Straßenfahrzeuges,
• 3 ein Ausführungsbeispiel eines erfindungsgemäßen Verfahrens,
• 4 ein weiteres Ausführungsbeispiel des erfindungsgemäßen Verfahrens,
• 5 ein weiteres Ausführungsbeispiel des erfindungsgemäßen Verfahrens und
• 6 ein Ausführungsbeispiel eines erfindungsgemäßen Steuergeräts.

The invention is illustrated in the following exemplary embodiments. Show it:

• 1 an embodiment of a hidden object,
• 2nd an embodiment of a road vehicle according to the invention,
• 3rd an embodiment of a method according to the invention,
• 4th another embodiment of the method according to the invention,
• 5 a further embodiment of the method according to the invention and
• 6 an embodiment of a control device according to the invention.

In the figures, the same reference numerals designate the same or functionally similar reference parts. For the sake of clarity, only the relevant reference parts are highlighted in the individual figures.

1 shows a scenery from the traffic. A section of a street in an urban area is shown. The street is multi-lane and runs between two rows of buildings 8th . A road vehicle 2nd , for example a car or a minibus, drives on the road. Along one side of the road vehicle 2nd there are two parked vehicles 8th , for example a car and a truck. Between the parked vehicles 8th and the row of buildings runs along a pedestrian path. A pedestrian is shown walking between the parked vehicles 8th located and intends to use the space between the parked vehicles 8th to cross towards the street. Through the parked vehicles 8th is a detection area FOV of an imaging sensor 4th of the road vehicle 2nd limited. The pedestrian is covered by the parked truck and is therefore not covered by the imaging sensor 4th detected. The pedestrian is therefore a hidden object 1 .

However, the pedestrian can use microphones 3rd of the road vehicle 2nd localized and classified as pedestrians. For example, in audio signals of the microphones 3rd Footsteps or voices of the pedestrian can be recorded. Together with image signals from the imaging sensor 4th then there is a high probability that there is space between the parked vehicles 8th the pedestrian as a hidden object 1 located.

2nd shows an example of the road vehicle according to the invention 2nd . On a roof of the road vehicle 2nd is as an imaging sensor 4th For example, a 360 ° view camera is arranged with which image signals are obtained. Front and rear is on the road vehicle 2nd A microphone on each side 3rd arranged with which audio signals are obtained. The image signals and the audio signals become a control device 50 provided with which the imaging sensor 4th and the microphones 3rd are connected in a signal-conducting manner. The control unit 50 determined depending on the objects recognized as hidden 1 corresponding control signals B for actuators for the longitudinal and / or transverse guidance of the road vehicle 2nd to collide with the hidden objects 1 , for example due to the sudden appearance of the pedestrian 1 on the road to diminish or prevent. For example, the control signal B is a voltage level that is applied to brake actuators of the road vehicle 2nd is directed and as a result the road vehicle 2nd is braked.

The control unit 50 is in 6 shown. The control unit 50 includes a first interface 51 . Via the first interface 51 the audio signals and the image signals are obtained. The control unit also includes 50 a computing system 52 . The computing system 52 comprises at least one processor 53 and a memory 54 . The processor 53 and the memory 54 exchange data with each other. The processor 53 leads the artificial neural network 6 to determine the control signal B according to the inventive method. The control signal B is via a second interface 55 Actuators for the longitudinal and / or transverse guidance of the road vehicle 2nd provided. The control unit 50 is in a protective housing 56 built in with integrated cooling.

3rd schematically represents the method according to the invention, carried out by an artificial neural network 6 . The control unit 50 leads the artificial neural network 6 out. Audio signals from the microphones are used as input 3rd and image signals from the imaging sensor 4th receive. The artificial neural network 6 locates and classifies the objects 1 depending on the audio signals and compares the results obtained with the image signals to determine whether the objects 1 are covered. The output is the artificial neural network 6 a degree of concealment that indicates the likelihood that an object 1 is covered.

4th shows a further embodiment of the method according to the invention. A first artificial neural network 6 determined depending on audio signals from four microphones 3rd , an image signal of the imaging sensor 4th and the speed v of the road vehicle 2nd number N , Speed S , Position P , Size M and direction of movement D of objects 1 . The direction of movement D and the speed S of objects 1 are provided in an object list. This object list is calculated by calculating a delay in the audio signals as a function of those with the microphones 3rd obtained audio signals and the speed of the road vehicle v, for example by means of the Doppler effect. The object list and the number N , Speed S , Position P , Size M and direction of movement D of objects 1 become a second artificial neural network 6 provided, which validates the object list by means of fusion of audio signals and image signals.

5 shows a further embodiment of the method according to the invention, in which an artificial neural network 6 is used. The artificial neural network 6 receives audio signals from four microphones as input 3rd , an image signal from a camera as an imaging sensor 4th , an image signal of a 3D lidar as an imaging sensor 4th and the speed v of the road vehicle 2nd . The artificial neural network 6 carries out the method according to the invention and indicates a degree of masking as output. The artificial neural network determines depending on the degree of concealment 6 a corresponding control signal B.

Claims (7)

Method for recognizing objects (1) in road traffic for automated road vehicles (2) comprising the steps: Obtaining noise from objects in road traffic as audio signals of the objects (1) by means of microphones (3) which can be arranged on the road vehicle (2), Obtaining image signals of an environment of the road vehicle (2) by means of at least one imaging sensor (4) which can be arranged on the road vehicle (2), • Locating and classifying the objects (1) depending on the audio signals, the type of objects (1), number (N) of objects (1), speed (S) of objects (1), position (P) of objects (1 ), Size (M) of the objects (1) and / or direction of movement (D) of the objects (1) relative to the road vehicle (2) are determined as a function of the audio signals and a trained artificial neural network receives this specific information in an input layer, Assigning these localizations and classifications to the objects (1) in the vicinity of the road vehicle (2) depending on a comparison with the image signals, • Determining a degree of concealment for at least one of the objects (1) depending on the assignment and Outputting the degree of concealment, the artificial neural network in an output layer outputting a list of likely concealed objects validated by means of fusion of the audio signals and image signals. Procedure according to Claim 1 A control signal (B) is determined for a control unit (50) in order to automatically control the road vehicle (2) in order to reduce a collision with one of the concealed objects (1). Procedure according to one of the Claims 1 to 2nd , the method being computer-implemented. Method according to one of the preceding claims, wherein • an artificial neural network (6) receives the audio signals and the image signals in an input layer and outputs the objects (1) determined to be hidden in an output layer, whereby • the artificial neural network (6) between the input layer and the output layer comprises further layers, the number and connections of which are adapted to the recording of audio signals and image signals in a learning phase of the artificial neural network (6), and wherein • in the learning phase, the audio signals and / or image signals are prepared for monitored learning by means of filters, amplifiers and / or label labels and the artificial neural network (6) learns by changing weighting factors for individual connections in and / or between the layers which are concealed Determine objects (1). Control device (50) for an automated operable road vehicle (2) for regulating and / or automated control of longitudinal and / or transverse guidance of the road vehicle (2) depending on objects (1) determined to be concealed, comprising • at least one first interface (51) with a high data transmission rate in order to obtain audio signals and image signals, • at least one computing system (52) for processing the audio signals and image signals by means of at least one artificial intelligence, comprising o at least one processor (53) comprising a microarchitecture for parallel processing and execution of computing operations, to carry out the at least one artificial intelligence, and o at least one memory (54) with a high data transfer rate for data exchange with the processor (53), • wherein the computing system (52) is implemented, a method according to one of the Claims 1 to 4th to be carried out in real time in order to determine a degree of concealment of the objects (1), • a second interface (55) to, depending on the objects (1) determined to be concealed, control signals (B) for actuators for the longitudinal and / or transverse guidance of the road vehicle (2) to provide, and • a protective housing (56) with integrated cooling to protect the computing system (52) against vibrations, moisture, heat and / or cold and to dissipate waste heat of the computing system (52). Computer program product for recognizing objects (1) in road traffic comprising commands which cause a control unit (50) to follow Claim 5 a method according to one of the Claims 1 to 5 executes when the computer program runs on the control device (50). Automated road vehicle (2) for mobility services, comprising • at least two microphones (3) arranged on an outside of the road vehicle (2), • at least one imaging sensor (4) and on an outside of the road vehicle (2) • a control unit (50) after Claim 5 to control the longitudinal and / or transverse guidance of the road vehicle (2) as a function of objects (1) determined to be hidden.

Images