DE102019215118B3 - Control unit for automated driving functions for recognizing a lane closure and method and computer program for recognizing a lane closure - Google Patents

Abstract

Control unit (10) for automated driving functions for detecting a lane closure (R) by means of audio signals comprising a first interface (11) to at least one sound sensor (S) arranged on the vehicle (1) in order to receive audio signals from tire / road noises which when at least one tire of the vehicle (1) rolls on a road surface (D), a computing unit (12) comprising a classifier (K) is designed to classify and in dependence of the audio signals attached and non-attached components of the road surface (D) In the event of a classification of a non-attached component, output a signal indicating the end of the roadway (R), and a second interface (13) to provide the signal to actuators for longitudinal and / or lateral guidance of the vehicle (1).

Description

The invention relates to a control device for automated driving functions for recognizing a lane closure. The invention also relates to a method and a computer program for recognizing a lane closure.

Driver assistance systems which include a front camera are known from the prior art. With the front camera, the front area of an ego vehicle, for example vehicles and roads ahead, is recognized.

Often there are no lane markings on country roads or they are very dirty. A distinction between paved road and gravel road or planned alternative areas is very difficult with the known front cameras. For example, a gravel road is very difficult to see if the gravel is the same color as the asphalt.

However, other factors can also severely impair the camera sensor system, for example rain, strong sunlight / glare, contamination of the sensor system, image errors and / or incorrect detection due to errors in algorithms or due to optical errors.

The DE 10 2004 045 690 A1 discloses a device for determining external sound sources in the vicinity of a motor vehicle, comprising a plurality of microphones. A device for sound analysis identifies the sound source as a person, another motor vehicle, a siren or siren of a police or ambulance, a truck or the like by means of a classification process with regard to frequencies and volume.

The DE 10 2017 113 603 A1 discloses a device for addressing road noise, the type of road and the resulting noise to be suppressed being determined based on the noise level of the road.

This is where the invention comes in. The invention was based on the object of improving the detection of outer edges, for example gravel, gravel, grass, twigs, scrub, meadow, arable land or alternative areas.

The invention solves the problem by means of sound sensors that can be attached to the outer skin of the vehicle for acoustic detection of the outer edges. By recording tire-road noises by sound sensors outside the vehicle, characteristic acoustic features, for example gravel or gravel, noises from grass being driven over, twigs, scrub, meadow or field when crossing the ballast or gravel and thereby leaving the road an artificial intelligence.
In poor conditions such as snow, excessive sunlight or soiling of the sensors, the lane markings are not visible to the camera sensors. It is also difficult for adaptive cruise control systems to recognize the edge of the road if there are no marking lines, for example on country roads. By recording and evaluating the tire-road noise, touching / crossing the unpaved roadside, gravel or avoidance zones can be perceived.

The control device according to the invention for automated driving functions is designed to recognize a lane closure. The control device comprises a first interface to at least one sound sensor arranged on the vehicle in order to receive audio signals from tire / road noises that arise when at least one tire of the vehicle rolls on a road surface. In addition, the control device includes a computing unit. The computing unit includes a classifier. The classifier is designed to classify paved and unpaved components of the road surface as a function of the audio signals and, in the event of a classification of a non-paved component, to output a signal that indicates the end of the road. The control unit also includes a second interface in order to provide the signal to actuators for longitudinal and / or lateral guidance of the vehicle.

• • Aufnehmen von Reifen-Fahrbahn-Geräuschen, die beim Abrollen wenigstens eines Reifens eines Fahrzeuges auf einer Fahrbahndecke entstehen, mit wenigstens einem an dem Fahrzeug anordbaren Schallsensor und Erhalten von Audiosignalen,
• • Eingeben der Audiosignale in einen Klassifikator einer Recheneinheit eines Steuergeräts, der ausgeführt ist, in Abhängigkeit der Audiosignale befestigte und nicht befestigte Bestandteile der Fahrbahndecke zu klassifizieren,
• • im Falle einer Klassifikation eines nicht befestigten Bestandteils der Fahrbahndecke Ausgeben eines Signals, das auf den Fahrbahnabschluss hinweist, und
• • Bereitstellen des Signals für eine Längs- und/oder Querführung des Fahrzeuges,

wobei das Steuergerät ein erfindungsgemäßes Steuergerät ist.Another aspect of the invention relates to a method for recognizing a road closure by means of audio signals, comprising the method steps

• • Recording of tire-road noises that arise when at least one tire of a vehicle rolls on a road surface with at least one sound sensor that can be arranged on the vehicle and receiving audio signals,
• • Inputting the audio signals into a classifier of a processing unit of a control device, which is designed to classify fixed and non-fixed components of the road surface depending on the audio signals,
• • In the case of a classification of a non-paved component of the road surface, output of a signal that indicates the end of the road, and
• • Providing the signal for longitudinal and / or lateral guidance of the vehicle,

wherein the control device is a control device according to the invention.

According to one aspect of the invention, the method is based on the steps of detecting noise as a noise signal while driving with a microphone, evaluating the noise level using a trained artificial neural network and recognizing whether the vehicle has left the road and is driving on gravel / unpaved road with at least one tire . If the detection is positive, the longitudinal and / or transverse guidance is regulated and / or controlled in such a way that the vehicle leaves the unpaved road end and continues on a paved component of the road surface.

Another aspect of the invention relates to a computer program for recognizing a lane closure comprising commands which cause a control device according to the invention to carry out a method according to the invention when the computer program is running on the control device. The commands are implemented as software code sections, for example. The computer program has a technical effect when it is executed, namely the recognition of road closures and the associated increase in safety when driving.

The acoustic detection of the end of the lane is used to determine the position of the vehicle on the lane even if camera systems can no longer perceive the lane markings, for example due to heavy rain, glare, pollution of the sensors, errors in algorithms or optical errors. In addition, according to the invention, the determination of the position of the vehicle on a roadway using camera systems is checked for plausibility with the determination of the position of the vehicle using sound sensors and vice versa. This increases safety while driving.

Advantageous configurations of the invention emerge from the subclaims, the drawing and the description of preferred exemplary embodiments.

A control unit prepares the data from sensors as input signals, processes them by means of a processing unit, for example a programmable logic module, an FGPA or ASIC module or a computer platform, and provides logic and / or power levels as regulating or control signals. The signal that the processing unit outputs is a regulating or control signal. With this signal, actuators for longitudinal and / or lateral guidance of the vehicle are regulated and controlled via the second interface in order to keep the vehicle in the lane. An actuator for the lateral guidance is, for example, an electric motor of an electromechanical power steering system. The signal that the processing unit outputs and that indicates the end of the road is, in the case of a gravel road end, for example an electrical voltage signal for an actuator of a four-wheel drive of the vehicle in order to prevent the tires from spinning on gravel. This increases safety when driving if, for example, lane markings cannot be detected by camera systems due to poor environmental conditions and the vehicle encounters an unsecured component of the road surface. The control unit is signal-connected to the sound sensor via the first interface. The data exchange is wired or wireless, for example via radio technology, for example via a WLAN standard. According to a further aspect of the invention, the control unit is integrated into an on-board network of the vehicle, for example into a CAN bus. The control device is, for example, an electronic control device for automated driving functions, called an ECU in the English domain. According to one aspect of the invention, the domain is an ADAS (advanced driver assistance system) or AD (autonomous driving) domain ECU for assisted to fully automated, that is to say autonomous, driving.

The computing unit of the control device is implemented, for example, as a system-on-a-chip with a modular hardware concept, that is, all or at least a large part of the functions are integrated on one chip and can be expanded in a modular manner. The chip can be integrated into the control unit. The computing unit includes, for example, a multi-core processor and memory modules. The multi-core processor is configured for signal / data exchange with memory modules. For example, the multi-core processor includes a bus system. The memory modules form the main memory. The memory modules are, for example, RAM, DRAM SDRAM or SRAM. In a multi-core processor, several cores are arranged on a single chip, that is to say a semiconductor component. Multi-core processors achieve higher computing power and are more cost-effective to implement in a chip compared to multi-processor systems in which each individual core is arranged in a processor socket and the individual processor sockets are arranged on a motherboard. According to one aspect of the invention, the computing unit comprises at least one central processing unit, referred to in English as the central processing unit, abbreviated to CPU. The processing unit transforms a time spectrum of the audio signals into a frequency spectrum. Depending on the frequency spectrum, audio signals from paved components of the road surface can be distinguished from audio signals from non-paved components of the road surface. For example, the computing unit is programmed to carry out a Fourier transformation of the audio signals.

The computing unit also includes at least one graphics processor, referred to in English as a graphic processing unit, abbreviated to GPU. Graphics processors have a special micro-architecture for the parallel processing of sequences. According to one aspect of the invention, the graphics processor comprises at least one process unit which is specifically designed to carry out tensor and / or matrix multiplications. Tensor and / or matrix multiplications are the central arithmetic operations for deep learning. According to one aspect of the invention, the processing unit also includes hardware accelerators for artificial intelligence, for example so-called deep learning accelerators. According to a further aspect of the invention, the classifier is provided in the programming technique CUDA. This means that software code sections of the classifier are processed directly by the GPU.

Furthermore, the computing unit or the control device are configured to be modularly expanded with a plurality of, for example at least four, such chips.

The verge or road edge forms the end of the roadway, i.e. the end of the roadway, and prevents the edge of the roadway from breaking off. The upper part of the road surface is called the road surface. Examples of paved components of the road surface are asphalt surface courses, concrete pavements or paving stones. Examples of unpaved parts of the road surface are gravel or gravel or grass, twigs, scrub, meadow or arable land.

Tire-road noises are the noises that occur when the tire rolls on the road surface. Sources are airborne and structure-borne noise sources, for example air pumping or tire vibrations.

A sound sensor is a sensor that detects mechanical vibrations, for example caused by airborne sound waves, and converts them into a processable signal, for example an electrical signal such as an electrical voltage. According to one aspect of the invention, the sound sensor can be arranged in the vicinity of tires of the vehicle, for example in wheel arches. In this way, the tire-road noise is recorded as close as possible to the point of origin, i.e. with a low level of interference.

The electrical signal is called the audio signal. The audio signals are provided in available audio files compressed or uncompressed with the identification as meta information. For example, WAV is an uncompressed file format. Ogg-Vorbis or MP3 are each compressed file formats.

The sound sensor includes an analog and / or digital signal output. The forming takes place in two stages. In a first acoustic-mechanical conversion stage, the airborne sound is converted into the movement of an object according to a certain reception principle. In the second mechanical-electrical conversion stage, the movement of the object is converted into the electrical signal according to a specific converter principle. An acoustic sensor that detects airborne sound waves is a sound sensor. Examples of sound and / or acoustic sensors are an arrangement of a magnet and an electric coil, microphones, accelerometers, piezo sensors or strain gauges. A micro-electro-mechanical system, abbreviated to MEMS, comprising an arrangement of semiconductor elements that absorb vibrations, can also be used as a sound sensor. The sound sensor is arranged on the vehicle, for example in the vicinity of a tire, for example in a wheel arch.

According to one aspect of the invention, the sound sensor comprises a microphone. The microphone includes a microphone capsule and a transducer. The acoustic-mechanical conversion takes place in the microphone capsule. The microphone capsule includes, for example, a membrane that is excited to vibrate by airborne sound. The mechanical-electrical conversion takes place in the converter. The transducer is, for example, an electrodynamic transducer, such as in the case of a moving coil microphone, or an electrostatic transducer, such as in the case of a condenser microphone.

According to a further aspect of the invention, the sound sensor is implemented as a MEMS microphone. MEMS microphones are miniaturized microphones which, for example, are made using SMD technology, i.e. surface mounted devices, for direct use on the circuit board. MEMS microphones have small dimensions and are easy to process industrially, for example MEMS microphones can be assembled in a reflow soldering process. Compared to other microphones, MEMS microphones are less sensitive to high temperatures and are therefore particularly suitable for automotive applications. Alternatively, the sound sensor is an electret condenser microphone.

Machine learning refers to the learning process of an artificial system to recognize patterns or laws on the basis of training data inspired by the learning process of the human brain in order to be able to react appropriately to new information. Machine learning will realized by algorithms which are implemented in software code sections. An open source platform for programming machine learning is, for example, TensorFlow written in the programming languages Python, C ++ or CUDA, that is, compute unified device architecture. The platform is, for example, a Linux, macOS, Microsoft Windows, Android or JavaScript platform. Platt After the learning phase, unknown data can advantageously be assessed without first having to be learned by heart.

A classifier divides objects or situations into classes and recognizes patterns. A classifier is, for example, an algorithm that recognizes patterns and is called a pattern recognition algorithm. The classifier carries out a machine classification. The classification is learned from actual road closures and target road closures. The actual lane closure is the lane closure that the classifier receives as a classification result. The target lane closure is the actual, real lane closure, that is, paved and non-paved components of the road surface, which is entered into the classifier in the form of indicators. The marks are also called features, targets or labels. For example, an audio recording of gravel as an unsecured component of the road surface is labeled with the meta information “The audio signal comes from an outer edge of gravel”. In this way, the measured tire-road noise, which occurs when rolling on the unsecured part of the road surface, is related to the road closure information "gravel outer edge". By training the classifier with the actual lane terminations and the target lane terminations, the classifier is optimized in order to obtain the lane termination from audio signals from the sound sensor. The classifier learns an internal structure or topology by itself in order to assign minor fluctuations in the audio signal to the actual underlying lane closure.

According to the invention, the classifier is designed to classify aggregates. Aggregates are used as base layers in road and path construction.

According to a further aspect of the invention, the classifier is designed to classify sand, gravel, chippings, crushed stone, scraps, grass, twigs, scrub, meadow and / or fields. A bulk material - regardless of the rock - is called sand if the grain size has a diameter of less than 2 mm. Gravel and grit, in turn, are both mixtures of stones with a size of 2 to 63 mm. The difference again relates to the geometry of the individual grains. Gravel consists of round stones ("pebbles"), while grit consists of sharp-edged rubble stones. We speak of ballast when the bulk material consists of round or broken stones with a grain size between 32 and 63 mm. All amounts of rock with even larger grains are finally referred to as Schroppen. This means that various outer edges and evasive areas can be acoustically classified and thus recognized.

According to a further aspect of the invention, in a training phase of the classifier, tire-road noises are recorded that occur when rolling on road closures made of sand, gravel, gravel, crushed stone and / or scraps and / or when at least one tire of the vehicle crosses the road. For example, noises from grass, twigs, brush, meadow or field arise. According to a further aspect of the invention, in the training phase, gradual listening is trained in the order of paved road, gravel or gravel and grass, branches, scrub, meadow or field. This means that a creeping departure from the lane is recognized.

According to a further aspect of the invention, the classifier is a random forest classifier, a support vector machine or an artificial neural network comprising a plurality of layers, the layers comprising completely connected and / or convolutional layers.

Random Forest is a classifier comprising uncorrelated decision trees that grow after a certain randomization during a learning process. For a classification, each tree in this forest is allowed to make a decision and the class with the most votes decides the final classification. Advantages of the random forest include that it trains relatively quickly due to the short training and / or construction times of a single decision tree, that evaluations based on several trees can be parallelized and that important classes such as median or hard shoulder can be recognized.

A support vector machine is a classifier that divides a set of objects into classes in such a way that the widest possible area around class boundaries remains free of objects. A support vector machine is advantageous, among other things, when repeatedly driving on the same route.

An artificial neural network is a collection of individual information processing units, called neurons, which are arranged in layers in a network architecture. At the beginning of the training, each neuron has a random one Initial weight. Training data is fed into the artificial neural network in a so-called forward pass. Each neuron weights its input signals with its weight and forwards the result to neurons in other layers. The overall result is calculated in the output layer. The deviation between the actual end of the lane and the target end of the lane is calculated in a backward feed, the so-called backward pass.

The internal structure of the artificial neural network is adapted in the reverse feed. The share that each neuron has in this deviation is calculated, and the weight of each neuron is then changed in the direction that minimizes the deviation, or error. Weights are increased, decreased or set to zero. The error is a function of the weights. The deviation between the actual end of the lane and the target end of the lane is evaluated by a cost function. When feeding backwards, the gradient of the error is fed backwards according to the individual weights. In this way, you know whether and to what extent the difference between the actual lane closure and the target lane closure is minimized if the respective weight is increased or decreased. By minimizing the deviation in the training phase, for example using the least squares method, the cross-entropy known from information theory or the gradient descent method, the weights are changed. Then do the next run, measure the error again and adjust the weights, and so on. The artificial neural network thus learns increasingly better to infer the known target road closures from the training data.

The artificial neural network comprises layers with fully connected networks, referred to in English as a fully connected network. In a fully connected network, each neuron in one layer is connected to all neurons in the previous layer. Each connection has its own weight. The artificial neural network alternatively or additionally comprises convolutional layers. In a convolutional neural network, also called a convolution network, a filter is applied to a layer of neurons regardless of their position with the same weights. The convolutional neural network comprises several pooling layers between the convolutional layers. Pooling layers change the width and height of a two-dimensional layer. Pooling layers are also used for higher-dimensional layers.

According to a further aspect of the invention, the classifier is validated with image signals of the lane closure while the vehicle is traveling. The image signals are obtained with imaging sensors arranged on the vehicle, for example camera, lidar or radar sensors. The image signals correspond to validation data with which the classifier is tested during or at the end of the learning process.

According to a further aspect of the invention, the control device is designed to merge the audio signals with image signals obtained by means of imaging sensors arranged on the vehicle in order to determine a position of the vehicle on the roadway. Safety is increased by a fusion of camera signals and the lane markings determined by means of audio signals. For example, the results of the evaluation of the audio signals are checked for plausibility with the lane markings obtained from the camera signals, and vice versa. Furthermore, the redundancy is increased by the further possibility of determining the lane marking.

According to a further aspect of the invention, the signal is made available to a vehicle driver, for example via a man-machine user interface, for example optically, acoustically or tactilely. The vehicle driver is thus notified of the detected lane closure. This is particularly advantageous if the vehicle driver forms the fall-back level in an automated driving system. In the case of the SAE J3016 standard, this is the case up to SAE Level 3. According to a further aspect of the invention, such a notification occurs if, after the ballast or gravel has been exceeded, noises from grass, branches, scrub, meadow or fields can be heard on at least one tire . In this case, an emergency stop of the vehicle occurs during AD operation, for example SAE level 4 or 5.

A further aspect of the invention relates to a vehicle with a control device according to the invention and at least one sound sensor arranged on an outer skin of the vehicle and connected to the control device via a first interface. According to one aspect of the invention, the vehicle is an automatically operated vehicle, for example a partially automated, highly automated or fully automated / autonomously driving vehicle. The sound sensor is, for example, a MEMS microphone.

• 1: Ausführungsbeispiel eines Fahrzeuges, teilweise auf einem nicht befestigten Bestandteil einer Fahrbahndecke fahrend,
• 2: Ausführungsbeispiel einer Fahrbahndecke mit Fahrbahnabschluss,
• 3: Ausführungsbeispiel eines erfindungsgemäßen Steuergeräts,
• 4: schematische Darstellung des erfindungsgemäßen Verfahrens dargestellten Ausführungsbeispiele erläutert.

The invention is based on the in the figures

• 1 : Exemplary embodiment of a vehicle, partially driving on a non-paved part of a road surface,
• 2 : Example of a road surface with a roadway closure,
• 3 : Embodiment of a control device according to the invention,
• 4th : Schematic representation of the inventive method illustrated embodiments illustrated.

In the figures, the same reference symbols denote the same or functionally similar reference parts. In the figures, only the reference parts relevant to the respective understanding are given.

This in 1 depicted vehicle 1 is a car. On an outer skin of the vehicle 1 is a sound sensor S. arranged. The vehicle 1 drives on a roadway F. . The roadway F. is for example a country road. The vehicle 1 partly drives on a fixed part of a road surface D. , for example asphalt, partly on a non-paved component, for example gravel. The non-attached component forms a roadway closure R. , for example a roadside.

2 shows that the components of the roadway that are not fastened R. Visually difficult to distinguish from the paved component due to the dark color of the asphalt and shadows in the border area.

Using the in 3 control unit shown 10 becomes the end of the road R. acoustically recognized with the sound sensor S. and a computing unit 12 of the control unit.

To this end, the control unit includes 10 a first interface 11 to the sound sensor S. to get tire-road noise. Tire-road noises from paved components of the road surface that are driven over differ from tire-road noises from non-paved components of the road surface that are run over. A classifier K a computing unit 12 of the control unit 10 classifies the audio signals received into audio signals from fixed and non-fixed components of the driving belt corner D. . In the case of the classification of a non-attached component, the computing unit generates 12 a signal for longitudinal and / or lateral control of the vehicle 1 to the vehicle 1 from the non-paved component to the paved component of the road surface D. to maneuver.

With the control unit 10 will that be in 4th Method shown for recognizing a lane closure R. performed by means of audio signals. In a first process step V1 are tire-road noises that occur when at least one tire of the vehicle rolls over the road surface D. arise with the at least one on the vehicle 1 arranged sound sensor S. recorded and audio signals received. In a second process step V2 are the audio signals in the classifier K the arithmetic unit 12 of the control unit 10 entered and paved and unpaved components of the road surface depending on the audio signals D. classified. In the case of a classification of a non-paved component of the road surface D. is in a third process step V3 a signal that points to the end of the road R. indicates, issued. In a fourth process step V4 becomes the signal for longitudinal and / or lateral guidance of the vehicle 1 provided.

List of reference symbols

1

vehicle

10

Control unit

11

first interface

12

Arithmetic unit

13

second interface

R.

Lane closure

F.

roadway

D.

Road surface

S.

Sound sensor

K

Classifier

V1-V4

Process step

Claims (7)

Control unit (10) for automated driving functions for recognizing a lane closure (R) by means of audio signals, comprising • a first interface (11) to at least one sound sensor (S) arranged on the vehicle (1) in order to receive audio signals from tire-lane noises, which arise when at least one tire of the vehicle (1) rolls on a road surface (D), • a computing unit (12) comprising a classifier (K) designed to classify attached and non-attached components of the road surface (D) as a function of the audio signals and in the case of a classification of an unsecured component, to output a signal indicating the end of the roadway (R), and • a second interface (13) to provide the signal to actuators for longitudinal and / or lateral guidance of the vehicle (1) , where the classifier (K) is designed to classify aggregates. Control unit (10) Claim 1 , the classifier (K) being designed to classify sand, gravel, chippings, crushed stone, scraps, grass, twigs, scrub, meadow and / or fields. Control device (10) according to one of the preceding claims, wherein the classifier (K) comprises a random forest classifier, a support vector machine or an artificial neural network comprising several layers, the layers comprising completely connected and / or convolutional layers. Control device (10) according to one of the preceding claims, designed to fuse the audio signals with image signals obtained by means of imaging sensors arranged on the vehicle (1) to determine a position of the vehicle (1) on the roadway (F). Method for recognizing a lane closure (R) by means of audio signals comprising the method steps • Recording of tire-road noises that arise when at least one tire of a vehicle (1) rolls on a road surface (D) with at least one sound sensor (S) that can be arranged on the vehicle (1) and receiving audio signals (V1), • Input of the audio signals into a classifier (K) of a computing unit (12) of a control unit (10) which is designed to classify (V2) fixed and non-fixed components of a road surface (D) as a function of the audio signals, • In the case of a classification of a non-paved component of the road surface (D), output of a signal that indicates the end of the road (R) (V3), and • Providing the signal for longitudinal and / or lateral guidance of the vehicle (1) (V4), the control device (10) being a control device (10) according to one of the preceding claims. Procedure according to Claim 5 , wherein the signal is provided to a vehicle operator. Computer program for recognizing a lane closure (R) comprising commands which cause a control device (10) to follow one of the Claim 1 to 5 a procedure according to Claim 6 or 7th executes when the computer program is running on the control unit (10).

Images