GB2611559A - Method and device to detect traffic hazards based on sound events - Google Patents

Abstract

System and method onboard a vehicle passively recording surrounding background environmental noise using at least two microphones 2, carrying out noise cancellation, localising the source of hazard based on time delay or phase shift the sound arrives at the different microphones, and classifying the nature of the hazard using a trained neural network. The driver is warned of the location and the nature of hazard. The vehicle further broadcast the warning to surround vehicles and a centralised server using transceiver 3. Other onboard sensor data (e.g. camera, LiDAR) are used to refine and enhance the position of the hazard source. Also claimed is a device onboard a vehicle having at least two microphones, a transceiver 3, a data acquisition unit 4 for obtaining other onboard sensor signals, and a processor 5 for processing audio signal from the microphones 2.

Description

Method and Device to Detect Traffic Hazards Based on Sound Events

Background of the Invention

Depending on the background noise, reflections, absorption behaviour etc., sound events in traffic situations can be detected over wider distances up to several hundred metres or even kilometres. One specific advantage for using sound events as indicators for potential hazards is that no direct line of sight between the receiving unit (e.g. microphone) or a person and the sound source is required since airborne sound propagation follows a different physical basis.

Acoustic methods can hence be used to detect potential hazards in road traffic that are not detected by LIDAR, radar, camera (e.g. children behind parked vehicles, etc.). An increase in safety, especially the safety of vulnerable road users, can be achieved by augmenting traffic hazard detection based on sound events.

This invention presents a method and a device to detect and localize traffic hazards and risks based on sound events by passively recording traffic sound on vehicles, detecting pre-defined sound patterns in pre-defined hazard categories, communicate the ego position of the vehicle and the relative position of detected sound source to other vehicles and traffic participants and allow for enhanced localisation, warning and improved database for hazard situations.

State of the Art Several technical realisations of acoustic warning and localisation systems are state of the art, all making use of the finite propagation speed of sound and the ability to localise sound sources using two or more microphones, similar to the spatial resolution ability of human hearing.

These technical realisations are also applied in traffic situations, e.g.: * for the warning and localisation of emergency vehicles (V. Tran and W. Tsai, "Acoustic-Based Emergency Vehicle Detection Using Convolutional Neural Networks," in IEEE Access, vol. 8, pp. 75702-75713, 2020, doi: 10.1109/ACCESS.2020.2988986), * for the detection and localisation of high-impulse noise such as firearm shooting (P. Naz et al., Acoustic detection and localization of small arms, influence of urban conditions, art. no. 69630E, in Proceedings of SPIE -The International Society for Optical Engineering * April 2008, DOI: 10.1117/12.783450) or * for traffic monitoring in tunnels and other infrastructure (F. Graf, M. Gruber, Rapid Incident Detection in Tunnels Through Acoustic Monitoring -Operating Experiences in Austrian Road Tunnels, Proceedings of the 9th International Conference 'Tunnel Safety and Ventilation' 2018, Graz).

(Deep) neuronal networks for traffic risk detection are known and published e.g.: * for radar-based systems (EP 3 828 592 Al, Deep Neuronal Network for Detecting Obstacle Instances Using Radar Sensors in Autonomous Machine Applications) * for lidar-based systems (EP 3 832 532 A2, Multi-View Deep Neuronal Network for Lidar Perception) * for camera-based systems (C. Pena-Caballero et al., Real-Time Road Hazard Information System, in MDPI Infrastructure Journal, 2020, 5, 75; doi:10.3390/infrastructures5090075) The present invention goes beyond the state of the art by presenting a method and a device to define, detect, localise and communicate hazardous traffic sound events and evaluate data from multiple vehicles and traffic participants to refine the position of the hazard source, the data-driven models and to provide data for centralised analytics.

Description of the Invention

The sound signal of a traffic situation received via a microphone 2 mounted on a vehicle 1 depends on several parameters. Among others, the noise level, signal amplitude, phase and frequency are affected e.g. by: * Static (time-invariant) vehicle data of the vehicle model used such as the o Powertrain (including internal combustion engine or e-motor) NVH characteristics o Type of tire o The modal behaviour of the vehicle structure * Dynamic data such as the o Velocity o Acceleration or braking behaviour * Environmental data, such as the o Weather conditions (rain, snow, ...) o wind * Microphone characteristics and sensitivity maps It is obvious that the proposed method and device can also be applied for motorcycles, trucks, bicycles and all other types of vehicles. It is not limited to cars.

The method comprises the following steps: In a first step, a set of sound events are defined that indicate hazard potential (e.g. horns, squealing brakes, children's voices, sirens, ...). Those sound events are analysed concerning their acoustical properties (frequency spectra, temporal signal course, typical sound pressure level at source, etc.) and described by a set of reduced parameters. Similar sound events are categorised, set in context and assigned a hazard potential metric. This step also incorporate prior knowledge of realistic traffic sound sources, e.g. that human-based sound sources are only considered if they come from 1-2 m above ground.

In a second step, a data-driven model of traffic sound events with hazard potential is set up using physically informed Al. To create (training) data for these sound events, experimental data may serve a as a basis and the number of experimental data can be extended by means of numerical methods. Only physically realistic traffic sound events are used for training, typically in a supervised learning approach. The parameter sensitivity and severity are considered in this data-driven model. The creation of the model and the machine learning procedure is done offline, a model update can be provided e.g. via update over the air. The data-driven model (filter) is implemented on the calculation unit 4.

A third step of the proposed method comprises the minimisation and reduction of the background noise caused by wind noise, tire noise and other static and dynamic noise sources by considering existing knowledge of vehicle data. State of the art methods to model disturbing noise sources and providing a noise model are used to reduce background noise that is not related to the traffic hazard sound events.

Static vehicle data is inserted into this model by means of powertrain, chassis or tire characteristics (e.g. from product data sheets or experimental/numerical investigations). Dynamic vehicle data is integrated using e.g. OBD-2 interfaces or any other suitable interface to obtain dynamic vehicle data. An obvious process for handling dynamic data is the equalization of the velocity-dependent Doppler frequency shift. Environmental data may be included using web-interfaces and web-based weather data services for a given position. Microphone characteristics are considered typically by means of product data sheets.

Several state of the art approaches in noise, vibration, harshness (NVH) are known to reduce complexity and model-order, resulting in numerical models with fast calculation time. The output of this third step is a filtered/cleaned sound signal received by the microphones where disturbing deterministic noise sources are reduced as much as possible.

In a forth step, the data-driven model is applied to the filtered microphone signals to detect hazard sound events and categories of sound events incorporate the prior knowledge. The output of this step is the temporal detection of noticeable sound events for all microphones and the plausibility check if the detected sound event comes from the same sound source (e.g. signal similarity, delay difference in the range of microphone distance for the given sound speed).

In a fifth step, acoustic localisation is for the used microphones is applied according to state of the art methods for acoustic localisation with multiple microphones. The time delay or phase shift of recorded signals for microphones 2 in certain distance due to the finite speed of sound is evaluated. The output of this step is an estimation of the relative position of the hazard source. The ego vehicle is warned that a hazard source has been detected and the information about direction of the hazard source and the distance to the hazard source is provided.

In a sixth step, the ego position of the vehicle during the is occurrence of the traffic hazard sound event is determined (GPS, dGPS, SLAM, etc.). Based on the amplitude and characteristic of the received sound event, the range and direction where this sound event can be detected is estimated. The ego position of the vehicle, the orientation of the vehicle with respect to north direction and the relative position of risk sound events and noise category is transmitted to other vehicles in the close vicinity (range within the sound event can be detected by other traffic participants) using a communication unit 3. The output of this step is that the other vehicles in the close range are warned and that information about the position of the hazard source is provided.

In a seventh step, the information about position estimation of several vehicles is fused and sensor fusion with other sensor data (e.g. camera, LIDAR etc.) is carried out. Situation data is fed back to allow for further learning approaches and for model improvement. The output of this step is the refinement of the position of the hazard source, enriched training data as well as data for visualization (e.g. heat maps) of hazard events over time that can be exploited for traffic management or city planning etc. The proposed device to detect traffic hazards based on sound events consists of at least two microphones 2 mounted inside the structure or on the outer surface of the structure of a vehicle 1. With those microphones 2, the sound (traffic noise) is recorded over time and the sound signals are provided to the computation unit 4. An increasing number of microphones 2 and increasing inter-microphone distances allow for higher resolution acoustic localisation and improved reliability of the hazard sound source localisation. It can be advantageous to position microphones close to dynamic sound sources resulting from the vehicle 1 (e.g. close to a tire or close to aero-acoustic noise sources such as side mirror or A-pillar) to obtain a high-quality signal in the vicinity of the dynamic noise source.

The data acquisition unit 4 acquires static vehicle data and dynamic vehicle data (e.g. via OBD-2 interface) and environmental data (e.g. via internet access by the transmitting/receiving unit 3).

The computation unit 5 processes the signals of the microphones 2. Static vehicle data, dynamic data, environmental data as well as parameters and characteristic diagrams of the vehicle 1 and the microphones 2 are processed and the method to determine and localise hazard sound events is executed.

The output of the computation unit 5 is transmitted to other vehicles and to central services using the transmitting/receiving unit 3 For better understanding, two figures are provided: Fig. 1: shows a flow chart of the seven steps that define the proposed method Fig. 2: shows the device integrated into a vehicle 1. The device consists of two or more microphones 2, a transmitting/receiving unit 3 as well as a data acquisition unit 4 and a computation unit 5.

Claims (2)

1. Claims 1. Method to define, detect, localise and communicate hazardous traffic sound events comprising in a first step the sound event definition and categorisation, in a second step the model set-up and the training of the model by means of data-driven approaches, in a third step the on-board microphone measurements and background noise reduction, in a forth step the hazard sound event detection based on background noise filtered microphone signals, in a fifth step the acoustic localisation of the hazard source based on the time delay or phase shift of two or more microphones 2, in a sixth step the ego vehicle land the driver are warned and the localised hazard source position is transmitted to other vehicles for warning, and in a seventh step additional data fusion with other in-vehicle sensors is carried out to refine the position of the hazard source and information on time of occurrence, hazard sound category and absolute position of the hazard sound source are transmitted to a central server for further for centralised data analytics.
2. 2. Device to detect, localise and communicate hazardous traffic sound events on a moving vehicle 1 comprising two or more microphones 2 mounted in or outside of the structure of the vehicle 1, a transmitting/receiving unit 3 to communicate with internet and other traffic participants, a data acquisition unit 4 to acquire static vehicle data, dynamic vehicle data and environmental data, and a computation unit 5 to processes the signals of the microphones 2.