DE102019215654A1 - System and method for voice communication with a vehicle owner and an automated road vehicle comprising such a system - Google Patents

Abstract

System (10) for voice communication with a vehicle owner (1) comprising at least one acoustic sensor (11) comprising a fastening arrangement (12) for installing the acoustic sensor (11) in a body of a road vehicle (2), an evaluation device (13) for processing the Signals of the acoustic sensor (11) comprising a machine learning program (14) which is trained to recognize words and / or sentences (3) spoken by traffic participants (4) outside the vehicle in the signals of the acoustic sensor (11) and to recognize the recognized words and / or to filter sets (3) from the signals of the acoustic sensor (11) when the machine learning program (14) is being executed on the evaluation device (13), an interface (15) by means of which the system (10) from the vehicle external road users ( 4) is activated, and a terminal (16) which can be used as a vehicle internal terminal and which transmits the signal is connected to the evaluation device (13), the Sy stem (10) provides the spoken words and / or sentences (3) of the external road user (4) to the vehicle owner (1) by means of the terminal (16).

Description

The invention relates to a system and method for voice communication with a vehicle owner. The invention also relates to an automated road vehicle comprising such a system.

Acoustic sensors for detecting external noises outside of vehicles are known from the prior art. For example, the DE 10 2016 006 802 A1 a method and a device for detecting at least one special signal emanating from an emergency vehicle.

Road vehicles are often consciously or not consciously parked illegally, that is, improperly, for example in urban areas, for example in cities. This affects or hinders road traffic. For example, emergency services and / or rescue vehicles are hindered by incorrectly parked vehicles or the flow of traffic is impaired. In some cases, high fines are required for such impairments or disabilities or the illegally parked vehicle is towed away.

One object of the invention was to resolve the impairments or handicaps described without having to impose fines or tow vehicles.

The invention solves this problem by establishing voice communication with the vehicle owner of the road vehicle, in particular via so-called external speech communication, abbreviated ESC.

In one aspect, the invention provides a system for voice communication with a vehicle owner. The system includes at least one acoustic sensor. The acoustic sensor comprises a fastening arrangement for installing the acoustic sensor in a body of a road vehicle. The system also includes an evaluation device. The evaluation device processes the signals from the acoustic sensor. The evaluation device comprises a machine learning program. The machine learning program is trained to recognize words and / or sentences spoken by road users outside the vehicle in the signals of the acoustic sensor and to filter the recognized words and / or sentences from the signals of the acoustic sensor when the machine learning program is executed on the evaluation device. The system also includes an interface. The system is activated by road users who are external to the vehicle by means of the interface. The system also includes a terminal that can be used as a vehicle internal terminal. The terminal is connected to the evaluation device in a signal-transmitting manner. In the activated state, the system provides the spoken words and / or sentences of the external road user to the vehicle owner by means of the vehicle's internal terminal.

According to a further aspect, the invention provides a method for voice communication between a law enforcement officer and a vehicle owner. In the case of a driverless road vehicle that includes a system according to the invention, the law enforcement officer activates the system via its interface. By means of the acoustic sensor and the evaluation device, spoken words and / or sentences of the law enforcement officer are filtered from ambient noise. The vehicle owner is called using the terminal. The law enforcement officer speaks to the vehicle owner using the system.

According to a further aspect, the invention provides a road vehicle that can be operated in an automated manner. The road vehicle comprises a system according to the invention. A first acoustic sensor is arranged in a front left area of the road vehicle. A second acoustic sensor is arranged in a front right area of the road vehicle. A third acoustic sensor is arranged in a rear left area of the road vehicle. A fourth acoustic sensor is arranged in a rear right area of the road vehicle. This arrangement with four acoustic sensors enables a 360 ° recording of ambient noise and the speech of the external road user.

According to a further aspect of the invention, the system is activated via the voice of the external road user. When the system is not active, the system is in a sleep or hibernation or standby state. The system is woken up via the language of the external road user, also known as wake up on standby.

Further refinements of the invention emerge from the subclaims, the drawing and the description of preferred exemplary embodiments.

Road vehicles are road vehicles, for example motor vehicles, for example motor vehicles with internal combustion engines and / or electric machines. Examples of road vehicles are cars, commercial vehicles, trucks or buses. According to one aspect of the invention, the road vehicle comprises technical equipment for self-driving, that is to say driverless or fully automated, autonomous driving. The ADAS, which means advanced driver assistance system, or the AD, das This means autonomous driving, Domain ECU, i.e. electronic control unit, perceives a vehicle's surroundings by means of surroundings detection sensors, derives a trajectory planning from it and determines corresponding control signals that are provided to the vehicle actuators in order to regulate the longitudinal and / or lateral guidance of the road vehicle. The acoustic sensor of the device according to the invention is an example of a surroundings detection sensor. Further surroundings detection sensors are, for example, optical sensors, for example cameras or lidar, or radar sensors. According to one aspect of the invention, the signals from the acoustic sensor are merged with signals from further surroundings detection sensors in order to localize and / or classify objects in road traffic.

Vehicle owner is a person who has general power of disposal over the road vehicle and uses it for their own account. According to one aspect of the invention, a vehicle owner is an operator of a fleet of road vehicles. In the case of a vehicle fleet, the term vehicle owner also includes a call center through which the fleet is managed.

External road users are, for example, pedestrians, passers-by, law enforcement officers or police officers.

An acoustic sensor is a sensor that detects mechanical vibrations, for example caused by airborne sound waves, and converts them into a processable signal, for example an electrical signal such as an electrical voltage. The acoustic sensor includes an analog and / or digital signal output. The forming takes place in two stages. In a first acoustic-mechanical conversion stage, the airborne sound is converted into the movement of an object according to a certain reception principle. In the second mechanical-electrical conversion stage, the movement of the object is converted into the electrical signal according to a specific converter principle. Examples of acoustic sensors are an arrangement of a magnet and an electrical coil, microphones, accelerometers, piezo sensors or strain gauges. A micro-electromechanical system, abbreviated to MEMS, comprising an arrangement of semiconductor elements that absorb vibrations, can also be used as an acoustic sensor.

According to one aspect of the invention, the acoustic sensor comprises a protective grille, an acoustically permeable membrane and a sound channel, which, taking into account airborne and / or structure-borne noise, aeroacoustics, flow and fluid dynamics, electronics and mechanics, are coordinated with one another in such a way that the acoustic sensor can also difficult environmental and flow conditions that exist in automotive applications is functional. The acoustic sensor is, for example, an acoustic sensor as in the patent application with the file number 10 2019 206 331.4 disclosed, the content of which is incorporated by reference in the present disclosure.

According to one aspect of the invention, the acoustic sensor is directed outwards relative to the body by means of the fastening arrangement in the fastened state. Air-borne sound enters the fastening arrangement essentially axially, that is to say axially or at a certain angle to the axial.

According to a further aspect of the invention, the fastening arrangement of the acoustic sensor comprises a decoupling component. By means of the decoupling component, the acoustic sensor can be installed insensitive to vibrations and is therefore particularly suitable for automotive applications. According to one aspect of the invention, the decoupling component is manufactured from a two-component material that generates an acoustically and / or vibrationally effective impedance jump. The two-component material comprises a relatively soft material with a relatively small impedance and a relatively hard material with a relatively large impedance. The soft material is arranged in front of the hard material in the direction of air flow. The decoupling component is, for example, a molded part. According to a further aspect of the invention, the decoupling component is made from vibration-damping materials of different densities, for example from mixed-cell polyurethane foams. The decoupling component has, for example, a high mechanical load capacity and / or good insulating properties.

According to one aspect of the invention, the evaluation device of the acoustic sensor is arranged on a printed circuit board. According to one aspect of the invention, a microphone is also arranged on the circuit board. The circuit board is also called a printed circuit board. According to one aspect of the invention, the circuit board comprises further components for processing the signals from the acoustic sensor. The components of the circuit board include, for example, logic modules such as ASICS or FPGAs. For example, one component implements a high-pass filter that allows airborne sound waves with frequencies greater than 300 Hz to pass. Using filter functions, spoken words and / or sentences are filtered from ambient noises detected by the acoustic sensor. According to one aspect of the invention, the filtered words and / or sentences are amplified by means of amplifier circuits on the circuit board, for example in terms of their volume amplitude. The dynamic range of a signal is limited by means of compressor functions. The components are, for example, mounted directly on the surface of the circuit board, for example soldered, and are also called surface mounted devices, abbreviated to SMD. The evaluation device or components of the circuit board are designed for Fourier transformation, beamforming and / or obtaining a cepstrum in order to determine the direction from which noises and / or speech are detected, an approximation or removal of noises and / or speech and / or intensities of noises and / or determine language. The system thus determines the direction and / or distance from which the spoken words and / or sentences are coming.

According to a further aspect of the invention, the acoustic sensor is implemented as a MEMS microphone. MEMS microphones are miniaturized microphones which, for example, are made using SMD technology for direct use on the circuit board. MEMS microphones have small dimensions and are easy to process industrially, for example MEMS microphones can be assembled in a reflow soldering process.

The evaluation device comprises one or more central processor units. According to a further aspect of the invention, the evaluation device comprises one or more graphic processor units with microarchitectures that optimize matrix and / or tensor multiplications that are carried out when processing machine learning models, in particular when processing artificial neural networks. For example, the evaluation device includes hardware accelerators for machine learning models. The machine learning models are, for example, gradient-based learning models. According to one aspect of the invention, the machine learning model is an algorithm which is implemented in a computer program. The processor units execute instructions from the computer program. The evaluation device also includes a memory. The memory can consist of one or more non-transitory computer-readable media and comprises at least a program storage area and a data storage area. The program memory area and the data memory area can comprise combinations of different types of memory, for example read-only memory, ROM = read-only memory, and random access memory, RAM = random access memory, e.g. B. dynamic RAM DRAM or synchronous DRAM SDRAM.

The terminal device is connected to a network, for example a telecommunications network, for example a 5G technology network, or by means of other types of radio technology, for example WLAN, to the Internet. According to one aspect of the invention, the terminal is a mobile terminal.

According to one embodiment, the terminal provides the spoken words and / or sentences to the vehicle owner in a telephone call. Voice communication with the vehicle owner is thus established in real time. As an alternative to a telephone call, a short text message is transmitted by means of the terminal. According to one aspect of the invention, the machine learning program is trained and executed to convert recognized words and / or sentences into texts that are transmitted by means of the short text message.

According to a further aspect of the invention, the system comprises an optical detection device. The optical detection device detects the vehicle external road user in the activated state of the system. The evaluation device authenticates the external road user based on signals from the optical detection device. The system provides the spoken words and / or sentences of the external road user as a function of the authentication.

The optical detection device is, for example, a camera or a lidar or a system comprising cameras and lidars. According to one aspect of the invention, the camera is a 3D camera in order to obtain depth information. According to one aspect of the invention, the optical detection device is an optical detection device which is arranged on the vehicle as part of a driver assistance system or a driving robot, for example as a camera of an interior and / or exterior monitoring system. For example, a badge of the external road user or their face, also called FaceID, is recorded. According to one aspect of the invention, the evaluation device executes an image object recognition algorithm, for example an artificial neural network trained in object recognition, in order to authenticate an external road user, for example as a law enforcement officer, for example as a police officer.

According to a further aspect of the invention, the evaluation device merges the signals from the acoustic sensor and the signals from the optical detection device.

The terminal provides the spoken words and / or sentences to the vehicle owner in a video call. The vehicle owner thus sees the external road user. The authentication is thus checked for plausibility.

According to a further aspect of the invention, a memory of the evaluation device stores classes of authenticated road users, their spoken words and / or sentences should be provided. This allows the vehicle owner to choose which types of external road users he would like to communicate with. For example, the vehicle owner can use the memory of the evaluation device to choose that he only wants to communicate with external road users who authenticate themselves as police officers via a badge. According to a further aspect of the invention, the time within which the authentication is to be active can be set by the vehicle owner.

According to a further aspect of the invention, the authentication can be activated and / or deactivated by the vehicle owner. In the case of the deactivated authentication, the vehicle owner can be reached at any time by any person via the vehicle by means of external speech recognition through the system according to the invention. This is particularly advantageous if a vehicle has been damaged. Even if the vehicle owner parks his vehicle for a short time in the no-parking zone, he can be reached by all road users via voice communication if authentication is deactivated.

According to a further embodiment of the invention, the interface is a user interface. The user interface is arranged on the body or on a window of the vehicle. The user interface, also called human machine interface, abbreviated to HMI, is, for example, a switch or a push button, also called a button. As an alternative or in addition, the user interface is a graphical user interface in the form of a vehicle window which is used as a touchscreen.

According to one aspect of the invention, the machine learning program comprises a convolutional neural network, a recurrent neural network or a convolutional recurrent neural network. In a convolutional neural network, a neuron of one layer is only connected to nearest neurons of the previous layer with the same set of weights for neuron connections. A recurrent neural network comprises connections from neurons of a first layer of neurons to neurons of the same and / or a second layer of neurons. Recurrent neural networks, which comprise connections from neurons of a first layer of neurons to neurons of a second layer of neurons, in simplified terms correspond to a network comprising a first network and a second network. The second network is a copy of the first network. The first network gives information to the second network. The information obtained from processing in the first network can thus be linked to a task that the second network is to carry out. For example, the information of the first network that processes image data is that people are arranged on an image and that person A should be analyzed further. The task of the second network is to identify people's faces. With the information from the first network, the second network will identify person A's face. As a further example, the first network predicts the position of an object, forwards this prediction to the second network, which also receives information about the actual position of the object at this point in time, and so on.

In the case of recurrent neural networks that comprise several copies of a first network, the problem of long-term dependencies, referred to in English as long-term dependencies, occurs. A detailed description of these long-term dependencies can be found in Gers Felix A., Schmidhuber Jürgen A. and Cummins Fred A., "Learning to Forget: Continual Prediction with LSTM", Neural Comput. , October 2000, volume 12th , Number 10 , Pp. 2451-2471. To put it simply, the problem is that between the first network and a copy of the first network that requires information from the first network to carry out its task, there are several copies of the first network, that is to say a gap. If there is a gap, recurrent neural networks are unable to link information.

This problem is solved by long-short-term memory networks, LSTMs for short. Long-short-term memory networks are a special type of recurrent neural network that avoid long-term dependencies. In simple terms, long-short-term memory networks are programmed to store information about longer gaps. A detailed discussion of long-short-term memory networks, abbreviated LSTM, can be found in S. Hochreiter, J. Schmidhuber, “Long Short-Term Memory”, Neural Computation 9 (8): 1735-1780, 1997 in Chapter 4th "Long Short-Term Memory". According to one aspect of the invention, the machine learning program LSTM comprises architecture components. This means that the machine learning program is optimized for speech recognition, in which the saving of words is important.

According to a further embodiment of the invention, the law enforcement officer contacts the vehicle owner by means of voice communication in the event that the driverless road vehicle has been parked improperly, a paid parking period has expired, the road vehicle has been damaged or there is a risk of delay. The law enforcement officer informs the vehicle owner about the condition of the Road vehicle and / or asks the vehicle owner to move the road vehicle.

According to a further aspect of the invention, the acoustic sensor and / or the evaluation device or the circuit board comprising the acoustic sensor and the evaluation device are arranged in a housing. The housing protects the circuit board and its components from mechanical and / or thermal influences. The housing comprises fastening means, for example screws, in order to fasten the housing and the device to a control device or to a road vehicle.

In summary, the vehicle owner of a road vehicle that has been illegally parked is contacted directly by the security guard by means of the system and method according to the invention. In order to prevent misuse, the law enforcement officer authenticates himself on the road vehicle, for example via a service badge or FaceID, for example by means of a camera located in the vehicle. To activate the system, a user interface, for example a button, is actuated. If the authentication was successful, a direct connection to a stored telephone of the vehicle owner is established. The law enforcement officer can request the vehicle owner to move the road vehicle immediately. There is therefore no fine. The vehicle owner can activate or deactivate the authentication himself. The system can also be activated via the voice of the external road user. The system is woken up via the language of the external road user, also known as wake up on standby.

The invention is exemplified in 1 explains an embodiment of a system according to the invention 10 shows. The system 10 includes, for example, four acoustic sensors 11 . For example, there are two acoustic sensors 11 on the two outer edges of the front bumper of a road vehicle 2 and two acoustic sensors 11 on the two outer edges of a rear bumper of the road vehicle 2 arranged. The road vehicle 2 is, for example, a car that has been parked in a no-parking zone. A vehicle owner 1 of the road vehicle 2 has the road vehicle 2 leave. The acoustic sensors 11 are each by means of fastening arrangements 12th in the body of the road vehicle 2 built-in. The acoustic sensors 11 classify and localize ambient noises, for example sirens from emergency vehicles or speech, that is, individual words and / or sentences 3 , by law enforcement officers or other road users outside the vehicle 4th , for example pedestrians.

The system also includes 10 an evaluation device 13th who have favourited a trained machine learning model 14th executes. Using the machine learning model 14th will be the words and / or sentences 3 of the external road user 4th in the by means of the acoustic sensors 11 detected ambient noise recognized, filtered and amplified. Via an end device 16 , for example in the road vehicle 2 is installed, the words and / or sentences are given to the vehicle owner via a phone call 1 forwarded.

The system 10 is via an interface 15th activated. the interface 15th is for example a user interface in the form of a button.

The system also includes an optical sensing device 17th . The optical detection device 17th is for example a camera. By means of the optical detection device 17th becomes the external road user 4th authenticated, for example as a law enforcement officer via a registration of an official badge.

List of reference symbols

1

Vehicle owner

2

Road vehicle

3

Word / sentence

4th

external road users

10

system

11

Acoustic sensor

12th

Mounting arrangement

13th

Evaluation device

14th

Machine learning model

15th

interface

16

Terminal

17th

optical detection device

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 102016006802 A1 [0002]

Claims (10)

A system (10) for voice communication with a vehicle owner (1) comprising • at least one acoustic sensor (11) comprising a fastening arrangement (12) for installing the acoustic sensor (11) in a body of a road vehicle (2), An evaluation device (13) for processing the signals of the acoustic sensor (11) comprising a machine learning program (14) which is trained to use words and / or sentences (3) spoken in the signals of the acoustic sensor (11) by road users (4) outside the vehicle ) to recognize and to filter the recognized words and / or sentences (3) from the signals of the acoustic sensor (11) when the machine learning program (14) is executed on the evaluation device (13), • an interface (15) by means of which the system (10) is activated by road users (4) external to the vehicle, and A terminal (16) which can be used as a vehicle internal terminal and which is connected to the evaluation device (13) to transmit the signal, the system (10) receiving the spoken words and / or sentences (3) of the external road user (4) in the activated state Vehicle owner (1) by means of the terminal (16) provides. System (10) according to Claim 1 wherein the terminal (16) provides the spoken words and / or sentences (3) to the vehicle owner (1) in a telephone call. System (10) according to Claim 1 or 2 , comprising an optical detection device (17) which detects the external road user (4) in the activated state of the system (10), the evaluation device (13) authenticating and authenticating the external road user (4) based on signals from the optical detection device (17) the system (10) provides the spoken words and / or sentences (3) of the external road user (4) as a function of the authentication. System (10) according to Claim 3 , wherein the evaluation device (13) merges the signals of the acoustic sensor (11) and the signals of the optical detection device (17) and the terminal device (16) provides the spoken words and / or sentences (3) to the vehicle owner (1) in a video call. System (10) according to Claim 3 or 4th , wherein a memory of the evaluation device (13) stores classes of authenticated road users whose spoken words and / or sentences (3) are to be provided. System (10) according to one of the Claims 1 to 5 wherein the interface (15) is a user interface which is arranged on the body or on a window of the vehicle (2). System (10) according to one of the Claims 1 to 4th wherein the machine learning program (14) comprises a convolutional neural network, a recurrent neural network or a convolutional recurrent neural network. Method for voice communication between a law enforcement officer (4) and a vehicle owner (1), the law enforcement officer (4) in a driverless road vehicle (2) having a system (10) according to one of the Claims 1 to 7th comprises, the system (10) activated via its interface (15), spoken words and / or sentences (3) of the security guard (4) are filtered from ambient noise by means of the acoustic sensor (11) and the evaluation device (13), the vehicle owner (1 ) is called by means of the terminal (16) and the security guard (4) speaks to the vehicle owner (1) by means of the system (10). Procedure according to Claim 8 In the event that the driverless road vehicle (2) has been parked illegally, a paid parking period has expired, the road vehicle (2) has been damaged or there is a risk of delay, the law enforcement officer (4) contacts the vehicle owner (1) by means of voice communication and informs the vehicle owner (1) of the state of the road vehicle (2) and / or requests the vehicle owner (1) to move the road vehicle (2). Automated operable road vehicle (2) comprising system (10) according to one of the Claims 1 to 7th , wherein a first acoustic sensor (11) in a front left area of the road vehicle (2), a second acoustic sensor (11) in a front right area of the road vehicle (2), a third acoustic sensor (11) in a rear left area of the road vehicle ( 2) and / or a fourth acoustic sensor (11) are arranged in a rear right area of the road vehicle (2).

Images