EP3375204B1 - Traitement de signal audio dans un véhicule - Google Patents

Traitement de signal audio dans un véhicule Download PDF

Info

Publication number
EP3375204B1
EP3375204B1 EP16788664.7A EP16788664A EP3375204B1 EP 3375204 B1 EP3375204 B1 EP 3375204B1 EP 16788664 A EP16788664 A EP 16788664A EP 3375204 B1 EP3375204 B1 EP 3375204B1
Authority
EP
European Patent Office
Prior art keywords
signal
audio signal
audio
limited
vehicle
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP16788664.7A
Other languages
German (de)
English (en)
Other versions
EP3375204A1 (fr
Inventor
David Scheler
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Volkswagen AG
Original Assignee
Volkswagen AG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Volkswagen AG filed Critical Volkswagen AG
Publication of EP3375204A1 publication Critical patent/EP3375204A1/fr
Application granted granted Critical
Publication of EP3375204B1 publication Critical patent/EP3375204B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L21/0216—Noise filtering characterised by the method used for estimating noise
    • G10L21/0232—Processing in the frequency domain
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R3/00—Circuits for transducers, loudspeakers or microphones
    • H04R3/02—Circuits for transducers, loudspeakers or microphones for preventing acoustic reaction, i.e. acoustic oscillatory feedback
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R5/00—Stereophonic arrangements
    • H04R5/02—Spatial or constructional arrangements of loudspeakers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30—Control circuits for electronic adaptation of the sound field
    • H04S7/307—Frequency adjustment, e.g. tone control
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L2021/02082—Noise filtering the noise being echo, reverberation of the speech
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L21/0216—Noise filtering characterised by the method used for estimating noise
    • G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L21/0216—Noise filtering characterised by the method used for estimating noise
    • G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
    • G10L2021/02166—Microphone arrays; Beamforming
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2430/00—Signal processing covered by H04R, not provided for in its groups
    • H04R2430/03—Synergistic effects of band splitting and sub-band processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2499/00—Aspects covered by H04R or H04S not otherwise provided for in their subgroups
    • H04R2499/10—General applications
    • H04R2499/13—Acoustic transducers and sound field adaptation in vehicles
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R5/00—Stereophonic arrangements
    • H04R5/04—Circuit arrangements, e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
    • H04S2400/05—Generation or adaptation of centre channel in multi-channel audio systems

Definitions

  • the present invention relates to a method for audio signal processing in a vehicle and a corresponding audio signal processing apparatus for a vehicle.
  • the present invention relates to audio signal processing with echo cancellation, for example for speech processing.
  • speech dialogue systems are used to assist the driver or occupants.
  • speech dialogue systems serve to control electronic devices without the need for a haptic operator operation.
  • the electronic devices may include, for example, a vehicle computer or a multimedia system of the vehicle. Spoken speech from the driver or occupant is received via a hands-free microphone and fed to speech recognition.
  • An application of microphones in the vehicle interior for e.g. Voice control, telephony or vehicle interior communication can potentially be affected by acoustic coupling of loudspeaker outputs of the vehicle sound system. This can lead to recognition errors in the case of speech recognition, to far-end echo in the case of hands-free telephony, and to feedback in the case of vehicle interior communication. Depending on the application, the consequences are disturbed communication, increased distraction or even disturbing noises and echoes.
  • the audio signals reproduced via the vehicle sound system can comprise, for example, music, traffic news, radio broadcasts, outputs of a navigation system or the (artificial) language of a speech dialogue system.
  • the disturbance of the speech recognition can lead to recognition errors which can make the dialogue inefficient and an increased distraction from the driving task can cause. This can cause dissatisfaction or annoyance to the driver or occupant.
  • a simple solution to the aforementioned problem is to mute the audio playback of, for example, a radio during the voice dialogue or telephone call in the vehicle.
  • the muting of audio playback is often perceived by vehicle users as disturbing and unnecessary.
  • important information can be missed by, for example, a navigation system.
  • the audio playback volume may be temporarily reduced.
  • the extent of the interference by the audio playback is then lower, but generally still so large that it can not be dispensed with a further cleanup of the microphone signal.
  • Couplings are also reduced by constructive-acoustic measures.
  • microphones with a suitable directional characteristic can be used, microphones and loudspeakers inside the vehicle can be suitably arranged relative to one another, or acoustic conditions in the vehicle can be suitably exploited.
  • signal processing components are used to clean up the microphone signals.
  • the signal components coupled into the microphones by the loudspeakers of the vehicle sound system are estimated and removed from the microphone signals.
  • Such methods are referred to as echo cancellation or echo cancellation.
  • transition frequency f c is on the order of 1 kHz.
  • the sum of the left and right channels is high-pass filtered to obtain the frequencies above f c as a monophonic signal.
  • the left (or right) speaker signal is the sum of the low frequencies of the left (or right) channel and high frequencies of both channels.
  • Two echo cancellers are needed: a high frequency mono echo canceller and a low frequency stereo echo canceller.
  • the US 2012/232890 A1 relates to a speech recognition system in which an n-channel system sound is converted by means of a monophonic unit into a single-channel acoustic signal.
  • the single-channel acoustic signal is output to a weighting unit and to an echo canceller.
  • the n-channel system sound is output through multiple speakers in, for example, a vehicle.
  • a microphone is used to receive voice input from a user as well as multichannel system sound output from the speakers and to the echo canceller.
  • a common type of echo cancellation is linear echo cancellation.
  • LEM impulse response Loudspeaker Enclosure Microphone
  • a linear echo cancellation algorithm adaptively estimates the LEM impulse response for each possible microphone-speaker pair. Based on the LEM impulse response, the injected loudspeaker sound components in each microphone signal are then calculated and subtracted therefrom. The rate of adaptation and effective echo cancellation are limited and generally in competition with each other.
  • the disclosed EP 1936939 A1 an echo cancellation in which the microphone signal is divided into subband signals and sub-sampled.
  • a reference audio signal is output through a speaker.
  • the reference audio signal is also sub-sampled and sub-sampled subband signals of the reference audio signal are stored.
  • echoes in the microphone subband signals are estimated and the estimated echoes are subtracted from the microphone subband signals to obtain improved microphone subband signals.
  • the multi-channel audio signal may be, for example, a stereo signal or a surround signal in the vehicle.
  • the estimation problem Due to correlations between the different audio source signals, the estimation problem is mathematically underdetermined. As a consequence, in the event of a sudden appearance of audio source signals, the effectiveness of echo cancellation may be greatly reduced. It may even occur that the LEM estimate diverges, for example when there are changes in the surround sound image. This can occur, for example, when so-called phantom sound sources appear, disappear or shift in the surround panorama.
  • an echo canceller which cooperates with a sound output device with a multi-channel audio unit.
  • the sound output device outputs output sound signals as analog signals of multiple channels through a plurality of speakers.
  • a microphone detects an outside sound and generates an input sound signal as an analog signal.
  • the outside sound echoes the output sound signals.
  • the echo canceller has an echo cancellation function for removing the echo from the input tone signal.
  • the echo canceller receives the output sound signals from the sound output device.
  • the general interfering signals may also include multichannel audio playback. This is for example in the DE 102009051508 A1 considered.
  • a microphone array is used instead of a single microphone. Through the microphone array a multi-channel speech signal is recorded, which is passed instead of a simple speech signal to an echo cancellation unit.
  • the multichannel speech signal recorded by the microphone array is post-processed before being input to the echo canceling unit in a unit downstream of the microphone array for processing the microphone signals by delayed summation of the signals. As a result, the signals of the authorized speakers are separated and all other speaker signals and noise are reduced.
  • the echo canceling unit evaluates the propagation time of the various channels of the multi-channel speech signal and removes all portions of the signal that do not emanate from the authorized speaker's location according to their propagation time.
  • using a microphone array or multiple microphones increases costs, requires more installation space, and requires powerful computational resources.
  • this object is achieved by a method for audio signal processing in a vehicle according to claim 1 and an audio signal processing apparatus for a vehicle according to claim 7.
  • the dependent claims define embodiments of the invention.
  • a method for audio signal processing in a vehicle is provided.
  • a mono audio signal is generated based on a multi-channel audio source signal.
  • the multi-channel audio source signal is, for example, a stereo signal or a surround signal which is to be output in the vehicle via a plurality of loudspeakers of the vehicle.
  • the mono audio signal is limited to a frequency range between a predetermined lower frequency and a predetermined upper frequency.
  • the mono audio signal can be limited, for example, with a bandpass filter to the frequency range between the predetermined lower frequency and the predetermined upper frequency. Limiting the mono audio signal to the frequency domain produces a limited mono audio signal.
  • the limited mono audio signal is output through the plurality of speakers in the vehicle.
  • this voice audio signal contains the limited mono audio signal output via the plurality of loudspeakers.
  • An influence of this limited mono audio signal output via the several loudspeakers on the speech audio signal received via the microphone is compensated by means of the limited mono audio signal. For example, an echo compensation can be performed, which takes into account only the mono audio signal.
  • a complex echo cancellation taking into account a multi-channel audio signal is therefore not required. Instead, only a single-channel echo cancellation is required, which can be realized with relatively low processing power.
  • the echo cancellation taking into account only one echo signal is very reliable, even if the mono audio signal is output through several different speakers, as with a mono audio signal no changes in the multi-channel sound image can occur.
  • the interfering mono audio signal can be largely or completely removed from the voice audio signal.
  • the predetermined lower frequency has a value in the range of 100 Hz to 300 Hz
  • the predetermined upper frequency has a value in the range of 4 kHz to 8 kHz.
  • a speech recognizer which is used, for example, for voice control or voice input in a vehicle, in many cases evaluates audio signals in a limited frequency range of, for example, 100 Hz to 8 kHz in order to recognize the voice input from a user. Therefore, echo cancellation is required only in this limited frequency range. Therefore, the predetermined lower frequency is preferably 100 Hz and the predetermined upper frequency is 8 kHz. As a result, an undisturbed speech signal can be provided to the speech recognizer in the limited frequency range relevant for him.
  • a plurality of limited channel-specific audio signals are generated as a function of the multi-channel audio source signal.
  • a channel-specific audio signal relates, for example, to an audio signal which is determined by the multi-channel audio signal source specifically for a loudspeaker associated with the respective channel. For a stereo source signal, this may include, for example, an audio signal for the right speaker or an audio signal for the left speaker.
  • a respective limited channel-specific audio signal of the plurality of limited channel-specific audio signals is therefore assigned to a respective audio signal of the multi-channel audio source signal.
  • a respective limited channel-specific audio signal is limited to a frequency range which only includes frequencies below the predetermined lower frequency and frequencies above the predetermined upper frequency.
  • a respective limited channel-specific audio signal is formed by a corresponding frequency limitation from the respectively assigned audio signal of the multi-channel audio source signal.
  • the audio signals of the multi-channel audio signal are respectively limited or filtered so that they only include frequencies below the predetermined lower frequency and / or frequencies above the predetermined upper frequency.
  • the plurality of limited channel-specific audio signals are output through the plurality of speakers in the vehicle, so that the effect of multi-channel audio reproduction such as stereo reproduction or surround reproduction can be achieved.
  • an audio reproduction in the vehicle is modified so that in the frequency range between the predetermined lower frequency and the predetermined upper frequency, the multi-channel audio source signal is reproduced in one channel (mono) and in the remaining frequency range multi-channel.
  • the mono audio signal and the plurality of limited channel-specific audio signals may be generated from the multi-channel audio source signal according to the following embodiment.
  • the multi-channel audio source signal is divided into a center signal component which is the same on all channels and a respective side signal component per audio channel of the multi-channel audio source signal.
  • the limited mono audio signal is generated from the center signal component and the plurality of limited channel-specific audio signals are generated from the respective side signal components.
  • the mid-signal component for example, can be used directly as a mono audio signal or suitably scaled used as a mono audio signal.
  • the side signal components may be used directly as the limited channel specific audio signals or in suitably scaled form.
  • the center signal component in the case of a stereo signal, can be formed, for example, from the sum of the right and left audio source signals.
  • the side signal components may be coded together in a difference signal from the difference between the right and left audio source signals and further processed.
  • the center signal component and the side signal components can be generated and processed in a simple manner.
  • the center signal component is formed by averaging respective samples of the audio channels of the multi-channel audio source signal.
  • the respective side signal components are formed by subtracting the center signal component from the respective audio signals of the multi-channel audio source signal. This generation of the center signal component and the side signal components is possible for audio source signals with an arbitrary number of channels.
  • an implementation in, for example, a digital signal processor can be realized in a simple manner.
  • the speech audio signal received via the microphone is limited to a frequency range between the predetermined lower frequency and the predetermined upper frequency.
  • the echo cancellation is applied to the thus limited voice audio signal using the limited mono audio signal.
  • the influence of the limited mono audio signal outputted through the plural speakers to the limited voice audio signal is compensated.
  • the speech recognizer operates generally only in the frequency range between the predetermined lower frequency and the predetermined upper frequency, echo cancellation in a speech audio signal limited thereto is sufficient.
  • spurious signals outside this frequency range are already eliminated before the echo cancellation and have therefore, does not affect echo cancellation and speech recognition, which makes both echo cancellation and speech recognition more reliable.
  • the reproduction of an audio signal is more important to some occupants of the vehicle than to others.
  • audio outputs of a navigation system are more important to the driver than to the remaining occupants
  • audio outputs of a video displayed in the rear of the vehicle are more important for rear passengers than for the driver and front passenger.
  • a plurality of weighting factors associated with the respective loudspeakers may be generated in response to the multi-channel audio source signal.
  • the limited mono audio signal is weighted for each loudspeaker with the weighting factor assigned to the respective loudspeaker. Thereby, a center of gravity of the audio output in the vehicle can be appropriately shifted.
  • the weighted output will not affect the quality of the echo cancellation.
  • the echo cancellation may adjust to the new weight in a relatively short time, for example within a few seconds or minutes.
  • the following weighting may be used.
  • the speaker in the driver's area may output 70% of the mono audio signal, and the remaining three speakers may output only 10% of the mono audio signal, for example.
  • an audio signal processing apparatus for a vehicle.
  • the audio signal processing apparatus is capable of generating a mono audio signal based on a multi-channel audio source signal.
  • the audio signal processing device may have, for example, a summation device.
  • the audio signal processing apparatus is further capable of limiting the mono audio signal to a frequency range between a predetermined lower frequency and a predetermined upper frequency. This can be realized for example with a bandpass filter.
  • the predetermined lower frequency has a value in the range of 100 Hz to 300 Hz
  • the predetermined upper frequency has a value in the range of 4 kHz to 8 kHz.
  • the limited mono audio signal is output through several speakers in the vehicle.
  • the limited mono audio signal is output to a compensation device, for example to a Echo canceller.
  • the compensation device serves to compensate for an influence of the limited mono audio signal output via the plurality of loudspeakers on a voice audio signal received in the vehicle via a microphone by means of the limited mono audio signal.
  • the audio signal processing device is therefore suitable for carrying out the method described above and its embodiments and therefore also comprises the advantages described above.
  • FIG. 1 First, the environment of an audio signal processing device 15 according to the invention in a vehicle 10 will be described. In FIG. 2 Details of the audio signal processing device 15 will be described in conjunction with other components of the vehicle 10. FIG. 3 Finally, Fig. 12 schematically shows the operation of the audio signal processing apparatus 15. Like reference numerals in the figures refer to the same or similar components.
  • FIG. 1 shows a vehicle 10 in a plan view.
  • the vehicle 10 includes a voice recognition system 11.
  • voice commands or instructions from occupants of the vehicle 10 may be captured, processed, and executed.
  • configuration settings of the vehicle 10 or a multimedia system of the vehicle 10 may be changed via appropriate instructions.
  • an audio signal source such as CD or radio
  • a specific radio station can be selected or a title of a CD.
  • a telephone connection to a desired participants are set up or a navigation destination can be set in a navigation system of the vehicle 10.
  • corresponding commands or instructions are received by a driver 12 of the vehicle 10 via a microphone 13.
  • a spoken command of the driver 12 is forwarded by the microphone 13 as a speech audio signal to an audio signal processing device 15.
  • the operation of the audio signal processing device 15 will be described later with reference to FIG FIG. 2 will be described in detail.
  • the processed speech audio signal is supplied to the speech recognition system 11.
  • the speech recognition system 11 evaluates the speech audio signal and detects contained commands and instructions and executes them.
  • the speech recognition system can be coupled with a so-called dialogue system, which can lead a dialogue with the driver via questions and answers.
  • the vehicle 10 further includes an audio signal source 14.
  • the audio signal source 14 may include, for example, a broadcast receiver, a media player such as a CD player or an MP3 player, or a navigation system of the vehicle 10.
  • the audio signal source 14 outputs a multi-channel audio source signal.
  • the multi-channel audio source signal is supplied to and processed by the audio signal processing device 15, as described below with reference to FIG FIG. 2 will be described.
  • the processed multi-channel audio source signal is output from the audio signal processing device 15 to an amplifier 16.
  • the amplifier 16 amplifies the individual signals of the processed multi-channel audio source signal so that they can be reproduced via speakers 17-20 in an interior of the vehicle 10.
  • the vehicle 10 includes four loudspeakers 17-20.
  • the vehicle 10 may include any number of speakers, for example two, three or more than four.
  • the FIG. 1 are the speakers 17-20 assigned to the seats of the vehicle 10.
  • the loudspeaker 17 is associated with a driver's seat of the driver 12, the loudspeaker 18 with a passenger seat, the loudspeaker 19 with a rear right seat and the loudspeaker 20 with a rear left seat.
  • the driver 12 may voice instructions or commands to the speech recognition system 11. This will be in FIG. 1 represented by the dashed arrow between the driver 12 and the microphone 13. While the driver 12 is issuing commands and instructions, multi-channel audio source signals may be supplied from the audio signal source 14 be output through the speakers 17-20. The outputs of the speakers 17-20 also reach the microphone 13, as indicated by the corresponding dashed arrows between the speakers 17-20 and the microphone 13 in the FIG. 1 is shown. However, the outputs from the speakers 17-20 may interfere with speech intelligibility, such that the speech recognition system 11 may not or insufficiently recognize the commands and instructions from the driver 12.
  • FIG. 2 shows details of the audio signal processing device 15 and the speech recognition system 11, which help to reduce or compensate for the influence of the outputs from the loudspeakers 17-20 on the speech signal of the driver 12.
  • the audio signal source 14 in the example of FIG. 2 however, it is clear that the audio signal processing apparatus 15 described below can process any number of channels of a multi-channel audio signal source in the same way.
  • FIG. 2 shown components of the audio signal processing device 15 described.
  • components of the audio signal processing device 15 may not necessarily be designed as concrete components or assemblies, but may be partially or wholly reproduced programmatically and implemented by a suitable controller, such as a microprocessor or a digital signal processor.
  • the audio signal processing device 15 includes inputs via which the multi-channel audio source signal is received by the audio signal source 14.
  • a two-channel stereo audio source signal includes, for example, a left channel L and a right channel R, which are supplied to the audio signal processing device 15.
  • a middle signal component M and for each channel a side signal component S are generated from the two- or multi-channel audio source signal.
  • a common side signal component can be formed as the difference between the left channel L and the right channel R. Since, regardless of the number of side signal components, all side signal components are treated the same in the following FIG. 2 only one path for the side signal components S is shown. This one path can therefore comprise only one side signal component in the case of a stereo signal or several side signal components in the multi-channel case.
  • the middle signal component M may comprise, for example, a sum signal from all the supplied channels.
  • a respective side signal component S may, for example, comprise a difference signal between the respective audio signal of the respective channel of the multi-channel audio source signal and the mid-signal component.
  • the audio signal processing device 15 further comprises a first bandpass filter 23 and a notch filter or notch filter 22.
  • the first bandpass filter 23 has a predetermined lower frequency and a predetermined upper frequency.
  • the first bandpass filter 23 essentially passes only signals having a frequency between the predetermined lower frequency and the predetermined upper frequency. Signals with a frequency below the predetermined lower frequency and signals with a frequency above the predetermined upper frequency are substantially suppressed or at least greatly attenuated.
  • the attenuation may be, for example, 70 dB or more, and in a digital embodiment of the first bandpass filter, the signal above the predetermined upper frequency and below the predetermined lower frequency can be completely suppressed.
  • the notch filter 22 has a frequency response which is substantially inverse to the frequency response of the first bandpass filter 23. That is, the notch filter 22 essentially only passes signals having a frequency below the predetermined lower frequency or above the predetermined upper frequency.
  • the lower predetermined frequency may be 100 Hz
  • the upper predetermined frequency may be 8 kHz, for example.
  • the lower predetermined frequency may be selected in a range of 100 Hz to 300 Hz
  • the upper predetermined frequency may be selected in a range of 4 kHz to 8 kHz. The larger the frequency range between the lower predetermined frequency and the upper predetermined frequency is selected, the more reliable the speech recognition function.
  • reproduction of a multi-channel audio source signal is more affected the greater the frequency range between the lower predetermined frequency and the upper predetermined frequency is selected.
  • a corresponding notch filter 22 having the lower predetermined frequency and the upper predetermined frequency is to be provided for each of these plurality of side signal components.
  • a filtered or frequency-limited mid-signal component Mb is generated.
  • filtered or frequency-limited side signal components Sb are generated.
  • the filtered center signal component Mb and the filtered side signal component Sb are applied to a second signal converter 24, which generates filtered audio signals for the individual channels.
  • the filtered audio signal for a respective single channel may be formed, for example, by summing the filtered center signal component Mb and the corresponding filtered channel-specific side signal component Sb.
  • the filtered audio signals Lb, Rb are output from the audio signal processing device 15 and supplied to the amplifier 16 channel by channel.
  • the audio signal processing device 15 further comprises a second bandpass filter 26.
  • the second bandpass filter 26 has the same filter characteristic as the first bandpass filter 23.
  • the second bandpass filter 26 is coupled on the input side to the microphone 13 and on the output side with an echo canceller 25 of the speech recognition system 11.
  • the echo canceller 25 of the speech recognition system 11, the filtered center signal component Mb is further supplied. Based on the filtered center signal component Mb, the echo canceller 25 performs echo cancellation on the filtered voice signal from the microphone 13.
  • the speech signal processed by the echo canceler 25 is supplied to a speech recognizer 27 of the speech recognition system 11.
  • the audio signal processing device 15 comprises a weighting device 28, which is coupled to the multi-channel audio source signal and / or the audio signal source 14. Based on information from the multi-channel audio source signal or information from the audio signal source 14, the weighting device 28 provides weighting factors with which the filtered audio signals are weighted prior to being output from the second signal converter 24.
  • FIG. 3 shows a method 30 with method steps 31-37, which are carried out by the audio signal processing device 15 in conjunction with the speech recognition system 11.
  • illustrated processing steps can be performed with electronic tools, which include, for example, analog or digital circuits and processing devices.
  • processing devices For example, they may include microprocessors or digital signal processors.
  • the entire functionality of the audio signal processing device 15 can be integrated into, for example, an existing electronic device, for example a digital signal processor of the speech recognition system 11.
  • a multi-channel audio source signal for example a stereo signal or a surround signal
  • a frequency limited mono audio signal and frequency limited channel specific audio signals are generated by the first signal converter 21 and the filters 22 and 23.
  • the frequency-limited center signal component Mb described above may be, for example, the frequency-limited mono audio signal.
  • the frequency-limited side signal components Sb described above may be, for example, the frequency-limited channel-specific audio signals.
  • the frequency-limited mono audio signal and the frequency-limited channel-specific audio signals can also be formed in any other way from the multi-channel audio source signal, for example in a digital signal processor.
  • the limited mono audio signal is output through all the loudspeakers 17-20 and the limited channel specific audio signals are outputted over the loudspeaker associated with the respective channel.
  • the mono audio signal is limited to a relevant frequency range for speech recognition, for example to a frequency range of 100 Hz to 8 kHz.
  • the channel-specific audio signals are limited to a frequency range outside the relevant frequency range for speech recognition, that is, for example, to frequencies below 100 Hz and above 8 kHz. Due to the reduction of the multichannel nature of the audio reproduction within the frequency range relevant to the speech recognizer 27, only the single-channel mono audio signal is present as the disturbing signal for speech recognition. For the vehicle occupant, however, a feeling of spatial perception remains in the sound perception since the multichannel nature is retained for frequencies outside the range relevant for speech recognition.
  • the weighting device 28 may determine an audio centroid for the multi-channel audio source signals or the current signal source based on the information supplied thereto, and the limited mono audio signal corresponding to that audio centroid on the audio source Distribute audio channels. For example, if a voice output of a navigation system represents the multi-channel audio signal source, the limited mono audio signal, for example, for the speaker 17 may be weighted more heavily than for the speakers 18-20, since this information is more relevant to the driver 12 than for the rest of the vehicle occupants.
  • the weighting device 28 may consider further information of the vehicle 10, for example a current seat occupancy in the vehicle.
  • a speech audio signal is received via the microphone 13 in step 35.
  • the received speech audio signal is frequency limited by the second bandpass filter 26.
  • the echo canceller 25 is supplied with the limited mono audio signal and the limited voice audio signal.
  • the echo canceller 25 performs echo cancellation in the voice audio signal using the mono audio signal. Since both the speech audio signal and the mono audio signal are limited to the frequency range relevant to the speech recognition (eg 100Hz-8kHz), the echo cancellation can also be limited to this limited frequency range, whereby less noise occurs and the echo canceller 25 can be made simpler or less Computing power required.
  • single-channel echo cancellation requires only a single audio reference signal, namely the mono audio signal, and only needs to estimate an acoustic impulse response. As a result, system resources are saved in the echo cancellation, which are available for example for the speech recognizer 27.
  • the thus adjusted speech audio signal is supplied to the speech recognizer 27 and processed there to extract corresponding commands and instructions from the spoken speech.

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Acoustics & Sound (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Quality & Reliability (AREA)
  • Computational Linguistics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Otolaryngology (AREA)
  • Fittings On The Vehicle Exterior For Carrying Loads, And Devices For Holding Or Mounting Articles (AREA)
  • Circuit For Audible Band Transducer (AREA)
  • Stereophonic System (AREA)

Claims (8)

  1. Procédé (30) pour le traitement de signaux audio dans un véhicule (10), comprenant :
    - la génération d'un signal audio mono sur la base d'un signal de source audio multi-canal,
    - la limitation du signal audio mono à une plage de fréquences entre une fréquence inférieure prédéterminée et une fréquence supérieure prédéterminée,
    - l'émission du signal audio mono limité par l'intermédiaire de plusieurs haut-parleurs (17-20) dans le véhicule (10) et
    - la compensation d'une influence du signal audio mono limité émis par l'intermédiaire des plusieurs haut-parleurs sur un signal audio vocal reçu dans le véhicule (10) par l'intermédiaire d'un microphone (13), au moyen du signal audio mono,
    caractérisé en ce que la fréquence inférieure prédéterminée présente une valeur de l'ordre de 100 kHz à 300 kHz et la fréquence supérieure prédéterminée présente une valeur de l'ordre de 4 kHz à 8 kHz.
  2. Procédé selon la revendication 1, caractérisé en outre par :
    - la génération de plusieurs signaux audio limités spécifiques aux canaux en fonction du signal audio source multicanal de façon à ce qu'un signal audio limité spécifique à un canal parmi les plusieurs signaux audio limités spécifiques aux canaux corresponde à un signal audio correspondant du signal audio source multicanal et soit limité à une plage de fréquences en dessous de la fréquence inférieure prédéterminée et/ou au-dessus de la fréquence supérieure prédéterminée et
    - l'émission des plusieurs signaux audio spécifiques aux canaux par l'intermédiaire des plusieurs haut-parleurs (17-20) dans le véhicule (10).
  3. Procédé selon la revendication 2, caractérisé en ce que le signal audio source multicanal est réparti en une partie de signal centrale identique sur tous les canaux et une partie de signal latérale par canal audio du signal audio source multicanal et la partie de signal centrale correspondante est utilisée pour la génération des plusieurs signaux audio limités spécifiques aux canaux.
  4. Procédé selon la revendication 3, caractérisé en ce que le signal central est formée par le calcul d'une moyenne des échantillons des canaux audio du signal audio source multicanal et les parties de signaux latérales correspondantes sont calculées par soustraction du signal central des signaux audio correspondants du signal audio source multicanal.
  5. Procédé selon l'une des revendications précédentes, caractérisé en ce que le signal audio vocal reçue par l'intermédiaire du microphone (13) est limité à une plage de fréquences entre la fréquence inférieure prédéterminée et la fréquence supérieure prédéterminée et l'influence du signal audio mono limité émis par l'intermédiaire des plusieurs haut-parleurs (17-20) sur le signal audio vocal limité est compensée.
  6. Procédé selon l'une des revendications précédentes, caractérisé en outre par :
    - la génération de plusieurs facteurs de pondération correspondant aux haut-parleurs (17-20) concernés en fonction du signal audio source multicanal et
    - l'émission d'un signal audio mono limité pondéré par le signal audio source multicanal correspondant au haut-parleur concerné par l'intermédiaire du haut-parleur (17-20) concerné.
  7. Dispositif de traitement de signaux audio pour un véhicule (10) qui est conçu :
    - pour générer un signal audio mono sur la base d'un signal audio source multicanal,
    - pour limiter le signal audio mono à une plage de fréquences entre une fréquence inférieure prédéterminée et une fréquence supérieure prédéterminée,
    - pour émettre le signal audio mono limité par l'intermédiaire de plusieurs haut-parleurs (17-20) dans le véhicule (10) et
    - pour envoyer le signal audio mono limité à un dispositif de compensation (25) afin de compenser une influence du signal audio mono limité émis par l'intermédiaire des plusieurs haut-parleurs (17-20) sur un signal audio vocal reçu par l'intermédiaire d'un microphone (13) au moyen du signal audio mono limité,
    caractérisé en ce que la fréquence inférieure prédéterminée présente une valeur de l'ordre de 100 kHz à 300 kHz et la fréquence inférieure prédéterminée présente une valeur de l'ordre de 4kHz à 8 kHz.
  8. Dispositif de traitement de signaux audio selon la revendication 7, caractérisé en ce que le dispositif de traitement de signaux audio (15) est conçu pour l'exécution du procédé selon l'une des revendications 1 à 6.
EP16788664.7A 2015-11-10 2016-10-26 Traitement de signal audio dans un véhicule Active EP3375204B1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102015222105.9A DE102015222105A1 (de) 2015-11-10 2015-11-10 Audiosignalverarbeitung in einem Fahrzeug
PCT/EP2016/075831 WO2017080830A1 (fr) 2015-11-10 2016-10-26 Traitement de signal audio dans un véhicule

Publications (2)

Publication Number Publication Date
EP3375204A1 EP3375204A1 (fr) 2018-09-19
EP3375204B1 true EP3375204B1 (fr) 2019-06-19

Family

ID=57218876

Family Applications (1)

Application Number Title Priority Date Filing Date
EP16788664.7A Active EP3375204B1 (fr) 2015-11-10 2016-10-26 Traitement de signal audio dans un véhicule

Country Status (5)

Country Link
US (1) US10339951B2 (fr)
EP (1) EP3375204B1 (fr)
CN (1) CN108353229B (fr)
DE (1) DE102015222105A1 (fr)
WO (1) WO2017080830A1 (fr)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102015222105A1 (de) 2015-11-10 2017-05-11 Volkswagen Aktiengesellschaft Audiosignalverarbeitung in einem Fahrzeug
DE102018007433A1 (de) 2018-09-20 2019-02-28 Daimler Ag Audiosignalverarbeitungssystem für ein Fahrzeug
CN110246517B (zh) * 2019-07-08 2021-07-13 广州小鹏汽车科技有限公司 一种电台音乐识别方法、车载系统和车辆
JP7383942B2 (ja) * 2019-09-06 2023-11-21 ヤマハ株式会社 車載音響システムおよび車両
CN111739552A (zh) * 2020-08-28 2020-10-02 南京芯驰半导体科技有限公司 一种麦克风阵列波束成形的方法及系统
CN112309416B (zh) * 2020-10-20 2022-11-29 中国第一汽车股份有限公司 车载语音回音消除方法、系统、车辆和存储介质
CN113096681B (zh) * 2021-04-08 2022-06-28 海信视像科技股份有限公司 显示设备、多声道回声消除电路及多声道回声消除方法

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5828756A (en) * 1994-11-22 1998-10-27 Lucent Technologies Inc. Stereophonic acoustic echo cancellation using non-linear transformations
DE19533541C1 (de) * 1995-09-11 1997-03-27 Daimler Benz Aerospace Ag Verfahren zur automatischen Steuerung eines oder mehrerer Geräte durch Sprachkommandos oder per Sprachdialog im Echtzeitbetrieb und Vorrichtung zum Ausführen des Verfahrens
US7660425B1 (en) * 1999-05-25 2010-02-09 British Telecommunications Plc Acoustic echo cancellation
US6665645B1 (en) * 1999-07-28 2003-12-16 Matsushita Electric Industrial Co., Ltd. Speech recognition apparatus for AV equipment
JP4554044B2 (ja) 1999-07-28 2010-09-29 パナソニック株式会社 Ａｖ機器用音声認識装置
US20050213747A1 (en) 2003-10-07 2005-09-29 Vtel Products, Inc. Hybrid monaural and multichannel audio for conferencing
NO328256B1 (no) 2004-12-29 2010-01-18 Tandberg Telecom As Audiosystem
US8594320B2 (en) * 2005-04-19 2013-11-26 (Epfl) Ecole Polytechnique Federale De Lausanne Hybrid echo and noise suppression method and device in a multi-channel audio signal
JP4896449B2 (ja) * 2005-06-29 2012-03-14 株式会社東芝 音響信号処理方法、装置及びプログラム
EP1936939B1 (fr) 2006-12-18 2011-08-24 Harman Becker Automotive Systems GmbH Annulation d'écho de faible complexité
JP4916394B2 (ja) 2007-07-03 2012-04-11 富士通株式会社 エコー抑圧装置、エコー抑圧方法及びコンピュータプログラム
DE102009051508B4 (de) 2009-10-30 2020-12-03 Continental Automotive Gmbh Vorrichtung, System und Verfahren zur Sprachdialogaktivierung und -führung
US8219394B2 (en) * 2010-01-20 2012-07-10 Microsoft Corporation Adaptive ambient sound suppression and speech tracking
EP2466864B1 (fr) 2010-12-14 2019-02-27 Deutsche Telekom AG Décorrélation transparente des signaux de haut-parleurs dans des compensateurs d'écho à plusieurs canaux
JP5649488B2 (ja) * 2011-03-11 2015-01-07 株式会社東芝 音声判別装置、音声判別方法および音声判別プログラム
GB2510331A (en) * 2012-12-21 2014-08-06 Microsoft Corp Echo suppression in an audio signal
CN104956436B (zh) 2012-12-28 2018-05-29 株式会社索思未来 带有语音识别功能的设备以及语音识别方法
US9589573B2 (en) * 2013-07-12 2017-03-07 Cirrus Logic International Semiconductor Ltd. Wind noise reduction
DE102015222105A1 (de) 2015-11-10 2017-05-11 Volkswagen Aktiengesellschaft Audiosignalverarbeitung in einem Fahrzeug

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
WO2017080830A1 (fr) 2017-05-18
CN108353229B (zh) 2020-10-23
US20180358031A1 (en) 2018-12-13
US10339951B2 (en) 2019-07-02
DE102015222105A1 (de) 2017-05-11
CN108353229A (zh) 2018-07-31
EP3375204A1 (fr) 2018-09-19

Similar Documents

Publication Publication Date Title
EP3375204B1 (fr) Traitement de signal audio dans un véhicule
EP1595427B1 (fr) Procede et dispositif de separation de signaux sonores
DE69738288T2 (de) Einrichtung zur unterdrückung einer störenden komponente eines eingangssignals
DE10351793B4 (de) Adaptive Filtervorrichtung und Verfahren zum Verarbeiten eines akustischen Eingangssignals
DE69827731T2 (de) Anordnung zur aktiven Unterdrückung von akustischem Echo und Geräusch
DE3925589C2 (de) Verfahren und Anordnung zur Störbefreiung von Sprachsignalen
DE102018127071B3 (de) Audiosignalverarbeitung mit akustischer Echounterdrückung
DE10018666A1 (de) Vorrichtung und Verfahren zum geräuschabhängigen Anpassen eines akustischen Nutzsignals
DE60004888T2 (de) Sprachsverteilungssystem
DE102006027673A1 (de) Signaltrenner, Verfahren zum Bestimmen von Ausgangssignalen basierend auf Mikrophonsignalen und Computerprogramm
EP1251493A2 (fr) Procédé pour la réduction du bruit avec fréquence parasite auto-adaptative
EP1771034A2 (fr) Calibration d'un microphone dans un formeur de faisceaux-RGSC
DE102014214052A1 (de) Virtuelle Verdeckungsmethoden
DE102010007842A1 (de) Geräuschunterdrückungsvorrichtung, Geräuschunterdrückungsverfahren und Aufzeichnungsmedium
EP1189419B1 (fr) Procede et appareil pour eliminer l'interference d'un haut-parleur sur de signaux de microphone
DE102013011761A1 (de) Kraftfahrzeug mit einer Freisprecheinrichtung und Verfahren zur Erzeugung eines Frequenzganges für Freisprecheinrichtungen
DE112016004394T5 (de) Heulunterdrückungsvorrichtung
DE102018117558A1 (de) Adaptives nachfiltern
DE102014210760B4 (de) Betrieb einer Kommunikationsanlage
DE102019107173A1 (de) Verfahren und Vorrichtung zum Erzeugen und Ausgeben eines Audiosignals zum Erweitern des Höreindrucks bei Live-Veranstaltungen
DE10025655B4 (de) Verfahren zum Entfernen einer unerwünschten Komponente aus einem Signal und System zum Unterscheiden zwischen unerwünschten und erwünschten Signalkomponenten
EP1718113B1 (fr) Dispositif de prothèse auditive avec compensation du signal de rétroaction acoustique et électromagnetique
WO2007036443A1 (fr) Procede de reduction active du bruit et dispositif pour la mise en oeuvre dudit procede
DE102014214699A1 (de) Wiedergabe tieffrequenter Audiosignale durch Körperschall
DE69731133T2 (de) Übertragungssystem für korrelierte Signale

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20180611

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 502016005192

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: H04R0003020000

Ipc: G10L0021020800

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
RIC1 Information provided on ipc code assigned before grant

Ipc: H04S 7/00 20060101ALI20190215BHEP

Ipc: G10L 21/0208 20130101AFI20190215BHEP

Ipc: H04R 3/02 20060101ALI20190215BHEP

Ipc: G10L 21/0216 20130101ALN20190215BHEP

Ipc: H04R 5/04 20060101ALN20190215BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20190404

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

Free format text: NOT ENGLISH

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Free format text: LANGUAGE OF EP DOCUMENT: GERMAN

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1146511

Country of ref document: AT

Kind code of ref document: T

Effective date: 20190715

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 502016005192

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20190619

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190919

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190919

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190920

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191021

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191019

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200224

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 502016005192

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG2D Information on lapse in contracting state deleted

Ref country code: IS

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20191031

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20191026

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20191031

26N No opposition filed

Effective date: 20200603

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20191031

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20191031

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20191026

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20161026

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190619

REG Reference to a national code

Ref country code: AT

Ref legal event code: MM01

Ref document number: 1146511

Country of ref document: AT

Kind code of ref document: T

Effective date: 20211026

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20211026

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230523

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20231024

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231026

Year of fee payment: 8

Ref country code: DE

Payment date: 20231031

Year of fee payment: 8