EP3148217B1 - Verfahren zum betrieb eines binauralen hörsystems - Google Patents

Verfahren zum betrieb eines binauralen hörsystems Download PDF

Info

Publication number
EP3148217B1
EP3148217B1 EP16172902.5A EP16172902A EP3148217B1 EP 3148217 B1 EP3148217 B1 EP 3148217B1 EP 16172902 A EP16172902 A EP 16172902A EP 3148217 B1 EP3148217 B1 EP 3148217B1
Authority
EP
European Patent Office
Prior art keywords
signal
binaural
common gain
frequency band
hearing aid
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP16172902.5A
Other languages
English (en)
French (fr)
Other versions
EP3148217A1 (de
Inventor
Homayoun KAMKAR-PARSI
Henning Puder
Eghart Fischer
Martin Bouchard
Hala AS´AD
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sivantos Pte Ltd
Original Assignee
Sivantos Pte Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sivantos Pte Ltd filed Critical Sivantos Pte Ltd
Publication of EP3148217A1 publication Critical patent/EP3148217A1/de
Application granted granted Critical
Publication of EP3148217B1 publication Critical patent/EP3148217B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R25/00—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception
    • H04R25/40—Arrangements for obtaining a desired directivity characteristic
    • H04R25/407—Circuits for combining signals of a plurality of transducers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R25/00—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception
    • H04R25/43—Electronic input selection or mixing based on input signal analysis, e.g. mixing or selection between microphone and telecoil or between microphones with different directivity characteristics
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L21/0216—Noise filtering characterised by the method used for estimating noise
    • G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
    • G10L2021/02166—Microphone arrays; Beamforming
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2225/00—Details of deaf aids covered by H04R25/00, not provided for in any of its subgroups
    • H04R2225/43—Signal processing in hearing aids to enhance the speech intelligibility
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2410/00—Microphones
    • H04R2410/01—Noise reduction using microphones having different directional characteristics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2430/00—Signal processing covered by H04R, not provided for in its groups
    • H04R2430/20—Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R25/00—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception
    • H04R25/55—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception using an external connection, either wireless or wired
    • H04R25/552—Binaural

Definitions

  • the invention relates to a method for operating a binaural hearing system, said binaural hearing system comprising a first hearing aid and a second hearing aid, wherein the first hearing aid generates a first reference signal from a sound signal by a first microphone, wherein the second hearing aid generates a second reference signal from a sound signal by a second microphone, wherein the first reference signal and the second reference signal are both used to derive a first binaural beamformer signal.
  • the invention further relates to a binaural hearing system, comprising a first hearing aid and a second hearing aid, said binaural hearing system being configured to perform such a method.
  • binaural beamformers can provide noise reduction and preserve efficiently the binaural cues of the target speaker. Binaural cues enclosure all the acoustical information available to both ears of a listener for localizing a sound source. Now for an application in a binaural beamformer in which noise reduction is performed via the beamforming, the binaural cues of the target source are typically preserved, as the beamforming enhances sound from this direction. However, the typical sound environment does also comprise residual noise, which is to be reduced by the noise reduction, so that the binaural cues of the residual noise may be distorted.
  • this may happen independently of whether the residual noise of the sound environment being a directional noise source or a superposition of few directional noise sources, or diffuse background noise.
  • the distortion of the binaural cues of the residual noise causes a negative impact on the perception of the resulting acoustic scene.
  • US 2004/0252852 A1 teaches a binaural hearing system in which bandwise attenuation factors for the left ear signal and the right ear signal are determined, respectively, from the respective noise powers of said signals.
  • the attenuation factors control the mixing ratio of a binaural beamforming signal to be output to the user of the hearing system.
  • Each local signal may be added to the binaural beamforming signal according to an individually user-tailored coefficient for a better spatial perception.
  • the method shall preferably achieve said object with no restrictions on the acoustic environment or on a signal-to-noise-ratio (SNR).
  • the object is achieved by a method for operating a binaural hearing system, said binaural hearing system comprising a first hearing aid and a second hearing aid, wherein the first hearing aid generates a first reference signal from a sound signal by a first microphone, wherein the second hearing aid generates a second reference signal from a sound signal by a second microphone, wherein the first reference signal and the second reference signal are both used to derive a first binaural beamformer signal and a second binaural beamformer signal.
  • the first reference signal and the second reference signal are used to determine a common gain, and from the first reference signal and the common gain a first common gain signal is derived, wherein at least for a number of frequency bands the first binaural beamformer signal is compared with the first common gain signal for classification with respect to a noise reduction, and the first binaural beamformer signal and the first common gain signal are mixed in dependence of said classification in order to obtain a first output signal.
  • first microphone or a second microphone, respectively, shall comprise any type of sound transducer which is set up to and capable to receive an acoustical wave pattern and to transduce this acoustical wave pattern into an electrical signal.
  • first binaural beamformer signal in particular shall comprise a signal with non-trivial spatial sensitivity characteristics.
  • the binaural beamformer signal may in particular show a varying signal level for the probe reference sound source varying its angular position with respect to the assembly of the first reference microphone and the second reference microphone.
  • the first reference signal and the second reference signal in particular may be combined as linear combinations with different gain factors and possibly a delay between the two mentioned signals.
  • the spatial characteristics of the first binaural beamformer signal may vary over different frequency bands of the binaural hearing system.
  • the number of frequency bands for which the first binaural beamformer signal and the first common gain signal are compared and mixed in order to obtain a first output signal may depend on the implemented frequency decomposition given by a particular filtering process which is applied to the first reference signal and to the second reference signal, preferably in the same manner.
  • the total number and mutual overlap of frequency bands may depend on the particular decomposition or filtering process employed.
  • first common gain signal shall in particular comprise a signal whose spatial sensitivity characteristics are given only by the mechanical properties of the first reference microphone, or of the second microphone, respectively.
  • first common gain signal may in particular show the same variations in its signal level for the probe reference sound generator varying its angular position with respect to the first reference microphone as the first reference signal is showing when varying the angular position of the probe reference sound source.
  • the common gain for a given frequency band may be derived as a scalar factor using the first reference signal and the second reference signal, and the scalar factor may be directly applied to the first reference signal in order to obtain the first common gain signal in said frequency band.
  • the first common gain signal and the second common gain signal each may show an approximately omnidirectional sensitivity characteristic.
  • a strong noise reduction - in terms of the target speaker SNR improvement - can be achieved by a highly directional beamforming, i.e., the spatial sensitivity characteristics of the beamformer signals are narrowly directed towards the target speakers location. This means that a sound signal whose source is located spaced apart from the target speaker is increasingly suppressed by the spatial sensitivity characteristics with increasing angular distance to the target speaker. This procedure achieves a good SNR for the target speakers' speech vs.
  • the sound signals of intervening non-target speakers are not only attenuated as a result of the noise reduction.
  • the binaural cues of these non-target sound signals get distorted by the binaural beamforming, causing the hearing of the user of the binaural hearing system to locate the non-target speakers at a different angular position with respect to the target speaker - the target direction of the binaural beamforming - than they are located in reality.
  • the method recognizes that whenever a strong noise reduction in the binaural beamfroming is present in a given frequency band, the main signal energy stems from the target signal. However, if in a given frequency band, a lower level of noise reduction is present, the target signal is contributing less to the overall signal level. Even though the noise reduction is not as strong as for high-target-content frequency bands, it does still distort the binaural cues of non-target sound signals, in particular if the degree of noise reduction is lower not or not only due to a less directional beamforming but mainly due to the reduced contribution of the target signal. However, in such a frequency band, there is far less penalty for the target signal's sound quality when "further deteriorating" somehow the noise reduction, as there is less target signal contribution present.
  • the output signal is given by a mixture of the first binaural beamformer signal and the first common gain signal.
  • the degree of noise reduction is to be determined.
  • the comparison preferably is done in terms of a quantity which is a function of a degree of noise reduction.
  • a function such as the absolute signal power can be taken for both signals and the presence or degree of noise reduction may be then inferred from the hierarchy established by the comparison of the signal powers.
  • the power difference of the two signals to be classified or their complex coherence may be compared to a threshold value, the result of the comparison indicating a stronger or a weaker presence of noise reduction in the first binaural beamformer signal.
  • the use of not only the first common gain's signal components for restoring the binaural cues in the output signal, but also its inherent information on a degree of noise reduction in a frequency band, is of particular advantage: at first, one might be tempted to directly take into account the first reference signal for comparison with the first binaural beamformer signal.
  • the signal level of the first reference signal may strongly depend on the position of a sound signal's source, and its contribution to the residual noise level may be overestimated if the source is close to the first reference microphone and in the area which Is shadowed off from the second reference microphone by the head of the user.
  • the first binaural beamformer signal Is compared to the first common gain signal in a time-frequency domain, wherein the first output signal is obtained in the time-frequency domain.
  • the whole signal processing may be performed in the time-frequency domain.
  • the first reference signal and the second reference signal, and, if present, further input signals may be converted into the time-frequency domain via a fast Fourier transform (FFT) or a filterbank. This allows for tracking the temporal evolution of the spectral components of the signals involved.
  • FFT fast Fourier transform
  • the first output signal is obtained in the time-frequency domain, as well, and may be optionally processed further and transformed back into the time domain before outputting it via a speaker or similar.
  • the classification is performed with the signal components of a given frequency band in each of the involved frequency bands.
  • the signal powers of the first binaural beamformer signal and the first common gain signal may be compared to each other in absolute terms.
  • the mean signal powers may be taken as a mean value over a certain time window, e.g., one frame or a sequence of more frames, for a comparison,
  • the power difference may be taken as a suitable variable for comparison.
  • the power difference may be compared to certain thresholds such as the mean value of the power difference over the given time window or the maximum or the minimum value of the power difference in said time window,
  • the classification may be such that according to the result, a hierarchy or ordering of noise reduction content may be established, and the mixing of the first binaural beamfromer signal and the first common gain signal may be according to the hierarchy or ordering.
  • a similar hierarchy or ordering may also be established by using the complex coherence between the first binaural beamformer signal and the first common gain signal.
  • the complexe coherence of two signals is defined as the cross power spectral density (CPSD) of the two signals, normalized over the square root of the product of the two signals' auto power spectral densities (PSD).
  • CPSD cross power spectral density
  • PSD auto power spectral densities
  • the complex coherence may then be compared to a threshold value such as the mean value of the complex coherence over a given time window, and the comparison result may determine a mixing rule for the first output signal.
  • the complexe coherence will attain values between 0 and 1, such that a quantitative mixing rule in terms of a convex weighting coefficient for the first binaural beamformer signal and the first common gain signal may directly be taken as the complex coherence itself.
  • variable to use for the comparison of the first binaural beamformer signal and the first common gain signal and for their classification with respect to each other is that the variable may give a quantitative insight in the binaural noise reduction present in the binaural beamformer signal.
  • the variably chosen at least in principle may give such an insight in a monotonous, non-binary way.
  • the common gain is determined in each of said number of frequency bands from the first reference signal in the frequency band and the second reference signal in the frequency band, the first common gain signal in the frequency band is derived from the common gain in the frequency band and the first reference signal in the frequency band, and a second common gain signal in the frequency band is derived from the common gain in the frequency band and the second reference signal in the frequency band.
  • Defining an individual common gain for different frequency bands increases the sensitivity of the proposed method to properly detect noise reduction in a given frequency band and thus improves the classification of the first binaural beamformer signal and the first common gain signal accordingly, as well as the recovery of the binaural cues, since the performance of method in frequency bands with a low target signal content does not depend on the high-target-content frequency bands.
  • a phase of the first output signal is derived from a phase of the first reference signal.
  • the phase of the first output signal may be given by the phase of the first reference signal.
  • the interaural phase difference has a dominant effect on the perception of the binaural cues by a listener. The higher the frequencies, the less dominant is this effect.
  • a phase of the first output signal is derived from a phase of the first binaural beamformer signal.
  • the phase of the first output signal may be given by the phase of the first binaural beamformer signal. It may be of further advantage to decide whether or not to derive the phase of the first output signal from the phase of the first binaural beamformer signal according to the classification result of the first binaural beamformer signal and the first common gain signal in this frequency band. This is useful especially if either no significant noise reduction is present in the frequency band or if the interaural phase difference does not contribute significantly to the binaural cues in this frequency band.
  • the first binaural beamformer signal and the first common gain signal each are decomposed into a magnitude component and a phase component, and the magnitude component of the first binaural beamformer signal and the magnitude component of the first common gain signal are mixed in dependence of said classification in order to obtain a magnitude component of the first output signal.
  • the phase of the first output signal may be derived according to another classification due to a comparison of the first binaural beamformer signal and the first common gain signal, preferably in a different variable than the comparison for classifying the mixing of the magnitudes.
  • the mixing of the magnitude components can be done according to a classification obtained by comparing the absolute value of the complex coherence or the power difference, while the decision about which phase to take for the first output signal can be made by means of a classification employing the phase angle of the complex coherence.
  • the first hearing aid generates a first supplementary signal from a sound signal by a third microphone
  • the second hearing aid generates a second supplementary signal from the sound signal by a fourth microphone
  • the first supplementary signal and the second supplementary signal are both taken into account to derive the first binaural beamformer signal and the second binaural beamformer signal.
  • the notion of a third microphone, or a fourth microphone, respectively shall comprise any type of sound transducer which is set up to and capable to receive an acoustical wave pattern and to transduce this acoustical wave pattern into an electrical signal.
  • binaural hearing aids for a better spatial sound perception more than just one microphone in a single hearing aid may be employed.
  • the use of more than one microphone at one side, in combination with the microphone or microphones from the other side, allows for a better beamforming, i.e., a narrower directionality if required or a better signal-to-noise-ratio in beamforming noise reduction,
  • the third microphone is located within the first hearing aid slightly apart from the first microphone in order to be able to detect small time shifts with respect to the first reference microphone when a propagating sound signal impinges on the first hearing aid. Similar considerations may hold for the fourth microphone and its location within the second hearing aid.
  • the second binaural beamformer signal is compared with the second common gain signal for classification with respect to a noise reduction, and the second binaural beamformer signal and the second common gain signal are mixed in dependence of said classification in order to obtain a second output signal.
  • the first output signal may be used to derive a first playback signal which is converted into a first sound signal via a first loudspeaker of the first hearing aid
  • the second output signal may be used to derive a second playback signal which is converted into a second sound signal via a second loudspeaker of the second hearing aid.
  • the first playback signal may be derived from the first output signal without an additional input of the second output signal into the first playback signal.
  • the notion of a first loudspeaker or a second loudspeaker, respectively shall comprise any type of sound generator which is set up to and capable to receive an electrical signal and to transduce this electrical signal into an acoustical wave pattern of sound waves.
  • the playback signal is given by a mixture of the local binaural beamformer signal - containing signal components from the microphones of the other hearing aid - and the local common gain signal containing only local signal components, the information of the other hearing aid's microphones entering only via the common gain factor.
  • a binaural hearing system comprising a first hearing aid and a second hearing aid, said binaural hearing system comprising means to perform the method described above.
  • FIG 1 a schematical top view of a hearing situation 1 corresponding to a conversation is shown.
  • a user 2 of a state-of-the-art binaural hearing system (not shown) is surrounded by his conversational partners, given by the speakers 4, 6, 8, 10, 12, while directing his view towards the target speaker 4 for a given moment.
  • the state-of-the-art binaural hearing system is applying a noise reduction in which noise from directions other than the one of the target speaker 4, at least partially, is aimed to be reduced via the binaural beamforming of the binaural beamforming system, the target speaker 4 will be perceived by the user 2 in the proper direction.
  • the other, non-target speakers 6, 8, 10, 12, apart from having an attenuated signal volume in the output signal of the binaural beamforming hearing aid as perceived by the user 2, due to the binaural beamforming may show their binaural cues distorted when talking to the user 2 which is focused on the target speaker 4, leading to an improper perception of the acoustical localization of the non-target speakers 6, 8, 10, 12 in the perception of the user 2.
  • the user 2 can see the actual positions of two intervening non-target speakers 6, 12 as spatially well separated from the target speaker 4, but due to the state-of-the-art binaural beamforming, displayed by the beam 14, and the loss of binaural cues of the non-target speakers 6, 12 caused by the noise reduction processes, the user 2 "hears" contributions from the non-target speakers 6, 12 as if those were located much closer to the target speaker 4.
  • a method 18 for operating a binaural hearing system 20 is illustrated by means of a block diagram.
  • the method 18 is particularly useful in order to preserve binaural cues of a sound signal 22 when noise reduction is active in the binaural hearing system 20.
  • the binaural hearing system 20 comprises a first hearing aid 24 and a second hearing aid 26.
  • a first reference signal 28 is generated from the sound signal 22 by a first microphone 30, while in the second hearing aid 26, a second reference signal 32 Is generated from the sound signal 22 by a second microphone 34.
  • a first supplementary signal 36 is generated from the sound signal 22 by a third microphone 38, while in the second hearing aid 26 a second supplementary signal 40 is generated from the sound signal 22 by a fourth microphone 42.
  • the second reference signal 32 and the second supplementary signal 40 are transmitted from the second hearing aid 26 to the first hearing aid 24.
  • the first reference signal 28, the second reference signal 32, the first supplementary signal 36 and the second supplementary signal 42 are transformed into the time-frequency domain by a filter bank 44.
  • the signal components of the first and second reference and supplementary signals 28, 32, 36, 40 in each frequency band are used to form a first binaural beamformer signal 46.
  • the binaural information in the binaural beamformer signal 46 together with the first reference signal 28 and the second reference signal 32 are used to determine a common gain 48 in each frequency band given by the filter bank 40.
  • the first binaural beamformer signal 46 is used to derive the common gain 48 in order to include all possible acoustical information, i.e., also time or level differences that are only measurable with the help of the first and the second supplementary signal 36, 40.
  • the information content of these two signals is introduced into the common gain 48 by the first binaural beamformer signal 46.
  • the common gain 48 is applied to the first reference signal 28 by a mere scalar multiplication to obtain the first common gain signal 50.
  • a decomposition 52 is applied in order to decompose the first binaural beamformer signal 46 into its magnitude and phase components.
  • a decomposition 54 is applied in order to decompose the first common gain signal 50 into at least its magnitude component. If the phase component of the first common gain signal 50 is not further taken into account, the decomposition 54 can be achieved by merely taking the absolute value of the first common gain signal 50.
  • the first binaural beam former signal 46 and the first common gain signal 50 are now compared in a way still to be described.
  • the result of the comparison is a hierarchical classification 56 with respect to noise reduction, i.e., from the classification 56 quantitative information on whether or not a strong noise reduction is present in the first binaural beamformer signal 46 can be obtained.
  • the classification 56 the magnitudes of the first binaural beamformer signal 46 and the first common gain signal 50 are mixed in order to derive a first output signal 58.
  • the phase of the first output signal 58 is taken to be the phase 60 of the first reference signal 28 or the phase of the first binaural beamformer signal 46 according to the classification 56.
  • the first output signal 58 may be further processes (not shown in figure 3 ) to derive a first playback signal 62 which is converted into a sound signal 64 by a first loudspeaker 66 of the first hearing aid 24.
  • the magnitude of the first output signal 58 in a given frequency band and to a time-frequency frame may be taken as a linear mixture
  • a *
  • 1 stOut indicates the first output signal 58, 1stBinBF the first binaural beamformer signal 46, and 1 stComG the first common gain signal 50, and
  • the power difference of the first common gain signal 50 and the first binaural beamformer signal 46 as the relevant variable for the classification 56, one might establish a hierarchy by taking as the upper and lower limit for the power difference diff the maximum and minimum values max(diff), min(diff) over the given time window (e.g., a frame or a sequence of frames) as well as the temporal mean value mean(diff) of the power difference over the time window.
  • the upper and lower limit for the power difference diff the maximum and minimum values max(diff), min(diff) over the given time window (e.g., a frame or a sequence of frames) as well as the temporal mean value mean(diff) of the power difference over the time window.
  • the power difference diff may attain a value which falls into one of the following four intervals: 1 : max diff , max diff / 2 + mean diff / 2 , 2 : max diff / 2 + mean diff / 2, mean diff , 3 : mean diff , mean diff / 2 + min diff / 2 , 4 : mean diff / 2 + min diff / 2, min diff .
  • the phase of the first output signal 58 in these examples each time may be given by the phase 60 of the first reference signal 28 for lower frequency bands (up to 1.5 kHz or 2 kHz) and by the phase of the first binaural beamformer signal 46 otherwise.
  • the main Idea is that when the condition C ⁇ T is met, more magnitude content of the first binaural beamformer signal 46 In the lower frequency bands than in the higher frequency bands is processed into the first output signal 58.
  • This approach reflects the fact that most speech energy is concentrated at lower frequencies, so that the spatial preceptlon of an acoustical environment such as hearing situation 1 is typically dominated by the content in the lower frequency bands.
  • the complex coherence C between the first binaural beamformer signal 46 and the first common gain signal 50 is below a temporal mean value, one may infer that for the short time slot to which the complex coherence is taken, the first binaural beamformer signal 46 - which also indicates the degree of noise reduction by the spatial characteristics of the beam in the given frequency band - differs "more than normal" from the first common gain signal 50.
  • the first common gain signal 50 can be interpreted as a reference of the total signal level at both the first hearing aid 24 and the second hearing aid 26 in each particular frequency band for the degree of noise reduction present in the first binaural beamformer signal 46, while the reference is taken into account via the comparison of C with respect to its the temporal mean T. So an instantaneously "more than normal"-deviation of the first binaural beamformer signal 46 from the first common gain signal 50 indicates a higher degree of binaural noise reduction in the former.
  • the phase of the first output signal 58 in this example may be given for lower frequency bands (up to 1.5 kHz or 2 kHz) by the phase 60 of the first reference signal 28 or by the phase of the first binaural beamformer signal 46, depending on the phase angle of C, and only by the phase of the first binaural beamformer signal 46 for higher frequencies.
  • a noise reduction process which is based on a binaural beamforming process suppressing sounds from sound sources located in different directions than the target sound source may distort the binaural cues of non-target sound signals, i.e., sound signal components whose source is not located in the target direction. Even though these sound signals are suppressed by the binaural beamforming anyway, and might not be perceived as "conversationally relevant", they still might have an important impact on the user's 2 perception of the acoustical scene in his hearing environment. Distorted binaural cues of these non-target sound signals then may lead to a mismatch of the acoustical perception of the non-target sound sources and their actual positions as seen by the user.

Landscapes

  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Neurosurgery (AREA)
  • Otolaryngology (AREA)
  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Stereophonic System (AREA)

Claims (12)

  1. Verfahren (18) zum Betrieb eines binauralen Hörsystems (20), wobei das binaurale Hörsystem (20) eine erste Hörhilfe (24) und eine zweite Hörhilfe (26) umfasst,
    wobei die erste Hörhilfe (24) ein erstes Referenzsignal (28) aus einem Schallsignal (22) von einem ersten Mikrofon (30) erzeugt,
    wobei die zweite Hörhilfe (26) ein zweites Referenzsignal (32) aus einem Schallsignal (22) von einem zweiten Mikrofon (34) erzeugt,
    wobei das erste Referenzsignal (28) und das zweite Referenzsignal (32) jeweils verwendet werden, um ein erstes binaurales Strahlbildungssignal (46) und ein zweites binaurales Strahlbildungssignal abzuleiten, wobei das erste Referenzsignal (28) und das zweite Referenzsignal (32) verwendet werden, um eine gemeinsame Verstärkung (48) zu bestimmen, und aus dem ersten Referenzsignal (28) und der gemeinsamen Verstärkung (48) ein erstes gemeinsames Verstärkungssignal (50) abgeleitet wird,
    dadurch gekennzeichnet, dass
    mindestens für eine Anzahl von Frequenzbändern
    - das erste binaurale Strahlbildungssignal (46) mit dem ersten gemeinsamen Verstärkungssignal (50) zur Klassifizierung (56) in Bezug auf eine Menge verglichen wird, die eine Funktion eines Grades der Geräuschreduktion ist, und
    - das erste binaurale Strahlbildungssignal (46) und das erste gemeinsame Verstärkungssignal (50) in Abhängigkeit von der Klassifizierung (56) gemischt werden, um ein erstes Ausgabesignal (58) zu erhalten.
  2. Verfahren (18) nach Anspruch 1,
    wobei das erste binaurale Strahlbildungssignal (46) mit dem ersten gemeinsamen Verstärkungssignal (50) in einer Zeit-Frequenz-Domäne verglichen wird und
    wobei das erste Ausgabesignal (58) in der Zeit-Frequenz-Domäne erhalten wird.
  3. Verfahren (18) nach Anspruch 1 oder 2,
    wobei zur Klassifizierung (56) des ersten binauralen Strahlbildungssignals (46) und des ersten gemeinsamen Verstärkungssignals (50) mindestens eine der Variablen Signalleistung, durchschnittliche Signalleistung und komplexe Kohärenzfunktion berücksichtigt wird.
  4. Verfahren (18) nach einem der vorstehenden Ansprüche,
    wobei für mindestens eine Anzahl von Frequenzbändern
    - die gemeinsame Verstärkung (48) in jedem aus der Anzahl von Frequenzbändern vom ersten Referenzsignal (28) in dem Frequenzband und dem zweiten Referenzsignal (32) im Frequenzband bestimmt wird,
    - das erste gemeinsame Verstärkungssignal (50) im Frequenzband aus der gemeinsamen Verstärkung (48) im Frequenzband und dem ersten Referenzsignal (28) im Frequenzband abgeleitet wird, und
    - ein zweites gemeinsames Verstärkungssignal im Frequenzband aus der gemeinsamen Verstärkung (48) im Frequenzband und dem zweiten Referenzsignal im Frequenzband abgeleitet wird.
  5. Verfahren (18) nach einem der vorstehenden Ansprüche,
    wobei in mindestens einem Frequenzband unterhalb von 2 kHz eine Phase des ersten Ausgabesignals (58) aus einer Phase (60) des ersten Referenzsignals (28) abgeleitet wird.
  6. Verfahren (18) nach einem der vorstehenden Ansprüche,
    wobei in mindestens einem Frequenzband eine Phase des ersten Ausgabesignals (58) aus einer Phase des ersten binauralen Strahlbildungssignals (46) abgeleitet wird.
  7. Verfahren (18) nach einem der vorstehenden Ansprüche,
    wobei für die Anzahl von Frequenzbändern
    - das erste binaurale Strahlbildungssignal (46) und das erste gemeinsame Verstärkungssignal (50) jeweils in eine Größenkomponente und eine Phasenkomponente zerlegt werden, und
    - die Größenkomponente des ersten binauralen Strahlbildungssignals (46) und die Größenkomponente des ersten gemeinsamen Verstärkungssignals (50) in Abhängigkeit von der Klassifizierung (56) gemischt werden, um eine Größenkomponente des ersten Ausgabesignals (58) zu erhalten.
  8. Verfahren (18) nach einem der vorstehenden Ansprüche,
    wobei die erste Hörhilfe (24) ein erstes zusätzliches Signal (36) aus einem Schallsignal (22) von einem dritten Mikrofon (38) erzeugt,
    wobei die zweite Hörhilfe (26) ein zweites zusätzliches Signal (40) aus dem Schallsignal (22) von einem vierten Mikrofon (42) erzeugt, und
    wobei das erste zusätzliche Signal (36) und das zweite zusätzliche Signal (40) beide berücksichtigt werden, um das erste binaurale Strahlbildungssignal (46) und das zweite binaurale Strahlbildungssignal abzuleiten.
  9. Verfahren (18) nach Anspruch 8,
    wobei das erste zusätzliche Signal (36) und das zweite zusätzliche Signal (40) zum Bestimmen der gemeinsamen Verstärkung (48) berücksichtigt werden.
  10. Verfahren (18) nach einem der vorstehenden Ansprüche,
    wobei aus dem zweiten Referenzsignal (32) und der gemeinsamen Verstärkung (48) ein zweites gemeinsames Verstärkungssignal abgeleitet wird, und
    wobei mindestens für eine Anzahl von Frequenzbändern
    - das zweite binaurale Strahlbildungssignal mit dem zweiten gemeinsamen Verstärkungssignal verglichen wird zur Klassifizierung mit Bezug auf eine Menge, die eine Funktion eines Grades von Geräuschreduktion ist, und
    - das zweite binaurale Strahlbildungssignal und das zweite gemeinsame Verstärkungssignal in Abhängigkeit von der Klassifizierung gemischt werden, um ein zweites Ausgabesignal zu erhalten.
  11. Verfahren nach Anspruch 10, wobei ein erstes Wiedergabesignal (62) aus dem ersten Ausgabesignal (58) ohne eine zusätzliche Eingabe des zweiten Ausgabesignals in das erste Wiedergabesignal (62) abgeleitet wird, und
    wobei das erste Ausgabesignal (62) durch einen ersten Lautsprecher (66) der ersten Hörhilfe (24) in ein Schallsignal (64) umgewandelt wird.
  12. Binaurales Hörsystem (20), umfassend eine erste Hörhilfe (24) und eine zweite Hörhilfe (26), wobei das binaurale Hörsystem (20) Mittel zum Durchführen des Verfahrens (18) nach einem der vorstehenden Ansprüche umfasst.
EP16172902.5A 2015-09-24 2016-06-03 Verfahren zum betrieb eines binauralen hörsystems Active EP3148217B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP15186710 2015-09-24

Publications (2)

Publication Number Publication Date
EP3148217A1 EP3148217A1 (de) 2017-03-29
EP3148217B1 true EP3148217B1 (de) 2019-01-09

Family

ID=54199594

Family Applications (1)

Application Number Title Priority Date Filing Date
EP16172902.5A Active EP3148217B1 (de) 2015-09-24 2016-06-03 Verfahren zum betrieb eines binauralen hörsystems

Country Status (2)

Country Link
EP (1) EP3148217B1 (de)
DK (1) DK3148217T3 (de)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10555094B2 (en) 2017-03-29 2020-02-04 Gn Hearing A/S Hearing device with adaptive sub-band beamforming and related method
DK3672283T3 (da) * 2018-12-21 2022-04-11 Sivantos Pte Ltd Fremgangsmåde til forbedring af den rumlige høreperception for et binauralt høreapparat

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7206421B1 (en) * 2000-07-14 2007-04-17 Gn Resound North America Corporation Hearing system beamformer
US8139787B2 (en) * 2005-09-09 2012-03-20 Simon Haykin Method and device for binaural signal enhancement
GB0609248D0 (en) * 2006-05-10 2006-06-21 Leuven K U Res & Dev Binaural noise reduction preserving interaural transfer functions
WO2010051606A1 (en) * 2008-11-05 2010-05-14 Hear Ip Pty Ltd A system and method for producing a directional output signal
US20150172807A1 (en) * 2013-12-13 2015-06-18 Gn Netcom A/S Apparatus And A Method For Audio Signal Processing

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
DK3148217T3 (en) 2019-04-15
EP3148217A1 (de) 2017-03-29

Similar Documents

Publication Publication Date Title
EP2360943B1 (de) Strahlformung in Hörgeräten
Hadad et al. The binaural LCMV beamformer and its performance analysis
Marquardt et al. Theoretical analysis of linearly constrained multi-channel Wiener filtering algorithms for combined noise reduction and binaural cue preservation in binaural hearing aids
US9992587B2 (en) Binaural hearing system configured to localize a sound source
EP3013070B1 (de) Hörgerätesystem
Van den Bogaert et al. Speech enhancement with multichannel Wiener filter techniques in multimicrophone binaural hearing aids
US9613610B2 (en) Directional sound masking
Arweiler et al. The influence of spectral characteristics of early reflections on speech intelligibility
US9681246B2 (en) Bionic hearing headset
US20090202091A1 (en) Method of estimating weighting function of audio signals in a hearing aid
Thiemann et al. Speech enhancement for multimicrophone binaural hearing aids aiming to preserve the spatial auditory scene
Marquardt et al. Interaural coherence preservation for binaural noise reduction using partial noise estimation and spectral postfiltering
US10003893B2 (en) Method for operating a binaural hearing system and binaural hearing system
Reindl et al. Analysis of two generic wiener filtering concepts for binaural speech enhancement in hearing aids
Kates et al. Integrating a remote microphone with hearing-aid processing
As' ad et al. A robust target linearly constrained minimum variance beamformer with spatial cues preservation for binaural hearing aids
EP3672283B1 (de) Verfahren zur verbesserung der räumlichen wahrnehmung einer binauralen hörvorrichtung
Marquardt et al. Optimal binaural LCMV beamformers for combined noise reduction and binaural cue preservation
EP3588979B1 (de) Verfahren zur verbesserung der signalrichtwirkung in einem hörgerät
EP3148217B1 (de) Verfahren zum betrieb eines binauralen hörsystems
Gößling et al. Performance analysis of the extended binaural MVDR beamformer with partial noise estimation
Bissmeyer et al. Adaptive spatial filtering improves speech reception in noise while preserving binaural cues
Farmani et al. Sound source localization for hearing aid applications using wireless microphones
Gößling et al. Binaural LCMV beamforming with partial noise estimation
Xiao et al. Effect of target signals and delays on spatially selective active noise control for open-fitting hearables

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20170725

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

RIN1 Information on inventor provided before grant (corrected)

Inventor name: PUDER, HENNING

Inventor name: FISCHER, EGHART

Inventor name: BOUCHARD, MARTIN

Inventor name: AS AD, HALA

Inventor name: KAMKAR-PARSI, HOMAYOUN

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: SIVANTOS PTE. LTD.

RIC1 Information provided on ipc code assigned before grant

Ipc: H04R 25/00 20060101AFI20180223BHEP

Ipc: G10L 21/0216 20130101ALN20180223BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20180424

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 21/0216 20130101ALN20180413BHEP

Ipc: H04R 25/00 20060101AFI20180413BHEP

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTC Intention to grant announced (deleted)
RIC1 Information provided on ipc code assigned before grant

Ipc: H04R 25/00 20060101AFI20180712BHEP

Ipc: G10L 21/0216 20130101ALN20180712BHEP

INTG Intention to grant announced

Effective date: 20180724

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

GRAL Information related to payment of fee for publishing/printing deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR3

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

GRAR Information related to intention to grant a patent recorded

Free format text: ORIGINAL CODE: EPIDOSNIGR71

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

INTC Intention to grant announced (deleted)
INTG Intention to grant announced

Effective date: 20181129

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 21/0216 20130101ALN20181126BHEP

Ipc: H04R 25/00 20060101AFI20181126BHEP

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 1088812

Country of ref document: AT

Kind code of ref document: T

Effective date: 20190115

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602016009119

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: NV

Representative=s name: E. BLUM AND CO. AG PATENT- UND MARKENANWAELTE , CH

REG Reference to a national code

Ref country code: DK

Ref legal event code: T3

Effective date: 20190408

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20190109

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1088812

Country of ref document: AT

Kind code of ref document: T

Effective date: 20190109

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190409

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190509

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190410

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190409

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190509

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602016009119

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20191010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20190630

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190603

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190630

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190603

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20160603

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190109

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: CH

Payment date: 20230702

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20240620

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240617

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DK

Payment date: 20240620

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20240621

Year of fee payment: 9