EP3214620B1 - Ein monaurales, intrusives sprachverständlichkeitsvorhersagesystem, hörhilfe system - Google Patents

Ein monaurales, intrusives sprachverständlichkeitsvorhersagesystem, hörhilfe system Download PDF

Info

Publication number
EP3214620B1
EP3214620B1 EP17158062.4A EP17158062A EP3214620B1 EP 3214620 B1 EP3214620 B1 EP 3214620B1 EP 17158062 A EP17158062 A EP 17158062A EP 3214620 B1 EP3214620 B1 EP 3214620B1
Authority
EP
European Patent Office
Prior art keywords
time
frequency
unit
speech intelligibility
signal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP17158062.4A
Other languages
English (en)
French (fr)
Other versions
EP3214620A1 (de
Inventor
Jesper Jensen
Jan Mark De Haan
Asger Heidemann Andersen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Oticon AS
Original Assignee
Oticon AS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oticon AS filed Critical Oticon AS
Publication of EP3214620A1 publication Critical patent/EP3214620A1/de
Application granted granted Critical
Publication of EP3214620B1 publication Critical patent/EP3214620B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R25/00—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception
    • H04R25/50—Customised settings for obtaining desired overall acoustical characteristics
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0316—Speech enhancement, e.g. noise reduction or echo cancellation by changing the amplitude
    • G10L21/0364—Speech enhancement, e.g. noise reduction or echo cancellation by changing the amplitude for improving intelligibility
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61N—ELECTROTHERAPY; MAGNETOTHERAPY; RADIATION THERAPY; ULTRASOUND THERAPY
    • A61N1/00—Electrotherapy; Circuits therefor
    • A61N1/18—Applying electric currents by contact electrodes
    • A61N1/32—Applying electric currents by contact electrodes alternating or intermittent currents
    • A61N1/36—Applying electric currents by contact electrodes alternating or intermittent currents for stimulation
    • A61N1/372—Arrangements in connection with the implantation of stimulators
    • A61N1/378—Electrical supply
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L21/0216—Noise filtering characterised by the method used for estimating noise
    • G10L21/0232—Processing in the frequency domain
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/038—Speech enhancement, e.g. noise reduction or echo cancellation using band spreading techniques
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
    • G10L25/06—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters the extracted parameters being correlation coefficients
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
    • G10L25/60—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for measuring the quality of voice signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R25/00—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception
    • H04R25/50—Customised settings for obtaining desired overall acoustical characteristics
    • H04R25/505—Customised settings for obtaining desired overall acoustical characteristics using digital signal processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R25/00—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception
    • H04R25/55—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception using an external connection, either wireless or wired
    • H04R25/552—Binaural
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R25/00—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception
    • H04R25/55—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception using an external connection, either wireless or wired
    • H04R25/554—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception using an external connection, either wireless or wired using a wireless connection, e.g. between microphone and amplifier or using Tcoils
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R25/00—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception
    • H04R25/70—Adaptation of deaf aid to hearing loss, e.g. initial electronic fitting
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61N—ELECTROTHERAPY; MAGNETOTHERAPY; RADIATION THERAPY; ULTRASOUND THERAPY
    • A61N1/00—Electrotherapy; Circuits therefor
    • A61N1/18—Applying electric currents by contact electrodes
    • A61N1/32—Applying electric currents by contact electrodes alternating or intermittent currents
    • A61N1/36—Applying electric currents by contact electrodes alternating or intermittent currents for stimulation
    • A61N1/36036—Applying electric currents by contact electrodes alternating or intermittent currents for stimulation of the outer, middle or inner ear
    • A61N1/36038—Cochlear stimulation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2225/00—Details of deaf aids covered by H04R25/00, not provided for in any of its subgroups
    • H04R2225/025—In the ear hearing aids [ITE] hearing aids
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2225/00—Details of deaf aids covered by H04R25/00, not provided for in any of its subgroups
    • H04R2225/41—Detection or adaptation of hearing aid parameters or programs to listening situation, e.g. pub, forest
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2225/00—Details of deaf aids covered by H04R25/00, not provided for in any of its subgroups
    • H04R2225/43—Signal processing in hearing aids to enhance the speech intelligibility
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2225/00—Details of deaf aids covered by H04R25/00, not provided for in any of its subgroups
    • H04R2225/55—Communication between hearing aids and external devices via a network for data exchange
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R25/00—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception
    • H04R25/55—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception using an external connection, either wireless or wired
    • H04R25/558—Remote control, e.g. of amplification, frequency
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R25/00—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception
    • H04R25/60—Mounting or interconnection of hearing aid parts, e.g. inside tips, housings or to ossicles
    • H04R25/604—Mounting or interconnection of hearing aid parts, e.g. inside tips, housings or to ossicles of acoustic or vibrational transducers
    • H04R25/606—Mounting or interconnection of hearing aid parts, e.g. inside tips, housings or to ossicles of acoustic or vibrational transducers acting directly on the eardrum, the ossicles or the skull, e.g. mastoid, tooth, maxillary or mandibular bone, or mechanically stimulating the cochlea, e.g. at the oval window

Definitions

  • the present disclosure deals with monaural, intrusive intelligibility prediction of noisy/processed speech signals comprising a target signal component based on simultaneous knowledge of a substantially noise-free ('clean') version of the target signal component.
  • the present disclosure further relates to a hearing aid comprising a monaural, intrusive intelligibility predictor unit, and to a binaural hearing aid system comprising first and second hearing aids, each comprising a monaural, intrusive intelligibility predictor unit, wherein the binaural hearing aid system is configured to establish a wireless link allowing the exchange of monaural speech intelligibility predictors or information derived therefrom between the first and second hearing aids.
  • a monaural speech intelligibility predictor unit :
  • a monaural speech intelligibility predictor unit adapted for receiving a target signal comprising speech in an essentially noise-free version s and in a noisy and/or processed version x , the monaural speech intelligibility predictor unit being configured to provide as an output a final monaural speech intelligibility predictor value d indicative of a listener's perception of said noisy and/or processed version x of the target signal, the monaural speech intelligibility predictor unit comprising
  • the monaural speech intelligibility predictor unit comprises a normalization and transformation unit adapted for providing normalized and/or transformed versions X ⁇ m ( S ⁇ m ) of said time-frequency segments X m ( S m ) .
  • the normalization and transformation unit is configured to apply one or more algorithms for row and/or column normalization and/or transformation operations to the time-frequency segments S m and/or X m .
  • the normalization and transformation unit is configured to provide at least one normalization and/or transformation operation of rows and at least one normalization and/or transformation operation of columns to the time-frequency segments S m and/or X m .
  • the monaural speech intelligibility predictor unit comprises a normalization and transformation unit configured to provide normalization and/or transformation of rows and columns of the time-frequency segments S m and X m , wherein the normalization and/or transformation of rows comprise(s) at least one of the following operations
  • the final monaural speech intelligibility calculation unit is configured to combine said intermediate speech intelligibility coefficients d m , or a transformed version thereof, by averaging over time, or by applying a MIN or MAX-function, or other algebraic or statistical function, to the intermediate speech intelligibility coefficients d m , or a transformed version thereof.
  • the first and second input units are configured to receive the noise free version of the target signal s (also termed the 'clean (version of the) target signal') and the noisy and/or processed version x of the target signal (termed the 'information signal x' ) , respectively, as a time variant (time domain/full band) signal s(n) and x(n) , respectively, n being a time index.
  • the first and second input units are configured to receive the clean target signal s and the information signal x , respectively, in a time-frequency representation s(k,m) and x(k,m), respectively, from another unit or device, k and m being frequency and time indices, respectively.
  • the first and second input units each comprises a frequency decomposition unit for providing a time-frequency representation s(k,m) and x(k,m) of the clean target signal s and the information signal x from a time domain version of the respective signals (s(n) and x(n), n being a time index).
  • the frequency decomposition unit comprises a band-pass filterbank (e.g., a Gamma-tone filter bank), or is adapted to implement a Fourier transform algorithm (e.g. a short-time Fourier transform (STFT) algorithm).
  • STFT short-time Fourier transform
  • the monaural speech intelligibility predictor unit comprises a voice activity detector unit for indicating whether or not or to what extent a given time-segment of the essentially noise-free version s and the noisy and/or processed version x , respectively, of the target signal comprises or is estimated to comprise speech, and providing a voice activity control signal indicative thereof.
  • the voice activity detector unit is configured to provide a binary indication identifying segments comprising speech or no speech.
  • the voice activity detector unit is configured to identify segments comprising speech with a certain probability.
  • the voice activity detector is applied to a time-domain signal (or full-band signal, s(n), x(n), n being a time index).
  • the voice activity detector is applied to a time-frequency representation of a signal ( s(k,m), x(k,m), or s j (m), x j (m), k and j being frequency indices (bin and sub-band, respectively), m being a time index) or a signal originating therefrom).
  • the voice activity detector unit is configured to identify time-frequency segments comprising speech on a time-frequency unit level (or e.g. in a frequency sub-band signal x j (m) ).
  • the monaural speech intelligibility predictor unit is adapted to receive (e.g. wirelessly receive) a voice activity control signal from another unit or device.
  • the monaural speech intelligibility predictor unit comprises a voice activity detector unit for identifying time-segments of the essentially noise-free version s and the noisy and/or processed version x , respectively, of the target signal comprising or estimated to comprise speech, and wherein the monaural speech intelligibility predictor unit is configured to provide modified versions of the essentially noise-free version s and the noisy and/or processed version x , respectively, of the target signal comprising only such time segments comprising speech or being estimated to comprise speech.
  • the first and second time-frequency segment division units are configured to base the generation of the time-frequency segments S m and X m , respectively, or normalized and/or transformed versions, S ⁇ m and X ⁇ m , thereof on the voice activity control signal, e.g. to generate said time-frequency segments in dependence of the voice activity control signal, e.g. only if speech is indicated to be present, or if the probability that the time-frequency segment in question contains speech is larger than a predefined value, e.g. 0.5).
  • a predefined value e.g. 0.5
  • the monaural speech intelligibility predictor unit comprises a hearing loss model unit configured to apply a frequency dependent modification of the said noisy and/or processed version x of the target signal reflecting a deviation from normal hearing, e.g. a hearing impairment, of a relevant ear of the listener to provide a modified noisy and/or processed version x of the target signal for use together with said essentially noise-free version s of the target signal as a basis for calculating the final monaural speech intelligibility predictor d.
  • a hearing loss model unit configured to apply a frequency dependent modification of the said noisy and/or processed version x of the target signal reflecting a deviation from normal hearing, e.g. a hearing impairment, of a relevant ear of the listener to provide a modified noisy and/or processed version x of the target signal for use together with said essentially noise-free version s of the target signal as a basis for calculating the final monaural speech intelligibility predictor d.
  • the hearing loss model unit is configured to add a statistically independent noise signal, which is spectrally shaped according to an audiogram of the relevant ear of the listener, to said noisy and/or processed version x of the target signal.
  • the first and second envelope extraction units each comprises an algorithm for implementing a Hilbert transform, or for low-pass filtering the magnitude of complex-valued STFT signals s(k,m) and x(k,m), etc.
  • the monaural speech intelligibility predictor unit comprises
  • the normalization and/or transformation unit is configured to apply one or more algorithms for row and/or column normalization and/or transformation to the time-frequency segments S m and/or X m , respectively.
  • the normalization and/or transformation unit is configured to apply one or more of the following algorithms to the time-frequency segments X m and S m , respectively, commonly denoted Z m , , where sub-script, time index m is skipped for simplicity in the following expressions:
  • a and b represent (e.g. any K) elements from time frequency segments S m (or S ⁇ m ) and X m (or X ⁇ m ) , respectively.
  • a and b represent elements from columns of time frequency segments S m (or S ⁇ m ) and X m (or X ⁇ m ) , respectively. In an embodiment, a and b represent elements from rows of time frequency segments S m (or S ⁇ m ) and X m (or X ⁇ m ) , respectively. In an embodiment, a and b represent all elements in time frequency segments S m (or S ⁇ m ) and X m (or X ⁇ m ) , respectively.
  • the intermediate intelligibility index d m is defined as
  • the duration of the speech active parts of the noisy and/or processed version x of the target signal is defined as a (possibly accumulated) time period where the voice activity control signal indicates that the noisy and/or processed version x of the target signal comprises speech.
  • a hearing aid is a hearing aid
  • a hearing aid adapted for being located at or in left and right ears of a user, or for being fully or partially implanted in the head of the user, the hearing aid comprising a monaural speech intelligibility predictor unit as described above, in the detailed description of embodiments, in the drawings and in the claims is furthermore provided by the present disclosure.
  • the hearing aid is configured to adaptively modify the processing of an input signal to the hearing aid to maximize the final monaural speech intelligibility predictor d. to enhance the user's intelligibility of an output signal of the hearing aid presented to the user
  • the hearing aid comprises
  • the configurable signal processing unit is adapted to control or influence the processing of the respective electric input signals, or one or more signals originating therefrom (e.g. a resulting beamformed signal) based on said final speech intelligibility predictor d provided by the monaural speech intelligibility predictor unit.
  • the configurable signal processing unit is adapted to control or influence the processing of the respective electric input signals based on said final speech intelligibility predictor d when the target signal component comprises speech, such as only when the target signal component comprises speech (as e.g. defined by a voice (speech) activity detector).
  • the configurable signal processing unit is adapted to control or influence the processing of the respective electric input signals to maximize the final speech intelligibility predictor d .
  • the hearing aid is adapted to provide a frequency dependent gain and/or a level dependent compression and/or a transposition (with or without frequency compression) of one or frequency ranges to one or more other frequency ranges, e.g. to compensate for a hearing impairment of a user.
  • the output unit comprises a number of electrodes of a cochlear implant or a vibrator of a bone conducting hearing aid.
  • the output unit comprises an output transducer.
  • the output transducer comprises a receiver (loudspeaker) for providing the stimulus as an acoustic signal to the user.
  • the output transducer comprises a vibrator for providing the stimulus as mechanical vibration of a skull bone to the user (e.g. in a bone-attached or bone-anchored hearing aid).
  • the input unit comprises an input transducer for converting an input sound to an electric input signal.
  • the input unit comprises a wireless receiver for receiving a wireless signal comprising sound and for providing an electric input signal representing said sound.
  • the hearing aid comprises a directional microphone system adapted to enhance a target acoustic source among a multitude of acoustic sources in the local environment of the user wearing the hearing aid.
  • the directional system is adapted to detect (such as adaptively detect) from which direction a particular part of the microphone signal originates.
  • the hearing aid comprises an antenna and transceiver circuitry for wirelessly receiving a direct electric input signal from another device, e.g. a communication device or another hearing aid.
  • a wireless link established by antenna and transceiver circuitry of the hearing aid can be of any type.
  • the wireless link is used under power constraints, e.g. in that the hearing aid comprises a portable (typically battery driven) device.
  • the hearing aid comprises a forward or signal path between an input transducer (microphone system and/or direct electric input (e.g. a wireless receiver)) and an output transducer.
  • the signal processing unit is located in the forward path.
  • the signal processing unit is adapted to provide a frequency dependent gain according to a user's particular needs.
  • the hearing aid comprises an analysis path comprising functional components for analyzing the input signal (e.g. determining a level, a modulation, a type of signal, an acoustic feedback estimate, etc.).
  • some or all signal processing of the analysis path and/or the signal path is conducted in the frequency domain.
  • some or all signal processing of the analysis path and/or the signal path is conducted in the time domain.
  • the hearing aid comprises an analogue-to-digital (AD) converter to digitize an analogue input with a predefined sampling rate, e.g. 20 kHz.
  • the hearing aid comprises a digital-to-analogue (DA) converter to convert a digital signal to an analogue output signal, e.g. for being presented to a user via an output transducer.
  • AD analogue-to-digital
  • DA digital-to-analogue
  • the hearing aid comprises a number of detectors configured to provide status signals relating to a current physical environment of the hearing aid (e.g. the current acoustic environment), and/or to a current state of the user wearing the hearing aid, and/or to a current state or mode of operation of the hearing aid.
  • one or more detectors may form part of an external device in communication (e.g. wirelessly) with the hearing aid.
  • An external device may e.g. comprise another hearing aid, a remote control, and audio delivery device, a telephone (e.g. a Smartphone), an external sensor, etc.
  • one or more of the number of detectors operate(s) on the full band signal (time domain).
  • one or more of the number of detectors operate(s) on band split signals ((time-) frequency domain).
  • the hearing aid further comprises other relevant functionality for the application in question, e.g. compression, noise reduction, feedback reduction, etc.
  • a monaural speech intelligibility predictor unit as described above, in the detailed description of embodiments, in the drawings and in the claims in a hearing aid to modify signal processing in the hearing aid aiming at enhancing intelligibility of a speech signal presented to a user by the hearing aid is furthermore provided by the present disclosure.
  • use of a monaural speech intelligibility predictor unit in a hearing aid in a noisy environment is provided (e.g. a car telephony situation, or other listening situation where a (e.g. substantially clean version of the) target speech signal is received wirelessly and acoustic noise is present at the user's ears) to enhance a user's intelligibility of speech in a noisy environment.
  • use of a monaural speech intelligibility predictor unit in an active ear protection device is provided.
  • a method of providing a monaural speech intelligibility predictor :
  • a method of providing a monaural speech intelligibility predictor for estimating a user's ability to understand an information signal x comprising a noisy and/or processed version of a target speech signal comprises
  • the method comprises subjecting a speech signal (a signal comprising speech) to a hearing loss model configured to model imperfections of an impaired auditory system to thereby provide said information signal x .
  • a speech signal e.g. signal x ' in FIG. 3A
  • a hearing loss model HMM in FIG. 3A
  • the resulting information signal x can be used as an input to the speech intelligibility predictor (MSIP in FIG. 3A ), thereby providing a measure of the intelligibility of the speech signal for an unaided hearing impaired person.
  • the hearing loss model is a generalized model reflecting a hearing impairment of an average hearing impaired user.
  • the hearing loss model is configurable to reflect a hearing impairment of a particular user, e.g. including a frequency dependent hearing loss (deviation of a hearing threshold from a(n average) hearing threshold of a normally hearing person).
  • a speech signal e.g. signal y in FIG. 3C
  • a signal processing e.g. SPU in FIG. 3C
  • HMM hearing loss model
  • the resulting information signal x can be used as an input to the speech intelligibility predictor (cf. e.g. MSIP in FIG.
  • Such scheme may e.g. be used to evaluate the influence of different processing algorithms (and/or modifications of processing algorithms) on the user's (estimated) intelligibility of the resulting information signal (cf. e.g. FIG. 3B ) or be used to online optimization of signal processing in a hearing aid (cf. e.g. 3C).
  • the method comprises adding noise to a target speech signal to provide said information signal x , which is used as input to the method of providing a monaural speech intelligibility predictor value.
  • the addition of a predetermined (or varying) amount of noise to an information signal can be used to - in a simple way - emulate a hearing loss of a user (to provide the effect of a hearing loss model).
  • the target signal is modified (e.g. attenuated) according to the hearing loss of a user, e.g. an audiogram.
  • noise is added to a target signal AND the target signal is attenuated to reflect a hearing loss of a user.
  • a binaural hearing (aid) system :
  • a (first) binaural hearing system comprising left and right hearing aids as described above, in the detailed description of embodiments and drawings and in the claims is furthermore provided.
  • each of the left and right hearing aids comprises antenna and transceiver circuitry for allowing a communication link to be established and information to be exchanged between said left and right hearing aids.
  • the binaural hearing system further comprises a binaural speech intelligibility prediction unit for providing a final binaural speech intelligibility measure d binaural of the predicted speech intelligibility of the user, when exposed to said sound input, based on the monaural speech intelligibility predictor values d left , d right of the respective left and right hearing aids.
  • the binaural hearing system is adapted to activate such approach when an asymmetric listening situation is detected or selected by the user, e.g. a situation where a speaker is located predominantly to one side of the user wearing the binaural hearing system, e.g. when sitting in a car.
  • the respective configurable signal processing units of the left and right hearing aids are adapted to control or influence the processing of the respective electric input signals based on said final binaural speech intelligibility measure d binaural . In an embodiment, the respective configurable signal processing units of the left and right hearing aids are adapted to control or influence the processing of the respective electric input signals to maximize said final binaural speech intelligibility measure d binaural .
  • the binaural hearing system further comprises an auxiliary device.
  • the system is adapted to establish a communication link between the hearing aid(s) and the auxiliary device to provide that information (e.g. control and status signals, possibly audio signals) can be exchanged or forwarded from one to the other.
  • information e.g. control and status signals, possibly audio signals
  • the auxiliary device is or comprises a remote control for controlling functionality and operation of the hearing aid(s).
  • the function of a remote control is implemented in a SmartPhone, the SmartPhone possibly running an APP allowing to control the functionality of the audio processing device via the SmartPhone (the hearing aid(s) comprising an appropriate wireless interface to the SmartPhone, e.g. based on Bluetooth or some other standardized or proprietary scheme).
  • a non-transitory application termed an APP
  • the APP comprises executable instructions configured to be executed on an auxiliary device to implement a user interface for a hearing aid or a hearing system described above in the 'detailed description of embodiments', and in the claims.
  • the APP is configured to run on cellular phone, e.g. a smartphone, or on another portable device allowing communication with said hearing aid or said hearing system.
  • a computer readable medium :
  • a tangible computer-readable medium storing a computer program comprising program code means for causing a data processing system to perform at least some (such as a majority or all) of the steps of any one of the methods described above, in the 'detailed description of embodiments' and in the claims, when said computer program is executed on the data processing system is furthermore provided by the present application.
  • Such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
  • Disk and disc includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
  • the computer program can also be transmitted via a transmission medium such as a wired or wireless link or a network, e.g. the Internet, and loaded into a data processing system for being executed at a location different from that of the tangible medium.
  • a transmission medium such as a wired or wireless link or a network, e.g. the Internet
  • a data processing system :
  • a data processing system comprising a processor and program code means for causing the processor to perform at least some (such as a majority or all) of the steps of the any one of the methods described above, in the 'detailed description of embodiments' and in the claims is furthermore provided by the present application.
  • a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out (steps of) the method described above, in the 'detailed description of embodiments' and in the claims is furthermore provided by the present application.
  • a 'hearing aid' refers to a device, such as e.g. a hearing instrument or an active ear-protection device or other audio processing device, which is adapted to improve, augment and/or protect the hearing capability of a user by receiving acoustic signals from the user's surroundings, generating corresponding audio signals, possibly modifying the audio signals and providing the possibly modified audio signals as audible signals to at least one of the user's ears.
  • a 'hearing aid' further refers to a device such as an earphone or a headset adapted to receive audio signals electronically, possibly modifying the audio signals and providing the possibly modified audio signals as audible signals to at least one of the user's ears.
  • Such audible signals may e.g.
  • acoustic signals radiated into the user's outer ears acoustic signals transferred as mechanical vibrations to the user's inner ears through the bone structure of the user's head and/or through parts of the middle ear as well as electric signals transferred directly or indirectly to the cochlear nerve of the user.
  • the hearing aid may be configured to be worn in any known way, e.g. as a unit arranged behind the ear with a tube leading radiated acoustic signals into the ear canal or with a loudspeaker arranged close to or in the ear canal, as a unit entirely or partly arranged in the pinna and/or in the ear canal, as a unit attached to a fixture implanted into the skull bone, as an entirely or partly implanted unit, etc.
  • the hearing aid may comprise a single unit or several units communicating electronically with each other.
  • a hearing aid comprises an input transducer for receiving an acoustic signal from a user's surroundings and providing a corresponding input audio signal and/or a receiver for electronically (i.e. wired or wirelessly) receiving an input audio signal, a (typically configurable) signal processing circuit for processing the input audio signal and an output means for providing an audible signal to the user in dependence on the processed audio signal.
  • an amplifier may constitute the signal processing circuit.
  • the signal processing circuit typically comprises one or more (integrated or separate) memory elements for executing programs and/or for storing parameters used (or potentially used) in the processing and/or for storing information relevant for the function of the hearing aid and/or for storing information (e.g. processed information, e.g.
  • the output means may comprise an output transducer, such as e.g. a loudspeaker for providing an air-borne acoustic signal or a vibrator for providing a structure-borne or liquid-borne acoustic signal.
  • the output means may comprise one or more output electrodes for providing electric signals.
  • the vibrator may be adapted to provide a structure-borne acoustic signal transcutaneously or percutaneously to the skull bone.
  • the vibrator may be implanted in the middle ear and/or in the inner ear.
  • the vibrator may be adapted to provide a structure-borne acoustic signal to a middle-ear bone and/or to the cochlea.
  • the vibrator may be adapted to provide a liquid-borne acoustic signal to the cochlear liquid, e.g. through the oval window.
  • the output electrodes may be implanted in the cochlea or on the inside of the skull bone and may be adapted to provide the electric signals to the hair cells of the cochlea, to one or more hearing nerves, to the auditory cortex and/or to other parts of the cerebral cortex.
  • a 'hearing system' refers to a system comprising one or two hearing aids
  • a 'binaural hearing system' refers to a system comprising two hearing aids and being adapted to cooperatively provide audible signals to both of the user's ears.
  • Hearing systems or binaural hearing systems may further comprise one or more 'auxiliary devices', which communicate with the hearing aid(s) and affect and/or benefit from the function of the hearing aid(s).
  • Auxiliary devices may be e.g. remote controls, audio gateway devices, mobile phones (e.g. SmartPhones), public-address systems, car audio systems or music players.
  • Hearing aids, hearing systems or binaural hearing systems may e.g. be used for compensating for a hearing-impaired person's loss of hearing capability, augmenting or protecting a normal-hearing person's hearing capability and/or conveying electronic audio signals to a person.
  • the electronic hardware may include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure.
  • Computer program shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.
  • the present application relates to the field of hearing aids or hearing aid systems.
  • the present disclosure relates to signal processing methods for predicting the intelligibility of speech, e.g., the output signal of a signal processing device such as a hearing aid.
  • the intelligibility prediction is made in the form of an index that correlates highly with the fraction of words that an average listener would be able to understand from some speech material. For situations where an estimate of absolute intelligibility, i.e., the actual percentage of words understood, is desired, this index may be transformed to a number in the range 0-100 percent, see e.g. [3] for one method to do this.
  • the method proposed here belongs to the class of so-called intrusive methods. Methods in this class are characterized by the fact that they make their intelligibility prediction by comparing the noisy - and potentially signal processed - speech signal, with a noise-free, undistorted version of the underlying speech signal, see [1, 2, 3] for examples of existing methods.
  • the assumption that a noise-free reference signal is available is reasonable in many practically relevant situations.
  • the stimuli are often created artificially by explicitly adding noise signal to noise-free speech signals - in other words, noise-free signals are readily available.
  • the proposed intelligibility prediction algorithm allows one to replace a costly and time-consuming listening test involving human subjects, with machine predictions.
  • Much of the signal processing of the present disclosure is performed in the time-frequency domain, where a time domain signal is transformed into the (time-)frequency domain by a suitable mathematical algorithm (e.g. a Fourier transform algorithm) or filter (e.g. a filter bank).
  • a suitable mathematical algorithm e.g. a Fourier transform algorithm
  • filter e.g. a filter bank
  • FIG. 1A schematically shows a time variant analogue signal (Amplitude vs time) and its digitization in samples, the samples being arranged in a number of time frames, each comprising a number N s of digital samples.
  • FIG. 1A shows an analogue electric signal (solid graph), e.g. representing an acoustic input signal, e.g. from a microphone, which is converted to a digital audio signal in an analogue-to-digital (AD) conversion process, where the analogue signal is sampled with a predefined sampling frequency or rate f s , f s being e.g.
  • Each (audio) sample x(n) represents the value of the acoustic signal at n by a predefined number N b of bits, N b being e.g. in the range from 1 to 16 bits.
  • a number of (audio) samples N s are arranged in a time frame, as schematically illustrated in the lower part of FIG. 1A , where the individual (here uniformly spaced) samples are grouped in time frames (1, 2, ..., N s )).
  • the time frames may be arranged consecutively to be non-overlapping (time frames 1, 2, ..., m, ..., M) or overlapping (here 50%, time frames 1, 2, ..., m, ..., M'), where m is time frame index.
  • a time frame comprises 64 audio data samples. Other frame lengths may be used depending on the practical application.
  • FIG. 1B schematically illustrates a time-frequency representation of the (digitized) time variant electric signal x(n) of FIG. 1A .
  • the time-frequency representation comprises an array or map of corresponding complex or real values of the signal in a particular time and frequency range.
  • the time-frequency representation may e.g. be a result of a Fourier transformation converting the time variant input signal x(n) to a (time variant) signal x(k,m) in the time-frequency domain.
  • the Fourier transformation comprises a discrete Fourier transform algorithm (DFT).
  • DFT discrete Fourier transform algorithm
  • the frequency range considered by a typical hearing device e.g.
  • a time frame is defined by a specific time index m and the corresponding K DFT-bins (cf. indication of Time frame m in FIG. 1B ).
  • a time frame m represents a frequency spectrum of signal x at time m.
  • a DFT-bin (k,m) comprising a (real) or complex value x(k,m) of the signal in question is illustrated in FIG. 1B by hatching of the corresponding field in the time-frequency map.
  • Each value of the frequency index k corresponds to a frequency range ⁇ f k , as indicated in FIG. 1B by the vertical frequency axis f .
  • Each value of the time index m represents a time frame.
  • the time ⁇ t m spanned by consecutive time indices depend on the length of a time frame (e.g. 25 ms) and the degree of overlap between neighbouring time frames (cf. horizontal t -axis in FIG. 1B ).
  • each sub-band comprising one or more DFT-bins (cf. vertical Sub-band j -axis in FIG. 1B ).
  • the j th sub-band (indicated by Sub-band j ( x j (m) ) in the right part of FIG. 1B ) comprises DFT-bins with lower and upper indices k1(j) and k2(j), respectively, defining lower and upper cut-off frequencies of the j th sub-band, respectively.
  • a specific time-frequency unit (j,m) is defined by a specific time index m and the DFT-bin indices k1(j)-k2(j), as indicated in FIG. 1B by the bold framing around the corresponding DFT-bins.
  • a specific time-frequency unit (j,m) contains complex or real values of the j th sub-band signal x j (m) at time m.
  • FIG. 2A symbolically shows an intrusive monaural speech intelligibility predictor unit (MSIP) providing a monaural speech intelligibility predictor d based on either
  • MSIP intrusive monaural speech intelligibility predictor unit
  • FIG. 2B shows a first embodiment an intrusive monaural speech intelligibility predictor unit (MSIP).
  • the intrusive monaural speech intelligibility predictor unit (MSIP) is adapted for receiving a target signal comprising speech in an essentially noise-free version s(n) and in a noisy and/or processed version x(n) , where n is a time index.
  • the monaural speech intelligibility predictor unit is configured to provide as an output a final monaural speech intelligibility predictor value d indicative of a listener's (user's) perception of the noisy and/or processed version x of the target signal.
  • AEU envelope extraction unit
  • AEU second envelope extraction unit
  • the monaural speech intelligibility predictor unit further comprises a first time-frequency segment division unit ( SDU ) for dividing said time-frequency sub-band representation s j (m) of the noise-free version s of the target signal into time-frequency segments S m corresponding to a number N of successive samples of the sub-band signals s j (m), and a second time-frequency segment division unit ( SDU ) for dividing the time-frequency sub-band representation x j (m) of the noisy and/or processed version x of the target signal into time-frequency segments X m corresponding to a number N of successive samples of the sub-band signals x j (m) .
  • the monaural speech intelligibility predictor unit further optionally comprises a first normalization and/or transformation unit ( N / TU ) adapted for providing normalized and/or transformed versions S ⁇ m of the time-frequency segments S m , and optionally a second normalization and/or transformation unit ( N / TU ) adapted for providing normalized and/or transformed versions X ⁇ m of the time-frequency segments X m .
  • the monaural speech intelligibility predictor unit further comprises an intermediate speech intelligibility calculation unit (ISIU ) adapted for providing intermediate speech intelligibility coefficients d m estimating an intelligibility of the time-frequency segment X m , wherein the intermediate speech intelligibility coefficients d m are based on the essentially noise-free, optionally normalized and/or transformed, time frequency segments S m , S ⁇ m , and the noisy and/or processed, optionally normalized and/or transformed, time-frequency segments X m , X ⁇ m .
  • ISIU intermediate speech intelligibility calculation unit
  • the monaural speech intelligibility predictor unit further comprises a final monaural speech intelligibility calculation unit (FSIU ) for calculating a final monaural speech intelligibility predictor d estimating an intelligibility of the noisy and/or processed version x of the target signal by combining, e.g. by averaging or applying a MIN or MAX-function, the intermediate speech intelligibility coefficients d m , or a transformed version thereof, over time.
  • FSIU final monaural speech intelligibility calculation unit
  • FIG. 2C shows a second embodiment an intrusive monaural speech intelligibility predictor unit (MSIP).
  • the embodiment of FIG. 2C comprises the same functional units as decribed in connection with FIG. 2B . Additionally, it comprises a voice activity detector unit ( VAD ) for indicating whether or not or to what extent a given time-segment of the essentially noise-free version s ( s'(n) in FIG. 2C ) and the noisy and/or processed version x ( x'(n) in FIG. 2C ), respectively, of the target signal comprises or is estimated to comprise speech, and providing a voice activity control signal indicative thereof.
  • VAD voice activity detector unit
  • the voice activity detector unit ( VAD ) itself is configured to provide modified versions of the essentially noise-free version s and the noisy and/or processed version x , respectively, of the target signal comprising only time segments comprising speech or being estimated to comprise speech (in FIG. 2C denoted s(n) and x(n) respectively).
  • the modified signals s(n) and x(n) may be created in respective separation units.
  • an optional hearing loss model is included (cf. FIG. 3A, 3B, 3C ).
  • a hearing loss model builds (at least) on an audiogram containing frequency dependent hearing thresholds of a user (or representative of a type of hearing loss).
  • The, perhaps, simplest hearing loss model consists of adding to the input signal x(n) a statistically independent noise signal, which is spectrally shaped according to the audiogram of the listener [5].
  • the proposed monaural, intrusive speech intelligibility predictor may be decomposed into a number of sub-stages as illustrated in FIG. 2B and 2C and discussed above. Each sub-stage is described in more detail in the following.
  • VAD Voice Activity Detection
  • Speech intelligibility relates to regions of the input signal with speech activity - silence regions do no contribute to SI.
  • the first step is to detect voice activity regions in the input signals. Since the noise-free speech signal s'(n) is available, voice activity is trivial. For example, in [3] the noise-free speech signal s'(n) was divided into successive frames. Speech-active frames were then identified as the ones with a frame-energy no less than e.g. 40 dB of the frame with maximum energy. The speech inactive frames, i.e., the ones with energy less than e.g., 40 dB of the maximum frame energy, are then discarded from both signals, x'(n) and s'(n). Let us denote the input signals with speech activity by x(n) and s(n), respectively, where n is a discrete-time index. A voice activity detector is shown in FIG. 2C as unit VAD.
  • the first step is to perform a frequency decomposition (cf. input unit IU in FIG. 2C ) of the signals x(n) and s(n) .
  • a frequency decomposition cf. input unit IU in FIG. 2C
  • This may be achieved in many ways, e.g., using a short-time Fourier transform (STFT), a band-pass filterbank (e.g., a Gamma-tone filter bank), etc.
  • STFT short-time Fourier transform
  • a band-pass filterbank e.g., a Gamma-tone filter bank
  • the temporal envelopes of each sub-band signal are extracted (cf. unit AEU in FIG. 2C ). This may, e.g., be achieved using a Hilbert transform, or by low-pass filtering the magnitude of complex-valued STFT signals, etc.
  • each frame is Fourier transformed using a fast Fourier transform (FFT) (potentially after appropriate zero-padding).
  • FFT fast Fourier transform
  • the resulting DFT bins may be grouped in perceptually relevant sub-bands. For example, one could use one-third octave bands (e.g. as in [4]), but it should be clear that any other sub-band division can be used (for example, the grouping could be uniform, i.e., unrelated to perception in this respect, cf. FIG. 1B ). In the case of one-third octave bands and a sampling rate of 10000 Hz, there are 15 bands which cover the frequency range 150-5000 Hz. Other numbers of bands and another frequency range can be used.
  • time-frequency tiles defined by these frames and sub-bands as time-frequency (TF) units (or STFT coefficients), cf. FIG. 1B .
  • TF time-frequency
  • STFT coefficients cf. FIG. 1B .
  • envelope representations may be implemented, e.g., using a Gammatone filterbank, followed by a Hilbert envelope extractor, etc., and functions f(x) may be applied to these envelopes in a similar manner as described above for STFT based envelopes.
  • the result of this procedure is a time-frequency representation in terms of sub-band temporal envelopes, x j (m) and s j (m), where j is a sub-band index, and m is a time index.
  • the time-frequency representations x j (m) and s j (m) into segments, i.e., spectrograms corresponding to N successive samples of all sub-band signals.
  • the corresponding segment S m for the noise-free reference signal is found in an identical manner.
  • time-segments could be used, e.g., segments, which have been shifted in time to operate on frame indices m - N /2 + 1 through m + N /2 .
  • each segment X m and S m may be normalized/transformed in various ways (below, we show the normalizations/transformations as applied to X m ; they are applied to S m in a completely analogously manner. The same normalization/transformation is applied to both X m and S m ). In particular, we consider the following row (R) normalizations/transformations
  • X ⁇ m g 2 g 1 h 2 h 1 X m (mean- and norm- standardization of the columns followed by mean- and norm-standardization of the rows), etc.
  • X ⁇ m g 2 g 1 h 2 h 1 X m (mean- and norm- standardization of the columns followed by mean- and norm-standardization of the rows), etc.
  • a particular combination of row- and column-normalizations/transformations is chosen and applied to all segments X m and S m of the noisy/processed and noise-free signal, respectively.
  • ISIU Intermediate Intelligibility Coefficients
  • the time-frequency segments S m or the normalized/transformed time-frequency segments S ⁇ m of the noise-free reference signal may now be used together with the corresponding noisy/processed segments X m , X ⁇ m to compute an intermediate intelligibility index d m , reflecting the intelligibility of the noisy/processed signal segment X m , X ⁇ m .
  • d m may be defined as
  • FIG. 3A shows an intrusive monaural speech intelligibility predictor unit in combination with a hearing loss model (HLM) and an evaluation unit (MSIP) (together constituting a modified monaural speech intelligibility predictor unit ( MSIP ')).
  • HLM hearing loss model
  • MSIP evaluation unit
  • the signal x'(n) is passed through hearing loss model (HLM) configured to model the imperfections of an impaired auditory system (e.g. the impaired auditory system of a particular user).
  • the hearing loss model unit ( HLM ) is e.g. based on an audiogram of an ear of a user (and possible other data related to a user's hearing ability).
  • the hearing loss model unit ( HLM ) is e.g. configured to apply a frequency dependent modification of the noisy and/or processed version x ' of the target signal reflecting a deviation from normal hearing, e.g.
  • an evaluation unit is shown to receive and evaluate the speech intelligibility predictor d and provide a processed predictor d' .
  • the evaluation unit (EVAL) may e.g. further process the speech intelligibility predictor value d, to e.g. graphically and/or numerically display the current and/or recent historic values, derive trends, etc.
  • the evaluation unit may propose actions to the user (or a communication partner or caring person), such as add directionality, move closer, speak louder, activate SI-enhancement mode, etc.
  • the evaluation unit may e.g. be implemented in a separate device, e.g. acting as a user interface to the speech intelligibility predictor unit (MSIP) and/or to a hearing aid including such unit., e.g. implemented as a remote control devise, e.g. as an APP of a smartphone (cf. FIG. 6A, 6B ).
  • FIG. 3B shows an intrusive monaural speech intelligibility predictor unit ( MSIP' ) in combination with a signal processing unit ( SPU ) and an evaluation unit (EVAL), e.g. of a hearing device.
  • the embodiment of FIG. 3B additionally comprises a number of input units (here 2: M1, M2 ) , e.g. microphones, for providing a time-variant electric input signal representing a sound input received at the input unit in question.
  • At least one (such as each) of the electric input signals comprises a target signal component (e.g. a speech component) and a noise signal component (termed Noisy target in FIG. 3B ).
  • the target signal component is assumed to originate from a target signal source in the environment of the device (e.g. a hearing device, see FIG. 3C ).
  • the embodiment of FIG. 3B further comprises a configurable signal processing unit (SPU) for processing the electric input signals (e.g. providing beamforming and/or noise reduction, frequency and level dependent amplification, level dependent compression, or the like) and providing a processed signal x' based on one or more of the electric input signals, which are inputs to the configurable signal processing unit (SPU).
  • the processed signal x ' from the configurable signal processing unit (SPU) is fed to the hearing loss model ( HLM ) unit of the monaural speech intelligibility predictor unit ( MSIP' ) .
  • HLM hearing loss model
  • MSIP' monaural speech intelligibility predictor unit
  • the hearing loss model unit comprises a model of a hearing loss of a user (e.g. the user of the device) and is configured to shape an input signal to provide an output signal x representing a processed (possibly hearing loss compensated) and (again) deteriorated signal, which is fed to the monaural speech intelligibility predictor (MSIP).
  • the embodiment of FIG. 3B further comprises an antenna and transceiver unit (Rx) for receiving a wireless signal (termed Clean target in FIG. 3B ) comprising the target signal and for extracting an essentially noise-free version s of the target signal, which is connected to the monaural speech intelligibility predictor (MSIP).
  • the final speech intelligibility predictor d from the monaural speech intelligibility predictor unit (MSIP') is fed to the evaluation unit (EVAL) whose modified predictor value d' is fed to the configurable signal processing unit (SPU).
  • the configurable signal processing unit (SPU) is adapted to control or influence the processing of the respective electric input signals based on the final speech intelligibility predictor d provided by the monaural speech intelligibility predictor unit and as modified by the evaluation unit (EVAL).
  • the configurable signal processing unit (SPU) is adapted to control or influence the processing of the respective electric input signals to maximize the final speech intelligibility predictor d . (e.g. controlled by the evaluation unit ( EVAL (max) ) .
  • the embodiment of FIG. 3B may e.g. further comprise an output unit for creating output stimuli configured to be perceivable by the user as sound based on an electric output either in the form of the processed signal x ' from the signal processing unit or a signal derived therefrom.
  • the output unit (cf. e.g. OT in FIG. 3C ) may e.g. comprise a loudspeaker for placement in an ear canal of a user, or a vibrator for being attached to the skull of a user, or electrodes for placement in cochlea of a user.
  • a hearing aid according to the present disclosure is provided.
  • the hearing aid may take the form or an air conducting hearing instrument, a bone-conducting hearing instrument, a cochlear implant prosthesis, an active ear-protection device, a headset, an earphone with active noise cancellation, etc.
  • FIG. 3C shows a first embodiment of a hearing device ( HD, e.g. a hearing aid) comprising an intrusive monaural speech intelligibility predictor unit (MSIP' ) comprising a hearing loss model part ( HLM ) and a predictor part (MSIP) configured to optimize a user's intelligibility (represented by index d ) of an output signal u of the hearing device ( HD ).
  • the embodiment of FIG. 3C is equivalent to the embodiment of FIG. 3B but further comprises an output unit comprising an output transducer ( OT ) in the form of a loudspeaker, which is directly connected to the output u of the signal processing unit (SPU).
  • 3C only comprises one the input unit (IT) comprising a microphone for picking up a noisy representation y' of the target signal hearing aid and converting it to an electric input signal y, which is fed to the configurable signal processing unit (SPU).
  • the antenna and transceiver unit (Rx) is adapted for receiving a wireless signal (termed s' in FIG. 3C ) comprising the target signal and for extracting an essentially noise-free version s of the target signal, which is fed to the predictor part (MSIP) of the intrusive monaural speech intelligibility predictor unit ( MSIP' ).
  • the monaural speech intelligibility predictor unit ( MSIP' ) provides an estimate of the intelligibility of the output signal by the user in the form of the (final) speech intelligibility predictor d, which is fed to a control part of the configurable signal processing unit (SPU) to modify signal processing to optimize d . in a feedback loop.
  • SPU configurable signal processing unit
  • FIG. 4A shows a first scenario for using a hearing aid (HD) comprising an intrusive monaural speech intelligibility predictor according to the present disclosure (as described in connection with FIG. 2A, 2B, 2C and FIG. 3A, 3B, 3C above) to improve a hearing aid user's ( U ) intelligibility of speech from a speaker ( TLK ) wearing a wireless microphone ( M ) , e.g. in a teaching or lecture situation.
  • the speaker's voice (the target signal) is picked up by the microphone ( M ) located close to the speaker's mouth.
  • the microphone ( M ) comprises a transmitter ( Tx ) for wirelessly transmitting the essentially noise-free version of the target signal s to a corresponding receiver ( Rx ) of the hearing aid worn by the user ( U ) using wireless link WLS (e.g. using FM or Bluetooth or other standardized or proprietary technology). Simultaneously, an acoustically propagated version of the target signal coloured (modified) by the location (e.g. a room with reflecting surfaces, e.g. walls) and mixed with possible noise (noise) from the environment is picked up (noisy signal x ) by one or more microphone of the hearing aid (HD).
  • WLS wireless link
  • an acoustically propagated version of the target signal coloured (modified) by the location e.g. a room with reflecting surfaces, e.g. walls
  • mixed with possible noise (noise) from the environment is picked up (noisy signal x ) by one or more microphone of the hearing aid (HD).
  • FIG. 4B shows a second (similar) scenario for using a hearing aid (HD) comprising an intrusive monaural speech intelligibility predictor according to the present disclosure to improve a hearing aid user's ( U ) intelligibility of speech from a remote speaker of a telephone conversation using a handsfree telephone set in a car ( CAR ) , where remote sound is wirelessly as well as acoustically transmitted to the hearing aid user.
  • the 'clean' target signal of the remote speaker received by the telephone ( CELL PHONE ) is wirelessly transmitted to and received by a receiver ( Rx ) of the hearing aid (HD) as clean signal s ( Clean target s in FIG.
  • a microphone (IT) of the hearing aid by a loudspeaker ( LOUDSPEAKER ) of the handsfree telephone set providing 'noisy signal' x ( noisy target x in FIG. 4B ).
  • the user is driving a car ( CAR, WHEEL ) while talking in a telephone via a handsfree telephone set.
  • the noise in the car cabin (indicated by noise sources N1, N2) is acoustically mixed with the 'target signal' played by a loudspeaker of the handsfree telephone set (or of the telephone itself) and picked up as noisy target signal x . by the hearing aid microphone(s) (IT).
  • the hearing aid (HD) used in the two scenarios of FIG. 4A, 4B may be a hearing aid according to the present disclosure, e.g. as described in connection with FIG. 3A, 3B, 3C , which is configured to adapt the processing of an acoustic signal picked up by a microphone of the hearing aid and processed by a signal processing device to optimize the user's speech intelligibility (based on a predictor of a monaural speech intelligibility predictor unit, as proposed by the present disclosure).
  • FIG. 5A shows a third scenario for using a hearing aid (HD) comprising an intrusive monaural speech intelligibility predictor according to the present disclosure to improve a hearing aid user's intelligibility of speech from a remote speaker of a telephone conversation using a handsfree telephone set in a car ( CAR, WHEEL ) , where sound from a remote communication partner is wirelessly transmitted to the hearing aid user ( U ).
  • the scenario of FIG. 5A is similar to the scenario of FIG. 4B apart from the fact that in the scenario of FIG. 5A the voice of the remote communication partner is NOT played by a loudspeaker in the car.
  • the clean target signal s is transmitted from the CELL PHONE to the hearing aid HD.
  • the background noise v' Noise v'
  • the background noise v' Noise v'
  • the background noise v' as captured is substantially equal to the noise v ed (Noise v ed ) that is present at the ear drum ( Ear drum ) of the user (cf. FIG. 5B , 5C ).
  • the assumption is of course better the closer to the ear drum the microphone is situated (and/or the more open the ear canal part is).
  • a microphone of the hearing aid is located in the ear canal, e.g. at the entrance of the ear canal or close to the ear drum (cf. e.g. IT 3 in FIG. 5C ).
  • FIG. 5B shows an embodiment of a hearing aid comprising an intrusive monaural speech intelligibility predictor for use in the (third) scenario of FIG. 5A .
  • the embodiment of a hearing aid ( HD ) shown in FIG. 5B comprises the same functional components as the embodiment shown in FIG. 3C .
  • the interconnection of the signal processing unit ( SPU ) and the wireless receiver ( Rx ) and the input transducer ( IT ) is different, however.
  • SPU signal processing unit
  • Rx wireless receiver
  • IT input transducer
  • the sound output of the loudspeaker ( OT ) of the hearing aid equal to the processed signal f(s) from the signal processing unit ( SPU ) , is acoustically mixed with 'environmental' (car cabin) noise v ed at the eardrum ( Ear drum, cf. Mixture of s and v ed in FIG. 5B ).
  • the basic idea of the embodiment of a hearing aid in FIG. 5B is to process the clean version s of the target signal so that the speech intelligibility d is maximized when the processed version of the clean target signal ( f(s) ) .
  • the processed version of the clean target signal ( f(s) ) can be adaptively controlled, whereas this is not the case for the car cabin noise v ed at the eardrum (which is given).
  • the loudspeaker (or alternatively an acoustic guide element) is located in the ear canal, preferably close to the ear drum to deliver the processed signal f(s) to the ear drum.
  • the microphone(s) of the hearing device which is(are) used to pick up background noise v' (cf. FIG. 5A, 5B ), is(are) located close to the ear drum, or at the entrance of the ear canal, or in pinna, or behind the ear.
  • the signal processing unit ( SPU ) is configured to iteratively modify signal processing of the clean target signal s received from wireless receiver unit ( Rx ) to provide processed version f(s) the clean target signal s that optimizes speech intelligibility of the (mixed) signal present at the ear drum of the user (in practice here approximated by maximizing the monaural speech intelligibility predictor d(f(s) + v,s) according to the resent disclosure).
  • SNR signal to noise ratio
  • reliance only on increasing gain of the clean target signal may, however, not be attractive or possible (e.g. due to acoustic feedback problems, maximum power output limitations of the loudspeaker, or uncomfortable levels of the user, etc.).
  • an appropriate frequency dependent shaping of the clean target signal is generally proposed and governed by the monaural speech intelligibility predictor (including the hearing loss model ( HLM ) preferably defining decisive aspects of a hearing impairment of the user of the hearing aid).
  • FIG. 5C illustrates an exemplary hearing aid ( HD ) formed as a receiver in the ear (RITE) type of hearing aid comprising a part ( BTE ) adapted for being located behind pinna and a part ( ITE ) comprising an output transducer (OT, e.g. a loudspeaker/receiver) adapted for being located in an ear canal ( Ear canal ) of the user (e.g. exemplifying a hearing aid (HD) as shown in FIG. 5A, 5B ).
  • the BTE-part ( BTE ) and the ITE-part ( ITE ) are connected (e.g. electrically connected) by a connecting element ( IC ) .
  • IC connecting element
  • the BTE part ( BTE ) comprises two input units comprising two (individually selectable) input transducers (e.g. microphones) ( IT 1 , IT 2 ) each for providing an electric input audio signal representative of an input sound signal from the environment (in the scenario of FIG. 5A , from the car cabin).
  • the hearing device of FIG. 5C further comprises two (individually selectable) wireless receivers ( WLR 1 , WLR 2 ) for providing respective directly received auxiliary audio and/or information signals.
  • the hearing aid ( HD ) further comprises a substrate ( SUB ) whereon a number of electronic components are mounted, including a configurable signal processing unit ( SPU ) , a monaural speech intelligibility predictor unit (MSIP), and a hearing loss model unit ( HLM, coupled to each other and input and output units via electrical conductors Wx ).
  • the configurable signal processing unit ( SPU ) provides an enhanced audio signal (cf. signal f(s) in FIG. 5B ), which is intended to be presented to a user.
  • an enhanced audio signal cf. signal f(s) in FIG. 5B
  • the ITE part comprises an output unit in the form of a loudspeaker (receiver) ( OT ) for converting an electric signal ( f(s) in FIG. 5B ) to an acoustic signal.
  • the ITE-part further comprises an input unit comprising an input transducer (e.g. a microphone) (IT 3 ) for providing an electric input audio signal representative of an input sound signal from the environment in the ear canal (here approximating the noise v ed from the car cabin at the ear drum ( Ear drum ) of the user ( U ) wearing the hearing aid ( HD )) .
  • an input transducer e.g. a microphone
  • the hearing aid may comprise only the input unit ( IT 3 ) located in or at the ear canal, or the input unit ( IT 3 ) located in or at the ear canal in combination with a an input unit located elsewhere, e.g. in a BTE-part.
  • the ITE-part further comprises a guiding element, e.g. a dome, ( DO ) for guiding and positioning the ITE-part in the ear canal of the user.
  • the hearing aid (HD) exemplified in FIG. 5C is a portable device and further comprises a battery (BAT) for energizing electronic components of the BTE- and ITE-parts.
  • BAT battery
  • the hearing aid ( HD ) comprises a directional microphone system (beamformer) adapted to enhance a target acoustic source among a multitude of acoustic sources in the local environment of the user wearing the hearing aid device.
  • the directional system is adapted to detect (such as adaptively detect) from which direction a particular part of the microphone signal originates.
  • the hearing aid of FIG. 5C may form part of a hearing aid and/or a binaural hearing aid system according to the present disclosure (cf. e.g. FIG. 7 ).
  • FIG. 6A shows an embodiment of a binaural hearing system comprising left and right hearing devices ( HD left , HD right ) , e.g. hearing aids, in communication with a portable (handheld) auxiliary device ( Aux ) functioning as a user interface ( UI ) for the binaural hearing aid system (cf. FIG. 6B ).
  • the binaural hearing system comprises the auxiliary device ( Aux, and the user interface UI ) .
  • wireless links denoted IA-WL e.g. an inductive link between the left and right hearing devices
  • WL-RF e.g. RF-links
  • Bluetooth between the auxiliary device Aux and the left HD left , and between the auxiliary device Aux and the right HD right , hearing aid, respectively) are indicated (implemented in the devices by corresponding antenna and transceiver circuitry, indicated in FIG. 6A in the left and right hearing devices as RF-IA-Rx / Tx-l and RF-IA-RxlTx-r, respectively).
  • FIG. 6B shows the auxiliary device ( Aux ) comprising a user interface ( UI ) in the form of an APP for controlling and displaying data related to the speech intelligibility predictors.
  • the user interface ( UI ) comprises a display (e.g. a touch sensitive display) displaying a screen of a Speech intelligibility SI-APP for controlling the hearing aid system and presenting information to the user.
  • the APP comprises a number of predefined action buttons regarding functionality of the binaural (or monaural) hearing system.
  • a user ( U ) has the option of influencing a mode of operation via the selection of a SI-prediction mode to be a Monaural SIP or Binaural SIP mode.
  • the un-shaded buttons are selected, i.e. SI-enhancement mode (where the processing is adapted to optimize speech intelligibility based on the (monaural or binaural) speech intelligibility predictor) together with a specific Car telephony mode (as described in connection with FIG. 5A, 5B , 5C ).
  • SI-enhancement mode where the processing is adapted to optimize speech intelligibility based on the (monaural or binaural) speech intelligibility predictor
  • a specific Car telephony mode as described in connection with FIG. 5A, 5B , 5C .
  • a show Current SI-estimate has been activated (resulting in a current predicted value of the binaural speech intelligibility predictor being displayed (in the form of the positive indicator ' ') together with an indication of the current noise level (indicated as 'HIGH')).
  • the grey shaded button Lecture mode (as described in connection with FIG. 4A, 4B )
  • FIG. 7 shows an embodiment of a binaural hearing aid system according to the present disclosure comprising a left and right hearing devices ( HD left, HD right ) , each comprising a monaural speech intelligibility predictor unit ( MSIP' ) whose individual predictor values d left and d right are exchanged between the hearing devices and used to influence or control signal processing of respective signal processing units ( SPU ) in the hearing devices to optimize binaural speech intelligibility of the user.
  • the left and right hearing devices ( HD left , HD right ) are e.g. hearing devices as shown in an discussed in connection with FIG. 3C .
  • Each of the left and right hearing aids comprises antenna and transceiver circuitry ( IA-Rx / Tx ) for allowing a communication link ( IA-WLS ) to be established and information ( d left , d right ) to be exchanged between said left and right hearing aids.
  • the binaural hearing aid system comprises a binaural speech intelligibility prediction unit for providing a final binaural speech intelligibility measure d binaural of the predicted speech intelligibility of the user when exposed to a sound input, wherein the final binaural speech intelligibility measure d binaural is determined in dependence of the final monaural speech intelligibility predictor values d left, d right of the respective left and right hearing aids.
  • the binaural speech intelligibility prediction unit may e.g. be implemented in one or both of the signal processing units ( SPU ) of the left and right hearing devices.
  • connection or “coupled” as used herein may include wirelessly connected or coupled.
  • the term “and/or” includes any and all combinations of one or more of the associated listed items. The steps of any disclosed method is not limited to the exact order stated herein, unless expressly stated otherwise.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • General Health & Medical Sciences (AREA)
  • Neurosurgery (AREA)
  • Otolaryngology (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computational Linguistics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Quality & Reliability (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Animal Behavior & Ethology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Biomedical Technology (AREA)
  • Stereophonic System (AREA)
  • Circuit For Audible Band Transducer (AREA)

Claims (16)

  1. Monaurale Sprachverständlichkeitsvorhersageeinheit (monaural speech intelligibility predictor unit - MSIP), angepasst zum Empfangen eines Zielsignals, das Sprache umfasst, in einer im Wesentlichen rauschfreien Version s und in einer verrauschten und/oder verarbeiteten Version x, wobei die monaurale Sprachverständlichkeitsvorhersageeinheit dazu konfiguriert ist, einen finalen monauralen Sprachverständlichkeitsvorhersagewert d, der die Wahrnehmung eines Zuhörers der verrauschten und/oder verarbeiteten Version x des Zielsignals angibt, als eine Ausgabe auszugeben, wobei die monaurale Sprachverständlichkeitsvorhersageeinheit (MSIP) Folgendes umfasst
    • eine erste Eingabeeinheit (input unit - IU) zum Bereitstellen einer Zeit-Frequenz-Darstellung s(k,m) der rauschfreien Version s des Zielsignals, wobei k ein Frequenzbereichsabschnitt ist, k=1, 2, ..., K und m ein Zeitindex ist;
    • eine zweite Eingabeeinheit (IU) zum Bereitstellen einer Zeit-Frequenz-Darstellung x(k,m) der verrauschten und/oder verarbeiteten Version x des Zielsignals, wobei k ein Frequenzbereichsabschnitt ist, k=1, 2, ..., K und m ein Zeitindex ist;
    • eine erste Einhüllendenextraktionseinheit (envelope extraction unit - AEU) zum Bereitstellen einer Zeit-Frequenz-Teilbanddarstellung sj(m) der rauschfreien Version s des Zielsignals, die temporale Einhüllende, oder Funktionen davon, der Frequenz-Teilbandsignale sj(m) des rauschfreien Zielsignals darstellt, wobei j ein Frequenz-Teilbandindex ist, j=1, 2, ..., J and m der Zeitindex ist;
    • eine zweite Einhüllendenextraktionseinheit (AEU) zum Bereitstellen einer Zeit-Frequenz-Teilbanddarstellung xj(m) der verrauschten und/oder verarbeiteten Version x des Zielsignals, die temporale Einhüllende, oder Funktionen davon, der Frequenz-Teilbandsignale xj(m) der verrauschten und/oder verarbeiteten Version des Zielsignals darstellt, wobei j=1, 2, ..., J und m der Zeitindex ist;
    • eine erste Zeit-Frequenz-Segmentteilungseinheit (segment division unit - SDU) zum Teilen der Zeit-Frequenz-Teilbanddarstellung sj(m) der rauschfreien Version s des Zielsignals in Zeit-Frequenz-Segmente Sm entsprechend einer Anzahl N von aufeinanderfolgenden Abtastungen der Teilbandsignale;
    • eine zweite Zeit-Frequenz-Segmentteilungseinheit (SDU) zum Teilen der Zeit-Frequenz-Teilbanddarstellung xj(m) der verrauschten und/oder verarbeiteten Version x des Zielsignals in Zeit-Frequenz-Segmente Xm entsprechend einer Anzahl N von aufeinanderfolgenden Abtastungen der Teilbandsignale;
    wobei die erste und die zweite Zeit-Frequenz-Segmentteilungseinheit dazu konfiguriert sind, die Zeit-Frequenz-Darstellungen sj(m) bzw. xj(m) in Segmente in der Form von Spektrogrammen entsprechend N aufeinanderfolgenden Abtastungen von allen Teilbandsignalen zu teilen, wobei das m. Segment durch die Matrix JxN definiert ist, Z m = z 1 m − N + 1 ⋯ z 1 m ⋮ ⋮ z J m − N + 1 ⋯ z J m ,
    Figure imgb0078
     wobei z, Z für s, S bzw. x, X steht;
    • eine Normalisierungs- und Umwandlungseinheit (normalization and transformation unit - N/TU), die dazu konfiguriert ist, zumindest einen Normalisierungs- und/oder Umwandlungsbetrieb von Zeilen und zumindest einen Normalisierungs- und/oder Umwandlungsbetrieb von Spalten der Zeit-Frequenz-Segmente Sm und Xm bereitzustellen;
    • eine Zwischensprachverständlichkeitsberechnungseinheit (intermediate speech intelligibility calculation unit - ISIU), die dazu angepasst ist, Zwischensprachverständlichkeitskoeffizienten dm bereitzustellen, die eine Verständlichkeit des Zeit-Frequenz-Segments Xm schätzen, wobei die Zwischensprachverständlichkeitskoeffizienten dm auf im wesentlichen rauschfreien, normalisierten und/oder umgewandelten Zeit-Frequenz-Segmenten S̃m und den verrauschten und/oder verarbeiteten, normalisierten und/oder umgewandelten Zeit-Frequenz-Segmenten X̃m basieren;
    • eine finale monaurale Sprachverständlichkeitsberechnungseinheit (final monaural speech intelligibility calculation unit - FSIU) zum Berechnen eines finalen monauralen Sprachverständlichkeitsvorhersagers d, der eine Verständlichkeit der verrauschten und/oder verarbeiteten Version x des Zielsignals durch Kombinieren, z. B. durch Mitteln von, oder durch Anwenden einer MIN- oder MAX-Funktion auf, die Zwischensprachverständlichkeitskoeffizienten dm , oder eine umgewandelte Version davon, im Zeitverlauf schätzt.
  2. Monaurale Sprachverständlichkeitsvorhersageeinheit (MSIP) nach Anspruch 1, umfassend eine Stimmaktivitätsdetektoreinheit zum Angeben, ob oder ob nicht und in welchem Maße ein gegebenes Zeitsegment der im Wesentlichen rauschfreien Version s bzw. der verrauschten und/oder verarbeiteten Version x des Zielsignals Sprache umfasst oder als Sprache umfassend eingeschätzt ist, und zum Bereitstellen eines Stimmsteuersignals, das dies angibt.
  3. Monaurale Sprachverständlichkeitsvorhersageeinheit (MSIP) nach Anspruch 1 oder 2, umfassend eine Stimmaktivitätsdetektoreinheit zum Identifizieren von Zeitsegmenten der im Wesentlichen rauschfreien Version s bzw. der verrauschten und/oder verarbeiteten Version x des Zielsignals, die Sprache umfasst oder als Sprache umfassend eingeschätzt ist, und wobei die monaurale Sprachverständlichkeitsvorhersageeinheit dazu konfiguriert ist, modifizierte Versionen der im Wesentlichen rauschfreien Version s bzw. der verrauschten und/oder verarbeiteten Version x des Zielsignals bereitzustellen, wobei die modifizierten Versionen nur derartige Zeitsegmente umfassend, die Sprache umfassend und für die eingeschätzt ist, dass sie Sprache umfassen.
  4. Monaurale Sprachverständlichkeitsvorhersageeinheit (MSIP) nach einem der Ansprüche 1-3, umfassend eine Hörverlustmodelleinheit, die dazu konfiguriert ist, eine Modifizierung der verrauschten und/oder verarbeiteten Version x des Zielsignals, die eine Abweichung vom normalen Hören eines relevanten Ohres des Zuhörers widerspiegelt, anzuwenden, um eine modifizierte verrauschte und/oder verarbeitete Version x des Zielsignals zur Verwendung zusammen mit der im Wesentlichen rauschfreien Version s des Zielsignals als eine Grundlage zum Berechnen des finalen monaurale Sprachverständlichkeitsvorhersagers d bereitzustellen.
  5. Monaurale Sprachverständlichkeitsvorhersageeinheit (HLM) nach einem der Ansprüche 1-4, wobei die Hörverlustmodelleinheit dazu konfiguriert ist, ein statistisch unabhängiges Rauschsignal, das gemäß einem Audiogramm des relevanten Ohres des Zuhörers spektral geformt ist, zu der verrauschten und/oder verarbeiteten Version x des Zielsignals hinzuzufügen.
  6. Monaurale Sprachverständlichkeitsvorhersageeinheit (MSIP) nach einem der Ansprüche 1-5, dazu angepasst, die temporalen Einhüllendensignale xj(m) bzw. sj(m) als Folgendes zu extrahieren z j m = f ∑ k − k 1 j k 2 j z k m 2 ,
    Figure imgb0079
     wobei z für x oder s steht, j=1, ..., J und m=1, ..., M, k1(j) und k2(j) die DFT-Bereichsindizes entsprechend niedrigeren und höheren Eckfrequenzen des j. Teilbands bezeichnet, J die Anzahl von Teilbändern ist und M die Anzahl von Signalrahmen in dem betreffenden Signal ist, und f(·) eine Funktion ist.
  7. Monaurale Sprachverständlichkeitsvorhersageeinheit (MSIP) nach Anspruch 6, wobei die Funktion f(·)=f(w), wobei w für ∑ k k 1 j k 2 j z k m 2
    Figure imgb0080
     steht, aus den folgenden Funktionen ausgewählt ist:
    • f(w)=w mit Darstellung der Identität
    • f(w)=w2 mit Bereitstellung von Leistungseinhüllenden,
    • f(w)=2·log w oder f(w)=wβ, 0 < β < 2, mit Ermöglichung des Modellierens der kompressiven Nichtlinearität der gesunden Cochlea,
    oder Kombinationen daraus.
  8. Monaurale Sprachverständlichkeitsvorhersageeinheit (MSIP) nach einem der Ansprüche 1-7, umfassend
    • eine erste Normalisierungs- und/oder Umwandlungseinheit, die zum bereitstellen von normalisierten und/oder umgewandelten Versionen S̃m der Zeit-Frequenz-Segmente Sm angepasst ist;
    • eine zweite Normalisierungs- und/oder Umwandlungseinheit, die zum bereitstellen von normalisierten und/oder umgewandelten Versionen X̃m der Zeit-Frequenz-Segmente Xm angepasst ist.
  9. Monaurale Sprachverständlichkeitsvorhersageeinheit (MSIP) nach Anspruch 8, wobei die erste und die zweite Normalisierungs- und/oder Umwandlungseinheit dazu konfiguriert sind, einen oder mehrere der folgenden Algorithmen auf das Zeit-Frequenz-Segment Xm bzw. Sm, allgemein bezeichnet als Zm , anzuwenden, wobei der tiefgestellte Zeitindex m zur Vereinfachung in dem folgenden Ausdruck weggelassen wird:
    • Normalisierung von Zeilen auf Null bedeutet: g 1 Z = Z − μ z r 1 _ T ,
    Figure imgb0081
     wobei μ z r
    Figure imgb0082
     ein Vektor J×1 ist, dessen j'. Eintrag der Mittelwert der j'. Zeile von Z ist, daher die Hochstellung r in μ z r ,
    Figure imgb0083
     wobei 1 für einen Vektor N × 1 von Einsen steht, und wobei die Hochstellung T Matrixtranspositionen bezeichnet;
    • Normalisierung von Zeilen zu Einheitsnorm: g 2 Z = D r Z Z ,
    Figure imgb0084
     wobei D r Z = diag ⌊ 1 / Z 1 , : Z 1 , : H ⋯ 1 / Z J , : Z J , : H ⌋ ,
    Figure imgb0085
    wobei diag(·) eine Diagonalmatrix mit den Elementen der Argumente auf der Hauptdiagonale ist und wobei Z (j,:) die j'. Zeile von Z bezeichnet, sodass Dr (Z) eine Diagonalmatrix J × J mit der umgekehrten Norm von jeder Zeile der Hauptdiagonale ist und Nullen andernorts, die Hochstellung H hermitesche Transposition bezeichnet und Vormultiplikation mit Dr (Z) die Zeilen der resultierenden Matrix zu Einheitsnorm normalisiert;
    • Fourier-Transformation, angewendet für jede Zeile g 3 Z = ZF ,
    Figure imgb0086
     wobei F eine Fourier-Matrix N × N ist;
    • Fourier-Transformation, angewendet für jede Zeile, gefolgt von Berechnen der Größe der resultierenden komplexwertigen Elemente g 4 = ZF
    Figure imgb0087
     wobei |·| (die elementweisen Größen berechnet;
    • der Identitätsoperator g 5 Z = Z .
    Figure imgb0088
    • Normalisierung von Spalten auf Null bedeutet: h 1 Z = Z − 1 _ μ z c T ,
    Figure imgb0089
     wobei μ z c
    Figure imgb0090
     ein Vektor N × 1 ist, dessen i. Eintrag der Mittelwert der i. Zeile von Z ist und wobei 1 einen Vektor J×1 von Einsen bezeichnet;
    • Normalisierung von Spalten zu Einheitsnorm: h 2 Z = ZD c Z ,
    Figure imgb0091
     wobei D c Z = diag ⌊ 1 / Z : , 1 H Z : , 1 ⋯ 1 / Z : , N H Z : , N ⌋ ,
    Figure imgb0092
     wobei Z(:,n) die n'. Zeile von Z bezeichnet, sodass Dc (Z) eine Diagonalmatrix N × N mit der umgekehrten Norm von jeder Spalte an der Hauptdiagonale ist und Nullen andernorts, und wobei eine Nachmultiplikation mit Dc (Z) die Zeilen der resultierenden Matrix auf Einheitsnorm normalisiert.
  10. Monaurale Sprachverständlichkeitsvorhersageeinheit (MSIP) nach einem der Ansprüche 1-9, wobei die Zwischensprachverständlichkeitsberechnungseinheit (ISIU) dazu angepasst ist, die Zwischensprachverständlichkeitskoeffizienten dm in Abhängigkeit von einem Abtastungskorrelationskoeffizient d(a,b) der Elemente in zwei Kx1-Vektoren a und b zu bestimmen, wobei d(a,b) durch Folgendes definiert ist: d a b = ∑ k = 1 K a k − μ a b k − μ b ∑ k − 1 K a k − μ a 2 b k − μ b 2 ,
    Figure imgb0093
     wobei μ a = 1 K ∑ k 1 K a k
    Figure imgb0094
     und μ b = 1 K ∑ k = 1 K b k ,
    Figure imgb0095
     wobei k der Index des Vektoreintrags ist und K die Vektorabmessung ist, und wobei a und b K Elemente aus Zeit-Frequenz-Segmenten Sm oder S̃m bzw. Xm oder X̃m darstellen.
  11. Monaurale Sprachverständlichkeitsvorhersageeinheit (MSIP) nach Anspruch 10, wobei der Zwischenverständlichkeitsindex dm definiert ist als
    • der mittlere Abtastungskorrelationskoeffizient aller Spalten in Sm bzw. Xm oder S̃m bzw. X̃m, d. h. d m = 1 N ∑ j = 1 N d S ˜ m : , n , X ˜ m : , n ,
    Figure imgb0096
     wobei n ein Spaltenindex ist, oder als
    • der mittlere Abtastungskorrelationskoeffizient aller Zeilen in Sm und Xm oder S̃m und X̃m, d. h. d m = 1 J ∑ j = 1 J d S ˜ m j , : T , X ˜ m j , : T ,
    Figure imgb0097
     wobei j ein Zeilenindex ist, oder als
    • der Abtastungskorrelationskoeffizient aller Elemente in Sm und Xm oder S̃m und X̃m, d. h. d m = d S ˜ m : , X ˜ m : ,
    Figure imgb0098
     wobei die Notation Sm(:) und Xm(:) oder S̃m(:) und X̃m(:) für NJx1-Vektoren, gebildet durch Stapeln der Spalten der jeweiligen Matrizen, darstellt.
  12. Monaurale Sprachverständlichkeitsvorhersageeinheit (MSIP) nach einem der Ansprüche 1-11, wobei die finale Sprachverständlichkeitsberechnungseinheit (FSIU) dazu angepasst ist, den finalen Sprachverständlichkeitsvorhersager d aus den Zwischensprachverständlichkeitskoeffizienten dm, gegebenenfalls umgewandelt durch eine Funktion u(dm), als einen Mittelwert im Zeitverlauf des Informationssignals x zu berechnen: d = 1 M ∑ m = 1 M u d m
    Figure imgb0099
     wobei M die Dauer in Zeiteinheiten der sprachaktiven Teile der verrauschten und/oder verarbeiteten Version x des Zielsignals darstellt.
  13. Monaurale Sprachverständlichkeitsvorhersageeinheit (MSIP) nach Anspruch 12, wobei die Funktion u(dm) definiert ist als u d m = log 1 1 − d m 2 ,
    Figure imgb0100
     oder als u d m = d m .
    Figure imgb0101
  14. Hörhilfe (hearing aid - HD), dazu angepasst, an oder in dem linken oder rechten Ohr eines Benutzers angeordnet zu sein oder vollständig oder teilweise in den Kopf des Benutzers implantiert zu sein, wobei die Hörhilfe eine monaurale Sprachverständlichkeitsvorhersageeinheit (MSIP) nach einem der Ansprüche 1-13 umfasst.
  15. Hörhilfe (HD) nach Anspruch 14, dazu konfiguriert, die Verarbeitung eines Eingangssignals (y) zu der Hörhilfe (HD) adaptiv zu modifizieren, um den monauralen Sprachverständlichkeitsvorhersager d zu maximieren, um die Verständlichkeit eines Ausgabesignals (u) der Hörhilfe (HD), das dem Benutzer (U) dargelegt wird, durch den Benutzer zu verbessern.
  16. Verfahren zum Bereitstellen eines monauralen Sprachverständlichkeitsvorhersagers zum Schätzen der Fähigkeit eines Benutzers, ein Informationssignal x bestehend aus einer verrauschten und/oder verarbeiteten Version eines Zielsprachsignals zu verstehen, wobei das Verfahren Folgendes umfasst:
    • Bereitstellen einer Zeit-Frequenz-Darstellung s(k,m) der rauschfreien Version s des Zielsignals, wobei k ein Frequenzbereichsabschnitt ist, k=1, 2, ..., K und m ein Zeitindex ist;
    • Bereitstellen einer Zeit-Frequenz-Darstellung x(k,m) der verrauschten und/oder verarbeiteten Version x des Zielsignals, wobei k ein Frequenzbereichsabschnitt ist, k=1, 2, ..., K und m ein Zeitindex ist;
    • Bereitstellen einer Zeit-Frequenz-Teilbanddarstellung sj(m) der rauschfreien Version s des Zielsignals, die temporale Einhüllende, oder Funktionen davon, der Frequenz-Teilbandsignale sj(m) des rauschfreien Zielsignals darstellt, wobei j ein Frequenz-Teilbandindex ist, j=1, 2, ..., J and m der Zeitindex ist;
    • Bereitstellen einer Zeit-Frequenz-Teilbanddarstellung xj(m) der verrauschten und/oder verarbeiteten Version x des Zielsignals, die temporale Einhüllende, oder Funktionen davon, der Frequenz-Teilbandsignale xj(m) der verrauschten und/oder verarbeiteten Version des Zielsignals darstellt, wobei j=1, 2, ..., J und m der Zeitindex ist;
    • Teilen der Zeit-Frequenz-Teilbanddarstellung sj(m) der rauschfreien Version s des Zielsignals in Zeit-Frequenz-Segmente Sm entsprechend einer Anzahl N von aufeinanderfolgenden Abtastungen der Teilbandsignale;
    • Teilen der Zeit-Frequenz-Teilbanddarstellung xj(m) der verrauschten und/oder verarbeiteten Version x des Zielsignals in Zeit-Frequenz-Segmente Xm entsprechend einer Anzahl N von aufeinanderfolgenden Abtastungen der Teilbandsignale;
    wobei die erste und die zweite Zeit-Frequenz-Segmentteilungseinheit dazu konfiguriert sind, die Zeit-Frequenz-Darstellungen sj(m) bzw. xj(m) in Segmente in der Form von Spektrogrammen entsprechend N aufeinanderfolgenden Abtastungen von allen Teilbandsignalen zu teilen, wobei das m. Segment durch die Matrix JxN definiert ist, Z m = z 1 m − N + 1 ⋯ z 1 m ⋮ ⋮ z J m − N + 1 ⋯ z J m ,
    Figure imgb0102
     wobei z, Z für s, S bzw. x, X steht;
    • Bereitstellen von zumindest einem Normalisierungs- und/oder Umwandlungsbetrieb von Zeilen und zumindest einem Normalisierungs- und/oder Umwandlungsbetrieb von Spalten der Zeit-Frequenz-Segmente Sm und Xm ;
    • Bereitstellen von Zwischensprachverständlichkeitskoeffizienten dm, die eine Verständlichkeit des Zeit-Frequenz-Segments Xm schätzen, wobei die Zwischensprachverständlichkeitskoeffizienten dm auf im wesentlichen rauschfreien, normalisierten und/oder umgewandelten Zeit-Frequenz-Segmenten S̃m und den verrauschten und/oder verarbeiteten, normalisierten und/oder umgewandelten Zeit-Frequenz-Segmenten X̃m basieren;
    • Berechnen eines finalen monauralen Sprachverständlichkeitsvorhersagers d, der eine Verständlichkeit der verrauschten und/oder verarbeiteten Version x des Zielsignals durch Kombinieren, z. B. durch Mitteln von, oder durch Anwenden einer MIN- oder MAX-Funktion auf, die Zwischensprachverständlichkeitskoeffizienten dm, oder eine umgewandelte Version davon, im Zeitverlauf schätzt.
EP17158062.4A 2016-03-01 2017-02-27 Ein monaurales, intrusives sprachverständlichkeitsvorhersagesystem, hörhilfe system Active EP3214620B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP16157993 2016-03-01

Publications (2)

Publication Number Publication Date
EP3214620A1 EP3214620A1 (de) 2017-09-06
EP3214620B1 true EP3214620B1 (de) 2019-09-18

Family

ID=55451081

Family Applications (1)

Application Number Title Priority Date Filing Date
EP17158062.4A Active EP3214620B1 (de) 2016-03-01 2017-02-27 Ein monaurales, intrusives sprachverständlichkeitsvorhersagesystem, hörhilfe system

Country Status (4)

Country Link
US (1) US10176821B2 (de)
EP (1) EP3214620B1 (de)
CN (1) CN107147981B (de)
DK (1) DK3214620T3 (de)

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3402217A1 (de) * 2017-05-09 2018-11-14 GN Hearing A/S Sprachverständlichkeitsbasierte hörgeräte und zugehörige verfahren
EP3457716A1 (de) * 2017-09-15 2019-03-20 Oticon A/s Bereitstellung und übertragung eines audiosignals
EP3471440B1 (de) 2017-10-10 2024-08-14 Oticon A/s Hörgerät mit einem sprachverständlichkeitsschätzer zur beeinflussung eines verarbeitungsalgorithmus
DK3582513T3 (da) * 2018-06-12 2022-01-31 Oticon As Høreanordning omfattende adaptiv lydkildefrekvensreduktion
US11228849B2 (en) 2018-12-29 2022-01-18 Gn Hearing A/S Hearing aids with self-adjustment capability based on electro-encephalogram (EEG) signals
DK3675525T3 (da) * 2018-12-29 2023-08-14 Research Foundation Of The City Univ Of New York Rfcuny Høreapparater med selvjusteringsevne baseret på elektroencefalogram (EEG) signaler
US11087778B2 (en) * 2019-02-15 2021-08-10 Qualcomm Incorporated Speech-to-text conversion based on quality metric
US11615801B1 (en) * 2019-09-20 2023-03-28 Apple Inc. System and method of enhancing intelligibility of audio playback
US11153695B2 (en) * 2020-03-23 2021-10-19 Gn Hearing A/S Hearing devices and related methods
WO2021214556A1 (en) * 2020-04-20 2021-10-28 Hearx Ip (Pty) Ltd Method and system for predicting or detecting conductive hearing loss risk in a person
EP4106349A1 (de) * 2021-06-15 2022-12-21 Oticon A/s Hörgerät mit sprachverständlichkeitsschätzer
CN114900399A (zh) * 2022-05-09 2022-08-12 中山大学 一种基于深度学习相位差相关信号检测方法和系统

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2372700A1 (de) * 2010-03-11 2011-10-05 Oticon A/S Sprachverständlichkeitsprädikator und Anwendungen dafür
DK2563045T3 (da) * 2011-08-23 2014-10-27 Oticon As Fremgangsmåde og et binauralt lyttesystem for at maksimere en bedre øreeffekt
CN103730131B (zh) * 2012-10-12 2016-12-07 华为技术有限公司 语音质量评估的方法和装置
US9443533B2 (en) * 2013-07-15 2016-09-13 Rajeev Conrad Nongpiur Measuring and improving speech intelligibility in an enclosure
EP2928210A1 (de) * 2014-04-03 2015-10-07 Oticon A/s Binaurales Hörgerätesystem mit binauraler Rauschunterdrückung
EP3155618B1 (de) * 2014-06-13 2022-05-11 Oticon A/S Mehrbandiges rauschverminderungssystem und methodologie für digitale audiosignale
EP3057335B1 (de) * 2015-02-11 2017-10-11 Oticon A/s Hörsystem mit binauralem sprachverständlichkeitsprädiktor

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
US10176821B2 (en) 2019-01-08
EP3214620A1 (de) 2017-09-06
US20170256269A1 (en) 2017-09-07
CN107147981B (zh) 2021-02-02
CN107147981A (zh) 2017-09-08
DK3214620T3 (da) 2019-11-25

Similar Documents

Publication Publication Date Title
EP3214620B1 (de) Ein monaurales, intrusives sprachverständlichkeitsvorhersagesystem, hörhilfe system
EP3514792B1 (de) Verfahren zur optimierung eines algorithmus zur sprachverbesserung mit einem algorithmus zur vorhersage der sprachverständlichkeit
EP3203473B1 (de) Monaurale sprachverständlichkeitsprädiktoreinheit, hörgerät und binaurales hörsystem
EP4418690A2 (de) Hörgerät mit einem rauschunterdrückungssystem
EP3267698B1 (de) Hörgerätesystem mit einem eeg-aufzeichnungs- und -analysesystem
EP3509325B1 (de) Hörgerät mit strahlformerfiltereinheit mit einer glättungseinheit
EP3220661B1 (de) Verfahren zur vorhersage der verständlichkeit von verrauschter und/oder erweiterter sprache und binaurales hörsystem
EP3057335B1 (de) Hörsystem mit binauralem sprachverständlichkeitsprädiktor
EP3300078B1 (de) Stimmenaktivitätsdetektionseinheit und hörgerät mit einer stimmenaktivitätsdetektionseinheit
EP3873109A1 (de) Hörgerätesystem zur schätzung von akustischen übertragungsfunktionen
EP3681175A1 (de) Hörgerät mit direkter schallkompensation
US11856357B2 (en) Hearing device comprising a noise reduction system
US11632635B2 (en) Hearing aid comprising a noise reduction system
EP3118851A1 (de) Verbesserung von verrauschter sprache auf basis statistischer sprach- und rauschmodelle
EP3833043A1 (de) Hörsystem mit personalisiertem strahlformer
EP4106346A1 (de) Hörgerät mit adaptiver filterbank

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20180306

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 21/0232 20130101ALI20190305BHEP

Ipc: G10L 21/02 20130101ALI20190305BHEP

Ipc: G10L 25/60 20130101AFI20190305BHEP

Ipc: H04R 25/00 20060101ALI20190305BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20190415

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602017007063

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1182251

Country of ref document: AT

Kind code of ref document: T

Effective date: 20191015

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DK

Ref legal event code: T3

Effective date: 20191118

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20190918

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191218

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191218

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191219

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1182251

Country of ref document: AT

Kind code of ref document: T

Effective date: 20190918

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200120

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200224

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602017007063

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG2D Information on lapse in contracting state deleted

Ref country code: IS

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200119

26N No opposition filed

Effective date: 20200619

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20200229

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200227

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200227

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200229

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190918

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240201

Year of fee payment: 8

Ref country code: GB

Payment date: 20240201

Year of fee payment: 8

Ref country code: CH

Payment date: 20240301

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20240201

Year of fee payment: 8

Ref country code: DK

Payment date: 20240201

Year of fee payment: 8