EP2673778B1 - Nachbearbeitung mit medianfilterung von rauschunterdrückungsverstärkungen - Google Patents

Nachbearbeitung mit medianfilterung von rauschunterdrückungsverstärkungen Download PDF

Info

Publication number
EP2673778B1
EP2673778B1 EP12707413.6A EP12707413A EP2673778B1 EP 2673778 B1 EP2673778 B1 EP 2673778B1 EP 12707413 A EP12707413 A EP 12707413A EP 2673778 B1 EP2673778 B1 EP 2673778B1
Authority
EP
European Patent Office
Prior art keywords
gains
raw
processing
banded
post
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP12707413.6A
Other languages
English (en)
French (fr)
Other versions
EP2673778A1 (de
Inventor
Glenn N. Dickins
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dolby Laboratories Licensing Corp
Original Assignee
Dolby Laboratories Licensing Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dolby Laboratories Licensing Corp filed Critical Dolby Laboratories Licensing Corp
Publication of EP2673778A1 publication Critical patent/EP2673778A1/de
Application granted granted Critical
Publication of EP2673778B1 publication Critical patent/EP2673778B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L21/0216—Noise filtering characterised by the method used for estimating noise
    • G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
    • G10L2021/02166—Microphone arrays; Beamforming

Definitions

  • the present disclosure relates generally to signal processing, in particular of audio signals.
  • Acoustic signal processing is applicable today to improve the quality of sound signals such as from microphones.
  • many devices such as handsets operate in the presence of sources of echoes, e.g., loudspeakers.
  • signals from microphones may occur in a noisy environment, e.g., in a car or in the presence of other noise.
  • there may be sounds from interfering locations e.g., out-of-location conversation by others, or out-of-location interference, wind, etc. Acoustic signal processing is therefore an important area for invention.
  • An acoustic noise reduction system typically includes a noise estimator and a gain calculation module to determine suppression probability indicators, e.g., as a set of noise reduction gains that are determined, for example, on a set of frequency bands, and applied to the (noisy) input audio signal after transformation to the frequency domain and banding to the set of frequency bands to attenuate noise components.
  • the acoustic noise reduction system may include one microphone input, or a plurality of microphone inputs and downmixing, e.g., beamforming to generate one input audio signal.
  • the acoustic noise reduction system may further include echo reduction, and may further include out-of-location signal reduction.
  • Such statistical outliers might occur in other types of processing in which an input audio signal is transformed and banded.
  • Such other types of processing include perceptual domain-based leveling, perceptual domain-based dynamic range control, and perceptual domain-based dynamic equalization that takes into account the variation in the perception of audio depending on the reproduction level of the audio signal. See, for example, International Application PCT/US2004/016964 , published as WO 2004111994 .
  • the gains determined for each band for leveling and/or dynamic equalization include statistical outliers, e.g., isolated values, and such outliers might cause artifacts such as musical noise.
  • Median filtering the gains e.g., noise reduction gains, or leveling and/or dynamic equalization gains across frequency bands can reduce musical noise artifacts.
  • DE4405723A1 describes the use of median filtering for the reduction of "musical tones" which may occur in the context of spectral subtraction.
  • Gain values may vary significantly across frequencies, and in such a situation, running a relatively wide median filter along frequency bands has the risk of disrupting the continuity of temporal envelope, which is the inherent property for many signals and is crucial to perception as well. Whilst offering greater immunity to the outliers, a longer median filter can reduce the spectral selectivity of the processing, and potentially introduce greater discontinuities or jumps in the gain values across frequency and time.
  • Embodiments of the present invention include a method, an apparatus, and logic encoded in one or more computer-readable tangible media to carry out the method.
  • One embodiment includes a method of applying post-processing to raw banded, as recited in claim 1.
  • One embodiment includes an apparatus to post-process raw banded, as recited in claim 11.
  • One embodiment includes a tangible computer-readable storage , as recited in claim 10.
  • One embodiment includes program logic that when executed by at least one processor causes carrying out a method as described herein.
  • Particular embodiments may provide all, some, or none of these aspects, features, or advantages. Particular embodiments may provide one or more other aspects, features, or advantages, one or more of which may be readily apparent to a person skilled in the art from the figures, descriptions, and claims herein.
  • One aspect of the invention is processing of one or more input audio signals, including input processing to generate raw gains for noise reduction, or for other forms of input signal improvement.
  • the processing includes applying post-processing to the raw gains, including median filtering of raw gains for gain smoothing.
  • Another aspect of the invention is the post-processing that includes the median filtering of raw gains determined by input processing, e.g., for noise reduction or for other input processing.
  • a median filter replaces a particular raw gain value with the median of a predefined number of raw gain values, e.g., by the median of the particular raw gain value and a predefined set of neighboring raw gain values.
  • a median filter has one or more properties, e.g., the number of valued of which the median is determined, and the end conditions.
  • At least one of the propertied may be data dependent. Therefore, in some examples described herein, there may be a first median filter for one type of data, e.g., data likely to be noise, and a different median filter for another type of data, e.g., data likely to be voice.
  • FIG. 1 shows one example of processing of a set of one or more input audio signals, e.g., microphone signals 101 from differently located microphones, including an embodiment of the present invention.
  • the processing is by time frames of a number, e.g., M samples.
  • there is only one input e.g., one microphone
  • there is a plurality, denoted P of inputs e.g., microphone signals 101.
  • An input processor 105 accepts sampled input audio signal(s) 101 and forms a banded instantaneous frequency domain amplitude metric 119 of the input audio signal(s) 101 for a plurality B of frequency bands.
  • the metric 119 is mixed-down from the input audio signal.
  • the amplitude metric represents the spectral content.
  • the spectral content is in terms of the power spectrum.
  • the invention is not limited to processing power spectral values. Rather, any spectral amplitude dependent metric can be used. For example, if the amplitude spectrum is used directly, such spectral content is sometimes referred to as spectral envelope. Thus, the phrase "power (or other amplitude metric) spectrum" is sometimes used in the description.
  • the input processor 105 determines a set of raw banded gains 111 to apply to the instantaneous amplitude metric 119.
  • the input processing further includes determining a signal classification of the input audio signal(s), e.g., an indication of whether the input audio signal(s) is/are likely to be voice or not as determined by a voice activity detector (VAD), and/or an indication of whether the input audio signal(s) is/are likely to be wind or not as determined by a wind activity detector (WAD), and/or an indication that the signal energy is rapidly changing as indicated, e.g., by the spectral flux exceeding a threshold.
  • VAD voice activity detector
  • WAD wind activity detector
  • a feature of embodiments of the present invention includes applying post-processing to the raw gains to improve the quality of the output.
  • the post-processing includes median filtering of the raw gains determined by the input processing.
  • a median filter considers a set of raw gains and outputs the gain that is the median of the set of raw gains.
  • a set of B raw gains is determined every frame, so that there is a time sequence of sets of B raw gains over B frequency bands.
  • the median filter extends across frequency.
  • FIG. 2 shows diagrammatically sets of raw banded gains, one set for each of the present time, one frame back, two frames back, three frames back, etc., and further shows the coverage of an example median filter that includes five raw gain values centered around a frequency band b c in the present frame.
  • filter width we mean the width of the filter in the frequency band domain.
  • the post-processing produces a set of post-processed gains 125 that are applied to the instantaneous power (or other amplitude metric) 119 to produce output, e.g., as a plurality of processed frequency bins 133.
  • An output synthesis filterbank 135 (or for subsequent coding, a transformer/remapper) converts these frequency bins to desired output 137.
  • Input processing element 105 includes an input analysis filterbank, and a raw gain calculator.
  • the input analysis filterbank for the case of one input audio signal 101, includes a transformer to transform the samples of a frame into frequency bins, and a banding element to form frequency bands, most of which include a plurality of frequency bins.
  • the input analysis filterbank for the case of a plurality of input audio signals 101, includes a transformer to transform the samples of a frame of each of the input audio signals into frequency bins, a downmixer, e.g., a beamformer to downmix the plurality into a single signal, and a banding element to form frequency bands, most of which include a plurality of frequency bins.
  • the transformer implements short time Fourier transform (STFT).
  • STFT short time Fourier transform
  • DFT discrete finite length Fourier transform
  • FFT fast Fourier transform
  • Other embodiments use different transforms.
  • the B bands are at frequencies whose spacing is monotonically non-decreasing.
  • a reasonable number, e.g., 90% of the frequency bands include contribution from more than one frequency bin, and in particular embodiments, each frequency band includes contribution from two or more frequency bins.
  • the bands are monotonically increasing in a log-like manner.
  • the bands are on a psycho-acoustic scale, that is, the frequency bands are spaced with a scaling related to psycho-acoustic critical spacing, such banding called "perceptually-banding" herein.
  • the band spacing is around 1 ERB or 0.5 Bark, or equivalent bands with frequency separation at around 10% of the center frequency.
  • a reasonable range of frequency spacing is from 5-20% or approximately 0.5-2 ERB.
  • the input processing includes noise reduction
  • a plurality of input audio signals are accepted by the input processor, and the input processing includes reducing out-of location signals.
  • An example of input processing that includes reducing out-of location signals is described in concurrently filed International Application No PCT/US2012/024370 , titled COMBINED SUPPRESSION OF NOISE, ECHO, and OUT OF-LOCATION SIGNALS that also claims priority of U.S. Provisional Application No. 61/441,611 filed 10 February 2011 to inventors Dickins et al. titled "COMBINED SUPPRESSION OF NOISE, ECHO, AND OUT-OF-LOCATION SIGNALS," the contents of both which are incorporated herein by reference.
  • the resulting raw banded gains achieve simultaneous echo reduction and noise reduction.
  • the input processing also includes echo reduction.
  • echo reduction One example of input processing that includes echo reduction is described in concurrendy filed International Application No. PCT/US.
  • one or more reference signals also are included and used to obtain an estimate of some property of the echo, e.g., of the power (or other amplitude metric) spectrum of the echo. The resulting raw banded gains achieve simultaneous echo reduction and noise reduction.
  • the post-processed gains are accepted by an element 123 that modifies the gains to include additional echo suppression.
  • the result is a set of post-processed gains 125 that are applied to the input signal or signals. e.g., that are used to process, in the frequency domain, .as frequency bins, the input audio signal if there is one input, or a downmix of the input audio signals if there is a plurality of input audio signals, e.g., from differently located microphones.
  • Gain application module 131 accepts the banded post-processed gains 125 and applies such gains to the input audio signal or signals.
  • the processed data 133 may then be converted back to the sample domain by an output synthesis filterbank 135 to produce a frame of M signal samples 137.
  • the signal 133 is subject to transformation or remapping, e.g., to a form ready for coding according to some coding method.
  • the invention is not limited to the input processing and gain calculation described in International Application No. PCT/US2012/024370 , U.S. 61/441,611 , or even to noise reduction.
  • the input processing is to reduce noise (and possibly echo and out of location signals)
  • the input processing may be, additionally or primarily, to carry out one or more of perceptual domain-based leveling, perceptual domain-based dynamic range control, and perceptual domain-based dynamic equalization that take into account the variation in the perception of audio depending on the reproduction level of the audio signal, as described, for example, in commonly owned WO 2004111994 .
  • the raw banded gains calculated per WO 2004111994 are post-processed, including median filtering, to determine post-processed gains 125 to apply to the (transformed) input.
  • FIG. 3A shows a simplified block diagram of a post-processor 121 that includes a median filter 305 according to an embodiment of the present invention.
  • the post-processor 121 accepts raw gains 111 and in embodiments in which the post-processing changes according to signal classification, one or more signal classification indicators 115, e.g., the outputs of one or more of a VAD and a WAD. While not included in all embodiments, some embodiments of the post-processor include a minimum gain processor 303 to ensure that the gains do not fall below a predefined, possibly frequency-dependent value.
  • some embodiments of the post-processor include a smoothing filter 307 that processes the gains after median filtering to smooth frequency-band-to-frequency-band variations, and/or to smooth time variations.
  • FIG. 3B shows a simplified flowchart of a method of post-processing 310 that includes in 311 accepting raw gains, and in embodiments in which the post-processing changes according to signal classification, one or more signal classification indicators 115.
  • the post-processing includes median filtering 315 according to embodiments of the present invention. The inventors have found that median filtering is a powerful nonlinear smoothing technique, which works well for eliminating undesired outliers when compared with only using a smoothing method.
  • Some embodiments include in step 313 ensuring that the gains do not fall below a predefined minimum, which may be frequency band dependent. Some embodiments further include, in step 317, band-to-band and/or time smoothing, e.g., linear smoothing using, e.g., a weighted moving average.
  • band-to-band and/or time smoothing e.g., linear smoothing using, e.g., a weighted moving average.
  • a median filter 315 of raw banded gain values is characterized by: 1) the number of raw banded gains to include to determine the median value, 2) the frequency band positions of the raw banded gains that are included; 3) the edge conditions, i.e., the conditions used to extend the raw banded gains to allow calculation of the median at the edges of time and frequency band; and 4) how the characterization of the median filter is affected by the signal classification, e.g., one or more of the presence of voice and the presence of wind.
  • Some embodiments include a mechanism to control one or more of the median filtering characteristics over frequency and/or time based on signal classification. For example, in one embodiment that includes voice activity detection, one or more of the median filtering characteristics vary in accordance to whether the input is ascertained by a VAD to be voice or not. In one embodiment that includes wind activity detection, one or more of the median filtering characteristics vary in accordance to whether the input is ascertained by a WAD to be wind or not.
  • Examples of different edge conditions include (a) extrapolating of interior values for the edges; (b) using the minimum gain value to extend the raw banded gains at the edges, (c) using a zero gain value to extend the raw banded gains at the edges (d) duplicating the central filter position value to extend the raw banded gains at the edges, and (e) using a maximum gain value to extend the raw banded gains at the edges.
  • the post-processor 121 includes a minimum gain processor 303 that carries out step 313 to ensure the gains do not fall below a predefined minimum gain value.
  • the minimum gain processor ensures minimum values in a frequency-band dependent manner.
  • the manner of prevention minimum is dependent on the activity classification 115, e.g., whether voice or not.
  • the range of the maximum suppression depth or minimum gain may range from -80dB to -5dB and be frequency dependent.
  • the suppression depth was around -20dB at low frequencies below 200Hz, varying to be around -10dB at 1kHz and relaxing to be only -6dB at the upper voice frequencies around 4kHz.
  • Gain ' b,MIN is increased, e.g., in a frequency-band dependent way (or in another embodiment, by the same amount for each band b).
  • the amount of increase in the minimum is larger in the mid-frequency bands, e.g., bands between 500 Hz to 2 kHz.
  • the post-processor 121 includes a smoothing filter 307, e.g., a linear smoothing filter that carries out one or both of frequency band-to-band smoothing and time smoothing. In some embodiments, such smoothing is varied according to signal classification 115.
  • a smoothing filter 307 e.g., a linear smoothing filter that carries out one or both of frequency band-to-band smoothing and time smoothing. In some embodiments, such smoothing is varied according to signal classification 115.
  • smoothing 317 uses a weighted moving average with a fixed kernel.
  • One example uses a binomial approximation of a Gaussian weighting kernel for the weighted moving average.
  • a 5-point binomial smoother has a kernel 1 16 1 4 6 4 1 .
  • the factor 1/16 may be left out, with scaling carried out in one point or another as needed.
  • a 3-point binomial smoother has a kernel 1 4 1 2 1 .
  • Many other weighted moving average filters are known, and any such filter can suitably be modified to be used for the band-to-band smoothing of the gain.
  • the band-to-band smoothing is controlled by the signal classification.
  • a VAD e.g., a spatially-selective VAD is included, and if the VAD determines there is voice, the degree of smoothing is increased when noise is detected.
  • 5-point band-to-band weighted average smoothing is carried out in the case the VAD indicates voice is detected, else, when the VAD determines there is no voice, no smoothing is carried out.
  • time smoothing of the gains also is included.
  • Gain b is the current time-frame gain
  • Gain b,Smoothed is the time-smoothed gain
  • Gain b,Smoothed Pr ev is Gain b,Smoothed from the previous M -sample frame.
  • ⁇ b is a time constant which may be frequency band dependent and is typically in the range of 20 to 500ms.
  • a value of 50ms was used.
  • the amount of time smoothing is controlled by the signal classification of the current frame.
  • the signal classification of the current frame is used to control the values of first order time constants used to filter the gains over time in each band.
  • one embodiment stops time smoothing in the case voice is detected.
  • the parameters of post-processing are controlled by the immediate signal classifier (VAD, WAD) value that has low latency and is able to achieve a rapid transition of the post-processing from noise into voice (or other desired signal) mode.
  • VAD immediate signal classifier
  • WAD voice-to-live
  • the speed with which more aggressive post-processing is reinstated after detection of voice, i.e., at the trail out, has been found to be less important, as it affects intelligibility of voice to a lesser extent.
  • the band-to-band median filtering is controlled by the signal classification.
  • a VAD is included, and if the VAD determines it is likely that there is no voice, a 7 point T-shaped median filter with 5-point band-to-band and 3-point time median filtering is carried out, with edge processing including extending minimum gain values or a zero value at the edges to compute the median value. If the VAD determines it is likely that voice is present, in a first version, a 5-point T-shaped time-frequency median filtering is carried out with three frequency bands in the current time frame, and using two previous time frames, and in a second embodiment, a three point memoryless frequency-band only median filter, with the edge values extrapolated at the edges to calculate the median, is used.
  • the median value is the median value, such that the median filter is a median filter.
  • the post-processing e.g., the median filtering depends on the classification of the signal, and one such classification, in some embodiments, is whether there is wind or not.
  • a WAD is included, and if the WAD determines there is no wind, and a VAD indicates there is no voice, fewer raw gain values are included in the median filter.
  • the median filtering should be shorter, e.g., by using 3-point band-to-band median filter, with extrapolating the edge values applied at the edges.
  • more median filtering can be used, e.g., 5-point band-to-band median filtering is carried out, with edge processing including extending minimum gain values or a zero value at the edges to compute the median value. If the WAD indicated wind is likely, and the VAD indicates voice is unlikely, even more median filtering can be used, e.g., a 7-point band-to-band median filtering can be carried out, with edge processing including extending minimum gain values or a zero value at the edges to compute the median value.
  • An acoustic noise reduction system typically includes a noise estimator and a raw gain calculation module to determine a set of noise reduction gains that are determined, for example, on a set of frequency bands, and applied to the (noisy) input audio signal after transformation to the frequency domain and banding to the set of frequency bands to attenuate noise components.
  • the acoustic noise reduction system may include one microphone, or a plurality of inputs from differently located microphones and downmixing, e.g., beamforming to generate one input audio signal.
  • the acoustic noise reduction system may further include echo reduction, and may further include out-of-location signal reduction.
  • FIG. 4 shows one example of an apparatus configured to determine a set of post-processed gains for suppression of noise, and in some versions, simultaneous echo suppression, and in some versions, simultaneous suppression of out-of-location signals.
  • the inputs include a set of one or more input audio signals 101, e.g., signals from differently located microphones, each in sets of M samples per frame.
  • input audio signals 101 e.g., signals from differently located microphones, each in sets of M samples per frame.
  • there are two or more input audio signals e.g., signals from spatially separated microphones.
  • one or more reference signals 103 are also accepted, e.g., in frames of M samples.
  • a first input processing stage 403 determines a banded signal power (or other amplitude metric) spectrum 413 denoted P' b , and a banded measure of the instantaneous power 417 denoted Y ' b .
  • P' b a banded signal power (or other amplitude metric) spectrum 413
  • Y ' b a banded measure of the instantaneous power 417 denoted Y ' b .
  • each of the spectrum 413 and instantaneous banded measure 417 is of the inputs after being mixed down by a downmixer, e.g., a beamformer.
  • the first input processing stage 403 When echo suppression is included, the first input processing stage 403 also determines a banded power spectrum estimate of the echo 415, denoted E' b . the determining being from a previously calculated power spectrum estimates of the echo using a filter with a set of adaptively determined filter coefficients. In those versions that include out-of-location signal suppression, the first input processing stage 403 also determines spatial features 419 in the form of banded location probability indicators 419 that are usable to spatially separate a signal into the components originating from the desired location and those not from the desired direction.
  • the quantities from the first stage 403 are used in a second stage 405 that determines raw gains, and that post-processes the raw gains, including the median filtering of embodiments of the present invention, to determine the banded post-processed gains 125.
  • Embodiments of the second stage 405 include a noise power (or other amplitude metric) spectrum calculator 421 to determine a measure of the noise power (or other amplitude metric) spectrum, denoted E' b , and a signal classifier 423 to determine a signal classification 115, e.g., one or more of a voice activity detector (VAD), a wind activity detector, and a power flux calculator.
  • VAD voice activity detector
  • FIG. 4 shows the signal classifier 423 including a VAD.
  • FIG. 5 shows one embodiment 500 of the elements of FIG. 4 in more detail, and includes, for the example embodiment of noise, echo, and out-of-location noise suppression, the suppressor 131 that applied the post-processed gains 125 and the output synthesizer (or transformer or remapper) 135 to generate the output signal 137.
  • the first stage processor 403 of FIG. 4 includes elements 503, 505, 507, 509, 511, 513, 515, 517, 521, 523, 525, and 527 of FIG. 5 .
  • the input(s) frame(s) 101 are transformed by inputs transformer(s) 503 to determine transformed input signal bins, the number of frequency bins denoted by N.
  • the frequency domain signals from the input transformers 503 are accepted by a banded spatial feature calculator to determine banded location probability indictors, each between 0 and 1.
  • the signals are combines by combiner 511, in one embodiment a summer, to produce a combined reference input.
  • Y' b is used as a good-enough approximation to P' b .
  • the updating is triggered by a voice activity signal denoted S as determined by a voice activity detector (VAD) 525 using P' b (or Y' b ) , N' b , and E' b .
  • S When S exceeds a threshold, the signal is assumed to be voice.
  • VAD voice activity detector
  • the VAD derived in the echo update voice-activity detector 525 and filter updater 527 serves the specific purpose of controlling the adaptation of the echo prediction.
  • a VAD or detector with this purpose is often referred to as a double talk detector.
  • the echo filter coefficient updating of updater 527 is gated, with updating occurring when the expected echo is significant compared to the expected noise and current input power, as determined by the VAD 525 and indicated by a low value of local signal activity S.
  • the input transformers 503, 511 determine the short time Fourier transform (STFT).
  • STFT short time Fourier transform
  • the following transform and inverse pair is used for the forward transform in elements 503 and 511, and in output synthesis element 135.
  • analysis and synthesis windows also known as prototype filters, can be of length greater or smaller than the examples given herein.
  • the downmixer is a beamformer 507 designed to achieve some spatial selectivity towards the desired position.
  • the beamformer 507 is a linear time invariant process, i.e., a passive beamformer defined in general by a set of complex-valued frequency-dependent gains for each input channel.
  • a passive beamformer 107 that determines the simple sum of the two input channels.
  • beamformer 507 weights the sets of inputs (as frequency bins) by a set of complex valued weights.
  • the beamforming weights of beamformer 107 are determined according to maximum-ratio combining (MRC).
  • the beamformer 507 uses weights determined using zero-forcing. Such methods are well known in the art.
  • the offset Y' min is added to avoid a zero level power spectrum (or other amplitude metric spectrum) estimate.
  • Y' min can be measured, or can be selected based on a priori knowledge.
  • Y' min for example, can be related to the threshold of hearing or the device noise threshold.
  • One embodiment includes time smoothing of the instantaneous echo from echo prediction filter 517 to determine the echo spectral estimate E' b .
  • the parameter ⁇ N,b is best expressed in terms of the rate over time at which minimum follower will track. That rate can be expressed in dB/sec, which then provides a mechanism for determining the value of ⁇ N,b .
  • the range is 1 to 30dB/sec. In one embodiment, a value of 20dB/sec is used.
  • different approaches for noise estimation may be used. Examples of such different approached include but are not limited to alternate methods of determining a minimum over a window of signal observation, e.g., a window of 1 and 10 seconds. In addition or alternate to the minimum, such different approaches might also determine the mean and variance of the signal during times that it is classified as likely to be noise or that voice is unlikely.
  • the one or more leak rate parameters of the minimum follower are controlled by the probability of voice being present as determined by voice activity detecting (VAD).
  • S voice activity detecting
  • a typical value for ⁇ N is 1.4 (+3dB).
  • a range of values 1 to 4 can be used.
  • X' sens is set to avoid unstable adaptation for small reference signals. In one embodiment X' sens is related to the threshold of hearing.
  • the choice of value for S thresh depends on the number of bands. S thresh is between 1 and B, and for one embodiment having 24 bands to 8kHz, a suitable range was found to be between 2 and 8, with a particular embodiment using a value of 4.
  • Embodiments of the present invention use spatial information in the form of one or more measures determined from one or more spatial features in a band b that are monotonic with the probability that the particular band b has such energy incident from a spatial region of interest. Such quantities are called spatial probability indicators.
  • the one or more spatial probability indicators are functions of one or more banded weighted covariance matrices of the input audio signals.
  • the w b,n provide an indication of how each bin is weighted for contribution to the bands.
  • the one or more covariance matrices are smoothed over time.
  • ratio a quantity that is monotonic with the ratio of the banded magnitudes R b 11 ′ R b 22 ′ .
  • it is the determined, or estimated (a priori) value of the noise power (or other frequency domain amplitude metric) in band b for the microphone and related electronics. That is, the minimum sensitivity of any preprocessing used.
  • Phase ′ b tan ⁇ 1 R b 21 ′ .
  • the spatial feature Denote by the spatial feature "coherence” a quantity that is monotonic with R b 21 ′ R b 12 ′ R b 11 ′ R b 22 ′ .
  • related measures of coherence could be used such as 2 R b 21 ′ R b 12 ′ R b 11 ′ R b 11 ′ + R b 22 ′ R b 22 ′ or values related to the conditioning, rank or eigenvalue spread of the covariance matrix.
  • One feature of some embodiments of the noise, echo and out-of-location signal suppression is that, based on the a priori expected or current estimate of the desired signal features-the target values, e.g., representing spatial location, gathered from statistical data-each spatial feature in each band can be used to create a probability indicator for the feature for the band b.
  • the distributions of the expected spatial features for the desired location are modeled as Gaussian distributions that present a robust way of capturing the region of interest for probability indicators derived from each spatial feature and band.
  • RPI' b ratio probability indicator
  • PPI' b phase probability indicator
  • CPI' b coherence probability indicator
  • ⁇ Ratio' b Ratio' b - Ratio target b
  • Ratio target b is determined from either prior estimates or experiments on the equipment used, e.g., headsets, e.g., from data such as shown in FIG. 9A.
  • the function f Rb ( ⁇ Ratio' ) is a smooth function.
  • the Width Ratio , b is related to but does not need to be determined from actual data. It is set to cover the expected variation of the spatial feature in normal and noisy conditions, but also needs only be as narrow as is required in the context of the overall system to achieve the desired suppression.
  • Phase target b Phase' b - Phase target b
  • Phase target b Phase' b - Phase target b
  • Phase target b is determined from either prior estimates or experiments on the equipment used, e.g., headsets, obtained, e.g., from data.
  • f P b ( ⁇ Phase' ) is a smooth function.
  • f R b ⁇ Phas e b ′ exp ⁇ ⁇ Phas e b ′ Widt h Phase , b 2
  • Width Phase,b is a width tuning parameter expressed in units of phase.
  • Width Phase,b is related to but does not need to be determined from actual data.
  • CP I b ′ R b 21 ′ R b 12 ′ + ⁇ 2 R b 11 ′ R b 22 ′ + ⁇ 2 CFacto r b
  • CFactor b is a tuning parameter that may be a constant value in the range of 0.1 to 10; in one embodiment, a value of 0.25 was found to be effective.
  • FIG. 6 shows one example of the calculation in element 529 of the raw gains, and includes a spatially sensitive voice activity detector (VAD) 621, and a wind activity detector (WAD) 623.
  • VAD voice activity detector
  • WAD wind activity detector
  • Alternate versions of noise reduction may not include the WAD, or the spatially sensitive VAD, and further may not include echo suppression or other reduction.
  • the embodiment shown in FIG. 6 includes additional echo suppression, which may not be included in simpler versions.
  • the spatial probability indicators are used to determine what is referred to as the beam gain, a statistical quantity denoted BeamGain' b that can be used to estimate the in-beam and out-of-beam power from the total power, e.g., using an out-of-beam spectrum calculator 603, and further, can be used to determine the out-of-beam suppression gain by a spatial suppression gain calculator 611.
  • BeamGain' b a statistical quantity denoted BeamGain' b that can be used to estimate the in-beam and out-of-beam power from the total power, e.g., using an out-of-beam spectrum calculator 603, and further, can be used to determine the out-of-beam suppression gain by a spatial suppression gain calculator 611.
  • the probability indicators are scaled such that the beam gain has a maximum value of 1.
  • Some embodiments use BeamGain min of 0.01 to 0.3 (-40dB to -10dB).
  • One embodiment uses a BeamGain min of 0.1.
  • Power' b,InBeam and Power' b , OutOfBeam are statistical measures used for suppression.
  • One version of gain calculation uses a spatially-selective noise power spectrum calculator 605 that determines an estimate of the noise power (or other metric of the amplitude) spectrum.
  • One embodiment of the invention uses a leaky minimum follower, with a tracking rate determined by at least one leak rate parameter.
  • the leak rate parameter need not be the same as for the non-spatially-selective noise estimation used in the echo coefficient updating. Denote by N' b,S the spatially-selective noise spectrum estimate.
  • N b , S ′ min Powe r b , OutOfBeam ′ , 1 + ⁇ b N b , S Pr ev ′ , where N' b,S Pr ev is the already determined, i.e., previous value of N' b,S .
  • the leak rate parameter ⁇ b is expressed in dB/s such that for a frame time denoted T, (1 + ⁇ b ) 1 / T is between 1.2 and 4 if the probability of voice is low, and 1 if the probability of voice is high.
  • the noise estimate is updated only if the previous noise estimate suggests the noise level is greater, e.g., greater than twice the current echo prediction. Otherwise the echo would bias the noise estimate.
  • the gain calculator 529 includes an element 613 to calculates a probability indicator 614, expressed as a gain for the intermediate signal, e.g., the frequency bins Y n based on the spatially-selective estimates of the noise power (or other frequency domain amplitude metric) spectrum, and further on the instantaneous banded input power Y' b in a particular band.
  • this probability indicator is referred to as a gain, denoted Gain N . It should be noted however that this gain Gain N is not directly applied, but rather combined with additional gains, i.e., additional probability indicators in a gain combiner 615 to achieve a single gain to apply to achieve a single suppressive action.
  • the element 613 is shown with echo suppression, and in some versions does not include echo suppression.
  • Gai n N ′ max 0, Y b ′ ⁇ ⁇ N ′ N b , S Y b ′ GainExp
  • Y' b the instantaneous banded power (or other frequency domain amplitude metric)
  • N' b,S the banded spatially-selective (out-of-beam) noise estimate
  • ⁇ ' N a scaling parameter, typically in the range of 1 to 4.
  • ⁇ ' N 1.5.
  • Some embodiments of input processing for noise reduction include not only noise suppression, but also simultaneous suppression of echo.
  • element 613 includes echo suppression and in gain calculator 529, the probability indicator for suppressing echoes is expressed as a gain denoted Gain' b,N + E .
  • the suppression gain expressions above can be generalized as functions on the domain of the ratio of the instantaneous input power to the expected undesirable signal power, sometimes called "noise" for simplicity.
  • the undesirable signal power is the sum of the estimated (location-sensitive) noise power and predicted or estimated echo power. Combining the noise and echo together in this way provides a single probability indicator in the form of a suppressive gain that causes simultaneous attenuation of both undesirable noise and of undesirable echo.
  • an additional scaling of the probability indicator or gain is used, such additional scaling based on the ratio of input audio signal to echo power alone.
  • f A ( ⁇ ) , f B ( ⁇ ) a pair of suppression gain functions, each having desired properties for suppression gains, e.g., as described above, including, for example being smooth.
  • each of f A ( ⁇ ), f B ( ⁇ ) has sigmoid function characteristics.
  • f A Y b ′ N b , S ′ + E b ′ one can instead use a pair of probability indicators, e.g., gains f A Y b ′ N b , S ′ , f B Y b ′ E b ′ and determine a combined gain factor from f A Y b ′ N b , S ′ and f B Y b ′ E b ′ , which allows for independent control of the aggressiveness and depth for the response to noise and echo signal power.
  • f A Y b ′ N b , S ′ + E b ′ can be applied for both noise and echo suppression
  • f B Y b ′ E b ′ can be applied for additional echo suppression.
  • the two functions f A Y b ′ N b , S ′ , f B Y b ′ E b ′ or in another embodiment, the two functions f A Y b ′ N b , S ′ + E b ′ , f B Y b ′ E b ′ are combined as a product to achieve a combined probability indicator, as a suppression gain.
  • the spatial suppression gain 612 is combined with other suppression gains in gain combiner 615 to form an overall probability indicator expressed as a suppression gain.
  • additional smoothing is applied.
  • Gai n b , RAW ′ 0.1 + 0.9 Gai n b , S ′ ⁇ Gai n b , N + E ′ .
  • Gain' b,RAW suppresses noise and echo equally. As discussed above, it may be desirable to not eliminate noise completely, but to completely eliminate echo.
  • Gai n b , RAW ′ 0.1 + 0.9 Gai n b , S ′ ⁇ f A Y b ′ N b , S ′ + E b ′ ⁇ f B Y b ′ E b ′ , where f A Y b ′ N b , S ′ + E b ′ achieves (relatively) modest suppression of both noise and echo, while f B Y b ′ E b ′ suppresses the echo more.
  • f A ( ⁇ ) suppresses only noise
  • f B ( ⁇ ) suppresses the echo.
  • this noise and echo suppression gain is combined with the spatial feature probability indicator or gain for forming a raw combined gain, and then post-processed by a post-processor 625 and by the post processing step to ensure stability and other desired behavior.
  • the gain function f B Y b ′ E b ′ specific to the echo suppression is applied as a gain after post-processing by post-processor 625.
  • Some embodiments of gain calculator 529 include a determiner of the additional echo suppression gain and a combiner 627 of the additional echo suppression gain with the post-processed gain to result in the overall B gains to apply. The inventors discovered that such an embodiment can provide a more specific and deeper attenuation of echo, since the echo probability indicator or gain f B Y b ′ E b ′ is not subject to the smoothing and continuity imposed by the post-processing.
  • FIG. 7 shows a flowchart of a method 700 of operating a processing apparatus 100 to suppress noise and out-of-location signals and in some embodiments echo in a number P ⁇ 1 of signal inputs 101, e.g., from differently located microphones.
  • method 700 includes processing a Q ⁇ 1 reference inputs 102, e.g., Q inputs to be rendered on Q loudspeakers, or signals obtained from Q loudspeakers.
  • method 700 comprises: accepting 701 in the processing apparatus a plurality of sampled input audio signals 101, and forming 703, 707, 709 a mixed-down banded instantaneous frequency domain amplitude metric 417 of the input audio signals 101 for a plurality of frequency bands, the forming including transforming 703 into complex-valued frequency domain values for a set of frequency bins.
  • the forming includes in 703 transforming the input audio signals to frequency bins, downmixing, e.g., beamforming 707 the frequency data, and in 709 banding.
  • the method includes calculating the power (or other amplitude metric) spectrum of the signal.
  • the downmixing can be before transforming, so that a single mixed-down signal is transformed.
  • the system may make use of an estimate of the banded echo reference, or a similar representation of the frequency domain spectrum of the echo reference provided by another processing component or source within the realized system.
  • the method includes determining in 705 banded spatial features, e.g., location probability indicators 419 from the plurality of sampled input audio signals.
  • the method includes accepting 713 one or more reference signals and forming in 715 and 717 a banded frequency domain amplitude metric representation of the one or more reference signals.
  • the representation in one embodiment is the sum.
  • the method includes predicting in 721 a banded frequency domain amplitude metric representation of the echo 415 using adaptively determined echo filter coefficients.
  • the predicting in one embodiment further includes voice-activity detecting-VAD-using the estimate of the banded spectral amplitude metric of the mixed-down signal 413, the estimate of banded spectral amplitude metric of noise, and the previously predicted echo spectral content 415.
  • the coefficients are updated or not according to the results of voice-activity detecting.
  • Updating uses an estimate of the banded spectral amplitude metric of the noise, previously predicted echo spectral content 415, and an estimate of the banded spectral amplitude metric of the mixed-down signal 413.
  • the estimate of the banded spectral amplitude metric of the mixed-down signal is in one embodiment the mixed-down banded instantaneous frequency domain amplitude metric 417 of the input audio signals, while in other embodiments, signal spectral estimation is used.
  • the method 700 includes: a) calculating in 723 raw suppression gains including an out-of-location signal gain determined using two or more of the spatial features 419, and a noise suppression gain determined using spatially-selective noise spectral content; and b) combining the raw suppression gains to a first combined gain for each band.
  • the noise suppression gain in some embodiments includes suppression of echoes, and its calculating 723 also uses the predicted echo spectral content 415.
  • the method 700 further includes in 725 carrying out spatially-selective voice activity detection determined using two or more of the spatial features 419 to generate a signal classification, e.g., whether voice or not.
  • a signal classification e.g., whether voice or not.
  • wind detection is used such that the signal classification further includes whether the signal is wind or not.
  • the method 700 further includes carrying out post-processing on the first combined gains of the bands to generate a post-processed gain 125 for each band.
  • the post-processing includes ensuring minimum gain, e.g., in a band dependent manner.
  • the post-processing includes carrying out median filtering of the combined gains, e.g., to ensure there are no outlier gains.
  • Some embodiments of post-processing include ensuring smoothness by carrying out time and/or band-to-band smoothing.
  • the post-processing 725 is according to the signal classification, e.g., whether voice or not, or whether wind or not, and in some embodiments, the characteristics of the median filtering vary according to the signal classification, e.g., whether voice or not, or whether wind or not.
  • the method includes calculating in 726 an additional echo suppression gain.
  • the additional echo suppression gain is included in the first combined gain which is used as a final gain for each band, and in another embodiment, the additional echo suppression gain is combined with the results of applying post-processing to the first combined gain to generate a final gain for each band.
  • the method includes applying in 727 the final gain, including interpolating the gain for bin data to carry out suppression on the bin data of the mixed-down signal to form suppressed signal data 133, and applying in 729 one or both of a) output synthesis and transforming to generate output samples, and b) output remapping to generate output frequency bins.
  • noise reduction is only one example of input processing that determines gains that can be post-processed by the post-processing method that includes median filtering described in embodiments of the present invention.
  • a processing system-based apparatus A processing system-based apparatus
  • FIG. 8 shows a simplified block diagram of one processing apparatus embodiment 800 for processing one or more of audio inputs 101, e.g., from microphones (not shown).
  • the processing apparatus 800 is to determine a set of gains, to post-process the gains including median filtering the determined gains, and to generate audio output 137 that has been modified by application of the gains.
  • One version achieves one or more of perceptual domain-based leveling, perceptual domain-based dynamic range control, and perceptual domain-based dynamic equalization that takes into account the variation in the perception of audio depending on the reproduction level of the audio signal.
  • Another version achieved noise reduction.
  • One noise reduction version includes echo reduction, and in such a version, the processing apparatus also accepts one or more reference signals 103, e.g., from one or more loudspeakers (not shown) or from the feed(s) to such loudspeaker(s).
  • the processing apparatus 800 is to generate audio output 137 that has been modified by suppressing, in one embodiment noise and out-of-location signals, and in another embodiment also echoes as specified in accordance to one or more features of the present invention.
  • the apparatus for example, can implement the system shown in FIG. 6 , and any alternates thereof, and can carry out, when operating, the method of FIG. 7 including any variations of the method described herein.
  • Such an apparatus may be included, for example, in a headphone set such as a Bluetooth headset.
  • the audio inputs 101, the reference input(s) 103 and the audio output 137 are assumed to be in the form of frames of M samples of sampled data.
  • a digitizer including an analog-to-digital converter and quantizer would be present.
  • a de-quantizer and a digital-to-analog converter would be present.
  • the embodiment shown in FIG. 8 includes a processing system 803 that is configured in operation to carry out the suppression methods described herein.
  • the processing system 803 includes at least one processor 805, which can be the processing unit(s) of a digital signal processing device, or a CPU of a more general purpose processing device.
  • the processing system 803 also includes a storage subsystem 807 typically including one or more memory elements.
  • the elements of the processing system are coupled, e.g., by a bus subsystem or some other interconnection mechanism not shown in FIG. 8 .
  • Some of the elements of processing system 803 may be integrated into a single circuit, using techniques commonly known to one skilled in the art.
  • the storage subsystem 807 includes instructions 811 that when executed by the processor(s) 805, cause carrying out of the methods described herein.
  • the storage subsystem 807 is configured to store one or more tuning parameters 813 that can be used to vary some of the processing steps carried out by the processing system 803.
  • the system shown in FIG. 8 can be incorporated in a specialized device such as a headset, e.g., a wireless Bluetooth headset.
  • a headset e.g., a wireless Bluetooth headset.
  • the system also can be part of a general purpose computer, e.g., a personal computer configured to process audio signals.
  • the post-processing e.g., the median filtering is controlled by signal classification as determined by a VAD.
  • VAD signal classification
  • the invention is not limited to any particular type of VAD, and many VADs are known in the art.
  • the inventors When applied to suppression, the inventors have discovered that suppression works best when different parts of the suppression system are controlled by different VADs, each such VAD custom designed for the functions of the suppressor in which it is used in, rather than having an "optimal" VAD for all uses. Therefore, in some versions of the input processing for noise reduction, a plurality of VADs, each controlled by a small set of tuning parameters that separately control sensitivity and selectivity, including spatial selectivity, such parameters tuned according to the suppression elements in which the VAD is used.
  • Each of the plurality of the VADs is an instantiation of a universal VAD that determines indications of voice activity from Y' b .
  • the universal VAD is controlled by a set of parameters and uses an estimate of noise spectral content, the banded frequency domain amplitude metric representation of the echo, and the banded spatial features.
  • the set of parameters includes whether the estimate of noise spectral content is spatially selective or not.
  • the type of indication of voice activity that a particular instantiation determines is controlled by a selection of the parameters.
  • BeamGain' b BeamGain min + (1- BeamGain min ) RPI' b ⁇ PPI' b ⁇ CPI' b
  • BeamGainExp is a parameter that for larger values increases the aggressiveness of the spatial selectivity of the VAD, and is 0 for a non-spatially-selective VAD
  • N' b ⁇ N' b,S denotes either the total noise power (or other frequency domain amplitude metric) estimate N' b , or
  • ⁇ N , ⁇ E are between 1 and 4.
  • BeamGainExp is between 0.5 to 2.0 when spatial selectivity is desired, and is 1.5 for one embodiment of a spatially-selective VAD, e.g., used to control post-processing in some embodiments of the invention.
  • RPI' b , PPI' b , and CPI' b are, as above, three spatial probability indicators, namely the ratio probability indicator, the phase probability indicator, and the coherence probability indicator.
  • the above expression also controls the operation of the universal voice activity detecting method.
  • a binary decision or classifier can be obtained by considering the test S > S thresh as indicating the presence of voice. It should also be apparent that the value S can be used as a continuous indicator of the instantaneous voice level.
  • an improved useful universal VAD for operations such as transmission control or controlling the post processing could be obtained using a suitable "hang over" or period of continued indication of voice after a detected event. Such a hang over period may vary from 0 to 500ms, and in one embodiment a value of 200ms was used. During the hang over period, it can be useful to reduce the activation threshold, for example by a factor of 2/3. This creates increased sensitivity to voice and stability once a talk burst has commenced.
  • the noise in the above expression is N' b,S determined using an out-of-beam estimate of power (or other frequency domain amplitude metric).
  • Y sens is set to be around expected microphone and system noise level, obtained by experiments on typical components.
  • processor may refer to any device or portion of a device that processes electronic data, e.g., from registers and/or memory to transform that electronic data into other electronic data that, e.g., may be stored in registers and/or memory.
  • a "computer” or a “computing machine” or a “computing platform” may include one or more processors.
  • the methodologies described herein are, in some embodiments, performable by one or more processors that accept logic, instructions encoded on one or more computer-readable media. When executed by one or more of the processors, the instructions cause carrying out at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken is included. Thus, one example is a typical processing system that includes one or more processors. Each processor may include one or more of a CPU or similar element, a graphics processing unit (GPU), field-programmable gate array, application-specific integrated circuit, and/or a programmable DSP unit.
  • GPU graphics processing unit
  • DSP programmable DSP unit
  • the processing system further includes a storage subsystem with at least one storage medium, which may include memory embedded in a semiconductor device, or a separate memory subsystem including main RAM and/or a static RAM, and/or ROM, and also cache memory.
  • the storage subsystem may further include one or more other storage devices, such as magnetic and/or optical and/or further solid state storage devices.
  • a bus subsystem may be included for communicating between the components.
  • the processing system further may be a distributed processing system with processors coupled by a network, e.g., via network interface devices or wireless network interface devices.
  • the processing system requires a display, such a display may be included, e.g., a liquid crystal display (LCD), organic light emitting display (OLED), or a cathode ray tube (CRT) display.
  • a display e.g., a liquid crystal display (LCD), organic light emitting display (OLED), or a cathode ray tube (CRT) display.
  • the processing system also includes an input device such as one or more of an alphanumeric input unit such as a keyboard, a pointing control device such as a mouse, and so forth.
  • the term storage device, storage subsystem, or memory unit as used herein, if clear from the context and unless explicitly stated otherwise, also encompasses a storage system such as a disk drive unit.
  • the processing system in some configurations may include a sound output device, and a network interface device.
  • a non-transitory computer-readable medium is configured with, e.g., encoded with instructions, e.g., logic that when executed by one or more processors of a processing system such as a digital signal processing device or subsystem that includes at least one processor element and a storage subsystem, cause carrying out a method as described herein. Some embodiments are in the form of the logic itself.
  • a non-transitory computer-readable medium is any computer-readable medium that is not specifically a transitory propagated signal or a transitory carrier wave or some other transitory transmission medium. The term "non-transitory computer-readable medium" thus covers any tangible computer-readable storage medium.
  • Non-transitory computer-readable media include any tangible computer-readable storage media and may take many forms including non-volatile storage media and volatile storage media.
  • Non-volatile storage media include, for example, static RAM, optical disks, magnetic disks, and magneto-optical disks.
  • Volatile storage media includes dynamic memory, such as main memory in a processing system, and hardware registers in a processing system.
  • the storage subsystem thus a computer-readable storage medium that is configured with, e.g., encoded with instructions, e.g., logic, e.g., software that when executed by one or more processors, causes carrying out one or more of the method steps described herein.
  • the software may reside in the hard disk, or may also reside, completely or at least partially, within the memory, e.g., RAM and/or within the processor registers during execution thereof by the computer system.
  • the memory and the processor registers also constitute a non-transitory computer-readable medium on which can be encoded instructions to cause, when executed, carrying out method steps.
  • While the computer-readable medium is shown in an example embodiment to be a single medium, the term "medium” should be taken to include a single medium or multiple media (e.g., several memories, a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions.
  • a non-transitory computer-readable medium e.g., a computer-readable storage medium may form a computer program product, or be included in a computer program product.
  • the one or more processors operate as a standalone device or may be connected, e.g., networked to other processor(s), in a networked deployment, or the one or more processors may operate in the capacity of a server or a client machine in server-client network environment, or as a peer machine in a peer-to-peer or distributed network environment.
  • the term processing system encompasses all such possibilities, unless explicitly excluded herein.
  • the one or more processors may form a personal computer (PC), a media playback device, a headset device, a hands-free communication device, a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a game machine, a cellular telephone, a Web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.
  • PC personal computer
  • PDA personal digital assistant
  • embodiments of the present invention may be embodied as a method, an apparatus such as a special purpose apparatus, an apparatus such as a data processing system, logic, e.g., embodied in a non-transitory computer-readable medium, or a computer-readable medium that is encoded with instructions, e.g., a computer-readable storage medium configured as a computer program product.
  • the computer-readable medium is configured with a set of instructions that when executed by one or more processors cause carrying out method steps.
  • aspects of the present invention may take the form of a method, an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects.
  • the present invention may take the form of program logic, e.g., a computer program on a computer-readable storage medium, or the computer-readable storage medium configured with computer-readable program code, e.g., a computer program product.
  • embodiments of the present invention are not limited to any particular implementation or programming technique and that the invention may be implemented using any appropriate techniques for implementing the functionality described herein. Furthermore, embodiments are not limited to any particular programming language or operating system.
  • an element described herein of an apparatus embodiment is an example of a means for carrying out the function performed by the element for the purpose of carrying out the invention.
  • the short time Fourier transform is used to obtain the frequency bands
  • the invention is not limited to the STFT.
  • Transforms such as the STFT are often referred to as circulant transforms.
  • Most general forms of circulant transforms can be represented by buffering, a window, a twist (real value to complex value transformation) and a DFT, e.g., FFT.
  • a complex twist after the DFT can be used to adjust the frequency domain representation to match specific transform definitions.
  • the invention may be implemented by any of this class of transforms, including the modified DFT (MDFT), the short time Fourier transform (STFT), and with a longer window and wrapping, a conjugate quadrature mirror filter (CQMF).
  • MDFT modified DFT
  • STFT short time Fourier transform
  • CQMF conjugate quadrature mirror filter
  • MDCT Modified discrete cosine transform
  • MDST modified discrete sine transform
  • any one of the terms comprising, comprised of or which comprises is an open term that means including at least the elements/features that follow, but not excluding others.
  • the term comprising, when used in the claims should not be interpreted as being limitative to the means or elements or steps listed thereafter.
  • the scope of the expression a device comprising A and B should not be limited to devices consisting of only elements A and B.
  • Any one of the terms including or which includes or that includes as used herein is also an open term that also means including at least the elements/features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising.
  • Coupled when used in the claims, should not be interpreted as being limitative to direct connections only.
  • the terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other, but may be.
  • the scope of the expression “a device A coupled to a device B” should not be limited to devices or systems wherein an input or output of device A is directly connected to an output or input of device B. It means that there exists a path between device A and device B which may be a path including other devices or means in between.
  • “coupled to” does not imply direction.
  • a device A is coupled to a device B
  • a device B is coupled to a device A
  • Coupled may mean that two or more elements are either in direct physical or electrical contact, or that two or more elements are not in direct contact with each other but yet still co-operate or interact with each other.

Landscapes

  • Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Quality & Reliability (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computational Linguistics (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Circuit For Audible Band Transducer (AREA)
  • Cable Transmission Systems, Equalization Of Radio And Reduction Of Echo (AREA)
  • Soundproofing, Sound Blocking, And Sound Damping (AREA)

Claims (11)

  1. Verfahren (310) zum Anwenden einer Nachbearbeitung an rohen Band-Verstärkungen (111), um die rohen Band-Verstärkungen (111) zum Anwenden an ein oder mehrere Audiosignale (119) zu verbessern, um eine Verringerung des Rauschens und/oder eine Verringerung von Außerstandort-Signalen und/oder eine Verringerung von Echos und/oder eine wahrnehmungsdomänenbasierte Einpegelung und/oder eine wahrnehmungsdomänenbasierte Dynamikbereichssteuerung und/oder eine wahrnehmungsdomänenbasierte dynamische Entzerrung auszuführen, wobei das Verfahren (310) Folgendes umfasst:
    Anwenden einer Nachbearbeitung (315) an rohen Band-Verstärkungen (111), um nachbearbeitete Band-Verstärkungen (125) zum Anwenden an das eine oder die mehreren Audiosignale (119) zu erzeugen, wobei die rohen Band-Verstärkungen (111) durch eine Eingangsverarbeitung von einem oder mehreren Eingangsaudiosignalen (101) bestimmt werden, um die rohen Band-Verstärkungen (111) bei mehreren Frequenzbändern zu erzeugen, manche der Bänder mehr als einen Frequenz-Bin umfassen, die rohen Band-Verstärkungen (111) bestehen, um eine Verringerung des Rauschens und/oder eine Verringerung von Außerstandort-Signalen und/oder eine Verringerung von Echos und/oder eine wahrnehmungsdomänenbasierte Einpegelung und/oder eine wahrnehmungsdomänenbasierte Dynamikbereichssteuerung und/oder eine wahrnehmungsdomänenbasierte dynamische Entzerrung auszuführen,
    wobei das Erzeugen einer gewissen nachbearbeiteten Verstärkung (125) für ein gewisses Frequenzband zumindest eine Medianfilterung (315) unter Verwendung von rohen Verstärkungswerten für Frequenzbänder angrenzend zu dem gewissen Frequenzband beinhaltet, wodurch sich mediangefilterte Verstärkungen ergeben,
    wobei die Nachbearbeitung gemäß einer oder mehreren Eigenschaften stattfindet, einschließlich einer Endbedingung und einer Breite für die Medianfilterung (315), und
    wobei die Endbedingung der Medianfilterung und/oder die Breite der Medianfilterung von einer Signalklassifikation (115) des einen oder der mehreren Eingangsaudiosignale (101) abhängt bzw. abhängen.
  2. Verfahren nach Anspruch 1, wobei die Nachbearbeitung ferner eine Frequenzband-zu-Frequenzband-Glättung und/oder eine Glättung im Laufe der Zeit (317) der mediangefilterten Verstärkungen umfasst und optional wobei die Frequenzband-zu-Frequenzband-Glättung und/oder die Glättung im Laufe der Zeit (317) von einer Signalklassifikation (115) abhängt bzw. abhängen.
  3. Verfahren nach Anspruch 1 oder 2, wobei die Signalklassifikation (115) beinhaltet, ob das eine oder die mehreren Eingangsaudiosignale (101) wahrscheinlich Wind sind oder nicht.
  4. Verfahren nach einem der Ansprüche 1 bis 3, wobei die Signalklassifikation (115) beinhaltet, ob das eine oder die mehreren Eingangsaudiosignale (101) wahrscheinlich Sprache sind oder nicht, oder wobei die Signalklassifikation (115) beinhaltet, ob das eine oder die mehreren Eingangsaudiosignale (101) wahrscheinlich Rauschen sind oder nicht.
  5. Verfahren nach einem der Ansprüche 1 bis 4, wobei die Frequenzbänder auf einer Wahrnehmungsskala oder logarithmischen Skala liegen.
  6. Verfahren nach einem der Ansprüche 1 bis 5, wobei die Eingangsverarbeitung die rohen Band-Verstärkungen (111) zur Verringerung des Rauschens bestimmen soll.
  7. Verfahren nach einem der Ansprüche 1 bis 5, wobei die Eingangsverarbeitung die rohen Band-Verstärkungen (111) aus mehr als einem Eingangsaudiosignal (101) zur Verringerung des Rauschens und von Außerstandort-Signalen bestimmen soll.
  8. Verfahren nach einem der Ansprüche 1 bis 5, wobei die Eingangsverarbeitung die rohen Band-Verstärkungen (111) aus dem einen oder den mehreren Eingangsaudiosignalen (101) und einem oder mehreren Referenzsignalen (103) bestimmen soll, wobei die bestimmten Verstärkungen (111) zur Verringerung des Rauschens und von Echos sind.
  9. Verfahren nach einem der Ansprüche 1 bis 8, wobei die Eingangsverarbeitung die rohen Band-Verstärkungen (111) für eine wahrnehmungsdomänenbasierte Einpegelung und/oder eine wahrnehmungsdomänenbasierte Dynamikbereichssteuerung und/oder eine wahrnehmungsdomänenbasierte dynamische Entzerrung bestimmen soll.
  10. Greifbares computerlesbares Speichermedium, das Anweisungen umfasst, die bei Ausführung durch mindestens einen Prozessor eines Verarbeitungssystems bewirken, dass ein Verfahren nach einem der Ansprüche 1 bis 9 ausgeführt wird.
  11. Vorrichtung (121) zum Nachbearbeiteten von rohen Band-Verstärkungen (111) zum Anwenden an einem Audiosignal (119), wobei die rohen Band-Verstärkungen (111) durch eine Eingangsverarbeitung von einem oder mehreren Eingangsaudiosignalen (101) bestimmt werden, wobei die Vorrichtung (121) einen oder mehrere Prozessoren und ein greifbares computerlesbares Speichermedium umfasst, das Anweisungen umfasst, die bei Ausführung durch mindestens einen Prozessor des einen oder der mehreren Prozessoren bewirken, dass ein Verfahren nach einem der Ansprüche 1 bis 9 ausgeführt wird.
EP12707413.6A 2011-02-10 2012-02-08 Nachbearbeitung mit medianfilterung von rauschunterdrückungsverstärkungen Active EP2673778B1 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201161441611P 2011-02-10 2011-02-10
PCT/US2012/024372 WO2012109385A1 (en) 2011-02-10 2012-02-08 Post-processing including median filtering of noise suppression gains

Publications (2)

Publication Number Publication Date
EP2673778A1 EP2673778A1 (de) 2013-12-18
EP2673778B1 true EP2673778B1 (de) 2018-10-10

Family

ID=45809594

Family Applications (2)

Application Number Title Priority Date Filing Date
EP12707413.6A Active EP2673778B1 (de) 2011-02-10 2012-02-08 Nachbearbeitung mit medianfilterung von rauschunterdrückungsverstärkungen
EP12707412.8A Active EP2673777B1 (de) 2011-02-10 2012-02-08 Kombinierte lärmunterdrückung und standortexterne signale

Family Applications After (1)

Application Number Title Priority Date Filing Date
EP12707412.8A Active EP2673777B1 (de) 2011-02-10 2012-02-08 Kombinierte lärmunterdrückung und standortexterne signale

Country Status (4)

Country Link
EP (2) EP2673778B1 (de)
JP (1) JP6002690B2 (de)
CN (2) CN103348408B (de)
WO (2) WO2012109385A1 (de)

Families Citing this family (41)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9313597B2 (en) 2011-02-10 2016-04-12 Dolby Laboratories Licensing Corporation System and method for wind detection and suppression
US9538286B2 (en) 2011-02-10 2017-01-03 Dolby International Ab Spatial adaptation in multi-microphone sound capture
US8804977B2 (en) 2011-03-18 2014-08-12 Dolby Laboratories Licensing Corporation Nonlinear reference signal processing for echo suppression
CN103325380B (zh) 2012-03-23 2017-09-12 杜比实验室特许公司 用于信号增强的增益后处理
CN104050969A (zh) 2013-03-14 2014-09-17 杜比实验室特许公司 空间舒适噪声
GB201309773D0 (en) 2013-05-31 2013-07-17 Microsoft Corp Echo removal
GB201309779D0 (en) 2013-05-31 2013-07-17 Microsoft Corp Echo removal
GB201309777D0 (en) * 2013-05-31 2013-07-17 Microsoft Corp Echo suppression
GB201309771D0 (en) 2013-05-31 2013-07-17 Microsoft Corp Echo removal
JP6087762B2 (ja) * 2013-08-13 2017-03-01 日本電信電話株式会社 残響抑圧装置とその方法と、プログラムとその記録媒体
TWI498884B (zh) * 2013-09-09 2015-09-01 Pegatron Corp 具有過濾背景音功能的電子裝置及其方法
EP2854133A1 (de) * 2013-09-27 2015-04-01 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Erzeugung eines Abwärtsmischsignals
CN104681034A (zh) 2013-11-27 2015-06-03 杜比实验室特许公司 音频信号处理
EP3080975B1 (de) * 2013-12-12 2017-07-12 Koninklijke Philips N.V. Echounterdrückung
US10079941B2 (en) 2014-07-07 2018-09-18 Dolby Laboratories Licensing Corporation Audio capture and render device having a visual display and user interface for use for audio conferencing
CN104200817B (zh) * 2014-07-31 2017-07-28 广东美的制冷设备有限公司 语音控制方法和系统
JP6195548B2 (ja) * 2014-08-19 2017-09-13 日本電信電話株式会社 信号解析装置、方法、及びプログラム
US9953661B2 (en) * 2014-09-26 2018-04-24 Cirrus Logic Inc. Neural network voice activity detection employing running range normalization
WO2016112968A1 (en) * 2015-01-14 2016-07-21 Widex A/S Method of operating a hearing aid system and a hearing aid system
CA2990891A1 (en) 2015-06-30 2017-01-05 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forchung E.V. Method and device for associating noises and for analyzing
US9401158B1 (en) * 2015-09-14 2016-07-26 Knowles Electronics, Llc Microphone signal fusion
CN107493247B (zh) * 2016-06-13 2021-10-22 中兴通讯股份有限公司 一种自适应均衡方法、装置及均衡器
US9906859B1 (en) * 2016-09-30 2018-02-27 Bose Corporation Noise estimation for dynamic sound adjustment
CN106782591B (zh) * 2016-12-26 2021-02-19 惠州Tcl移动通信有限公司 一种在背景噪音下提高语音识别率的装置及其方法
CA3080399A1 (en) 2017-10-30 2019-05-09 The Research Foundation For The State University Of New York System and method associated with user authentication based on an acoustic-based echo-signature
CN107886965B (zh) * 2017-11-28 2021-04-20 游密科技(深圳)有限公司 游戏背景音的回声消除方法
CN108447500B (zh) * 2018-04-27 2020-08-18 深圳市沃特沃德股份有限公司 语音增强的方法与装置
US10847178B2 (en) * 2018-05-18 2020-11-24 Sonos, Inc. Linear filtering for noise-suppressed speech detection
JP7195344B2 (ja) * 2018-07-27 2022-12-23 ドルビー ラボラトリーズ ライセンシング コーポレイション パーベイシブ・リステニングのための強制ギャップ挿入
CN111145770B (zh) * 2018-11-02 2022-11-22 北京微播视界科技有限公司 音频处理方法和装置
CN109949820B (zh) * 2019-03-07 2020-05-08 出门问问信息科技有限公司 一种语音信号处理方法、装置及系统
CN110097884B (zh) * 2019-06-11 2022-05-17 大众问问(北京)信息科技有限公司 一种语音交互方法和装置
CN110136739B (zh) * 2019-07-03 2021-04-09 四川大学 语音信号中冲激性噪声的去除方法
CN112420073B (zh) * 2020-10-12 2024-04-16 北京百度网讯科技有限公司 语音信号处理方法、装置、电子设备和存储介质
CN112235693B (zh) * 2020-11-04 2021-12-21 北京声智科技有限公司 麦克风信号处理方法、装置、设备及计算机可读存储介质
AU2022218336A1 (en) * 2021-02-04 2023-09-07 Neatframe Limited Audio processing
GB202101561D0 (en) * 2021-02-04 2021-03-24 Neatframe Ltd Audio processing
CA3211264A1 (en) * 2021-03-11 2022-09-15 Sascha Disch Audio decorrelator, processing system and method for decorrelating an audio signal
US11849291B2 (en) * 2021-05-17 2023-12-19 Apple Inc. Spatially informed acoustic echo cancelation
CN113689878A (zh) * 2021-07-26 2021-11-23 浙江大华技术股份有限公司 回声消除方法、回声消除装置及计算机可读存储介质
EP4198976B1 (de) * 2021-12-17 2023-10-25 GN Audio A/S Windgeräuschunterdrückungssystem

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2913105B2 (ja) * 1989-03-10 1999-06-28 日本電信電話株式会社 音響信号検出方法
DE4405723A1 (de) * 1994-02-23 1995-08-24 Daimler Benz Ag Verfahren zur Geräuschreduktion eines gestörten Sprachsignals
JP3484801B2 (ja) * 1995-02-17 2004-01-06 ソニー株式会社 音声信号の雑音低減方法及び装置
US5587998A (en) * 1995-03-03 1996-12-24 At&T Method and apparatus for reducing residual far-end echo in voice communication networks
US6351731B1 (en) * 1998-08-21 2002-02-26 Polycom, Inc. Adaptive filter featuring spectral gain smoothing and variable noise multiplier for noise reduction, and method therefor
JP3435357B2 (ja) * 1998-09-07 2003-08-11 日本電信電話株式会社 収音方法、その装置及びプログラム記録媒体
KR100467020B1 (ko) 2002-07-26 2005-01-24 삼성전자주식회사 자기 정렬된 접합영역 콘택홀을 갖는 반도체 장치 및 그제조 방법
JP4247037B2 (ja) * 2003-01-29 2009-04-02 株式会社東芝 音声信号処理方法と装置及びプログラム
MXPA05012785A (es) 2003-05-28 2006-02-22 Dolby Lab Licensing Corp Metodo, aparato y programa de computadora para el calculo y ajuste de la sonoridad percibida de una senal de audio.
DE602004017603D1 (de) * 2004-09-03 2008-12-18 Harman Becker Automotive Sys Sprachsignalverarbeitung für die gemeinsame adaptive Reduktion von Störgeräuschen und von akustischen Echos
US8280730B2 (en) * 2005-05-25 2012-10-02 Motorola Mobility Llc Method and apparatus of increasing speech intelligibility in noisy environments
JP4671303B2 (ja) * 2005-09-02 2011-04-13 国立大学法人北陸先端科学技術大学院大学 マイクロホンアレイ用ポストフィルタ
WO2007049643A1 (ja) * 2005-10-26 2007-05-03 Nec Corporation エコー抑圧方法及び装置
US8068619B2 (en) * 2006-05-09 2011-11-29 Fortemedia, Inc. Method and apparatus for noise suppression in a small array microphone system
US8005237B2 (en) * 2007-05-17 2011-08-23 Microsoft Corp. Sensor array beamformer post-processor
JP4850191B2 (ja) * 2008-01-16 2012-01-11 富士通株式会社 自動音量制御装置及びそれを用いた音声通信装置
CN101510426B (zh) * 2009-03-23 2013-03-27 北京中星微电子有限公司 一种噪声消除方法及系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
WO2012109384A1 (en) 2012-08-16
JP2014510452A (ja) 2014-04-24
EP2673778A1 (de) 2013-12-18
CN103354937A (zh) 2013-10-16
EP2673777A1 (de) 2013-12-18
CN103348408B (zh) 2015-11-25
WO2012109385A1 (en) 2012-08-16
EP2673777B1 (de) 2018-12-26
CN103348408A (zh) 2013-10-09
JP6002690B2 (ja) 2016-10-05
CN103354937B (zh) 2015-07-29

Similar Documents

Publication Publication Date Title
EP2673778B1 (de) Nachbearbeitung mit medianfilterung von rauschunterdrückungsverstärkungen
EP2880655B1 (de) Perzentilfilterung einer rauschunterdrückungsverstärkung
US8712076B2 (en) Post-processing including median filtering of noise suppression gains
US11694711B2 (en) Post-processing gains for signal enhancement
US9173025B2 (en) Combined suppression of noise, echo, and out-of-location signals
US10482896B2 (en) Multi-band noise reduction system and methodology for digital audio signals
US6377637B1 (en) Sub-band exponential smoothing noise canceling system
US8861745B2 (en) Wind noise mitigation
US20070174050A1 (en) High frequency compression integration
JP2005514668A (ja) スペクトル出力比依存のプロセッサを有する音声向上システム

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20130910

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAX Request for extension of the european patent (deleted)
17Q First examination report despatched

Effective date: 20150318

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: DOLBY LABORATORIES LICENSING CORPORATION

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602012052000

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G10L0021020000

Ipc: G10L0021020800

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 21/0208 20130101AFI20180323BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIN1 Information on inventor provided before grant (corrected)

Inventor name: DICKINS, GLENN N.

INTG Intention to grant announced

Effective date: 20180504

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 1052179

Country of ref document: AT

Kind code of ref document: T

Effective date: 20181015

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602012052000

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20181010

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1052179

Country of ref document: AT

Kind code of ref document: T

Effective date: 20181010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190110

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190110

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190210

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190210

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190111

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602012052000

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

26N No opposition filed

Effective date: 20190711

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190208

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20190228

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190228

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190208

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190208

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20120208

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181010

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230512

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240123

Year of fee payment: 13

Ref country code: GB

Payment date: 20240123

Year of fee payment: 13

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20240123

Year of fee payment: 13