EP2372707B1 - Transformation spectrale adaptative pour signaux vocaux acoustiques - Google Patents

Transformation spectrale adaptative pour signaux vocaux acoustiques Download PDF

Info

Publication number
EP2372707B1
EP2372707B1 EP10156530A EP10156530A EP2372707B1 EP 2372707 B1 EP2372707 B1 EP 2372707B1 EP 10156530 A EP10156530 A EP 10156530A EP 10156530 A EP10156530 A EP 10156530A EP 2372707 B1 EP2372707 B1 EP 2372707B1
Authority
EP
European Patent Office
Prior art keywords
spectral
representation
representations
speech
frequency
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP10156530A
Other languages
German (de)
English (en)
Other versions
EP2372707A1 (fr
Inventor
Jochen Withopf
Patrick Hannon
Mohamed Krini
Gerhard Uwe Schmidt
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SVOX AG
Original Assignee
SVOX AG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SVOX AG filed Critical SVOX AG
Priority to EP10156530A priority Critical patent/EP2372707B1/fr
Publication of EP2372707A1 publication Critical patent/EP2372707A1/fr
Application granted granted Critical
Publication of EP2372707B1 publication Critical patent/EP2372707B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0316—Speech enhancement, e.g. noise reduction or echo cancellation by changing the amplitude
    • G10L21/0364—Speech enhancement, e.g. noise reduction or echo cancellation by changing the amplitude for improving intelligibility

Definitions

  • the present invention generally relates to speech synthesis technology.
  • Fig. 1 a spectrogram of the utterance "Sauerkraut is served" is depicted. The phoneme /s/ is spoken at approximately 0, 0.8, and 1.4 seconds. At these times almost 100% of the signal energy is above 4 kHz.
  • the speech intelligibility may still be sufficient because the listener is often able to predict missing phonemes from syntax and the context of what is being said. However, errors arise if such a prediction is not possible, e.g., because names or unknown words of a foreign language are transmitted. Furthermore, the phoneme /s/ is important in the English language to indicate the plural and possessive pronouns.
  • the compression factor c can also be set frequency dependently to give higher or lower compression in certain regions.
  • this system needs to transmit side information about the mapping laws that have been used to ensure correct demapping. Additionally, the receiver must be able to handle this side information and to perform the demapping.
  • US 5 771 299 aims to compress or expand the spectral envelope of an input signal.
  • the goal is to move signal information from a region of the spectrum that is outside of the audible range of hearing aid users into another range that is still audible for the user.
  • the system uses an LPC analysis filter and transmits the filter coefficients to the synthesis filter directly. Then, with the help of all-pass filters non-integer delays are introduced in the analysis and/or the synthesis filters. Delays larger than 1 compress the spectral envelope while delays smaller than 1 expand the spectral envelope.
  • the main problem with this system is that the voice specific characteristics such as formant positions are not preserved, since they are also compressed/expanded as part of the signal processing. This has the effect of enhancing audibility for the hearing impaired, but not of preserving the audio quality of the original input speech signal. Further disadvantages are delays of the output signal.
  • US 2009 0074197 discloses a similar transposition but with user-dependent information for spatial hearing.
  • the goal is to perform a frequency transposition that moves frequency regions of the input signal to user specific ranges that are measured using built-in mechanisms of the hearing aid.
  • a method according to EP 1 333 700 A2 applies a perception-based nonlinear transposition function to the input spectrum.
  • the same transposition function is applied over the whole signal so that artifacts resulting from switching between non-transposition and transposition processing are avoided.
  • the use of a single function over the entire speech signal ignores time-varying phoneme-specific characteristics in the signal.
  • transforming the input signal to and then from the perception based scale to apply the transposition function reduces spectral resolution. As a result, characteristics of parts of the spectrum, which would otherwise only be subjected to a linear section of the transposition function, are not accurately preserved.
  • US 2006 0226016 aims to correct the phase of a transpositioned spectrum, wherein the transposition system is always active. Such processing of voiced segments has a negative effect on the phase and harmonic characteristics of the signal.
  • the input signal is subjected to a high pass (or band pass) filter, after which the spectral envelope of the high pass signal is estimated using an all-pole model.
  • a warping function is then applied to the all-pole model, translating the poles to lower frequencies using both non-linear and linear warping factors.
  • An excitation signal (the prediction error signal) is then applied to the newly transposed all-pole model and shaped according to this transposed spectral envelope to create a synthesized transposed signal.
  • the synthesized signal with an optional amplification and the original low-pass signal are then summed to create a signal containing compressed and transposed segments of the original spectrum.
  • the LPC analysis and voice synthesis step is costly and the quality of such signals usually sounds quite artificial without proper postprocessing. This postprocessing is also expensive and this method is therefore not suitable for a system focusing on improved audio quality compared to systems for the hearing-impaired.
  • US 6 577 739 describes a proportional compression factor to the frequency spectrum generated by an FFT of the input signal.
  • the factors are set to be between 0.5 and 0.99, and are applied using different methods.
  • linear interpolation and assuming a compression factor of 0.5, the contribution of two input FFT bins are calculated and applied to one output IFFT pin.
  • Another method would use an IFFT length double that of the input FFT length.
  • a frequency transposition can be applied as well. This method again places a compressed frequency range of an input signal in another range in the output signal, which is still available to the communication channel or the end user of a hearing aid device.
  • CA 2 569 221 describes an attempt to retain information in frequency ranges outside of a band pass threshold.
  • the input signal is converted to the frequency domain via the FFT algorithm or a polyphase filterbank.
  • a compression function is then applied.
  • an amplification is applied in accordance to the energy level of the uncompressed portion of the input signal.
  • the system can also comprise sending the compressed signal to an automatic speech recognition module. This method can have negative compression effects by giving portions of the input spectrum either too much or too little emphasis.
  • WO 2008/0123886 first defines a pass band for the output signal, and then defines a threshold, generally lower than the pass band, above which the frequency compression is applied. In the frequency region above the threshold, the highest frequency still of interest is identified and the appropriate compression is then applied. After the compression, the new peak power is normalized in a manner proportional to the compression, or is simply halved by -3 dB. Additionally, this method provides for the ability to expand a received signal, compressed or otherwise and synthetically reconstruct high frequency portions of the signal that may or may not have been present in the original transmitted signal. This can have a negative effect on the audio quality of the signal. This method still neglects the time-varying speaker dependent characteristics of speech sounds and applies the same compression function to each frame.
  • WO 20051015952 discloses a method of enhancing sound for a hearing-impaired listener which comprises manipulating the frequency of high frequency components of the sound in a high frequency band.
  • the object of the present invention is to improve at least one out of controllability, precision, signal quality, processing load, and computational complexity.
  • the inventive method for adaptive spectral transformation for acoustic speech signals comprises the steps of
  • Selected spectral representations are classified by assigning them to spectral class representation.
  • the spectral transformation applied to a selected spectral representation is adapted to this selected spectral representation because the applied spectral transformation enhances the assigned spectral class representation.
  • the adaptations to enhance the spectral class representations are made in a setup procedure and include heuristic steps in order to find transformations which enhance the spectral class representations.
  • the adapted advantageous transformations for each cluster centre respectively for each class of the code book ensure enhancement of the spectral representation of all the spectral representations assigned to this class of the code book. The controllability, precision and signal quality are enhanced while the processing load and computational complexity are reduced.
  • Best results can easily be found by the use of an appropriate code book, respectively by selecting an appropriate speech corpus and an appropriate clustering algorithm.
  • Clustering of spectral representation with sufficient energy in the 4 to 8 kHz band allows specific enhancement of fricatives such as /s/, /sh/, /ch/, /z/ or /f/.
  • the step of finding transformations which enhance fricatives of different classes allows linking of very specific transformations to the classes of the code book.
  • An English speech corpus can for example be taken from the TIMIT acoustic-phonetic continuous speech data base.
  • This speech data base has been designed by the Massachusetts Institute of Technology (MIT), SRI International (SRI) and Texas Instruments, Inc. (TI) for acoustic-phonetic studies. It contains utterances of 630 male and female speakers in American English, divided into eight dialect regions. From each person, ten phonetically rich sentences have been recorded and tagged with phonetic information.
  • the training data can be reduced to 70 speakers from all dialect regions and pauses can been removed with the help of phoneme tags. The reduced training data has a duration of approximately 23 minutes.
  • a second data set can been extracted that consists for example of 10 minutes speech data from 30 speakers.
  • Frequency compression is compressing the bandwidth for example from a bandwidth of 0 to 8 kHz to an bandwidth of 0 to 4 kHz preferably with a compression only at the upper or lower end of the bandwidth, optionally linear at least in the middle frequency range, corresponding to no compression when the slope is equal to 1.
  • Formant boosting takes place using formant-dependent functions to increase the contrast between the formants and the non-formant frequencies in the frequency spectrum.
  • the formant boosting gain function linked to each spectral class representation of the code book is amplifying at least one preferably two or three of the formants at low frequencies.
  • Assigning the at least one selected spectral representation to one of a set of cluster centres includes
  • Calculating distance measures becomes very simple by first calculating feature vectors to the spectral representations.
  • the distance measures are just distances between the feature vectors.
  • the feature vectors are calculated from spectral envelope representations by a filtering transformation, preferably with a mel-filterbank, wherein the mel-filterbank optionally uses overlapping triangular windows with widths variable with frequency.
  • a code book including at least eight (for example for fricative enhancement), preferably thirty-two, optionally 128 cluster centres. With higher class numbers more different enhancement problems can be solved each in a different way by the linked at least one spectral transformations.
  • the spectral class representations for the cluster centres are preferably averaged spectral representations averaged over spectral representations of corresponding cluster elements.
  • a reduction to spectral class representations of special interest can be made by applying the clustering algorithm on the bases of a preselected sub-corpus of the speech corpus.
  • the sub-corpus can for example be reduced to spectral representations of the speech corpus which have a spectral centroid lying above a given threshold frequency, preferably of 3 kHz.
  • Transformations are only needed for spectral representation which can be improved.
  • a selection can be made by calculating the spectral centroid of each spectral input representation and selecting spectral input representations which have a spectral centroid lying above a threshold, frequency preferably above 3 kHz. Such a selection fits to a code book base on a sub-corpus of the speech corpus with the same threshold frequency.
  • a selection can also be made by detecting at least one of speech activity and background noise level. Spectral input representations with speech to be transformed will be selected with appropriate selection criteria.
  • the method for adaptive spectral transformation for acoustic speech signals can be implemented in different fields.
  • the acoustic speech signal is windowed and of each window a spectral representation is deduced.
  • the spectral representations of windows of an acoustic speech signal are provided by a system. Therefore the method of this invention starts with the step of receiving spectral input representations, which can be provided by the same system or by another system. At least some of the received spectral input representations are selected and enhanced by adapted transformations and the enhanced spectral representations are provided in the form of at least one spectral output representation.
  • the combination of untransformed and transformed spectral representations allow synthesizing an enhanced acoustic speech signal.
  • the invention can be implemented in a computer program comprising program code means for performing all the steps of the disclosed methods when said program is run on a computer.
  • the described solutions preserve and enhance voice specific characteristics such as formants and their respective positions. This has the effect of enhancing audibility, while preserving the audio quality of the original input speech signal.
  • the current invention also avoids unnecessary delays of the output signal.
  • Various compression functions are applied to the speech signal, which takes into account the time-varying phoneme-specific characteristics in the signal and allows for spectral sharpening through adaptive formant boosting. Also, unnecessary transformations of the input signal, which could reduce spectral resolution are avoided.
  • the effect of phase distortion on the harmonic section of the input signal can be avoided in the current invention by limiting the compression processing to only unvoiced segments of speech.
  • a focus on efficient algorithms and output signal synthesis allows the current invention to also avoid costly postprocessing to achieve high quality audio output.
  • transforming the input signal from the time domain to the frequency domain is not bound to the FFT (Fast Fourier Transform) algorithm. Compression functions are intended to be designed such that they are able to be efficiently stored in memory and do not inadvertently give portions of the input spectrum undesired emphasis.
  • Fig. 1 shows for the acoustic speech "Sauerkraut is served once a week” the energy distribution in time and frequency, a time series of the energy above 4 kHz and a time series of the amplitude.
  • the fricatives "s", “ce” and “k” have quite some energy in the frequency range from 4 to 8 kHz.
  • a time-domain input signal as for example "Sauerkraut is served once a week" is windowed before being transferred to the frequency domain via a fast Fourier transform (FFT) algorithm, discrete Fourier transform (DFT), discrete cosine transform (DCT), polyphase filterbank, wavelet transform, or some other time to frequency domain transformation.
  • FFT fast Fourier transform
  • DFT discrete Fourier transform
  • DCT discrete cosine transform
  • polyphase filterbank discrete cosine transform
  • wavelet transform or some other time to frequency domain transformation.
  • the windowed time-domain signal has the form of spectral representations.
  • the spectral representations can be reduced to feature vectors.
  • Fig. 2 shows an example of clusters of two dimensional feature vectors.
  • the procedure is similar for feature vectors with higher dimensionality.
  • the spectral representations or the feature vectors of the cluster centres are defined on the bases of spectral representations of windowed acoustic speech segments of a speech corpus by the clustering algorithm.
  • a code book includes the spectral class representations for the cluster centres, which are averages over the elements of the cluster.
  • the line with more details shows the average spectrum in dB, the line with less details shows a filtered spectral representation of the code book entry and the vertical line the spectral centroid fc .
  • the filtered spectral representations of the code book entries are the spectral class representations for the cluster centres, which are averages over the elements of the cluster.
  • the spectral representations or feature vectors of windowed frames of an input signal are subjected to a classification technique from the field of pattern recognition, such as the minimum mean squared error estimator.
  • the input frames are classified by finding the class corresponding to the smallest value of a cost function, d( v,c k ) in Equation 1, where v d is the D-dimensional set of features of the current frame and C dk is the set of features of a codebook entry k.
  • the codebook can be trained using feature vectors from a training set and any number of vector quantization algorithms, such as k-means [MacQueen, 1967] or the Linde-Buzo-Gray (LBG) algorithm[Linde, Buzo, & Gray, 1980].
  • LBG Linde-Buzo-Gray
  • LBG Linde-Buzo-Gray
  • LBG Linde-Buzo-Gray
  • LBG Linde-Buzo-Gray
  • LBG Linear discriminant analysis
  • PCA principal component analysis
  • SVM support vector machines
  • class enhancing algorithms can also be applied to decrease the intraclass variances and/or increase the interclass distances, which improves separability.
  • the feature vectors for training and testing are in this case a set of perception based features in the mel scale, although the feature vectors must not be limited to these specific features.
  • the features can also be designed to emphasize signal characteristics in or near the region of interest in the frequency spectrum.
  • the key concept here is that the feature vector has a reduced dimensionality with respect to the input signal frequency spectrum in order to reduce computational costs.
  • a spectral compression function and/or a formant boosting function are chosen from the relevant codebook entry.
  • a smoothing procedure can be applied to the chosen compression function in the frequency domain but in the time direction.
  • an IIR-smoothing function the time-variance of the applied compression functions can be reduced.
  • the spectral compression functions are functions- in the preferred case continuous and nonlinear - that apply compression rates in the frequency domain. Both the compression functions and the frequency spectrum can be either linearly or nonlinearly scaled, reflecting the occasional advantages of processing in more perception based scales like the logarithmic scale.
  • spectral compression maps a frequency interval of the input signal into a smaller frequency interval of the output.
  • this is equivalent to mapping a set of frequency pins from the input spectrum X(e j ⁇ ) into one frequency pin of the output spectrum Y (e j ⁇ ).
  • Fig. 5 illustrates an example for the relationship between input frequency f in and output frequency f out .
  • the curved line shows the compression characteristic, the horizontal and vertical lines indicate the frequency interval that is mapped.
  • Equation 2 also performs energy normalization with respect to the amount of frequency compression.
  • Energy normalization can take many forms, however, and could also be applied in a fashion based on the momentary broadband or frequency localized SNR.
  • Another option is to maintain the form of the precompression estimated spectral envelope and apply a gain or dampening factor to correct the energy level to correspond to the slope of the envelope in the region of interest.
  • the equation also preserves the phase of the original uncompressed signal, but other phase corrections and adjustments are also possible, such as using a random phase.
  • the compression functions can be continuous in nature and have various compression rates over frequency.
  • the compression function is linear in lower frequencies, corresponding to no compression when the slope is equal to 1. In the higher frequencies of interest, the compression rate increases gradually, with the rate and degree of increase dependent upon the feature-based classification.
  • Fig. 6 various examples of continuous nonlinear compression functions are shown. Those displayed with solid lines are linear with a slope of, or near, 1 in the lower frequency range, hence performing little to no compression. With increasing input frequency, the compression rate increases differently for each of the curves.
  • the dashed curves perform little to no compression in the middle frequency range, rather than the low frequencies. These curves compress frequencies in the lower and higher frequency extremes, and can be better understood as performing a transposition of the middle frequencies down into a lower region with almost no compression.
  • the appropriate compression function for each class of the code book is to be defined either manually, e.g., using subjective listening criteria, or automatically, e.g., applying unsupervised learning methods to find a preferred mapping to an output characteristic.
  • Speech can be made more accentuated, if formants are amplified. Especially in situations with background noise, we can expect a better localized SNR in these frequency regions, so the broadband SNR can be improved by applying a frequency dependent gain factor to the input spectrum. Ideally, the amplification should only be applied during speech activity. Otherwise, background noise will be amplified during pauses.
  • the current invention can receive information about speech activity from an external module, such as a noise detection and cancelation module.
  • the formant-boosting functions are also functions - (non)continuous, (non)linear, and on a (non)linear scale - designed such that a variable gain factor is applied to frequency ranges around formants in the spectrum.
  • Fig. 7 shows a formant boosting gain function and spectral representations (cluster mean and filtered cluster mean), where the line with more details shows the average spectrum in dB, the line with less details shows a filtered spectral representation of the code book entry.
  • a curve can be used to determine the percentage of the gain factor that is applied to the formant frequency range.
  • individual gain and dampening curves can be stored in a codebook and applied to those frequency ranges identified as formants in voiced speech.
  • the gain factor just described can be transformed into a dampening factor using another set of curves.
  • the boosting curves can be designed to have positive values for amplifying the formant frequencies and negative values for the dampening curves to reduce the magnitude of the valleys between formants.
  • FIG. 8 An overall block diagram of one possible incarnation of the system is seen in Fig. 8 .
  • Windows of a signal in the time domain are transformed to spectral representations by an analysis filter-bank.
  • an analysis filter-bank By extracting feature vectors a classification in relation to elements of a code book a signal processing is realized with the transformations linked to the code book elements.
  • the transformed spectral representations are transformed to windows of a signal in the time domain by a synthesis filter-bank.
  • Fig. 9 shows examples for processing of phonemes:
  • Fig. 10 shows an example of the sauerkraut utterance processed with the 128 class scheme.
  • the energy of the fricatives "s", “ce” and “k” is transformed below 4 kHz.
  • the system is also capable of receiving information from other signal processing modules, such as silence/unvoiced/voiced decisions from the noise estimation and reduction module.
  • the module should be capable of sending information to other modules, especially an ASR module, which can use the enhanced signal to improve recognition rates.
  • the ASR module could be retrained with the compressed output data. However the compressed signal alone achieves a change in recognition rates that can be useful.
  • a focus on efficient algorithms and output signal synthesis allows the current invention to also avoid costly postprocessing to achieve high quality audio output. Additionally, transforming the input signal from the time domain to the frequency domain is not bound to the FFT algorithm. Compression functions in the current invention are intended to be designed such that they are able to be efficiently stored in memory and do not inadvertently give portions of the input spectrum undesired emphasis.
  • Another invention is disclosed, which is new and inventive independent of the independent claims.
  • This further invention is related to onset sharpening, which can be used independently but is of course advantageously combinable with the previously described speech enhancement methods.
  • the onset sharpening method performs an onset sharpening, i.e., to introduce attenuation immediately before and/or amplification after speech onsets in order to make them more accentuated. This is improving speech quality and intelligibility, especially for speech signals corrupted with background noise that are to be enhanced with noise reduction methods. Noise reduction filters and their derivatives ( Fig. 11 ) tend to react too slow and therefore do remove desired signal components during speech onsets.
  • the speech onsets need to be found first. This is done based on a recursive Wiener noise reduction filter. There are no real-time constraints, so all of the following steps can be applied to the entire signal x(t) giving the necessary data for the next step.
  • An analysis filter bank is needed to transform the input signal x(t) into the frequency domain.
  • the attenuation factors of the recursive Wiener filter characteristic can be computed.
  • the following, non-frequency dependent, parameters have been used as filter parameters:
  • a mel-filterbank of 32 bands is applied to G rec ( ⁇ ⁇ , I), resulting in the 32 x M matrix G rec mel (m, I).
  • the actual onset detection is performed within each mel-band of G rec mel (m, I).
  • the moments when G rec mel (m, I) changes its value from G min to 1 (or close to 1) are of interest, i.e., the points when the filter opens.
  • These time instances can be found by taking the numerical derivative d G rec mel m l d l ⁇ G rec mel m l - G rec mel ⁇ m , l - 1 and comparing the resulting value with a threshold d G rec mel m l d l > ⁇
  • the mel-band m for time instance I is labeled as a speech onset.
  • the derivative of the noise reduction coefficient lies in the range of d G rec mel m l / dl ⁇ G min - 1 , 1 - G min and positive values indicate times when the filter is opening.
  • a threshold 0.2 has proved to give good detection results for various speech signals and SNRs. Because it could happen that the derivative is greater than for several consecutive frames, also a sliding time window of 100ms duration is applied. Within this time window, only one detection is allowed. Furthermore, if an onset has been detected in a certain mel-band, the neighboring mel-band towards lower frequencies of the same frame I is also marked as a speech onset.
  • the clean speech is mixed with background noise recorded in a car driving at a speed of 160 km/h to form an SNR of 1 dB during speech activity.
  • the onset sharpening can be performed. It consists of placing attenuation immediately before a detected speech onset and boosting the signal for a short time interval afterwards.
  • the term e/ ⁇ is used to normalize f (x) to a maximum value of 1.
  • the onset sharpening gain function g 08 (I) can be sampled.
  • onset sharpening e.g., a sinusoid.
  • the prototype function has been chosen with the stated parameters because it decays smoothly towards the ends and offers a steep slope around the zero crossing.
  • This prototype function is now placed at each detected speech onset time instance in the corresponding mel-band, giving the onset sharpening matrix for met-bands g os mel ( m , l ), which then is expanded into the onset sharpening matrix for the subbands g os (m,l).
  • the weighting of the filters contained in A that gives triangles of a broader bandwidth a lower amplitude is removed by the normalization containing the maximum operation.
  • Fig. 12 discloses an onset sharpening algorithm with the following recursive Wiener noise reduction filter being modified by the onset sharpening gain function.
  • g os ⁇ ⁇ l / 20 is the onset sharpening function in linear values. Since the noise reduction filter is applied multiplicative to the input spectrum, this filter modification could also be interpreted as a multiplication of the signal spectrum X ( e j ⁇ , l ) with the onset sharpening gain before (or after) applying the noise reduction filter.
  • the onset sharpening gain is built into the noise reduction characteristic to modify the spectral floor and the maximum gain of the filter (which is set to 1 in the recursive Wiener filter):
  • G rec mod 2 ⁇ ⁇ l max G min ⁇ ⁇ l ⁇ g os att ⁇ ⁇ l , g os amp ⁇ ⁇ l - B ⁇ e j ⁇ ⁇ ⁇ l 2 X ⁇ e j ⁇ ⁇ ⁇ l 2
  • G rec mod 3 ⁇ ⁇ l max G min ⁇ ⁇ l ⁇ g os att ⁇ ⁇ l , 1 - ⁇ ⁇ ⁇ ⁇ l g os amp ⁇ ⁇ l ⁇ G ⁇ ⁇ ⁇ , l - 1 ⁇ B ⁇ e j ⁇ ⁇ ⁇ l 2 X ⁇ e j ⁇ ⁇ ⁇ l 2 Inspection of noise reduction filters from the first and second modification shows that there are only small differences between the characteristics.
  • the parameter settings for ⁇ os , ⁇ offs . ⁇ os and the choice of the gain prototype function are of much greater influence. Therefore, the simpler modification G rec mod 1 ⁇ ⁇ l will be used for the evaluation.
  • the recursive Wiener filter with G rec mod 1 ⁇ ⁇ l has been compared to a standard recursive Wiener filter G rec ( ⁇ ⁇ , l ). This has mainly been done on the basis of a logarithmic spectral distance (LSD) measure. Comparison of the noise reduction filter coefficients for several characteristics gives a qualitative impression about the opening/closing properties of a filter. Finally, listening tests give a useful criterion that help to judge intelligibility and the amount of artifacts such as musical tones.
  • LSD logarithmic spectral distance
  • a noise reduction filter is computed for the disturbed signal x(t) and the two signal components are passed through this filter separately.
  • the distortion measures LSD speech and LSD noise can be calculated between the original and the filtered signal.
  • the speech component is passed through the filter unchanged, leading to a distance close to zero, whereas large distortions can be expected for the noise component. Based on these two measures, it is possible to judge on the noise suppression ability and on how much speech components are affected.
  • variable D gives the number of signal frames that are used for the calculation of the LSD, i.e., the number of signal frames with K l , > 0.
  • a set of 616 filters has been computed with all possible combinations of the parameters ⁇ os ⁇ [50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100] ms ⁇ offs ⁇ [0, -5, -10, -15, -20, -25, -30, -35] ms ⁇ os ⁇ [0, 2, 4, 6, 8, 10, 12] .
  • the signal that has been used is the "Sauerkraut is serve once a week" utterance used throughout this text, mixed with background noise from a car driving at a constant speed of 160km/h.
  • the SNR d ring speech activity is adjusted to 1 dB.
  • only the speech and only the noise components have been processed with these filters and the distances LSD speech os and LSD noise os have been calculated.
  • a recursive Wiener filter has been designed and the measures LSD speech rw and LSD noise rw have been evaluated.
  • LSD speech rw - LSD speech os and LSD noise os - LSD noise rw could be calculated They are defined such, that a positive value means that the onset sharpening approach gives better results in the LSD sense. This is apparently not always the case and again it can be seen that an improvement in noise suppression leads to worse results for speech and vice versa.
  • the gain factor ⁇ os basically only scales the distance measure.
  • T os 0
  • the modified filter reduces to the standard recursive Wiener filter and thus gives the same LSD.
  • ⁇ os ⁇ off ⁇ os 75 ⁇ ms , - 10 ⁇ ms , 3
  • Wiener filter opens more often, which potentially results in musical tones. It has also been seen that the recursive Wiener filter opens a bit later, which can be corrected by the onset sharpening modification.
  • Fig. 11 shows a procedure where the attenuation factor and its derivative together with the threshold are shown for mel-band 25 (corresponding to frequencies between 3.6 and 4.4 kHz). Because it could happen that the derivative is greater than ⁇ for several consecutive frames, also a sliding time window of 100ms duration is applied. Within this time window, only one detection is allowed. Furthermore, if an onset has been detected in a certain mel-band, the neighboring mel-band towards lower frequencies of the same frame I is also marked as a speech onset.
  • the signal that has been used is the utterance "Sauerkraut is served once a week" from the TIMIT database that has also been used before.
  • the clean speech is mixed with background noise recorded in a car driving at a speed of 160 km/h to form an SNR of 1 dB during speech activity.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Quality & Reliability (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Claims (15)

  1. Procédé pour la transformation spectrale adaptive pour des signaux vocaux acoustiques comprenant les étapes suivantes :
    réception d'au moins une représentation spectrale d'entrée, qui correspond à au moins une fenêtre d'un signal d'entrée de plage de temporisation pour une langage acoustique,
    sélection d'au moins une représentation spectrale d'entrée à transformer parmi les représentations spectrales d'entrée,
    attribution des représentations spectrales sélectionnées à un centre de cluster d'un jeu de centres de cluster, dans laquelle
    les centres de cluster sont définis par un algorithme de clustering sur la base des segments de représentations spectrales d'intervalles de temps d'un corpus linguistique,
    que des représentations spectrales de classe sont associées aux centres de cluster et sont des éléments d'un livre de codes, et
    que le livre de codes assigne au moins une transformation spectrale à chaque représentation spectrale de classe, qui améliore la représentation spectrales de classe correspondante,
    transformation de chaque représentation spectrale sélectionnée en une représentation spectrale de sortie, dans laquelle la transformation appliquée correspond à l'au moins une transformation spectrale assignés au centre de cluster, qui est attribué à la représentation spectrale respective sélectionnée, et
    mettre en place des représentations spectrales de sortie pour synthétiser un signal vocal acoustique.
  2. Procédé selon la revendication 1, dans lequel l'attribution des représentations spectrales sélectionnées à un centre de cluster d'un jeu de centres de cluster comprend le suivait:
    calculer des mesures de distances entre représentation spectrale sélectionnée et toutes les représentations spectrales de classe du livre de codes, et
    attribuer l'au moins une représentation spectrale sélectionnée au centre de cluster ayant les mesures de distance plus courtes entre la représentation spectrale sélectionnée et les représentations spectrales de classe du centre de cluster.
  3. Procédé selon la revendication 2, dans lequel le calcul des mesures de distance comprend le suivant :
    calculer les vecteurs de caractéristiques pour les représentations spectrales, et les mesures de distance sont des distances entre les vecteurs de caractéristiques.
  4. Procédé selon la revendication 3, dans lequel les vecteurs de caractéristiques sont calculés à parti des représentations spectrales par une transformation à filtrage, de préférence par un Mel banc de filtres, dans lequel le Mel banc de filtres utilise, le cas échéant, des fenêtres triangulaires chevauchants l'une par rapport à l'autre ayant des largeurs variables avec la fréquence.
  5. Procédé selon une quelconque des revendications 1 à 4, dans lequel le livre de codes comprend au moins huit, de préférence trente deux, le cas échéant 128 centres de cluster.
  6. Procédé selon une quelconque des revendications 1 à 5, dans lequel les représentations spectrales de classe pour les centres de cluster sont des représentations spectrales moyennées, qui ont été moyennées pour les classes respectives via des représentation spectrales des éléments de cluster correspondants.
  7. Procédé selon une quelconque des revendications 1 à 6, dans lequel la définition des centres de classe est effectuée par l'algorithme de clustering à la base d'un sous-corpus du corpus linguistique.
  8. Procédé selon la revendication 7, dans lequel la présélection d'un sous-corpus comprend la réduction aux représentations spectrales du corpus linguistique, qui ont un centre spectral, qui est au-dessus d'une fréquence d'un nombre, de préférence au-dessus de 3 kHz.
  9. Procédé selon une quelconque des revendications 1 à 8, dans lequel une transformation spectrale attribuée au moins une de chaque représentation de classe du livre de codes est une transformation spectrale de compression, qui imagine un intervalle de fréquence de la représentation spectrale sélectionnée à un intervalle de fréquence plus petit de la représentation spectrale de sortie.
  10. Procédé selon la revendication 9, dans lequel chaque transformation spectrale de compression attribuée à chaque représentation spectrale de classe du livre de codes d'une largeur de bande de 0 à 8 kHz est comprimée à une largeur de bande de 0 à 4 kHz, de préférence avec une compression seulement au bout supérieur ou inférieur de la largeur de bande, le cas échéant avec pas de compression linéairement correspondante au moins dans la largeur médiane de bande de fréquence, si la pente est égale à 1.
  11. Procédé selon une quelconque des revendications 1 à 7, dans lequel une des transformations spectrales attribuée à au moins une de chaque représentation de classe du livre de codes est une fonction d'amplification de formants.
  12. Procédé selon la revendication 11, dans lequel la fonction d'amplification de formants attribuée à chaque représentation spectrale de classe du livre de codes amplifie au moins un formant, de préférence deux ou trois, avec des fréquences basses.
  13. Procédé selon la revendication 8, dans lequel la sélection d'au moins une représentation spectrale sélectionnée comprend le calcul du centre spectrale de chaque représentation spectrale d'entrée, ainsi que la sélection de ceux représentations spectrales d'entrée, qui ont un centre spectral, qui est au-dessus d'une fréquence d'un nombre, de préférence au-dessus de 3 kHz.
  14. Procédé selon une quelconque des revendications 1 à 13, dans lequel la sélection d'au moins une représentation spectrale sélectionnée à transformer comprend déterminer au moins l'activité linguistique ou le niveau de bruit de fond ainsi que la sélection des représentations spectrales d'entrée comprenant de langage à transformer.
  15. Programme d'ordinateur comprenant des moyens de programme adaptés pour réaliser toutes les étapes d'une des revendications 1 à 14, quand le programme tourne à un ordinateur.
EP10156530A 2010-03-15 2010-03-15 Transformation spectrale adaptative pour signaux vocaux acoustiques Active EP2372707B1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP10156530A EP2372707B1 (fr) 2010-03-15 2010-03-15 Transformation spectrale adaptative pour signaux vocaux acoustiques

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP10156530A EP2372707B1 (fr) 2010-03-15 2010-03-15 Transformation spectrale adaptative pour signaux vocaux acoustiques

Publications (2)

Publication Number Publication Date
EP2372707A1 EP2372707A1 (fr) 2011-10-05
EP2372707B1 true EP2372707B1 (fr) 2013-03-13

Family

ID=42342646

Family Applications (1)

Application Number Title Priority Date Filing Date
EP10156530A Active EP2372707B1 (fr) 2010-03-15 2010-03-15 Transformation spectrale adaptative pour signaux vocaux acoustiques

Country Status (1)

Country Link
EP (1) EP2372707B1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105632485A (zh) * 2015-12-28 2016-06-01 浙江大学 一种基于语种识别系统的语言距离关系的获取方法

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9672843B2 (en) 2014-05-29 2017-06-06 Apple Inc. Apparatus and method for improving an audio signal in the spectral domain
US11170761B2 (en) 2018-12-04 2021-11-09 Sorenson Ip Holdings, Llc Training of speech recognition systems
US11017778B1 (en) 2018-12-04 2021-05-25 Sorenson Ip Holdings, Llc Switching between speech recognition systems
US10388272B1 (en) 2018-12-04 2019-08-20 Sorenson Ip Holdings, Llc Training speech recognition systems using word sequences
US10573312B1 (en) 2018-12-04 2020-02-25 Sorenson Ip Holdings, Llc Transcription generation from multiple speech recognition systems
US11488604B2 (en) 2020-08-19 2022-11-01 Sorenson Ip Holdings, Llc Transcription of audio

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5771299A (en) 1996-06-20 1998-06-23 Audiologic, Inc. Spectral transposition of a digital audio signal
US6577739B1 (en) 1997-09-19 2003-06-10 University Of Iowa Research Foundation Apparatus and methods for proportional audio compression and frequency shifting
EP1333700A3 (fr) 2003-03-06 2003-09-17 Phonak Ag Procédé de transposition de fréquence dans une prothèse auditive et une telle prothèse auditive
AU2003904207A0 (en) * 2003-08-11 2003-08-21 Vast Audio Pty Ltd Enhancement of sound externalization and separation for hearing-impaired listeners: a spatial hearing-aid
US7785459B2 (en) 2004-10-22 2010-08-31 Hewlett-Packard Development Company, L.P. Microneedles and methods of fabricating
US8086451B2 (en) 2005-04-20 2011-12-27 Qnx Software Systems Co. System for improving speech intelligibility through high frequency compression
ES1063647Y (es) 2006-07-28 2007-02-16 Pujante Antonio Gallego Brazalete para regular la temperatura corporal por exceso de calor
AU2008203351B2 (en) 2007-08-08 2011-01-27 Oticon A/S Frequency transposition applications for improving spatial hearing abilities of subjects with high frequency hearing loss
US8000487B2 (en) 2008-03-06 2011-08-16 Starkey Laboratories, Inc. Frequency translation by high-frequency spectral envelope warping in hearing assistance devices

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105632485A (zh) * 2015-12-28 2016-06-01 浙江大学 一种基于语种识别系统的语言距离关系的获取方法
CN105632485B (zh) * 2015-12-28 2019-05-31 浙江大学 一种基于语种识别系统的语言距离关系的获取方法

Also Published As

Publication number Publication date
EP2372707A1 (fr) 2011-10-05

Similar Documents

Publication Publication Date Title
Kingsbury et al. Robust speech recognition using the modulation spectrogram
US6889182B2 (en) Speech bandwidth extension
EP2372707B1 (fr) Transformation spectrale adaptative pour signaux vocaux acoustiques
EP1252621B1 (fr) Systeme et procede de modification de signaux vocaux
EP1739657B1 (fr) Amélioration d'un signal de parole
US20020128839A1 (en) Speech bandwidth extension
KR101199431B1 (ko) 대역폭 확장 시스템에서 고대역 에너지를 추정하기 위한 방법 및 장치
EP1638083B1 (fr) Extension de la largeur de bande de signaux audio à bande limitée
DE10041512B4 (de) Verfahren und Vorrichtung zur künstlichen Erweiterung der Bandbreite von Sprachsignalen
KR100726960B1 (ko) 음성 처리에서의 인위적인 대역폭 확장 방법 및 장치
US8504360B2 (en) Automatic sound recognition based on binary time frequency units
Karray et al. Towards improving speech detection robustness for speech recognition in adverse conditions
EP2984649B1 (fr) Extraction de caractéristiques acoustiques d'excitation relative
EP2058803A1 (fr) Reconstruction partielle de la parole
US8311842B2 (en) Method and apparatus for expanding bandwidth of voice signal
Pulakka et al. Speech bandwidth extension using gaussian mixture model-based estimation of the highband mel spectrum
Marzinzik Noise reduction schemes for digital hearing aids and their use for the hearing impaired
EP0780828A2 (fr) Procédé et système de reconnaissance de la parole
Riazati Seresht et al. Spectro-temporal power spectrum features for noise robust ASR
CN110931037A (zh) 改进的Mel频率尺度与耳语音共振峰结合的耳语音增强算法
Withopf et al. Phoneme-Dependent Speech Enhancement.
Thomsen et al. Speech enhancement and noise-robust automatic speech recognition
Macho et al. On the use of wideband signal for noise robust ASR
Xie Removing redundancy in speech by modeling forward masking
Geravanchizadeh et al. Improving the noise-robustness of Mel-Frequency Cepstral Coefficients for speaker verification

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

AX Request for extension of the european patent

Extension state: AL BA ME RS

17P Request for examination filed

Effective date: 20120315

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAC Information related to communication of intention to grant a patent modified

Free format text: ORIGINAL CODE: EPIDOSCIGR1

RIN1 Information on inventor provided before grant (corrected)

Inventor name: WITHOPF, JOCHEN

Inventor name: HANNON, PATRICK

Inventor name: SCHMIDT, GERHARD UWE

Inventor name: KRINI, MOHAMED

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 601218

Country of ref document: AT

Kind code of ref document: T

Effective date: 20130315

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602010005371

Country of ref document: DE

Effective date: 20130508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130613

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130613

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130624

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 601218

Country of ref document: AT

Kind code of ref document: T

Effective date: 20130313

REG Reference to a national code

Ref country code: NL

Ref legal event code: VDEP

Effective date: 20130313

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130614

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130715

Ref country code: MC

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20130331

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130713

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20130315

26N No opposition filed

Effective date: 20131216

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602010005371

Country of ref document: DE

Effective date: 20131216

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20140331

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20140331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20130315

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20100315

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130313

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 602010005371

Country of ref document: DE

Representative=s name: MURGITROYD & COMPANY, DE

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 7

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 8

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20231229

Year of fee payment: 15

Ref country code: GB

Payment date: 20240108

Year of fee payment: 15

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20240103

Year of fee payment: 15