EP1006511B1 - Sound processing method and device for adapting a hearing aid for hearing impaired - Google Patents

Abstract

A parameter characterizing a pitch is modified (61) by applying a multiplying factor to a value of the pitch. The parameter characterizing a voisement is modified (62) by a multiplying factor. A parameter characterizing the energy is modified (62) by a compression function. The parameter characterizing the spectral envelope is modified (64) by a compression rate of the frequency scale, and the length of interval of time taking into an account of a of synthesized phase (7) while multiplying a time interval of by a time factor. An Independent claim is included for: (a) a device for implementing a method

Description

The present invention relates to a method and a device for correcting sounds for the hearing impaired. It applies as well to the production of hearing aids, as software executable on personal computers or answering machines and in general to all devices intended to improve the listening comfort and the understanding of the speech of people with deafness.

The problem posed by the hard of hearing comes essentially from the specific and degraded nature of their auditory perception.

In his need to communicate, man has since the dawn of time built a mode of oral communication, speech, which is based on the average characteristics of production (voice) and perception (ear) of the signal sound. Common language is therefore that of the greatest number. Conversely, the hearing of the hard of hearing is far from the average and everyday language is difficult, if not at all, accessible to him.

Understanding everyday language is a prerequisite for integrating the hearing impaired into their community. In what can be considered as a reflex of social survival, any hearing impaired person is naturally led to form his own language and to implement procedures, techniques, and a communication strategy allowing him to transpose the common language into his particular language. A known and spectacular example is that of lip reading, which allows access to normal speech through a visual alphabet of lip position.

The twentieth century has seen a continuous effort in the design of machines to relieve and assist the hearing impaired.

Two classes of machines have been developed.

A first class is aimed at "mild" deafness and aims to correct hearing and make it normal as much as possible. This is what conventional prostheses widely available on the market do.

A second class is intended for heavier deafness and aims to transform speech into synthetic speech accessible to the hearing impaired person. In this category, most of the creations are aimed at the "deep deaf". A remarkable example is that of the cochlear implant which acts by means of electrodes by direct stimulation of the auditory nerve.

The present invention aims to provide a solution for people suffering from deafness called "intermediate". These people currently do not have adequate technical assistance. They are too affected to be served by the usual prostheses, but their hearing is sufficient to be able to do without devices for the deaf.

The usual prostheses generally use a method of selective amplification of speech as a function of frequency. In its implementation, an automatic regulation of the sound level acts on the gain of amplification, the aim being to give the best listening comfort and protection against instantaneous power peaks.

For reasons of commercial strategy and in response to patient demand, these prostheses are miniaturized to be worn behind the ear or as an insert, which leads to relatively poor performance that can only satisfy very coarse hearing corrections. Typically, only three frequency bands are defined for frequency correction. These prostheses unambiguously address the most frequent "mild" deafness. Heavier deafness can be relieved, but at the cost of painful inconveniences caused in particular by the amplification of the background noise, and the phenomenon of Larsen. On the other hand there is no possibility of correction in the frequency zones for which there is no hearing.

On the history of prostheses for the deaf one can usefully refer to the work of MJMTATO professor of ORL and MM VIGNERON and LAMOTTE cited in the article MJC LAFON having for title "Transposition and modulation", published in the audiophonology bulletin scientific annals of Franche Comté, Volume XII, N ° 3 & 4, monograph 164, 1996. These prostheses exploit the fact that the deaf are rarely completely deaf, and that a very weak balance of perception persists, often in the grave, which he has often been tried to take advantage.

This is how it is possible to restore, in a very rustic way, a perception of sound to the deaf by so-called “transposition” processes. from treble to bass. Unfortunately the understanding of language requires more than a simple perception and it turns out that the transmission of intelligibility is inseparable from a necessary "richness" of sound. Giving back this "richness" has become one of the main subjects of concern. It was thus that it was envisaged to create a speech of synthesis in order to restore the structural elements which form the support for the intelligibility of everyday language.

The technique implemented in 1952 by M J.M. TATO, consists in recording the speech said very quickly and in restoring it at half speed. This allows transposition of an octave in the bass, while preserving the structure of the initial speech. Tests have shown some benefit for the deaf.

However, the method has the drawback of being able to be used only in deferred time. The technique developed in 1971 by MM C. VIGNERON and M. LAMOTTE makes it possible to carry out an adaptation in “real time” by cutting time in intervals of 1/100 of seconds by removing one interval out of two, and by applying the process by M JMTATO on the remaining intervals. However, this system unfortunately has significant background noise.

The idea of building “natural” sounds is also present in a prosthesis also cited under the name GALAXIE in the article by M JC LAFON. This prosthesis implements a battery of filters and mixers distributed over six sub-bands and realizes a transposition in the bass usable for the deep deaf.

Unfortunately, these methods which intervene at the signal level present too much distortion and too great discomfort of listening to be able to be used by people suffering from intermediate deafness.

• 1- Il parait important de pouvoir transposer la globalité de la structure acoustique c'est à dire de ramener les éléments structurels de la parole porteurs de l'intelligibilité dans la zone de perception du malentendant.
• 2- Il parait aussi important de produire des sons « naturels »c'est à dire de reproduire une parole synthétique porteuse de l'information qui a une structure en harmonie avec l'acquis auditif du malentendant.
  Le brevet US 4 051 331 divulgue un procédé de correction auditive des malentendants où la restitution de la parole s'appuie sur des sources oscillatrices chacune centrées sur la fréquence centrale des formants.
• 3- Enfin il faut veiller à conserver la temporalité du signal de parole car le rythme est porteur d'information accessible au malentendant. L'idée à l'origine de l'invention est de pallier les inconvénients précités en utilisant un modèle paramétrique du signal de parole capable d'effectuer des transformations pertinentes en vue d'une correction auditive pour des malentendants en mettant en oeuvre une méthode capable de satisfaire les trois contraintes citées précédemment.

From the article by Mr. Jean Claude LAFON, three directions emerge which can be retained in the achievement of good prosthetic treatment.

• 1- It seems important to be able to transpose the whole of the acoustic structure, that is to say to bring the structural elements of speech carrying intelligibility back into the perception area of the hearing impaired.
• 2- It also seems important to produce “natural” sounds, that is to say to reproduce synthetic speech carrying information which has a structure in harmony with the hearing impaired person's hearing.
  US Pat. No. 4,051,331 discloses a hearing correction process for the hearing impaired in which speech restitution is based on oscillating sources each centered on the central frequency of the formants.
• 3- Finally, care must be taken to preserve the temporality of the speech signal because the rhythm carries information accessible to the hearing impaired. The idea behind the invention is to overcome the aforementioned drawbacks by using a parametric model of the speech signal capable of performing relevant transformations for hearing correction for the hearing impaired by implementing a method capable of to satisfy the three constraints mentioned above.

• Le paramètre caractérisant le pitch est modifié en appliquant à la valeur du pitch extrait un facteur multiplicateur,
• Le paramètre caractérisant l'énergie est modifié par une fonction de compression,
• Le paramètre caractérisant l'enveloppe spectrale est modifié par une compression homothétique de l'échelle des fréquences

Caractérisé en ce que

• Le paramètre caractérisant le voisement est donné sous la forme d'une fréquence de transition entre une bande voisée et une bande haute non voisée, qui est modifiée par un facteur multiplicateur afin de créer une bande haute modifiée sur laquelle on génère un bruit blanc pseudo aléatoire lors de la phase de synthèse ; et en ce que
• La durée de l'intervalle de temps pris en compte pour la phase de synthèse est modifié en multipliant l'intervalle de temps par un facteur temps.

To this end, the subject of the invention is a method for the hearing correction of the hearing impaired, of the type consisting in analyzing a speech signal in order to extract therefrom parameters characterizing the pitch, the voicing, the energy and the spectrum of the speech signal. , to modify the parameters to make speech intelligible to the hearing impaired, and to synthesize a speech signal perceptible by the hearing impaired from parameters modified in the following manner:

• The parameter characterizing the pitch is modified by applying a multiplier factor to the value of the extracted pitch,
• The parameter characterizing the energy is modified by a compression function,
• The parameter characterizing the spectral envelope is modified by a homothetic compression of the frequency scale

Characterized in that

• The parameter characterizing the voicing is given in the form of a transition frequency between a voiced band and an unvoiced high band, which is modified by a multiplying factor in order to create a modified high band on which pseudo-random white noise is generated during the synthesis phase; and in that
• The duration of the time interval taken into account for the synthesis phase is modified by multiplying the time interval by a time factor.

The subject of the invention is also a device for implementing the above method.

The method and the device according to the invention have the advantages of implementing the parametric models which are commonly used in vocoders for adapters to the hearing of the hearing impaired. This makes it possible to work no longer at the level of the sound signal, as do the prior techniques but at the level of the symbolic structure of the speech signal in order to preserve its intelligibility. The vocoders indeed have the advantage of using an alphabet which integrates the notions of "pitch", "spectrum", "voicing" and "energy" which are very close to the physiological model of the mouth and the ear. According to SHANNON theory, the information transmitted then carries the intelligibility of speech. The materialization of speech intelligibility in computer form thus opens up a new perspective. The intelligibility can thus be acquired during the analysis operation, and it is restored during the synthesis.

Thanks to the invention, the operation of synthesizing a parametric vocoder can therefore be adapted to the hearing characteristics of deaf. This technique, associated with more conventional methods, makes it possible to envisage a particularly general prosthetic process which can serve a very large population, and in particular people suffering from intermediate deafness.

As another advantage, the method and the device according to the invention offer great freedom in the adjustments, each parameter being able to be modified independently of the others without mutual impact, with a specific adjustment for each ear.

Other characteristics and advantages of the invention will become apparent from the following description given with reference to the appended drawings which represent:

FIG. 1, the parameters for modeling the speech signal used in the implementation of the invention.

Figure 2, a parametric model of speech signal production.

FIG. 3, the different steps necessary for implementing the method according to the invention in the form of a flowchart.

FIG. 4, a transformation curve during the synthesis of the speech signal of the energy of the speech signal measured during the process of analysis of the speech signal.

FIG. 5, an embodiment of a device for implementing the method according to the invention.

The speech signal processing method according to the invention is based on a parametric modeling of the speech signal of the type commonly used in the technique for producing HSX digital vocoders and a description of which can be found in the article. by MM P. Goumay, F. Charité, entitled "A 1200 bits / s HSX speech coder for very low bit rate communications", and published in IEEE Proceedings Workshop on Signal Processing System (Sips'98), Boston, October 8-10 1998.

• un paramètre de voisement qui décrit le caractère plus ou moins périodique des sons voisés ou aléatoire des sons non voisés du signal de parole,
• un paramètre définissant la fréquence fondamentale ou « PITCH » des sons voisés,
• un paramètre représentatif de l'évolution temporelle de l'énergie,
• et un paramètre représentatif de l'enveloppe spectrale du signal de parole.

This model is mainly defined by four parameters:

• a voicing parameter which describes the more or less periodic nature of the voiced or random sounds of the unvoiced sounds of the speech signal,
• a parameter defining the fundamental frequency or “PITCH” of the voiced sounds,
• a parameter representative of the time evolution of the energy,
• and a parameter representative of the spectral envelope of the speech signal.

The spectral envelope of the signal, or "spectrum", can be obtained by an autoregressive modeling using a linear prediction filter or by a short-term Fourier analysis synchronous with the pitch. These four parameters are estimated periodically on the speech signal from one to several times per frame depending on the parameter, for a frame duration typically between 10 and 30 ms.

The speech signal is reproduced in the manner shown in FIG. 2, by stimulating by pitch or by stochastic noise, a digital synthesis filter 1 which models the vocal tract by its transfer function according to whether the sound respectively is voiced or unvoiced.

A switch 2 transmits the pitch or the noise to the input of the synthesis filter 1.

An amplifier 3 of variable gain as a function of the energy of the speech signal is placed at the output of the synthesis filter 1.

In the case of a simple parametric model comprising a binary decision its voiced / its voiced, the synthesis procedure can be summarized to that represented in FIG. 2. However, the method according to the invention which is represented in FIG. 3 under the form of a flowchart is more complex and takes place in four stages, breaking down into a preprocessing stage 4, a stage 5 of analysis of the signal obtained in stage 4 for the extraction of the parameters characterizing the pitch, the voicing, the energy, and the spectrum of the speech signal, a step 6 during which the parameters obtained in step 5 are modified and a step 7 of synthesis of a speech signal composed from the modified parameters of l 'step 6.

Step 4 is that which is conventionally implemented in vocoders. It consists notably, after converting the speech signal into a digital signal, to reduce the background noise by using for example the method described by MDMalah, published in IEEE trans.Acoust., Speech Processing, Volume-12 N.6, pp. 1109-1121 1984, entitled "Speech enhancement using a minimum square error short time spectral amplitude estimator ", to cancel acoustic echoes using for example the method described in the article by MM.K. Murano, S. Unjani and F. Amano having for title:" Echo cancellation and applications "published in the journal IEEE Com. May, 28 (1), pp 49-55, January 1990, to perform an automatic gain control, or to pre-emphasize the signal.

The parametric processing of the speech signal obtained at the end of step 4 is carried out in step 5. It consists of cutting the speech signal into samples of constant analysis duration (typically 5 to 30 milliseconds) to perform on each of them the estimation of the parameters of the speech signal model. Using the HSX analysis model described in the article by Messrs. Gournay and F. Chartier cited above, the pitch and the voicing are estimated every 22.5 milliseconds. The voicing information is given in the form of a transition frequency between a voiced low band and an unvoiced high band. The signal energy is estimated every 5.625 milliseconds. During unvoiced periods of the signal, it is estimated over a duration of 45 samples (5.625 ms) and expressed in dB per sample. During voiced signal periods, it is estimated over a whole number of fundamental periods at least equal to 45 and expressed in dB per sample. The spectral envelope S (ω) is estimated every 11.25 milliseconds. It is obtained by linear prediction (LPC) by an auto-regressive modeling of a filter of order OLPC = 16 of transfer function: $${\text{S (ω) = 1 / | A (z) |}}^{\text{2}}\text{with z = exp (jω) and ω = 2 π f}$$ where A (z) is defined by:

• PitchAnalyse ;
• VoisementAnalyse ;
• EnergieAnalyse[i], i=O à 3 ;
• LpcAnalyse[k], k=I à 16.

In the following, the parameters resulting from the analysis are noted:

• PitchAnalyse;
• VoisementAnalyse;
• EnergieAnalyse [i], i = O to 3;
• LpcAnalyse [k], k = I to 16.

The synthesis procedure consists, for each analysis duration interval, in stimulating the synthesis filter giving S (ω) by the frequency-weighted sum (low band / high band defined by the frequency voicing), a pseudo-random white noise for the high band and a periodic Dirac comb signal of fundamental frequency equal to the pitch for the low band.

According to the invention, numerous transformations can be applied to the parameters resulting from the analysis of step 5. Each parameter can indeed be modified independently of the others, without reciprocal interaction. In addition, these transformations can be constant or only be activated under specific conditions (for example, triggering of the modification of the spectral envelope for certain configurations of energy distribution as a function of frequency, etc.) .

These modifications are carried out in steps 6 ₁ to 6 ₄ and they relate essentially to the value of the pitch which characterizes the fundamental frequency, the voicing, the energy and the spectral envelope.

For the course of step 6 _1, any transformation defining a new “pitch” value from the value of the analysis pitch obtained in step 5 is applicable.

Elementary transformation is homothety, defined by the relation: $$\text{PitchSynthèse = PitchAnalyse * FacteurPitch,}$$ with the following limitations: $$\text{0.25 <Pitch Factor <4.0}$$ $$\text{50 Hz <PitchSynthesis <400 Hz.}$$

The Pitch Factor factor is adjustable for the type of deafness considered.

As for the pitch, the voicing frequency can be modified by any transformation defining a "voicing frequency" for each value of the voicing frequency analyzed in step 5.

In the example of implementation of the invention, the chosen transformation is a homothety, defined by the relation: $$\text{VoisementSynthèse = VoisementAnalyse * FactorVoisement,}$$ with the following limitations: $$\text{0.25 <Voice Factor <4.}$$ $$\text{0 Hz <Voice Synthesis <4000 Hz.}$$

When the voicing transition frequency resulting from the VoisementAnalyse analysis is maximum (fully voiced signal, VoisementAnalyse = VoisementMaximum) the voicing frequency used in synthesis is unchanged (VoisementSynthèse = VoisementMaximum). Applying a multiplicative factor to it would indeed be completely arbitrary (VoisementAnalyse = VoisementMaximum does not mean an absence of voicing above VoisementMaxmum). For example VoisementMaximum can be set to 3625 Hz.

The Voice Factor factor is adjustable for the type of deafness considered.

The energy processing is carried out in step 6 ₃ As previously any transformation defining an energy from the energy of the speech signal analyzed in step 6 ₃ is applicable. In the example described below, the method according to the invention applies to energy a compression function with four linear segments as shown in the graph of FIG. 4.

The energy used in synthesis is given by the relation: $$\begin{gathered} \text{EnergieSynthèse [i] = Slope * EnergieAnalyse [i] + EnergieSynthèseSeuil -} \\ \text{* Slope EnergieAnalyseSeuil,} \end{gathered}$$ for i = O to 3, with $$\text{Slope = Low Slope for EnergieAnalyse <EnergieAnalyseSeuil;}$$ $$\text{Slope = High Slope for EnergieAnalyse> = EnergieAnalyseSeuil;}$$ and with the following limitations: $$\text{EnergieSynthèse <= EnergieSynthèseMax;}$$ $$\text{EnergieSynthèse = - Infinite for EnergieAnalyse <EnergieAnalyseMin.}$$ The EnergieAnalyseMin, EnergieSynthèseMax, SlopeLow, SlopeHigh and EnergieSynthèseSeuil treatment parameters are adjustable for the type of deafness considered.

The processing of the spectral envelope takes place in step 6 ₄ In this step any transformation defining a spectrum S '(ω) from the spectrum S (ω) analyzed in step 5 is applicable.

In the embodiment of the invention described below, the elementary transformation on the spectrum which is implemented is a homothetic compression of the frequency scale.

The frequency scale is compressed by a factor FactorSpectrum so that the useful bands before and after the treatment are respectively equal to [O..FECH / 2] and [O..FECH / (2 * FacteurSpectre)], where FECH is the sampling frequency of the system.

The implementation of this homothetic compression is very simple when the compression factor is an integer value. It then suffices to replace z by z ^{FacteurSpectre} in the expression of the filter any pole of synthesis, then to apply to the synthesized signal a low-pass filtering of cut-off frequency FECH / (2 * FacteurSpectre).

A first theoretical justification for the validity of the method described above consists in saying that this operation is equivalent to operating an oversampling of a factor FactorSpectrum of the impulse response of the vocal tract, by insertion of FactorSpectrum-1 null samples between each samples of the impulse response from the original vocal tract and then by low-pass filtering of the synthesized signal with a cutoff frequency equal to FECHI (2 * FacteurSpectre).

A second theoretical justification consists in considering that this operation is equivalent to duplicating and moving the poles of the transfer function.

Indeed, considering that the single pole OLPCs noted zi = pi.exp (2iπFi) of the transfer function 1 / A (z), the "SpectrumFactor * pole OLPCs" of 1 / A (Z ^{FactorSPectre} ) are then the " FactorSpectrum "complex roots of each of the zi. The poles conserved by the low-pass filtering operation are of the type z'i = p _i ^{I / FacteurSpectre} exp (2. I.π.Fi / FacteurSpectre) which shows that their resonance frequency has actually undergone homothetic compression a "FactorSpectrum" factor.

• LpcSynthèse[k] = 0 pour k=I à OLPC2, k non multiple de FacteurSpectre.
• LpcSynthèse [FacteurSpectre * k] = LpcAnalyse[k] pour k=I à OLPC ;

The LPC filter used in synthesis can therefore be expressed in the form; OLPC2 = FactorSpectrum * OLPC;

• LpcSynthesis [k] = 0 for k = I at OLPC2, k not multiple of FactorSpectrum.
• LpcSynthèse [FacteurSpectre * k] = LpcAnalyse [k] for k = I at OLPC;

It is possible to restrict the compression factor of the spectral envelope to be an integer between 1 and 4 such that: $$\text{1 <SpectrumFactor <4.}$$

The speech restored in step 7 can be further accelerated or slowed down by simple modification of the duration of the time interval taken into account for the synthesis phase.

In practice, this operation can take place by implementing a homothetic transformation procedure defined by the relation: $$\text{Tsynthesis = Tanalyse * TimeFactor}$$

If TimeFactor> 1, then it is a speech slowdown. If TimeFactor <1, then it is an acceleration of speech.

In addition to the preceding processing operations, a certain number of post-processing operations can be envisaged, consisting for example of performing bandpass filtering and linear equalization of the synthesized signal, or alternatively a multiplexing of the sound on the two ears.

The objective of the linear equalization operation is to compensate for the patient's audiogram by amplifying or attenuating certain frequency bands. As part of the prototype, the gain at 7 frequencies (0, 125, 250, 500, 1000, 2000 and 4000 Hz) can be adjusted over time between -80 and +10 dB according to the needs of the patient or the specifics of his audiogram. This operation can be carried out for example by filtering by a fast Fourier transform (FFT) in the manner described for example in the book by M.D Elliott having for title "Handbook of Digital signal processing" published in 1987 at Academic Press.

The multiplexing operation allows monophonic (for example a signal processed alone) or stereophonic (for example a signal processed on one channel and an unprocessed signal on another) reproduction. The stereophonic reproduction allows the hearing impaired to adapt the treatment for each of his ears (two linear equalizers to compensate for two different audiograms for example), and possibly to keep intact on one ear a form of signal to which he is accustomed and on which he can lean on, for example, to synchronize.

The device for implementing the method according to the invention which is shown in FIG. 5 comprises a first channel composed of an analysis device 8, a synthesis device 9 and a first equalizer 10 and a second channel comprising a second equalizer 11, the assembly of the two channels being coupled between a sound pickup device 13 and a pair of headphones 12 _a , 12 _b . The analysis 8 and synthesis devices 9 can be implemented using the known techniques for producing vocoders and in particular that of the HSX vocoders mentioned above. The outputs of the equalizers of the two channels are multiplexed by a multiplexer 14 to allow reproduction of the monophonic or stereophonic sound. A processing device 15 formed by a microprocessor or any equivalent device is coupled to the synthesis device 9 to ensure the modification of the parameters supplied by the analysis device 8.

A preprocessing device 16 interposed between the sound pickup device 13 and each of the two channels ensures the denoising and the conversion of the speech signal into digital samples. The denoised digital samples are applied respectively to the input of the equalizer 11 and to the input of the analysis device 8.

According to other embodiments of the device according to the invention, the processing device 15 can be integrated into the synthesis device 9, as it is also possible to integrate all of the analysis and synthesis treatments in the same software executable on a personal computer or on an answering machine for example.

Claims (3)

1. Method for the auditory correction of hearing impaired individuals, of the type consisting in analysing a speech signal (5) so as to extract therefrom parameters characterizing the pitch, the voicing, the energy and the spectrum of the speech signal, in modifying (6) the parameters so as to render the speech intelligible to a hearing impaired individual, and in synthesizing (7) a speech signal perceptible to the hearing impaired individual on the basis of the parameters modified in the following manner:

   • the parameter characterizing the pitch is modified (61) by applying a multiplier factor to the value of the pitch extracted,

   • the parameter characterizing the energy is modified (63) by a compression function,

   • the parameter characterizing the spectral envelope is modified (64) by a homothetic compression of the frequency scale

   characterized in that

   • the parameter characterizing the voicing is given in the form of a frequency of transition between a voiced band and an unvoiced high band, which is modified (62) by a multiplier factor so as to create a modified high band on which pseudo-random white noise is generated during the synthesis phase; and in that

   • the duration of the time interval taken into account for the synthesis phase is modified (7) by multiplying the time interval by a time factor.
2. Device for the implementation of the method according to Claim 1, characterized in that it comprises an analysis device (8) for extracting the parameters of the speech signal coupled to a synthesis device (9) as well as a processing device (14) coupled to the synthesis device (9) devised so as to modify the parameters provided by the analysis device (8) according to the characteristics of Claim 1 and to apply the modified parameters to the synthesis device (9) so as to reconstruct a speech signal with the modified parameters.
3. Device according to Claim 2, characterized in that the analysis device (8) and the synthesis device (9) comprise a linear prediction vocoder analysis and synthesis device.