CN116074717A - Consonant enhancement method and device for hearing loss - Google Patents

Abstract

The invention discloses a consonant enhancement method and device aiming at hearing loss. The method comprises the following steps: 1) Performing consonant detection on the voice S, dividing the voice into consonants and non-consonants, and dividing the consonants into different categories; 2) The enhancement parameters required by consonants of each category are respectively calculated through an auditory model and a loudness model, so that the enhancement parameters are used for processing the voice S', and the loudness L perceived by an auditory damage system is obtained by sequentially calculating the auditory damage model and the loudness model _HI Loudness L perceived by normal auditory system obtained by sequentially calculating normal auditory model and loudness model with voice S _NH Consistent; 3) Enhancing the consonants of different categories detected in the step 1) by using the consonant enhancement parameters set in the step 2); 4) And (3) compensating the non-consonant in the step (1) and the consonant processed in the step (3) by using a loudness compensation technology to generate a voice after compensating the voice S. The invention can obviously improve the speech intelligibility.

Description

Consonant enhancement method and device for hearing loss

Technical Field

The invention belongs to the technical field of hearing assistance, relates to a hearing compensation method, and particularly relates to a consonant enhancement method and device aiming at hearing loss.

Background

Hearing loss is one of the important public health problems, and currently 15 hundred million people worldwide suffer from hearing loss according to global hearing assessment reports issued by the world health organization 2021, of which 4.3 hundred million require rehabilitation services for hearing loss, and by 2050, hearing loss is increased to 25 hundred million, of which 7 hundred million require hearing rehabilitation. Hearing loss can cause communication impairment, thereby affecting interpersonal relationships and working capacity, resulting in social isolation and reduced quality of life. Studies have shown that the risk of fall, dementia, depression and death is higher in elderly people with hearing loss than in individuals without hearing loss. Children with hearing loss can suffer from slow development of language functions and have significant adverse effects on academic performance. From a socioeconomic perspective, the world health organization estimates that hearing loss is a worldwide loss of 7500 billion dollars annually, including the cost of the health sector, educational support costs, and economic losses due to productivity losses. Hearing loss affects physical and mental health and brings high socioeconomic costs, requiring early discovery and intervention.

The standard for hearing loss diagnosis is a hearing threshold (hearing threshold) obtained by playing a fixed frequency pure tone signal under specific environmental and equipment conditions, and measuring the lowest sound pressure level at which the subject can achieve a predetermined percentage of the correct perceived response. Hearing loss typically results in an elevated hearing threshold, which is manifested as undetectable faint sounds, because the threshold for acoustic nerve firing is raised, i.e., the acoustic nerve requires a greater intensity of sound to cause the same degree of firing; second, the dynamic range of loudness perception narrows due to the disappearance of the nonlinear compression mechanism of the cochlea. For normal hearing people, the outer hair cells of the cochlea respond differently to sounds of different sound intensities. Wherein when the input intensity is low, the change in loudness perceived by the human ear is amplified relative to the change in sound intensity; and when the intensity of the input sound is high, the degree of loudness enhancement is not much different from the degree of sound intensity increase, or even smaller. This mechanism allows a wider dynamic range of loudness for normal persons, higher sensitivity to sound at lower sound intensities, and more easily perceived differences in loudness. However, for hearing impaired patients, this mechanism may not work or even disappear completely due to the damage of the extra-cochlear hair cells, thus resulting in that the listener's perceived loudness changes to low-intensity sounds are no longer amplified, the dynamic range of loudness perception is narrowed, and the sensitivity to sound loudness changes is reduced.

The administration of appropriate hearing rehabilitation services allows the hearing of a vast majority of hearing impaired individuals to be compensated, thereby improving their speech communication and improving quality of life. The hearing aid is a hearing compensation device with the largest number of people at present, and the basic principle of the hearing compensation device is to map sound in the hearing dynamic range of a normal person to the hearing dynamic range of a hearing impaired patient by using a loudness compensation technology, so that the speech communication capacity of the hearing impaired patient is improved. However, for speech, the consonant energy in the speech is small and short, and the loudness of the consonant after the loudness compensation may not reach the hearing dynamic range of the hearing impaired patient, thus affecting the hearing compensation effect to some extent.

Disclosure of Invention

Aiming at the defects of the prior method, the invention provides a consonant enhancement method and equipment aiming at hearing loss. Vowels and consonants are the most basic two concepts in speech, vowels are sounds generated by air flow passing through the oral cavity without being blocked in the process of pronunciation, while consonants are air flow which comes out of the lung without vibrating vocal cords in the process of pronunciation, and are blocked by the whole or part of a pronunciation organ, so that the generated sounds are not clear enough. Consonants have small energy and low loudness compared to vowels, and may not reach within the hearing dynamic range of a hearing impaired patient even after loudness compensation. Studies of speech perception have shown that specific enhancement of consonants (either to enhance consonant intensity or to increase consonant duration) can significantly improve speech intelligibility. However, the parameters of consonant enhancement are also different for different types of consonants and different degrees of hearing loss. Therefore, it is important to study consonant enhancement methods for hearing loss.

The basic idea of the consonant enhancement method for hearing loss proposed in the present invention is: according to the loudness model and the hearing model, the enhancement parameters of consonants of different types under different hearing loss degrees are set, so that the loudness of consonants perceived by hearing impaired patients is consistent with the loudness of consonants perceived by normal people as much as possible.

The technical scheme of the invention is as follows:

a method of consonant enhancement for hearing loss, comprising the steps of:

1) Consonant detection is performed on the voice, the voice is divided into consonants and non-consonants, and the consonants are divided into different categories (no air supply, and wiping, sound supply, nasal sound, etc.);

2) Enhancement parameters required for different classes of consonants are calculated separately by the auditory model and the loudness model. The method aims at enabling the loudness perceived by a hearing-impaired system obtained by calculation after the speech processed by using the enhancement parameters sequentially uses a hearing-impaired model and a loudness model to be consistent with the loudness perceived by a normal hearing system obtained by calculation after the untreated stimulus passes through the normal hearing model and the loudness model. The loudness model can be found in international standard BS ISO 532-2:2017; the auditory model can be found in Kates J,2013.An auditory model for intelligibility and qualitypredictions[C/OL ]// Proceedings of Meetings on Acoustics ICA2013.Montreal, canada:

Acoustical Society of America:050184-050184；

3) Enhancing consonants of different categories detected in step 1) by using the consonant enhancement parameters set in step 2);

4) And (3) compensating the non-consonant in the step (1) and the consonant processed in the step (3) by using a traditional loudness compensation technology to generate compensated voice.

Further, consonants are classified into different categories according to pronunciation parts and pronunciation modes.

Further, the loudness model is mainly used for calculating short-time loudness of sound.

Further, the hearing model may simulate the processing of sound by the normal hearing system and the hearing impairment system by setting different parameters (i.e., hearing thresholds).

Further, the consonant enhancement parameters include consonant amplitude and consonant duration.

Further, the perceived enhanced consonant loudness of the hearing impaired patient is as follows: enhancing consonants, wherein the enhancement parameter is P (the enhancement is not performed under the initial condition), then compensating by adopting the traditional loudness compensation technology, inputting the compensated sound into an auditory model of hearing impairment for processing, and finally calculating the loudness L perceived by a hearing impaired patient from the processed signal through the loudness model _HI . Wherein the hearing model can realize the processing of sound by the normal hearing system and the hearing impaired system by setting parameters of Hearing Loss (HL) with different degrees; the input of the auditory model is a time domain signal (e.g., speech) and the output is a processed time domain signal (e.g., processed speech).

Furthermore, the mode of calculating the consonant loudness obtained by the perception of the normal person is as follows: inputting the consonant detected in the step 1) into a normal auditory model for processing, and calculating the short-time loudness L of the processed signal through a loudness model _NH 。

Further, the consonant enhancement parameters are adjusted to the enhanced consonant loudness L perceived by the hearing impaired patient _HI Consonant loudness L perceived with normal person _NH As uniform as possible. Assuming that the current consonant enhancement parameter P (initially set to 0, i.e., no enhancement is performed), if L _HI ＜L _NH And P is less than P _MAX Then increase the enhancement parameter P until L _HI ≥L _NH Or P is greater than or equal to P _MAX 。

A consonant enhancement device for hearing loss, comprising a consonant detection and classification module, a consonant enhancement parameter setting module, a consonant enhancement module, a loudness compensation module, and the like; wherein the method comprises the steps of

The consonant detection and classification is used for detecting the boundary of the consonant and classifying the detected consonant into different categories;

the consonant enhancement parameter module is used for determining enhancement parameters of consonants of different types under different hearing loss degrees by using a loudness model and an auditory model;

the consonant enhancement module is used for adjusting the amplitude and duration of consonants according to the consonant enhancement parameters;

the loudness compensation module is used for compensating sound according to the hearing loss condition, so that the overall loudness perceived by a hearing loss patient is as consistent as possible with that of a normal person.

Compared with the prior art, the invention has the following positive effects:

the invention uses a loudness model and an auditory model to adaptively set enhancement parameters of consonants of different types under different hearing loss degrees, and carries out enhancement treatment on the consonants. The result shows that the method can effectively improve the speech intelligibility of the hearing impaired patient.

Drawings

FIG. 1 is a frame diagram of a consonant enhancement method;

FIG. 2 is a consonant enhancement parameter setting flow chart;

FIG. 3 is a loudness model framework diagram;

FIG. 4 is a diagram of an auditory model framework;

FIG. 5 is a plot of auditory filter versus sound intensity;

FIG. 6 is a substrate film input-output nonlinear function;

FIG. 7 is a window function when certain consonant amplitude increases by 1.5 times the original;

FIG. 8 shows the recognition accuracy of a single word under different test conditions;

FIG. 9 shows consonant recognition accuracy under different test conditions.

Detailed Description

Specific implementations of the present invention will be disclosed in more detail below. Fig. 1 is a block diagram of a consonant enhancement method for hearing loss according to the present invention. The specific implementation steps of the method comprise consonant detection and classification, consonant enhancement parameter setting, consonant enhancement, loudness compensation and the like. The specific implementation process of each step is as follows:

1. detection and classification of consonants

The detection of consonants refers to slicing consonants from a continuous speech stream using manual labeling or machine automatic labeling. The consonant classification is based on consonant detection, and the detected consonants are classified into different categories according to pronunciation parts and pronunciation modes.

2. Consonant enhancement parameter setting

The method introduces a loudness model and an auditory model, adjusts the enhancement strategy by comparing the difference between the loudness of consonants perceived by normal hearing and hearing impaired patients, and determines enhancement parameters of consonants under different hearing loss degrees, and the flow is shown in figure 2.

For a given consonant and consonant enhancement parameter P, calculating the perceived loudness L of a normal listener and hearing impaired patient _NH And L is equal to _HI . Wherein for the loudness of consonants perceived by a normal listener, consonants are firstly input into a normal auditory model for processing, and then the processed signals are calculated by a loudness model, so that the perceived short-time loudness L can be obtained _NH The method comprises the steps of carrying out a first treatment on the surface of the The perceived consonant loudness of hearing impaired patients is obtained by firstly enhancing consonant under the consonant enhancement parameter P, then compensating by adopting the traditional loudness compensation technology, inputting the compensated sound into the hearing model of hearing impairment for processing, and finally calculating the loudness L perceived by hearing impaired patients by the processed signal through the loudness model _HI . If under the current enhanced parameter P, L _HI ＜L _NH And P is less than P _MAX Then the consonant enhancement parameters are gradually increased by p=p+δ (δ is the step size of the consonant enhancement parameter change) until L _HI ≥L _NH Or P is greater than or equal to P _MAX (P _MAX Maximum value of the consonant enhancement parameters to be set).

The loudness model introduced in the method is mainly used for calculating the loudness of sound perceived by normal people and hearing impaired patients with different hearing loss degrees. The calculation flow is shown in fig. 3, and mainly comprises the following stages: (1) Two determined filters respectively representing transmission processes of the external auditory meatus and the middle ear; the signals output after the auditory model processing are sequentially filtered and output by two filters, so that the simulated signals transmitted and output by the outer ear and the middle ear are obtained; (2) transitioning from spectrum to excitation mode; (3) Converting the excitation pattern to a specific loudness; (4) The instantaneous loudness, short-term loudness, and long-term loudness are calculated by determining the area of the region at a specific loudness. The above procedure may be calculated by the steps described in ISO 532-2:2017. Wherein the instantaneous loudness S _n Is a variable that is not directly perceived but meaningful. Short-term loudness S' _n Then the loudness that the listener can quickly determine, e.g. the loudness of a syllable in speech, the short-term loudness S 'at time n' _n The calculation of (2) is as follows:

wherein a is _a Is a variable related to the Attack time (Attack time) representing the effect of the instantaneous loudness on the short-term loudness as it gradually increases; a, a _r Is a variable related to the recovery Time (Release Time) and represents the effect of the instantaneous loudness on the short-term loudness as it gradually decreases.

The long-term loudness is the overall loudness of a certain longer time period, e.g. the loudness of a certain sentence, the long-term loudness S 'at time n' _n The' calculation is as follows:

wherein a is _al Is a variable related to the activation time and represents the effect of short-term loudness on long-term loudness as the short-term loudness gradually increases; a, a _rl Is a variable related to the recovery time and represents the effect of short-term loudness on long-term loudness as the short-term loudness gradually decreases.

The auditory model is mainly used for simulating the processing of sound by the periphery of human hearing. The computational model framework diagram is shown in fig. 4, and mainly comprises middle ear filter filtering, auditory filter bank filtering, dynamic range compression, signal attenuation caused by inner hair cell loss, inner hair cell release rate self-adaption and the like. Hearing loss may cause a threshold to rise, and may be accompanied by a loss of functions on the threshold, such as widening of an auditory filter, weakening or disappearance of input/output nonlinear compression characteristics, and the like. Parameters of the hearing model are adjusted according to the hearing threshold to simulate hearing peripheral processing of listeners in different hearing conditions. For mild or moderate hearing loss, the model attributes 20% of hearing loss to internal hair cell loss, adjusted primarily by the signal attenuation module; whereas 80% of the hearing loss is due to outer hair cell loss, mainly by adjusting the filter bandwidth and dynamic compression characteristics.

The variation in the bandwidth of an acoustic filter is generally affected by two factors: signal intensity and outer hair cell loss. The change in signal intensity and the bandwidth of the acoustic filter can be described approximately as a linear function as shown in fig. 5, where the bandwidth of the acoustic filter is nearly constant when the intensity is below 50dB SPL and increases as the intensity increases when the intensity exceeds 50dB SPL. The change in the bandwidth of the acoustic filter due to the outer hair cell loss can be expressed using the following formula:

where BW represents the ratio of the bandwidth of the auditory filter compared to the normal listener _OHC Represents the portion of the hearing loss that is attributed to the outer hair cell loss.

The weakening of the input-output nonlinear compression characteristic caused by the outer hair cell loss is mainly realized by adjusting a dynamic compression module in the model. As shown in fig. 6, the basement membrane input-output nonlinear function of a normal hearing listener is modeled in three segments, and the threshold of the default normal hearing listener is 0dB SPL, shown in bold. The sound intensity is in the range of 0 to 30dB SPL, the slope of the input/output function of the basement membrane is 1, the sound intensity is between 30dB SPL and 100dB SPL, the input signal is compressed by output, the slope is smaller than 1, and the input/output ratio of the function is called the compression rate. When the sound intensity is higher than 100dB SPL, the slope of the input-output function is restored to 1. While the outer hair cell loss reduces the compressibility, as indicated by the thin lines in fig. 6. The output corresponding to an input with a sound intensity of 100dB SPL is independent of the outer hair cell loss. In the figure, D represents the maximum outer hair cell loss, in which case the basement membrane input-output function becomes a linear function. When the outer hair cell loss exceeds D, the outer hair cell loss is set to D, and the remaining hearing loss is attributed to the inner hair cell loss.

3. Consonant enhancement

Consonant enhancement refers to the specific processing of consonants given consonant enhancement parameters. In the present method, consonant enhancement is achieved mainly by adjusting the amplitude of the consonant or the duration of the consonant.

The adjustment of consonant amplitude can be expressed by the following formula:

where P is the coefficient of consonant amplitude enhancement in dB. A is the amplitude of the sound before processing, and A' is the amplitude of the sound after processing. To ensure the naturalness of the processed speech, the method uses a cosine window to smooth the consonant, and fig. 7 shows the window function used when the magnitude of a certain consonant is increased by 1.5 times as much as the original.

The processing of the duration of consonants is to combine a Pitch synchronous superposition algorithm (Pitch-Synchronous Overlap And Add, PSOLA) and a waveform similarity algorithm (Waveform Similarity Overlap-Add, WSOLA) in a voice variable speed invariant Pitch algorithm to adopt different processing modes for different types of consonants. Wherein, the clear consonant uses PSOLA algorithm to extract fundamental frequency, duplicate the integer multiple of the fundamental tone to produce enhanced gospel; whereas voiced consonants are processed using the WSOLA algorithm. In order to ensure that the total duration of the sound is unchanged, the method moderately reduces the duration of the adjacent vowels while the duration of the consonants is increased.

4. Loudness compensation

The adaptation of hearing gain is an important step in obtaining hearing compensation through a hearing aid, and the hearing gain capable of compensating the hearing loss of a patient is directly deduced according to the hearing threshold of the patient through an adaptation formula, and the deduced hearing gain is different according to different adaptation formulas. According to the method, camfilter designed by a Camadapt adaptive formula is adopted for loudness compensation, and hearing-aid gain of each frequency band is calculated according to the hearing threshold of a hearing-impaired patient, so that the overall loudness perceived by the hearing-impaired patient after compensation is the same as that of a normal person as much as possible.

Advantages of the invention are described below in connection with specific embodiments.

The method is used for verifying the effect of a consonant enhancement method on a simulated hearing loss test. The results of the method will be compared with those of a traditional hearing compensation alone without hearing compensation.

1. Experimental setup

12 hearing normal subjects (10 men and 2 women) participated in the effect verification experiment in the sound-proof room environment. Wherein all normal hearing tests are conducted under simulated hearing loss conditions, which simulate the common moderate hearing loss conditions. The hearing loss simulation method is derived from an auditory model, the auditory model characterizes the processing of sound by the periphery of the human ear, and different hearing parameters are set, so that different hearing loss conditions can be simulated. Specifically, after setting the corresponding hearing loss, the hearing model is used to process the sound, and an attenuated sound signal is generated and then played to the normal test of hearing.

4 test conditions were set up in the experiment: the condition 1 is that the sound directly passes through the hearing loss model to generate the final voice without any compensation module; condition 2 is to use the traditional hearing compensation technology, namely, using Camfilter to carry out loudness compensation on the signals, and then generating voice through an hearing loss model; the condition 3 is that after the amplitude of consonant is enhanced, the Camfilter is used for loudness compensation of the signal, and finally, the hearing loss model is used for generating voice; and the condition 4 is that after the duration of consonants is enhanced, the Camfilter is used for loudness compensation of signals, and finally, the hearing loss model is used for generating voice.

All tested were tested under 4 conditions in a sound-proof room environment, and the correctness of the individual words and the correctness of the consonants were counted under each test condition.

2. Experimental results

Figures 8 and 9 show the individual words and consonants under 4 test conditions, respectivelyIs used for identifying the accuracy rate of the identification. The recognition accuracy under condition 1 (without compensation) is the lowest, and the recognition accuracy under condition 3 (consonant amplitude enhancement) and condition 4 (consonant duration enhancement) is higher than that under condition 1 (without compensation) and condition 2 (with only loudness compensation). The single factor repeated measurement analysis of variance result of the test condition shows that the test condition has obvious main effect

Multiple comparative analysis using Sidak correction showed that the accuracy under condition 1 was lower than all other conditions (all p<0.05 The accuracy of condition 3 is significantly higher than that of condition 2 (p<0.05). The experimental result shows that the consonant enhancement method aiming at the hearing loss can improve the speech recognition accuracy of the hearing-impaired patient, so that the method can be used as a feasible scheme for hearing compensation.

Although specific embodiments of, and the accompanying drawings for, the present invention are disclosed for illustrative purposes only and are for the purpose of aiding in the understanding of the present invention and the practice thereof, it will be understood by those skilled in the art that: various alternatives, variations and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the present invention should not be limited to the preferred embodiments and the disclosure of the drawings.

Claims (9)

1. A method of consonant enhancement for hearing loss, comprising the steps of:

1) Performing consonant detection on the voice S, dividing the voice into consonants and non-consonants, and dividing the consonants into different categories;

2) The enhancement parameters required by consonants of each category are respectively calculated through an auditory model and a loudness model, so that the enhancement parameters are used for processing the voice S', and the loudness L perceived by an auditory damage system is obtained by sequentially calculating the auditory damage model and the loudness model _HI Loudness L perceived by normal auditory system obtained by sequentially calculating normal auditory model and loudness model with voice S _NH Consistent;

3) Enhancing the consonants of different categories detected in the step 1) by using the consonant enhancement parameters set in the step 2);

4) And (3) compensating the non-consonant in the step (1) and the consonant processed in the step (3) by using a loudness compensation technology to generate a voice after compensating the voice S.

2. The method of claim 1, wherein the consonants are classified into different categories according to pronunciation location and pronunciation manner, including no air supply stop, stop wipe, pass, and nasal.

3.A method according to claim 1, characterized in that the loudness L is calculated _HI The method of (1) is as follows: firstly, enhancing consonants in the voice S by adopting the enhancement parameters; then, the enhanced voice is compensated by adopting a loudness compensation technology, the compensated voice is input into a hearing-impaired model for processing, and finally, the processed signal is input into the loudness model for calculation to obtain the perceived loudness L of the hearing-impaired patient _HI 。

4. A method according to claim 1, characterized in that the loudness L is calculated _NH The method of (1) is as follows: inputting the consonant detected in the step 1) into a normal hearing model for processing, and inputting the processed signal into a loudness model for calculating to obtain the loudness L _NH 。

5. A method according to claim 1 or 2 or 3, wherein the enhancement parameters are obtained by: 21 Calculating according to the current enhancement parameter P to obtain the loudness L _HI Sum loudness L _NH If the current L _HI <L _NH And P is<P _MAX Then the enhancement parameter P is enlarged; 22 Repeating step 21) until the loudness L is calculated from the current enhancement parameters P _HI Loudness of sound

L _NH Satisfy L _HI ≥L _NH Or P is greater than or equal to P _MAX 。

6. A method according to claim 1 or 2 or 3, characterized in that the normal auditory system or the hearing impaired model is obtained by setting the hearing threshold of the auditory model.

7. A consonant enhancement device for hearing loss, comprising a consonant detection and classification module, a consonant enhancement parameter setting module, a consonant enhancement module, a loudness compensation module, and the like; wherein the method comprises the steps of

The consonant detection and classification is used for detecting the boundary of the consonant and classifying the detected consonant into different categories;

the consonant enhancement parameter module is used for determining enhancement parameters of consonants of different types under different hearing loss degrees by using a loudness model and an auditory model;

the consonant enhancement module is used for adjusting the amplitude and duration of consonants according to the consonant enhancement parameters;

the loudness compensation module is used for compensating sound according to the hearing loss condition, so that the overall loudness perceived by a hearing loss patient is consistent with that of a normal person.

8. The consonant enhancement apparatus according to claim 7, wherein the consonant detection and classification classifies consonants into different categories according to pronunciation location and pronunciation manner, including no air supply stop, stop wipe, pass, and nasal.

9. The consonant enhancement device of claim 7, wherein the consonant enhancement parameter module obtains the enhancement parameters by: 21 Calculating according to the current enhancement parameter P to obtain the loudness L _HI Sum loudness L _NH If the current L _HI <L _NH And P is<P _MAX Then the enhancement parameter P is enlarged; 22 Repeating step 21) until the loudness L is calculated from the current enhancement parameters P _HI Loudness L _NH Satisfy L _HI ≥L _NH Or P is greater than or equal to P _MAX 。

Images