EP3113183B1 - Speech intelligibility improving apparatus and computer program therefor - Google Patents
Speech intelligibility improving apparatus and computer program therefor Download PDFInfo
- Publication number
- EP3113183B1 EP3113183B1 EP15755932.9A EP15755932A EP3113183B1 EP 3113183 B1 EP3113183 B1 EP 3113183B1 EP 15755932 A EP15755932 A EP 15755932A EP 3113183 B1 EP3113183 B1 EP 3113183B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- speech
- spectrum
- noise
- general outline
- envelope
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004590 computer program Methods 0.000 title claims description 7
- 238000001228 spectrum Methods 0.000 claims description 105
- 230000003595 spectral effect Effects 0.000 claims description 23
- 230000002194 synthesizing effect Effects 0.000 claims description 9
- 239000000284 extract Substances 0.000 claims description 5
- 238000000034 method Methods 0.000 description 24
- 238000012986 modification Methods 0.000 description 17
- 230000004048 modification Effects 0.000 description 17
- 238000009499 grossing Methods 0.000 description 12
- 230000008447 perception Effects 0.000 description 10
- 230000008569 process Effects 0.000 description 10
- 238000012545 processing Methods 0.000 description 9
- 230000001965 increasing effect Effects 0.000 description 8
- 230000008859 change Effects 0.000 description 7
- 238000007781 pre-processing Methods 0.000 description 7
- 238000007493 shaping process Methods 0.000 description 7
- 230000015572 biosynthetic process Effects 0.000 description 6
- 238000001914 filtration Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000003786 synthesis reaction Methods 0.000 description 6
- 230000006978 adaptation Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 230000006835 compression Effects 0.000 description 5
- 238000007906 compression Methods 0.000 description 5
- 238000000605 extraction Methods 0.000 description 4
- 230000006872 improvement Effects 0.000 description 4
- 238000005070 sampling Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 206010021403 Illusion Diseases 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000002411 adverse Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000009432 framing Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- NRNCYVBFPDDJNE-UHFFFAOYSA-N pemoline Chemical compound O1C(N)=NC(=O)C1C1=CC=CC=C1 NRNCYVBFPDDJNE-UHFFFAOYSA-N 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0316—Speech enhancement, e.g. noise reduction or echo cancellation by changing the amplitude
- G10L21/0324—Details of processing therefor
- G10L21/0332—Details of processing therefor involving modification of waveforms
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
- G10L21/0216—Noise filtering characterised by the method used for estimating noise
- G10L21/0232—Processing in the frequency domain
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
- G10L13/033—Voice editing, e.g. manipulating the voice of the synthesiser
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
- G10L19/0204—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders using subband decomposition
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/003—Changing voice quality, e.g. pitch or formants
- G10L21/007—Changing voice quality, e.g. pitch or formants characterised by the process used
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0316—Speech enhancement, e.g. noise reduction or echo cancellation by changing the amplitude
- G10L21/0364—Speech enhancement, e.g. noise reduction or echo cancellation by changing the amplitude for improving intelligibility
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R27/00—Public address systems
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
- G10L21/0216—Noise filtering characterised by the method used for estimating noise
- G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
- G10L2021/02165—Two microphones, one receiving mainly the noise signal and the other one mainly the speech signal
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
- G10L25/15—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters the extracted parameters being formant information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R2227/00—Details of public address [PA] systems covered by H04R27/00 but not provided for in any of its subgroups
- H04R2227/009—Signal processing in [PA] systems to enhance the speech intelligibility
Definitions
- the present invention relates to speech intelligibility improvement and, more specifically, to a technique of processing a speech signal such that the speech becomes highly intelligible even in a noisy environment.
- Such a broadcast is to transmit information to the public and, therefore, the information should desirably be correctly transmitted to the public.
- information is transmitted by speeches through an outdoor loudspeaker using an emergency municipal radio communication system, or through a speaker of a municipal sound truck. At the time of a disaster, it is particularly necessary to transmit such information rightly to the public.
- the simplest solution to such a problem is to turn up (amplify) the volume. Because of the limit of output device performance, however, the volume might not be sufficiently increased, or speech signals might be distorted and become harder to hear when the volume is increased. In addition, speeches in large volume would be unnecessarily loud for neighbors and passers-by, possibly causing a problem of noise pollution.
- Fig. 1 shows a typical example of prior art (Non-Patent Literature 1) for improving speech intelligibility without increasing the volume in a bad condition as described above.
- a conventional speech intelligibility improving apparatus 30 receives input of a speech signal 32 and outputs a modified speech signal 34 with improved intelligibility.
- Speech intelligibility improving apparatus 30 includes: a filtering unit (HPF) 40 mainly passing high-frequency band of speech signal 32 for enhancing high frequency range of voice signal 32; and a dynamic range compression unit (DRC) 42 for compressing dynamic range of waveform amplitude of the signal output from filtering unit 40, so as to make the waveform amplitude uniform in the time direction.
- HPF filtering unit
- DRC dynamic range compression unit
- Enhancement of high-frequency-range components of speech signal 32 by filtering unit 40 simulates unique utterance (Lombard speech) used by humans in a noisy environment and, hence, improvement in intelligibility is expected.
- the degree of enhancement of high-frequency-range components is adjusted continuously in accordance with characteristics of the input speech.
- dynamic range compressing unit 42 amplifies the waveform amplitude where the volume is locally small and attenuates the amplitude where the volume is large, so that the amplitude of speech waveform becomes uniform. In this manner, the speech becomes relatively more intelligible with indistinct sound reduced, without increasing the overall sound volume.
- this conventional approach does not include any method of adapting speech to noise. Therefore, there is no guarantee that high intelligibility can be maintained in various noisy environments. In other words, it is not always possible to address the changes in ambient noise mixed with the speech.
- Non-Patent Literature 2 A proposed solution to this problem is to generate a speech of higher intelligibility even in a noisy environment, by modifying speech spectrum in accordance with the noise characteristics (Non-Patent Literature 2). Constraints on spectrum modification, however, are rather lax and, hence, features essential in speech perception might possibly be modified by such modification of speech spectrum. Excessive modification caused in this manner may lead to undesirable degradation of voice quality, resulting in indistinct speeches.
- the present invention was made to solve such problems, and its object is to provide a speech intelligibility improving apparatus capable of synthesizing speeches highly intelligible in various environments, without unnecessarily increasing sound volume.
- the present invention provides a speech intelligibility improving apparatus according to independent claim 1.
- the present invention provides a computer program according to claim 4.
- One is a technique of speech adaptation to noise characteristics through spectrum shaping based on spectral envelope curve.
- the other is a technique of thinning out harmonics that do not have much influence to speech perception in noise and re-distributing energy of the thinned-out harmonics to other essential components.
- spectral envelope curve and “envelope surface” of spectrogram are used. These terms are different from the “spectral envelope” generally used in the art, and also different from mathematical “envelope curve” and “envelope surface.”
- the spectral envelope represents moderate variation in frequency direction with minute structure such harmonics included in speech spectrum removed, and is generally said to reflect human vocal tract characteristics.
- the "envelope curve” or the curve given as a cross-section at a specific time of the "envelope surface” in accordance with the present invention is a curve drawn in contact with, or close to and along, a plurality of local peaks of formant and the like of the general "spectral envelope” and it is given as more moderate curve than the spectral envelope.
- envelope of spectral envelope or a "general outline of peaks of spectral envelope.”
- the general "spectral envelope” will be denoted as “spectral envelope” and the curve in contact with local peaks of spectral envelope or the curve drawn along the peaks will be simply referred to as “envelope curve (of spectrum)”.
- envelope curve of spectrum
- spectrogram envelope a surface formed by spectral envelope of a spectrum constituting the spectrogram at each time point
- envelope surface the curved surface in contact with local peaks of spectrogram envelope or drawn along the peaks
- envelope curve or envelope surface may be extracted not through the spectral envelope.
- a curve represented as a cross-section at specific frequency of the "envelope surface” in accordance with the present specification (time change of spectrum at a certain frequency) is also referred to as an envelope curve here. It is needless to say that the "curve” and “curved surface” here encompass a straight line and a flat surface, respectively.
- the speech intelligibility is improved through the following steps.
- the present embodiment performs spectrum shaping while taking into consideration the significance of peaks of speech spectrum, such as formants, in speech perception, and simultaneously applies dynamic range compression to the temporal variation of spectrum, which is closely related to the auditory perception.
- Fig. 2 shows examples of speech spectrogram 60 and its envelope surface 62.
- envelope surface 62 is drawn 80 dB higher than the actual values for convenience, so as to facilitate viewing. Actually, these two are in such a relation that peaks of spectrogram 60 contact envelope surface 62 from below.
- the frequency axis is in Bark scale frequency, and the ordinate represents logarithmic power.
- perceptual or psycho-acoustic scale such as Mel scale, Bark scale or ERB scale, it becomes possible to extract an envelope surface with a high regard for spectrum in low frequency range, on which speech intelligibility much depends.
- Envelope surface 62 is taken to be a relatively moderate envelope relative to the variation of spectrogram 60 as mentioned above, and its change is more moderate in the time axis direction than in the frequency direction, as will be described later.
- n-th approximation of the envelope surface is represented as - X k,m (n)
- two-dimensional discrete inverse Fourier transform of its log is represented as - x u,v (n) .
- the envelope surface is updated in accordance with the following equation.
- ⁇ is a coefficient for accelerating convergence.
- - X k,m is given by X ⁇ k , m ⁇ ⁇ X ⁇ k , m n if X ⁇ k , m n > X ⁇ min , X ⁇ min if X ⁇ k , m n ⁇ X ⁇ min .
- - X min is a predetermined coefficient.
- L u,v ⁇ 1 if u ⁇ ⁇ f s and v ⁇ ⁇ NT f , 0 otherwise .
- f s sampling frequency of speech.
- T f represents frame period for analysis.
- N represents the total number of frames in a voice activity.
- FIG. 3 An envelope surface 62 of Fig. 2 , an envelope curve 72 of Fig. 3 and an envelope curve 92 of Fig. 4(A) are examples obtained in this manner.
- Figs. 3 and 4 show curves of cross-sections in the frequency direction and the time direction of the envelope surface, respectively, and hence, these are referred to as envelope curves.
- the speech is a synthesized speech and known. Therefore, such an envelope surface can be calculated in advance. If the speech is unknown and given on real-time basis, an envelope surface similar to the above can be obtained in the following manner.
- noise spectrum In order to adapt the envelope surface to noise, it is necessary to obtain noise spectrum.
- ambient noise is collected by a microphone, the power spectrum
- this smoothing is realized in accordance with the following equation.
- Y ⁇ k , m 1 ⁇ ⁇ Y ⁇ k , m ⁇ 1 + ⁇ Y k , m 2 0 ⁇ ⁇ ⁇ 1
- 2 shaped in accordance with - Y k,m is given by the following equation.
- emphasis of spectral peaks utilizing the envelope curve of speech spectrum is done simultaneously. This enhances formants and further improves intelligibility.
- (a) represents formant enhancement ( ⁇ >1) with the envelope curve of spectrum unchanged, while (b) corresponds to a speech spectrum modifying operation that makes the envelope curve parallel to the smoothed noise spectrum.
- Equation (7) (a) will be discussed in greater detail.
- a speech spectrogram (spectrum) 70 at a certain time point its envelope curve is assumed to be an envelope curve 72.
- Natural logarithm of the equation above is as follows.
- the curve 74 is modified to a curve 76 shown in Fig. 3(C) .
- This modification corresponds to emphasis of the peak portion by making deeper the trough portion of curve 74.
- the first term of the equation above means adding In - X k,m to the curve 76 shown in Fig. 3(C) in the log domain.
- the curve 76 of Fig. 3(C) moves upward by ln - X k,m along the log power axis.
- the peak of spectrum 80 is in contact with the same envelope curve as envelope curve 72 shown in Fig. 3(A) .
- ⁇ m ⁇ 1 if R m ⁇ ⁇ ⁇ ⁇ 0 , ⁇ R m otherwise .
- R m represents degree of spectrum modification.
- R m is given by the following equation.
- FIG. 5 shows an example of power spectrum of speech obtained by the modification described above.
- a noise signal 130 has smoothed spectrum 134.
- the above-described intelligibility improving process is done on a synthesized speech signal for utterance and a speech signal 132 is obtained. From Fig. 5 , we can see at first the effect attained by the use of Bark scale frequency when the envelope surface is extracted.
- the speech spectrum is adapted to noise spectrum mainly in a relatively low frequency range, and particularly in the frequency band of 4000 Hz or lower that influences intelligibility, the power of peaks of formant and the like of speech signal 132 of utterance becomes higher than the noise spectrum.
- the envelope curve 136 of spectrum of the speech signal in this band is parallel to and positioned above the smoothed spectrum 134 of the noise signal.
- the speech is synthesized such that the formant portions of speech (spectrum peak) that have much influence on intelligibility stand out from the noise spectrum. As a result, clear speech that is easily intelligible even in a noisy environment can be generated.
- Equation (7) realizes such a modification as shown in Fig. 4 on the variation of speech spectrogram in time direction.
- a cross-section 90 of a certain frequency of the spectrogram before the modification described above assume that a cross-section at the same frequency of the envelope surface of the spectrogram is represented by an envelope curve 92. Further, assume that a transitional portion 94 from consonant to vowel exists at a portion having relatively low power of cross-section 90.
- coefficients of Equation (5) are set, for example, in the following manner.
- the envelope curve is made to follow the rise and fall as shown in Fig. 4(A) and ⁇ is set to about 20 to about 40 Hz so that the transitional portion between consonant and vowel, for example, is emphasized as shown in (B) of the figure.
- the above-described spectrum shaping improves intelligibility of speech even in a noisy environment.
- the present embodiment aims to further enhance intelligibility by thinning out harmonics having only a slight influence on speech intelligibility, putting energy of the thinned-out harmonics on remaining harmonics and thereby increasing perceived volume and the intelligibility.
- the number of harmonics to be left is limited to a prescribed number or smaller.
- sinusoidal wave synthesis is used for speech synthesis.
- the coefficient ⁇ is positive, of the speech signal, only those harmonic components exceeding the level higher by ⁇ in logarithmic power than the smoothed spectrum of noise signal are synthesized, and other harmonic components are not synthesized. If the coefficient ⁇ is negative, only those harmonic components not lower than the level lower by the absolute value of ⁇ in logarithmic power than the smoothed spectrum of noise signal are synthesized, and other harmonic components are not synthesized.
- the harmonics on both sides of a harmonic positioned closest to each formant frequency are not thinned-out and not synthesized. This is based on a principle similar to so-called masking. Specifically, the harmonics next to the harmonic positioned closest to the formants do not have much influence on hearing. If the harmonic components become too thin, perception of voice pitch becomes difficult, and this is the reason why one of the neighboring harmonics is synthesized and the other is not.
- harmonic components 170, 172, 190, 174, 176, 178, 180 and 182 only satisfy Equation (12). Therefore, only these are the objects of synthesis, and other harmonic components are not synthesized. Further, harmonic components 190 and 180, which are to be the objects of synthesis, are not synthesized, since these are next to harmonic components 172 and 178 forming the formants, respectively. Harmonic components 170 and 176 on the opposite sides, respectively, are left.
- harmonic components 210, 212, 214, 216, 218 and 222 with power level increased are obtained as shown in Fig. 6(B) .
- the remaining harmonic components come to have power still higher than the noise spectrum and, SN ratio is improved near the formants.
- the total sum of energy of speech signal is unchanged and, therefore, physical sound volume is unchanged.
- a speech intelligibility improving apparatus 250 receives as inputs a synthesized speech signal 254 synthesized by a speech synthesizing unit 252 and a noise signal 256 representing ambient noise collected by a microphone 258, adapts synthesized speech signal 254 to noise signal 256, and thereby outputs a modified speech signal 260 that is more intelligible than the speech given by synthesized speech signal 254.
- Speech intelligibility improving apparatus 250 includes: a spectrogram extracting unit 290 receiving synthesized speech signal 254 and extracting its spectrogram
- Extraction of spectrogram by spectrogram extracting unit 290 can be realized by existing technique.
- Extraction of envelope surface by envelope surface extracting unit 292 uses the technique described in sections 1.1.1 and 1.1.2. This process can be realized by computer hardware and software, or by a dedicated hardware. Here, it is realized by computer hardware and software.
- a synthesized speech provided by speech synthesizing unit 252 is used as the object of modification as in the present embodiment, most of the spectrogram extraction and envelope surface extraction cay be done beforehand by calculation, since the speech signal is known in advance.
- Speech intelligibility improving apparatus 250 further includes: a pre-processing unit 294 performing pre-processing such as digitization and framing on noise signal 256 received from microphone 258 and outputting a noise signal consisting of a series of frames; a power spectrum calculating unit 296 extracting power spectrum from the framed noise signal output from pre-processing unit 294; a smoothing unit 298 smoothing time change of the power spectrum of noise signal extracted by power spectrum calculating unit 296, and thereby outputting a smoothed spectrum - Y k,m at time mT f (m-th frame) of the noise signal; a noise adapting unit 300 performing noise adaptation process described in section 1.1.3 above based on the spectrogram
- the output from sinusoidal speech synthesizing unit 305 is the modified speech signal 260, which is adapted to noise and has improved intelligibility. It is needless to say that the process of sampling the spectrum
- Speech intelligibility improving apparatus 250 operates in the following manner. Receiving an instruction of generating a speech, not shown, speech synthesizing unit 252 performs speech synthesis, outputs synthesized speech signal 254 and applies it to spectrogram extracting unit 290. Spectrogram extracting unit 290 extracts a spectrogram from synthesized speech signal 254, and applies it to envelope surface extracting unit 292 and noise adapting unit 300. Envelope surface extracting unit 292 extracts, from the spectrogram received from spectrogram extracting unit 290, an envelope surface and applies it to noise adapting unit 300.
- Microphone 258 collects ambient noise, converts it to noise signal 256 as an electric signal, and applies it to pre-processing unit 294.
- Pre-processing unit 294 digitizes the noise signal 256 received from microphone 258 frame by frame, each frame having a prescribed frame length and prescribed shift length, and applies the resulting signal as a series of frames to power spectrum calculating unit 296.
- Power spectrum calculating unit 296 extracts power spectrum from the noise signal received from pre-processing unit 294, and applies the power spectrum to smoothing unit 298. Smoothing unit 298 smoothes time sequence of the spectrum by filtering, and thereby calculates smoothed spectrum of noise, which is applied to noise adapting unit 300.
- Noise adapting unit 300 performs noise adaptation process on the spectrogram applied from spectrogram extracting unit 290 in accordance with the method described above, using the envelope surface of the spectrogram of synthesized speech 254 applied from envelope surface extracting unit 292 and the smoothed spectrum of noise signal applied from smoothing unit 298, outputs harmonic components obtained by sampling the spectrum
- Harmonics thinning unit 302 compares each harmonic output from noise adapting unit 300 with the smoothed spectrum of noise signal output from smoothing unit 298, performs the harmonics thinning process described above, and outputs only the remaining harmonics.
- Power re-distributing unit 304 re-distributes power of thinned-out harmonics to each harmonic of spectrogram after thinning output by thinning unit 302 and thereby raises the levels of remaining harmonics, and thus, outputs modified speech signal 260.
- the synthesized speech noise-adapted by noise adapting unit 300 has spectrum peaks emphasized and spectral feature at the transitional portions of speech emphasized. Further, its peak is adapted to the noise level and, hence, the speech intelligible even in a noisy environment can be generated. Further, harmonics thinning unit 302 thins out harmonics not having influence on intelligibility, and power re-distributing unit 304 re-distributes the power to remaining harmonics. As a result, only those portions of the speech which have influence on intelligibility come to have higher power while the total acoustic power is not changed. As a result, easily intelligible speech can be generated without unnecessarily increasing the sound volume.
- the above-described speech intelligibility improving apparatus 250 can substantially be realized by computer hardware and a computer program or programs co-operating with the computer hardware.
- programs executing the processes described in sections 1.1.1, 1.1.2 and 1.1.3 may be used for envelope surface extracting unit 292 and noise adapting unit 300.
- Fig. 8 shows an internal configuration of a computer system 330 realizing speech intelligibility improving apparatus 250 described above.
- computer system 330 includes a computer 340, and microphone 258 and a speaker 344 connected to computer 340.
- Computer 340 includes: a CPU (Central Processing Unit) 356; a bus 354 connected to CPU 356; a re-writable read only memory (ROM) 358 storing a boot-up program and the like; a random access memory (RAM) 360 storing program instructions, a system program and work data; an operation console 362 used, for example, by a maintenance operator; a wireless communication device 364 allowing communication with other terminals through radio wave; a memory port 366 to which a removable memory 346 can be attached; and a sound processing circuit 368 connected to microphone 258 and speaker 344, for performing a process of digitizing speech signals from microphone 258 and a process of analog-converting digital speech signals read from RAM 360 and applying the result to speaker 344.
- a CPU Central Processing Unit
- bus 354 connected to CPU 356
- ROM read only memory
- RAM random access memory
- an operation console 362 used, for example, by a maintenance operator
- a wireless communication device 364 allowing communication with other terminals through radio wave
- a computer program causing computer system 330 to function as speech intelligibility improving apparatus 250 in accordance with the above-described embodiment is stored in advance in a removable memory 346.
- the program is transferred to and stored in ROM 358.
- the program may be transferred to RAM 360 by wireless communication using wireless communication device 364 and then written to ROM 358.
- the program is read from ROM 358 and loaded to RAM 360.
- the program includes a plurality of instructions to cause computer 340 to operate as various functional units of speech intelligibility improving apparatus 250 in accordance with the above-described embodiment.
- Some of the basic functions necessary to realize the operation may be dynamically provided at the time of execution by the operating system (OS) running on computer 340, by a third party program, or by various programming tool kits or a program library installed in computer 340. Therefore, the program may not necessarily include all of the functions necessary to realize speech intelligibility improving apparatus 250 in accordance with the above-described embodiment.
- the program have only to include instructions to realize the functions of the above-described system by dynamically calling appropriate functions or appropriate program tools in a program tool kit from storage devices in computer 340 in a manner controlled to attain desired results. Naturally, the program only may provide all the necessary functions.
- the speech signal or the like is applied from microphone 258 to sound processing circuit 368, digitized by sound processing circuit 368 and stored in RAM 360, and processed by CPU 356.
- the modified speech signal obtained as a result of processing by CPU 356 is stored in RAM 360.
- sound processing circuit 368 reads the speech signal from RAM 360, analog-converts the same and applies the result to speaker 344, from which the speech is generated.
- the speech intelligibility improving apparatus 250 when a speech is to be generated in a noisy environment, the speech signal representing the speech to be generated can be modified both along the time-axis and the frequency-axis simultaneously based on the acoustic characteristics of noise, whereby the speech can be heard with high intelligibility even in a noisy environment.
- the speech signal representing the speech to be generated can be modified both along the time-axis and the frequency-axis simultaneously based on the acoustic characteristics of noise, whereby the speech can be heard with high intelligibility even in a noisy environment.
- formant peak when formant peak is to be emphasized, only the portion or portions having influence on hearing are emphasized and, therefore, unnecessary increase in the sound volume is avoided.
- the spectrum shaping technique in accordance with the present embodiment takes into consideration the importance of speech spectrum peaks such as formants in speech perception, and performs dynamic range compression with respect to time change of spectrum having close relation to speech perception. In this regard, this technique is much different from conventional approaches.
- the embodiment described above is directed to an apparatus for generating a synthesized speech in a noisy environment.
- the present invention is not limited to such an embodiment. It is needless to say that the present invention is applicable to modify actual speech of fresh voice to be more intelligible over noise, when the actual speech is to be transmitted over a speaker. In this situation, if it is possible, the actual speech should preferably be processed not on fully real-time basis but with a delay of some time. By doing so, it becomes possible to obtain the envelope surface of speech spectrogram for a longer time period and, hence, it becomes possible to modify the speech more effectively.
- one of the two harmonics on opposite sides of the harmonics positioned closest to a peak such as a formant is the object of deletion.
- the present invention is not limited to such an embodiment. Both of the two may be deleted, or both may not be deleted.
- the present invention is applicable to devices and equipment for reliably transmitting information by speech in a possibly noisy environment both indoors and outdoors.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Signal Processing (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Quality & Reliability (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Circuit For Audible Band Transducer (AREA)
Description
- The present invention relates to speech intelligibility improvement and, more specifically, to a technique of processing a speech signal such that the speech becomes highly intelligible even in a noisy environment.
- An announcement in public places such as train stations and underground shopping malls, speeches in actual voice, recorded voice or synthesized voice are emitted from a speaker through a transmission channel. Such a broadcast is to transmit information to the public and, therefore, the information should desirably be correctly transmitted to the public. Sometimes information is transmitted by speeches through an outdoor loudspeaker using an emergency municipal radio communication system, or through a speaker of a municipal sound truck. At the time of a disaster, it is particularly necessary to transmit such information rightly to the public.
- It is often difficult, however, to clearly hear and understand the contents of speeches in a public place such as a train station or an underground shopping mall. The reason for this difficulty is surrounding noise and acoustic transmission characteristics of the speaker. Particularly, outdoor transmission of information by speeches is adversely affected by long-path echo, wind and so on. Not only in the public places but also at home, when we listen to the radio or watch television, it is often difficult to clearly hear the speeches because of noise coming from outside and because of household noise.
- The simplest solution to such a problem is to turn up (amplify) the volume. Because of the limit of output device performance, however, the volume might not be sufficiently increased, or speech signals might be distorted and become harder to hear when the volume is increased. In addition, speeches in large volume would be unnecessarily loud for neighbors and passers-by, possibly causing a problem of noise pollution.
-
Fig. 1 shows a typical example of prior art (Non-Patent Literature 1) for improving speech intelligibility without increasing the volume in a bad condition as described above. Referring toFig. 1 , a conventional speechintelligibility improving apparatus 30 receives input of aspeech signal 32 and outputs a modifiedspeech signal 34 with improved intelligibility. Speechintelligibility improving apparatus 30 includes: a filtering unit (HPF) 40 mainly passing high-frequency band ofspeech signal 32 for enhancing high frequency range ofvoice signal 32; and a dynamic range compression unit (DRC) 42 for compressing dynamic range of waveform amplitude of the signal output from filteringunit 40, so as to make the waveform amplitude uniform in the time direction. - Enhancement of high-frequency-range components of
speech signal 32 by filteringunit 40 simulates unique utterance (Lombard speech) used by humans in a noisy environment and, hence, improvement in intelligibility is expected. The degree of enhancement of high-frequency-range components is adjusted continuously in accordance with characteristics of the input speech. On the other hand, dynamicrange compressing unit 42 amplifies the waveform amplitude where the volume is locally small and attenuates the amplitude where the volume is large, so that the amplitude of speech waveform becomes uniform. In this manner, the speech becomes relatively more intelligible with indistinct sound reduced, without increasing the overall sound volume. - The patent documents
US 2009/0281805 A1 andEP 1 850 328 A1 -
- NPL 1: T. Zorila, V. Kandia, and Y. Stylianou, "Speech-in-noise intelligibility improvement based on spectral shaping and dynamic range compression," in Proc. Interspeech, Portland Oregon, USA, 2012.
- NPL 2: C.H. Taal, R.C. Hendriks, R. Heusdens, "A speech preprocessing strategy for intelligibility improvement in noise based on a perceptual distortion measure, in Proc. ICASSP, pp. 4061-4064, 2012.
- In the existing system shown in
Fig. 1 , however, perceptual characteristics of speech are not considered in speech processing either by thefiltering unit 40 or by the dynamicrange compressing unit 42. Therefore, we cannot say that the system based on this prior art uses the optimal method for improving speech intelligibility. Specifically, while the enhancement of high frequency range of speech is based on global inclination of the speech spectrum and the dynamic range compression is based on the amplitude of the speech waveform, the former should be done in consideration of the significance of the spectral peaks such as formants in voice perception, and the latter should be done while paying attention to the fact that the waveform amplitude does not necessarily correspond to the speech power. - Further, this conventional approach does not include any method of adapting speech to noise. Therefore, there is no guarantee that high intelligibility can be maintained in various noisy environments. In other words, it is not always possible to address the changes in ambient noise mixed with the speech.
- A proposed solution to this problem is to generate a speech of higher intelligibility even in a noisy environment, by modifying speech spectrum in accordance with the noise characteristics (Non-Patent Literature 2). Constraints on spectrum modification, however, are rather lax and, hence, features essential in speech perception might possibly be modified by such modification of speech spectrum. Excessive modification caused in this manner may lead to undesirable degradation of voice quality, resulting in indistinct speeches.
- The present invention was made to solve such problems, and its object is to provide a speech intelligibility improving apparatus capable of synthesizing speeches highly intelligible in various environments, without unnecessarily increasing sound volume.
- According to a first aspect, the present invention provides a speech intelligibility improving apparatus according to
independent claim 1. - According to a second aspect, the present invention provides a computer program according to claim 4.
-
-
Fig. 1 is a block diagram showing a configuration of a conventional speech intelligibility improving apparatus. -
Fig. 2 is a graph showing a relation between speech spectrogram and envelope surface of the spectrogram used in an embodiment of the present invention. -
Fig. 3 includes graphs illustrating modifications of spectral distribution of a speech signal in accordance with an embodiment of the present invention. -
Fig. 4 includes graphs illustrating modifications of power variation at a specific frequency of speech signal spectrogram in accordance with an embodiment of the present invention. -
Fig. 5 is a graph illustrating a method of modifying spectral distribution envelope of a speech signal with noise-adaptation in an embodiment of the present invention. -
Fig. 6 includes graphs illustrating a method of boosting essential components using power of unnecessary harmonic components of a speech signal, in accordance with an embodiment of the present invention. -
Fig. 7 is a functional block diagram of a speech intelligibility improving apparatus in accordance with an embodiment of the present invention. -
Fig. 8 is a hardware block diagram of a computer implementing the speech intelligibility improving apparatus shown inFig. 7 . - In the following description and in the drawings, the same components are denoted by the same reference characters. Therefore, detailed description thereof will not be repeated. In the following description, basic concepts as a basis of an embodiment will be described first, and then, configurations and operations of the speech intelligibility improving apparatus in accordance with the embodiment will be described.
- In the embodiment described in the following, two techniques for improving speech intelligibility are used. One is a technique of speech adaptation to noise characteristics through spectrum shaping based on spectral envelope curve. The other is a technique of thinning out harmonics that do not have much influence to speech perception in noise and re-distributing energy of the thinned-out harmonics to other essential components.
- In the present specification, the terms spectral "envelope curve" and "envelope surface" of spectrogram are used. These terms are different from the "spectral envelope" generally used in the art, and also different from mathematical "envelope curve" and "envelope surface." The spectral envelope represents moderate variation in frequency direction with minute structure such harmonics included in speech spectrum removed, and is generally said to reflect human vocal tract characteristics. On the other hand, the "envelope curve" or the curve given as a cross-section at a specific time of the "envelope surface" in accordance with the present invention is a curve drawn in contact with, or close to and along, a plurality of local peaks of formant and the like of the general "spectral envelope" and it is given as more moderate curve than the spectral envelope. In this sense, this may be represented as "envelope of spectral envelope" or a "general outline of peaks of spectral envelope." Here, in order to distinguish the spectral envelope from the "envelope curve" in the present specification, the general "spectral envelope" will be denoted as "spectral envelope" and the curve in contact with local peaks of spectral envelope or the curve drawn along the peaks will be simply referred to as "envelope curve (of spectrum)". The same applies to the "envelope surface" of a spectrogram. In a spectrogram a surface formed by spectral envelope of a spectrum constituting the spectrogram at each time point is referred to as "spectrogram envelope," and the curved surface in contact with local peaks of spectrogram envelope or drawn along the peaks will be simply referred to as "envelope surface (of a spectrogram)." It is noted, however, that the envelope curve or envelope surface may be extracted not through the spectral envelope. A curve represented as a cross-section at specific frequency of the "envelope surface" in accordance with the present specification (time change of spectrum at a certain frequency) is also referred to as an envelope curve here. It is needless to say that the "curve" and "curved surface" here encompass a straight line and a flat surface, respectively.
- According to the technique of improving speech intelligibility through spectrum shaping based on envelope curve of spectrum, the speech intelligibility is improved through the following steps.
- (1) Extracting an envelope surface of speech spectrogram.
- (2) Modifying the spectrum to emphasize peaks such as formants of the spectrum, based on said envelope surface.
- (3) Modifying speech spectrum and time variation thereof in accordance with the envelope surface of spectrogram.
- (4) Further, adding such a modification to speech spectrum that makes the smoothed spectrum of noise becomes parallel to the envelope curve of speech spectrum, for each frame of the spectrogram.
- As described above, unlike the conventional method, the present embodiment performs spectrum shaping while taking into consideration the significance of peaks of speech spectrum, such as formants, in speech perception, and simultaneously applies dynamic range compression to the temporal variation of spectrum, which is closely related to the auditory perception.
-
Fig. 2 shows examples ofspeech spectrogram 60 and itsenvelope surface 62. InFig. 2 ,envelope surface 62 is drawn 80 dB higher than the actual values for convenience, so as to facilitate viewing. Actually, these two are in such a relation that peaks ofspectrogram 60contact envelope surface 62 from below. InFig. 2 , the frequency axis is in Bark scale frequency, and the ordinate represents logarithmic power. By using perceptual or psycho-acoustic scale such as Mel scale, Bark scale or ERB scale, it becomes possible to extract an envelope surface with a high regard for spectrum in low frequency range, on which speech intelligibility much depends.Envelope surface 62 is taken to be a relatively moderate envelope relative to the variation ofspectrogram 60 as mentioned above, and its change is more moderate in the time axis direction than in the frequency direction, as will be described later. - Consider, for a spectrogram |Xk,m|2 (where k represents a position of frequency range on the frequency axis of the spectrogram as an object, and m represents position on the time axis of the spectrogram as an object, or a frame number), finding an envelope surface -Xk,m (here, "-" represents a bar drawn over the character that immediately follows, in the equations shown below) in contact with the local peaks. Here, the following successive approximation is used.
- The n-th approximation of the envelope surface is represented as -Xk,m (n), and two-dimensional discrete inverse Fourier transform of its log is represented as -xu,v (n). The initial value -xu,v (0) is given by the following equation.
-
- For a prescribed value ε>0, convergence is determined using the following equation, in which M and N represent the number of data points and the total number of frames of the spectrum, respectively.
- In the present embodiment, the following equation is used for the term Lu,v in Equations (1), (2) and (3).
- An
envelope surface 62 ofFig. 2 , anenvelope curve 72 ofFig. 3 and anenvelope curve 92 ofFig. 4(A) are examples obtained in this manner.Figs. 3 and4 show curves of cross-sections in the frequency direction and the time direction of the envelope surface, respectively, and hence, these are referred to as envelope curves. - In the present embodiment, as will be described later, it is a precondition that the speech is a synthesized speech and known. Therefore, such an envelope surface can be calculated in advance. If the speech is unknown and given on real-time basis, an envelope surface similar to the above can be obtained in the following manner.
- (1) Successively calculating an envelope curve of currently analyzed frame of the spectrum.
- (2) Smoothing the time sequence of envelope curves obtained by the calculations in the time-axis direction, using a low-pass filter, for example.
- In order to adapt the envelope surface to noise, it is necessary to obtain noise spectrum. In the present embodiment, ambient noise is collected by a microphone, the power spectrum |Yk,m|2 thereof is successively calculated, and a spectrum -Yk,m smoothed in the time direction is obtained by using, for example, a low-pass filter. In the present embodiment, this smoothing is realized in accordance with the following equation.
- Equation (7) (a) will be discussed in greater detail. Referring to
Fig. 3(A) , for a speech spectrogram (spectrum) 70 at a certain time point, its envelope curve is assumed to be anenvelope curve 72. Equation (7) (a) can be represented asspectrum 70 shown inFig. 3(A) is modified to acurve 74 ofFig. 3(B) . InFig. 3(B) , the logarithmic power value of the peak ofcurve 74 is substantially 0. - Further, by multiplying this value by γ>1 in log domain, the
curve 74 is modified to acurve 76 shown inFig. 3(C) . This modification corresponds to emphasis of the peak portion by making deeper the trough portion ofcurve 74. - The first term of the equation above means adding In -Xk,m to the
curve 76 shown inFig. 3(C) in the log domain. As a result, thecurve 76 ofFig. 3(C) moves upward by ln-Xk,m along the log power axis. This results in aspectrum 80 shown inFig. 3(D) . The peak ofspectrum 80 is in contact with the same envelope curve asenvelope curve 72 shown inFig. 3(A) . - In Equation (8), Dk,m represents a ratio between the smoothed spectrum of noise and the envelope curve of speech spectrum. This value is raised to ζm-th power and multiplied by (a) as indicated by (b) of Equation (7) (in log domain, the difference between the smoothed spectrum of noise and the envelope curve of speech spectrum is multiplied by ζm and added to
spectrum 80 ofFig. 3(D) ). This is an operation to modifyspectrum 80 shown inFig. 3(D) such that the envelope curve of the spectrum becomes matches the smoothed spectrum of noise. Assuming that ζm = 1, for example, in log domain, it means that theenvelope curve 72 is subtracted fromspectrum 80 ofFig. 3(C) and the smoothed noise spectrum -Yk,m of noise is added. In order to avoid extreme modification, however, ζm for a specific ξ is defined as below.Fig. 5 shows an example of power spectrum of speech obtained by the modification described above. InFig. 5 , it is assumed that anoise signal 130 has smoothedspectrum 134. The above-described intelligibility improving process is done on a synthesized speech signal for utterance and aspeech signal 132 is obtained. FromFig. 5 , we can see at first the effect attained by the use of Bark scale frequency when the envelope surface is extracted. Specifically, the speech spectrum is adapted to noise spectrum mainly in a relatively low frequency range, and particularly in the frequency band of 4000 Hz or lower that influences intelligibility, the power of peaks of formant and the like ofspeech signal 132 of utterance becomes higher than the noise spectrum. Next, it is noted that theenvelope curve 136 of spectrum of the speech signal in this band is parallel to and positioned above the smoothedspectrum 134 of the noise signal. Thus, the speech is synthesized such that the formant portions of speech (spectrum peak) that have much influence on intelligibility stand out from the noise spectrum. As a result, clear speech that is easily intelligible even in a noisy environment can be generated. - In accordance with such a modification (in the frequency domain) of spectrum, Equation (7) realizes such a modification as shown in
Fig. 4 on the variation of speech spectrogram in time direction. Referring toFig. 4(A) , for across-section 90 of a certain frequency of the spectrogram before the modification described above, assume that a cross-section at the same frequency of the envelope surface of the spectrogram is represented by anenvelope curve 92. Further, assume that atransitional portion 94 from consonant to vowel exists at a portion having relatively low power ofcross-section 90. - If noise is substantially steady and power spectrum thereof does not much change over time, modification to make flat the
envelope curve 92 to match the noise is effected oncross-section 90 in the time direction of the spectrogram. As shown inFig. 4(B) , the spectrogram is modified such that anenvelope curve 102 is made flat in the time-axis direction. In atime change 100 after modification, the shape of atransitional portion 104 corresponding to thetransitional portion 94 from consonant to vowel shown inFig. 4(A) is pushed upward to be in contact withenvelope curve 102 from below. As a result, when a speech is synthesized based on the modifiedtime change 100, the transitional section as an important clue in consonant perception will relatively amplified/emphasized, and the speech intelligibility can be improved. - On the other hand, coefficients of Equation (5) are set, for example, in the following manner. For the frequency direction, τ is set to τ = 125 µs so that the envelope curve moderately comes to be in contact only with the spectral peak. This corresponds to representing the envelope curve of each frame of speech sampled at 16 kHz, using up to the 2-nd order cepstrum. On the other hand, for the time direction, the envelope curve is made to follow the rise and fall as shown in
Fig. 4(A) and η is set to about 20 to about 40 Hz so that the transitional portion between consonant and vowel, for example, is emphasized as shown in (B) of the figure. Further, γ is set to about γ = 1.3 to emphasize formants. - The above-described spectrum shaping improves intelligibility of speech even in a noisy environment. The present embodiment, however, aims to further enhance intelligibility by thinning out harmonics having only a slight influence on speech intelligibility, putting energy of the thinned-out harmonics on remaining harmonics and thereby increasing perceived volume and the intelligibility. Here, the number of harmonics to be left is limited to a prescribed number or smaller. For this purpose, sinusoidal wave synthesis is used for speech synthesis.
- First, presence/absence of harmonics in a frequency range in which speech is buried in noise does not much influence how the speech is heard. Therefore, in the present embodiment, thinning-out synthesis of harmonics is not performed for such a time frequency that satisfies Equation (12) below with respect to a prescribed constant θ.
- Further, in the present embodiment, even when the speech is not buried in noise, of the harmonics on both sides of a harmonic positioned closest to each formant frequency, one is not thinned-out and not synthesized. This is based on a principle similar to so-called masking. Specifically, the harmonics next to the harmonic positioned closest to the formants do not have much influence on hearing. If the harmonic components become too thin, perception of voice pitch becomes difficult, and this is the reason why one of the neighboring harmonics is synthesized and the other is not.
- In an example shown in
Fig. 6(A) , assume that the smoothed spectrum of noise is as represented byspectrum 160. If θ < 0, of the harmonic components shown inFig. 6 ,harmonic components harmonic components harmonic components Harmonic components - Further, energy of those harmonic components which are determined not to be synthesized is re-distributed to remaining harmonic components. As a result,
energy 200 is re-distributed toharmonic components Fig. 6(A) , and as a result,harmonic components Fig. 6(B) . As a result, the remaining harmonic components come to have power still higher than the noise spectrum and, SN ratio is improved near the formants. Here, the total sum of energy of speech signal is unchanged and, therefore, physical sound volume is unchanged. - The configuration of speech intelligibility improving apparatus in accordance with the present invention based on the principle above will be described in the following. Referring to
Fig. 7 , a speechintelligibility improving apparatus 250 in accordance with the present embodiment receives as inputs a synthesizedspeech signal 254 synthesized by aspeech synthesizing unit 252 and anoise signal 256 representing ambient noise collected by amicrophone 258, adapts synthesizedspeech signal 254 to noise signal 256, and thereby outputs a modifiedspeech signal 260 that is more intelligible than the speech given by synthesizedspeech signal 254. - Speech
intelligibility improving apparatus 250 includes: aspectrogram extracting unit 290 receiving synthesizedspeech signal 254 and extracting its spectrogram |Xk,m|2; and an envelopesurface extracting unit 292 extracting, based on the spectrogram |Xk,m|2 extracted byspectrogram extracting unit 290, the envelope surface |-Xk,m| thereof. Extraction of spectrogram byspectrogram extracting unit 290 can be realized by existing technique. Extraction of envelope surface by envelopesurface extracting unit 292 uses the technique described in sections 1.1.1 and 1.1.2. This process can be realized by computer hardware and software, or by a dedicated hardware. Here, it is realized by computer hardware and software. When a synthesized speech provided byspeech synthesizing unit 252 is used as the object of modification as in the present embodiment, most of the spectrogram extraction and envelope surface extraction cay be done beforehand by calculation, since the speech signal is known in advance. - Speech intelligibility improving apparatus 250 further includes: a pre-processing unit 294 performing pre-processing such as digitization and framing on noise signal 256 received from microphone 258 and outputting a noise signal consisting of a series of frames; a power spectrum calculating unit 296 extracting power spectrum from the framed noise signal output from pre-processing unit 294; a smoothing unit 298 smoothing time change of the power spectrum of noise signal extracted by power spectrum calculating unit 296, and thereby outputting a smoothed spectrum -Yk,m at time mTf (m-th frame) of the noise signal; a noise adapting unit 300 performing noise adaptation process described in section 1.1.3 above based on the spectrogram |Xk,m|2 of synthesized speech output from spectrogram extracting unit 290, the envelope surface |-Xk,m| of the synthesized speech output from envelope surface extracting unit 292 and smoothed spectrum -Yk,m of the noise signal output from smoothing unit 298, and outputting harmonic components obtained by sampling, at an interval of fundamental frequency of the speech, the spectrum |X'k,m|2 at time mTf of the adapted speech signal; a harmonic thinning unit 302 performing level comparison between each harmonic output from noise adapting unit 300 and the smoothed spectrum -Yk,m of noise andthinning out harmonics lower than a prescribed level (that is, SN ratio) in accordance with Equation (12) and thinning out one of the harmonics on opposite sides of the harmonic positioned closest to each formant frequency; a power re-distributing unit 304 uniformly re-distributing the power of thinned-out harmonic components to each of the harmonic components left after the thinning by harmonic thinning unit 302; and a sinusoidal speech synthesizing unit 305 synthesizing a speech from the remaining harmonics that received power re-distributed by power re-distributing unit 304. The output from sinusoidal
speech synthesizing unit 305 is the modifiedspeech signal 260, which is adapted to noise and has improved intelligibility. It is needless to say that the process of sampling the spectrum |X'k,m|2 at the interval of fundamental frequency of speech bynoise adapting unit 300 and the process of thinning out harmonics not having much influence on speech perception in a noisy environment byharmonics thinning unit 302 are applied only in a voiced section in which the speech has harmonic components. - Speech
intelligibility improving apparatus 250 operates in the following manner. Receiving an instruction of generating a speech, not shown,speech synthesizing unit 252 performs speech synthesis, outputs synthesizedspeech signal 254 and applies it to spectrogram extractingunit 290.Spectrogram extracting unit 290 extracts a spectrogram from synthesizedspeech signal 254, and applies it to envelopesurface extracting unit 292 andnoise adapting unit 300. Envelopesurface extracting unit 292 extracts, from the spectrogram received fromspectrogram extracting unit 290, an envelope surface and applies it tonoise adapting unit 300. -
Microphone 258 collects ambient noise, converts it to noise signal 256 as an electric signal, and applies it topre-processing unit 294.Pre-processing unit 294 digitizes thenoise signal 256 received frommicrophone 258 frame by frame, each frame having a prescribed frame length and prescribed shift length, and applies the resulting signal as a series of frames to powerspectrum calculating unit 296. Powerspectrum calculating unit 296 extracts power spectrum from the noise signal received frompre-processing unit 294, and applies the power spectrum to smoothingunit 298.Smoothing unit 298 smoothes time sequence of the spectrum by filtering, and thereby calculates smoothed spectrum of noise, which is applied tonoise adapting unit 300. -
Noise adapting unit 300 performs noise adaptation process on the spectrogram applied fromspectrogram extracting unit 290 in accordance with the method described above, using the envelope surface of the spectrogram of synthesizedspeech 254 applied from envelopesurface extracting unit 292 and the smoothed spectrum of noise signal applied from smoothingunit 298, outputs harmonic components obtained by sampling the spectrum |X'k,m|2 at each time after adaptation at the interval of fundamental frequency of speech, and applies the output toharmonics thinning unit 302. -
Harmonics thinning unit 302 compares each harmonic output fromnoise adapting unit 300 with the smoothed spectrum of noise signal output from smoothingunit 298, performs the harmonics thinning process described above, and outputs only the remaining harmonics.Power re-distributing unit 304 re-distributes power of thinned-out harmonics to each harmonic of spectrogram after thinning output by thinningunit 302 and thereby raises the levels of remaining harmonics, and thus, outputs modifiedspeech signal 260. - Because of the principle described above, the synthesized speech noise-adapted by
noise adapting unit 300 has spectrum peaks emphasized and spectral feature at the transitional portions of speech emphasized. Further, its peak is adapted to the noise level and, hence, the speech intelligible even in a noisy environment can be generated. Further,harmonics thinning unit 302 thins out harmonics not having influence on intelligibility, andpower re-distributing unit 304 re-distributes the power to remaining harmonics. As a result, only those portions of the speech which have influence on intelligibility come to have higher power while the total acoustic power is not changed. As a result, easily intelligible speech can be generated without unnecessarily increasing the sound volume. - The above-described speech
intelligibility improving apparatus 250 can substantially be realized by computer hardware and a computer program or programs co-operating with the computer hardware. Here, programs executing the processes described in sections 1.1.1, 1.1.2 and 1.1.3 may be used for envelopesurface extracting unit 292 andnoise adapting unit 300. -
Fig. 8 shows an internal configuration of acomputer system 330 realizing speechintelligibility improving apparatus 250 described above. - Referring to
Fig. 8 ,computer system 330 includes acomputer 340, andmicrophone 258 and aspeaker 344 connected tocomputer 340. -
Computer 340 includes: a CPU (Central Processing Unit) 356; abus 354 connected toCPU 356; a re-writable read only memory (ROM) 358 storing a boot-up program and the like; a random access memory (RAM) 360 storing program instructions, a system program and work data; anoperation console 362 used, for example, by a maintenance operator; awireless communication device 364 allowing communication with other terminals through radio wave; amemory port 366 to which aremovable memory 346 can be attached; and asound processing circuit 368 connected tomicrophone 258 andspeaker 344, for performing a process of digitizing speech signals frommicrophone 258 and a process of analog-converting digital speech signals read fromRAM 360 and applying the result tospeaker 344. - A computer program causing
computer system 330 to function as speechintelligibility improving apparatus 250 in accordance with the above-described embodiment is stored in advance in aremovable memory 346. After theremovable memory 346 is attached tomemory port 366 and a rewriting program ofROM 358 is activated by operatingoperation console 362, the program is transferred to and stored inROM 358. Alternatively, the program may be transferred to RAM 360 by wireless communication usingwireless communication device 364 and then written toROM 358. At the time of execution, the program is read fromROM 358 and loaded toRAM 360. - The program includes a plurality of instructions to cause
computer 340 to operate as various functional units of speechintelligibility improving apparatus 250 in accordance with the above-described embodiment. Some of the basic functions necessary to realize the operation may be dynamically provided at the time of execution by the operating system (OS) running oncomputer 340, by a third party program, or by various programming tool kits or a program library installed incomputer 340. Therefore, the program may not necessarily include all of the functions necessary to realize speechintelligibility improving apparatus 250 in accordance with the above-described embodiment. The program have only to include instructions to realize the functions of the above-described system by dynamically calling appropriate functions or appropriate program tools in a program tool kit from storage devices incomputer 340 in a manner controlled to attain desired results. Naturally, the program only may provide all the necessary functions. - In the present embodiment shown in
Figs. 2 to 7 , the speech signal or the like is applied frommicrophone 258 to soundprocessing circuit 368, digitized bysound processing circuit 368 and stored inRAM 360, and processed byCPU 356. The modified speech signal obtained as a result of processing byCPU 356 is stored inRAM 360. WhenCPU 356 instructssound processing circuit 368 to generate a speech,sound processing circuit 368 reads the speech signal fromRAM 360, analog-converts the same and applies the result tospeaker 344, from which the speech is generated. - The operation of
computer system 330 executing a computer program is well known and, therefore, description thereof will not be given here. - As described above, by the speech
intelligibility improving apparatus 250 in accordance with the above-described present embodiment, when a speech is to be generated in a noisy environment, the speech signal representing the speech to be generated can be modified both along the time-axis and the frequency-axis simultaneously based on the acoustic characteristics of noise, whereby the speech can be heard with high intelligibility even in a noisy environment. At the time of modifying the speech signal, when formant peak is to be emphasized, only the portion or portions having influence on hearing are emphasized and, therefore, unnecessary increase in the sound volume is avoided. - Further, the spectrum shaping technique in accordance with the present embodiment takes into consideration the importance of speech spectrum peaks such as formants in speech perception, and performs dynamic range compression with respect to time change of spectrum having close relation to speech perception. In this regard, this technique is much different from conventional approaches.
- The embodiment described above is directed to an apparatus for generating a synthesized speech in a noisy environment. The present invention, however, is not limited to such an embodiment. It is needless to say that the present invention is applicable to modify actual speech of fresh voice to be more intelligible over noise, when the actual speech is to be transmitted over a speaker. In this situation, if it is possible, the actual speech should preferably be processed not on fully real-time basis but with a delay of some time. By doing so, it becomes possible to obtain the envelope surface of speech spectrogram for a longer time period and, hence, it becomes possible to modify the speech more effectively.
- Further, in the above-described embodiment, when the power of those portions of speech signal which are buried in noise are to be re-distributed to portions having influence on hearing, one of the two harmonics on opposite sides of the harmonics positioned closest to a peak such as a formant is the object of deletion. The present invention, however, is not limited to such an embodiment. Both of the two may be deleted, or both may not be deleted.
- The embodiments as have been described here are mere examples and should not be interpreted as restrictive. The scope of the present invention is determined by each of the claims with appropriate consideration of the written description of the embodiments and embraces modifications within the meaning of the languages in the claims.
- The present invention is applicable to devices and equipment for reliably transmitting information by speech in a possibly noisy environment both indoors and outdoors.
-
- 30,250
- speech intelligibility improving apparatus
- 32, 132
- speech signal
- 34
- modified speech signal
- 40
- filtering unit
- 42
- dynamic range compressing unit
- 60
- spectrogram
- 62
- envelope surface
- 70, 80
- spectrum (spectrogram)
- 72, 92, 102, 136, 134
- envelope curve
- 130
- noise signal
- 256
- noise signal
- 258
- microphone
- 260
- modified speech signal
- 290
- spectrogram extracting unit
- 296
- power spectrum calculating unit
- 292
- envelope surface extracting unit
- 298
- smoothing unit
- 300
- noise adapting unit
- 302
- harmonics thinning unit
- 304
- power re-distributing unit
- 305
- sinusoidal speech synthesizing unit
- 330
- computer system
- 340
- computer
- 344
- speaker
Claims (4)
- A speech intelligibility improving apparatus for generating an intelligible speech, comprising:peak general outline extracting means for extracting, from a spectrum of a speech signal as an object, a general outline of peaks represented by a curve along a plurality of local peaks of a spectral envelope of the spectrum; wherein said peak general outline extracting means extracts, from the spectrogram of a speech signal as an object, a curved surface along a plurality of local peaks of an envelope of the spectrogram in time/frequency domain, and obtains said general outline of peaks at each time from the extracted curved surface; andspectrum modifying means for modifying the spectrum of said speech signal based on the general outline of peaks extracted by the peak general outline extracting means; andspeech synthesizing means for generating a speech based on the spectrum modified by said spectrum modifying means,
wherein said spectrum modifying means includesambient sound spectrum extracting means for extracting a spectrum from an ambient sound collected in an environment to which the speech is to be transmitted or in a similar environment, andmeans for modifying a spectrum of said speech signal based on said general outline of peaks extracted by said peak general outline extracting means and the ambient sound spectrum extracted by said ambient sound spectrum extracting means. - The speech intelligibility improving apparatus according to claim 1, wherein said peak general outline extracting means extracts said general outline of peaks based on perceptual or psycho-acoustic scale of frequency.
- The speech intelligibility improving apparatus according to claim 1, wherein said spectrum modifying means includes spectrum peak emphasizing means for emphasizing a peak of said speech signal, based on said general outline of peaks extracted by said peak general outline extracting means.
- A computer program causing, when executed by a computer, the computer to function as all means described in claims 1 to 3.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2014038786A JP6386237B2 (en) | 2014-02-28 | 2014-02-28 | Voice clarifying device and computer program therefor |
PCT/JP2015/053824 WO2015129465A1 (en) | 2014-02-28 | 2015-02-12 | Voice clarification device and computer program therefor |
Publications (3)
Publication Number | Publication Date |
---|---|
EP3113183A1 EP3113183A1 (en) | 2017-01-04 |
EP3113183A4 EP3113183A4 (en) | 2017-07-26 |
EP3113183B1 true EP3113183B1 (en) | 2019-07-03 |
Family
ID=54008788
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP15755932.9A Active EP3113183B1 (en) | 2014-02-28 | 2015-02-12 | Speech intelligibility improving apparatus and computer program therefor |
Country Status (4)
Country | Link |
---|---|
US (1) | US9842607B2 (en) |
EP (1) | EP3113183B1 (en) |
JP (1) | JP6386237B2 (en) |
WO (1) | WO2015129465A1 (en) |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TWI622978B (en) * | 2017-02-08 | 2018-05-01 | 宏碁股份有限公司 | Voice signal processing apparatus and voice signal processing method |
US10939862B2 (en) | 2017-07-05 | 2021-03-09 | Yusuf Ozgur Cakmak | System for monitoring auditory startle response |
US11883155B2 (en) | 2017-07-05 | 2024-01-30 | Yusuf Ozgur Cakmak | System for monitoring auditory startle response |
US11141089B2 (en) | 2017-07-05 | 2021-10-12 | Yusuf Ozgur Cakmak | System for monitoring auditory startle response |
JP6849978B2 (en) * | 2017-08-04 | 2021-03-31 | 日本電信電話株式会社 | Speech intelligibility calculation method, speech intelligibility calculator and speech intelligibility calculation program |
EP3573059B1 (en) * | 2018-05-25 | 2021-03-31 | Dolby Laboratories Licensing Corporation | Dialogue enhancement based on synthesized speech |
US11172294B2 (en) * | 2019-12-27 | 2021-11-09 | Bose Corporation | Audio device with speech-based audio signal processing |
EP4134954B1 (en) * | 2021-08-09 | 2023-08-02 | OPTImic GmbH | Method and device for improving an audio signal |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
ATE9415T1 (en) * | 1980-12-09 | 1984-09-15 | The Secretary Of State For Industry In Her Britannic Majesty's Government Of The United Kingdom Of Great Britain And | VOICE RECOGNITION SYSTEM. |
JPS61286900A (en) * | 1985-06-14 | 1986-12-17 | ソニー株式会社 | Signal processor |
US4827516A (en) * | 1985-10-16 | 1989-05-02 | Toppan Printing Co., Ltd. | Method of analyzing input speech and speech analysis apparatus therefor |
FR2715755B1 (en) * | 1994-01-28 | 1996-04-12 | France Telecom | Speech recognition method and device. |
JP3240908B2 (en) * | 1996-03-05 | 2001-12-25 | 日本電信電話株式会社 | Voice conversion method |
US6993480B1 (en) * | 1998-11-03 | 2006-01-31 | Srs Labs, Inc. | Voice intelligibility enhancement system |
US6904405B2 (en) * | 1999-07-17 | 2005-06-07 | Edwin A. Suominen | Message recognition using shared language model |
JP3770204B2 (en) * | 2002-05-22 | 2006-04-26 | 株式会社デンソー | Pulse wave analysis device and biological condition monitoring device |
EP1850328A1 (en) | 2006-04-26 | 2007-10-31 | Honda Research Institute Europe GmbH | Enhancement and extraction of formants of voice signals |
US20080312916A1 (en) | 2007-06-15 | 2008-12-18 | Mr. Alon Konchitsky | Receiver Intelligibility Enhancement System |
US9336785B2 (en) * | 2008-05-12 | 2016-05-10 | Broadcom Corporation | Compression for speech intelligibility enhancement |
JP5148414B2 (en) * | 2008-08-29 | 2013-02-20 | 株式会社東芝 | Signal band expander |
WO2011026247A1 (en) * | 2009-09-04 | 2011-03-10 | Svox Ag | Speech enhancement techniques on the power spectrum |
US9117455B2 (en) * | 2011-07-29 | 2015-08-25 | Dts Llc | Adaptive voice intelligibility processor |
-
2014
- 2014-02-28 JP JP2014038786A patent/JP6386237B2/en active Active
-
2015
- 2015-02-12 WO PCT/JP2015/053824 patent/WO2015129465A1/en active Application Filing
- 2015-02-12 US US15/118,687 patent/US9842607B2/en not_active Expired - Fee Related
- 2015-02-12 EP EP15755932.9A patent/EP3113183B1/en active Active
Non-Patent Citations (1)
Title |
---|
None * |
Also Published As
Publication number | Publication date |
---|---|
US20170047080A1 (en) | 2017-02-16 |
JP6386237B2 (en) | 2018-09-05 |
EP3113183A1 (en) | 2017-01-04 |
JP2015161911A (en) | 2015-09-07 |
EP3113183A4 (en) | 2017-07-26 |
US9842607B2 (en) | 2017-12-12 |
WO2015129465A1 (en) | 2015-09-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3113183B1 (en) | Speech intelligibility improving apparatus and computer program therefor | |
Ma et al. | Objective measures for predicting speech intelligibility in noisy conditions based on new band-importance functions | |
Van Kuyk et al. | An evaluation of intrusive instrumental intelligibility metrics | |
RU2552184C2 (en) | Bandwidth expansion device | |
US9318120B2 (en) | System and method for noise reduction in processing speech signals by targeting speech and disregarding noise | |
US8359195B2 (en) | Method and apparatus for processing audio and speech signals | |
Taal et al. | Speech energy redistribution for intelligibility improvement in noise based on a perceptual distortion measure | |
Kim et al. | Nonlinear enhancement of onset for robust speech recognition. | |
TWI451770B (en) | Method and hearing aid of enhancing sound accuracy heard by a hearing-impaired listener | |
TW201308316A (en) | Adaptive voice intelligibility processor | |
JP2002014689A (en) | Method and device for improving understandability of digitally compressed speech | |
US10176824B2 (en) | Method and system for consonant-vowel ratio modification for improving speech perception | |
JP5115818B2 (en) | Speech signal enhancement device | |
US7672842B2 (en) | Method and system for FFT-based companding for automatic speech recognition | |
Ngo et al. | Increasing speech intelligibility and naturalness in noise based on concepts of modulation spectrum and modulation transfer function | |
Chennupati et al. | Spectral and temporal manipulations of SFF envelopes for enhancement of speech intelligibility in noise | |
Zouhir et al. | A bio-inspired feature extraction for robust speech recognition | |
Nasreen et al. | Speech analysis for automatic speech recognition | |
JPH07146700A (en) | Pitch emphasizing method and device and hearing acuity compensating device | |
EP2063420A1 (en) | Method and assembly to enhance the intelligibility of speech | |
Goli et al. | Speech intelligibility improvement in noisy environments based on energy correlation in frequency bands | |
JP5745453B2 (en) | Voice clarity conversion device, voice clarity conversion method and program thereof | |
JP2005202335A (en) | Method, device, and program for speech processing | |
Fulop et al. | Signal Processing in Speech and Hearing Technology | |
Kang et al. | Optimization of a Real-Time Wavelet-Based Algorithm for Improving Speech Intelligibility |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20160905 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAX | Request for extension of the european patent (deleted) | ||
A4 | Supplementary search report drawn up and despatched |
Effective date: 20170626 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G10L 21/0364 20130101ALI20170620BHEP Ipc: G10L 25/15 20130101ALN20170620BHEP Ipc: G10L 21/0216 20130101ALN20170620BHEP Ipc: G10L 21/0232 20130101AFI20170620BHEP Ipc: H04R 27/00 20060101ALI20170620BHEP |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20180530 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602015033141 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: G10L0021007000 Ipc: G10L0021023200 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G10L 25/15 20130101ALN20190110BHEP Ipc: G10L 21/0364 20130101ALI20190110BHEP Ipc: G10L 21/0232 20130101AFI20190110BHEP Ipc: H04R 27/00 20060101ALI20190110BHEP Ipc: G10L 21/0216 20130101ALN20190110BHEP |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G10L 25/15 20130101ALN20190115BHEP Ipc: G10L 21/0364 20130101ALI20190115BHEP Ipc: H04R 27/00 20060101ALI20190115BHEP Ipc: G10L 21/0232 20130101AFI20190115BHEP Ipc: G10L 21/0216 20130101ALN20190115BHEP |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G10L 21/0216 20130101ALN20190124BHEP Ipc: G10L 21/0364 20130101ALI20190124BHEP Ipc: G10L 21/0232 20130101AFI20190124BHEP Ipc: G10L 25/15 20130101ALN20190124BHEP Ipc: H04R 27/00 20060101ALI20190124BHEP |
|
INTG | Intention to grant announced |
Effective date: 20190208 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP Ref country code: AT Ref legal event code: REF Ref document number: 1151945 Country of ref document: AT Kind code of ref document: T Effective date: 20190715 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602015033141 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20190703 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1151945 Country of ref document: AT Kind code of ref document: T Effective date: 20190703 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191104 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191003 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191003 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191103 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191004 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200224 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602015033141 Country of ref document: DE |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG2D | Information on lapse in contracting state deleted |
Ref country code: IS |
|
26N | No opposition filed |
Effective date: 20200603 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20200229 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200212 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200229 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200229 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200229 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20210226 Year of fee payment: 7 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20210217 Year of fee payment: 7 Ref country code: GB Payment date: 20210224 Year of fee payment: 7 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190703 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R119 Ref document number: 602015033141 Country of ref document: DE |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20220212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220228 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220212 Ref country code: DE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220901 |