JP2003525473A - Closed-loop multimode mixed-domain linear prediction speech coder - Google Patents

Abstract

(57) [Summary] A closed-loop multi-mode mixed-domain linear prediction (MDLP) speech coder has a high-rate time-domain coding mode and a low-rate frequency-domain coding mode. And a closed-loop mode selection mechanism for selecting the coding mode of the coder based on the audio content of the frame input to the coder. The frames of the transition speech (ie, from unvoiced speech to voiced speech or vice versa) are coded in a high rate time domain coding mode, eg, a CELP coding mode. Frames of voiced speech are coded in a low rate frequency domain coding mode, eg, a harmonic coding mode. The phase parameters are not coded by the frequency domain coding mode, but instead are modeled, for example, according to a quasi-phase model. In each audio frame coded in the frequency domain coding mode, the initial phase value is the initial phase value of the immediately preceding audio frame coded in the frequency domain coding mode. When the previous audio frame was encoded in the time domain coding mode, the initial phase value of the current audio frame is calculated from the decoded audio frame information of the audio frame encoded in the previous time domain. . Each speech frame coded in the frequency domain coding mode can be compared to a corresponding input speech frame to obtain a performance measure. When the performance measure is below a predetermined threshold, the input speech frame is coded in a time domain coding mode.

Description

Detailed Description of the Invention

BACKGROUND OF THE INVENTION I. FIELD OF THE INVENTION The present invention relates generally to the field of speech processing, and more particularly to a method and apparatus for coding speech in a closed-loop multimode mixed domain.

II. Background The transmission of voice by digital technology has become popular, especially in long distance digital wireless telephone applications. This has led to interest in determining the minimum amount of information that can be sent on the channel while maintaining the perceptual quality of the reconstructed speech. Achieving the voice quality of conventional analog telephones requires data rates on the order of 64 kilobits per second (kbps) when voice is simply sampled and sent in digital form. However, by using speech analysis followed by proper coding, transmission and recombining at the receiver, the data rate can be significantly reduced.

A device that employs a technique of compressing voice by extracting parameters related to a human voice generation model is called a voice coder. A speech coder divides the input speech signal into blocks of time, or analysis frames. Speech coders typically include an encoder and a decoder. The encoder parses the input speech frame and extracts certain relevant parameters and quantizes the parameters into a binary representation, i.e. a set of bits or binary data packets. The data packet is sent to the receiver and decoder on the communication channel. The decoder processes the data packet, unquantizes it to generate parameters, and re-synthesizes the speech frame using the unquantized parameters.

The function of a speech coder is to compress a digitized speech signal into a low bit rate signal by removing all of the inherent redundancy inherent in speech. Digital compression describes an input speech frame with a set of parameters,
It is implemented by employing quantization and representing this parameter with a set of bits. When the input speech frame has a large number of bits N _i and the data packet produced by the speech coder has a large number of bits N ₀ , the compression factor obtained by the speech coder is C _r = N _i / N ₀ . The challenge is to maintain the high voice quality of the decoded speech while obtaining the target compression factor. The performance of a speech coder depends on (1) how well the speech model, ie the combination of the analysis and synthesis processes described above, is performed, and (2) how the parameter quantization process is at a target bit rate of N ₀ bits per frame. It depends on how well it performs. The speech model thus aims to obtain the essence of the speech signal, ie the target speech quality, using a small set of parameters for each frame.

A speech coder is a time-domain coder, that is, an advanced time-resolution processing that encodes a small segment of speech (typically a subframe of 5 milliseconds (millisecond, ms)) at a time. By adopting this, it can be configured as a time domain coder that attempts to obtain a time domain speech waveform. For each sub-frame, various search algorithms known in the art are used to find a highly accurate representative from the codebook space.
Alternatively, the voice coder may be configured as a frequency domain coder,
A set of parameters is used to capture (analyze) the short-term speech spectrum of the input speech frame and a corresponding synthesis process is employed to try to reproduce the speech waveform from the spectral parameters. The parameter quantizer is described in the literature (A Gersho & RM
. Gray, Vector Quantization and Signal Compression (1992)) to preserve these parameters by presenting them using a stored representation of the code vector according to known quantization techniques.

A well-known time domain speech coder is a CELP (Code Excited Linear).
Predictive) coder, which is a reference by LB Rabiner & RW Schafer (D
igital Processing of Speech Signals 396-453 (1978)), which is hereby incorporated by reference in its entirety. In a CELP coder, the coefficients of a short-term formant filter are found by linear prediction (LP) analysis, and the short-term correlation, that is, redundancy, in a speech signal is removed. A short-term prediction filter is applied to the input speech frame to obtain the LP residual signal (resi
due signal), and the residual signal of this LP is quantized by modeling with a long-term prediction filter parameter and the following stochastic codebook. Therefore C
ELP coding separates the task of coding the time domain speech waveform into separate tasks: the task of coding the short term filter coefficients of LP and the task of coding the remainder of LP. Time domain coding is used at a fixed rate (ie, a rate that uses the same number of bits N ₀ for each frame) or a variable rate (ie, different bit rates for different types of frame content. Rate). A variable rate coder attempts to use only the amount of bits needed to code the codec parameters to a level suitable for achieving the target quality. An exemplary variable rate CELP coder is described in US Pat. No. 5,414,796, which is assigned to the assignee of the present invention and is hereby incorporated by reference in its entirety.

A time domain coder, such as a CELP coder, typically relies on a large number of bits N ₀ per frame to preserve the accuracy of the time domain speech waveform. Such coders typically deliver excellent voice quality when the number of bits N ₀ per frame is relatively high (eg, 8 kilobit seconds or more). However, at low bit rates (4 kbits or less), time domain coders do not maintain high quality and robust performance due to the limited number of bits available. Due to the limited codebook space at low bit rates, it removes the ability to match the waveforms found in traditional time domain coders and implements such coders in higher rate commercial applications. I succeeded in doing it.

[0008] Currently, research interest and active commercial demand are rapidly increasing, with high quality voice coders operating at moderate to low bit rates (ie, in the 2.4 to 4 kilobit-second range and below). Was developed. Applications include wireless telephony, satellite communications, internet telephony, various multimedia and voice streaming applications, voice mail, and other voice storage systems. Regarding the driving force, a large capacity is required, and robust performance in a situation where packets are lost is required. Efforts to standardize various recent voice codings have been devoted to another direct driving force that drives the research and development of low rate voice coding algorithms. A low-rate voice coder produces more channels, or users, for each acceptable applicable bandwidth, and combines the low-rate voice coder with an additional layer of proper channel coding to improve the overall coder It can be adapted to the bit budget specifications to ensure robust performance of the channel in the wrong circumstances.

In order to code at lower bit rates, various methods have been developed for coding the spectrum of the speech, ie in the frequency domain, in which the speech signal has a time-varying spectrum (time). -varying evolution of spectr
parsed as a). For example, the article by RJ McAulay & TF Quatieri (Sin
usoidal Coding, in Speech Coding and Synthesis ch. 4 (WB Kleijin & KK
. Paliwal eds., 1995). Spectral coders use a set of spectral parameters to precisely mimic a time-varying speech waveform,
The purpose is to model, or predict, the short-term speech spectrum of each input frame of speech. The spectral parameters are coded and the output frame of speech is generated using the decoded parameters. The synthesized speech produced does not match the original input speech waveform, but gives similar perceptual quality. Examples of frequency domain coders well known in the art include multi-band excitation coders (mu
ltiband excitation coder (MBE), sinusoidal transf
orm coder, STC) and harmonic coder (HC). Such a frequency domain coder provides a high quality parametric model with a compact set of parameters that can be accurately quantized with a small number of bits available at low bit rates.

Nevertheless, low bit rate coding puts a significant constraint on the limited coding resolution, ie the limited codebook space, limiting the effectiveness of a single coding mechanism, It prevents the coder from representing different types of speech segments under different background conditions of equal precision. For example, conventional low bit rate frequency domain coders do not send phase information for speech frames. Instead, the phase information is reconstructed by using a random, artificially generated initial phase value and a linear interpolation technique. For example, H. Yang et al. (Quadratic Phase Interpolation fo
r Voiced Speech Synthesis in the MBE Model, in 29 Electronic Letters 856
-57 (May 1993)). Since the phase information is artificially generated, the output speech produced by the frequency domain coder does not match the original input speech (even when the sinusoidal amplitude is completely preserved by the quantization-dequantization process). For example, most pulses are not synchronized). Therefore, it has been found that it is difficult for a frequency domain coder to employ a closed loop performance measure, such as signal-to-noise ratio (SNR) or perceptual SNR.

Multi-mode coding techniques have been employed to provide low rate speech coding in connection with the open loop mode decision process. One such multi-mode coding technique is described in Amitava Das, et al. (Multimode and Variable-Rate).
Coding of Speech, in Speech Coding and Synthesis ch. 7 (WB Kleijin &
KK Paliwal eds., 1995)). Conventional multi-mode coders apply different modes, i.e. coding-decoding algorithms, to different types of input speech frames. Each mode, i.e. the coding-decoding process, represents in a most efficient manner a certain type of speech segment, for example voiced speech, unvoiced speech or background noise (nonspeech). Specialized in. An external open loop mode decision mechanism examines the input speech frame to determine which mode to apply to the frame. Typically, open-loop mode decisions are made by extracting a number of parameters from the input frame, evaluating parameters for certain temporal and spectral characteristics, and based on the mode decisions for this evaluation. Thus, the mode decision is made without knowing in advance the extraction state of the output voice, ie how close the output voice will be to the input voice in terms of voice quality or other performance measure.

Based on the above, it is desirable to prepare a low bit rate frequency domain coder that estimates phase information more accurately. It is further advantageous to provide a multi-mode mixed domain coder to code certain speech frames in the time domain and other speech frames in the frequency domain based on the speech content of the frames.
It is even more desirable to provide a mixed domain coder that can code certain speech frames in the time domain and code other speech frames in the frequency domain according to a closed-loop coding mode decision mechanism. Therefore, there is a need for a closed-loop, multi-mode, mixed-domain speech coder that ensures time synchronism between the output speech produced by the coder and the original speech input to the coder.

SUMMARY OF THE INVENTION The present invention is directed to a closed-loop, multi-mode, mixed-domain speech coder that ensures time synchronism between the output speech produced by the coder and the original speech input to the coder. Accordingly, in one aspect of the invention, a multi-mode mixed domain speech processor is coupled to a coder having at least one time domain coding mode and at least one frequency domain coding mode, and a speech processor. And a closed-loop mode selection device configured to select the coding mode of the coder based on the frame content processed by.

In another aspect of the invention, a method of processing a frame includes applying an open-loop coding mode selection process to each successive input frame to create a time domain coding mode based on the audio content of the input frame. Or in the frequency domain coding mode, and if the speech content of the input frame indicates a steady-state voiced speech, coding the input frame in the frequency domain, and When the speech content indicates something other than steady-state voiced speech, the step of coding the input frame in the time domain is compared with the frame coded in the frequency domain and the input frame to determine the performance measure. Conveniently includes the steps of determining and coding the input frame in the time domain when the performance measure is below a predetermined threshold. Is.

In another aspect of the invention, a multi-mode mixed domain audio processor applies an open loop coding mode selection process to an input frame to time-domain code based on the audio content of the input frame. Mode or frequency domain coding mode, and means for coding the input frame in the frequency domain when the speech content of the input frame indicates a steady state voiced speech, and When the speech content of the above indicates something other than the voiced sound in the steady state, the performance measure is compared with the means for coding the input frame in the time domain and the frame coded in the frequency domain and the input frame. It is expedient to include means for determining and a means for coding the input frame in the time domain when the performance measure is below a predetermined threshold.

Detailed Description of the Preferred Embodiment In FIG. 1, a first encoder 10 receives audio samples s (n) in digital form and encodes the samples s (n) for transmission medium 12, ie, communication. Send to the first decoder 14 on channel 12. The decoder 14 decodes the coded audio samples and synthesizes the output audio signal S _SYNTH (n). For transmission in the opposite direction, the second encoder 16 encodes the audio sample s (n) in digital form and sends it on the communication channel 18. The second decoder 20 receives the encoded audio samples, decodes and synthesizes the output audio signal S _SYNTH (
n) is generated.

The audio samples s (n) can be obtained by various methods known in the art, such as pulse code modulation (PMC), compounded μ method, ie A method (companded μ-law, or A method). A-law) is used to represent a quantized audio signal in digital form. As is known in the art, audio samples s (n) are organized into frames of input data, each containing a predetermined number of audio samples s (n) in digital form. In the exemplary embodiment, a sampling rate of 8 kilohertz is employed, and each 20 millisecond frame contains 160 samples. In an embodiment described below, the data transmission rate is from 8 kbits (full rate) to 4 kbits (one half) per frame.
Rate), 2 kbits (1/4 rate), 1 kbits (1/8 rate). Alternatively, other data rates may be used. As used herein, the term "full rate" or "high rate" usually refers to a data rate of 8 kilobits per second or more, a "half rate" or The term "low rate" usually refers to data rates of 4 kilobit seconds or less. It is advantageous to change the data transmission rate, as a lower bit rate can be selectively adopted for frames containing relatively little audio information. Other sampling rates, frame sizes, and data transmission rates may be used, as will be appreciated by those skilled in the art.

Both the first encoder 10 and the second decoder 20 include a first speech coder or speech codec. Similarly, the second encoder 16 and the first decoder 14
Both include a second voice coder. A voice coder is a digital signal processor (digi
tal signal processor, DSP), application-specific integrated circuits
It will be appreciated that it may consist of integrated circuits (ASICs), discrete gate logic, firmware, or conventional programmable software modules and microprocessors. The software modules reside in RAM memory, flash memory, registers, or other form of writable storage medium known in the art. Instead,
A conventional processor, controller, or state machine may replace the microprocessor. An example of an ASIC specially designed for audio coding is US Pat. No. 5,727,123, which is assigned to the assignee of the present invention and is hereby fully incorporated by reference, and February 16, 1994. Filed and assigned to the assignee of the present invention,
And US patent application Ser. No. 08 / 197,417, which is hereby fully incorporated by reference
No. (Title of invention: VOCODER ASIC).

According to one embodiment, as shown in FIG. 2, multi-mode mixed-domain linear prediction that can be used in a speech coder.
, MDLP) encoder 100 includes mode determination module 102 and pitch estimation module 10.
4, linear prediction (LP) analysis module 106, LP analysis filter 1
08, LP quantization module 110, and MDLP residual encoder 112. The input speech frame s (n) has a mode determination module 102 and a pitch estimation module 104.
, LP analysis module 106 and LP analysis filter 108. The mode decision module 102 generates a mode index I _M and a mode M based on the periodicity of each input speech frame s (n) and other extraction parameters such as energy, spectral tilt, zero crossing rate, and so on. Various methods of classifying speech frames according to their periodicity are described in US patent application Ser. No. 08 / 815,354 (Title of Invention: METHOD
AND APPARATUS FOR PERFORMING REDUCED RATE VARIABLE RATE VOCODING), which was filed on March 11, 1997 and assigned to the assignee of the present invention,
It is taken here in its entirety as a reference. Such a method is based on the Telecommunication Industry Association Industr
y Interim Standards) TIA / EIA IS-127 and TIA / EIA IS-733.

[0020]

[Equation 1]

[0021]

[Equation 2]

The operation and configuration of various modules of the encoder 100 of FIG. 2 and the decoder 200 of FIG. 3, except for the MDLP residual encoder 112, are known in the art and are described in the above-referenced US Pat. No. 5,414,796 and LB. Rabiner. & RW Schafer (Digital Processing of Speech Signals 396-453 (1978)).

According to one embodiment, an MDLP encoder (not shown) is
The steps shown in the flowchart of FIG. 4 are executed. The MDLP encoder may be the MDLP residual encoder 112 of FIG. In step 300, the MDLP encoder determines whether the mode M is full rate (FR), quarter rate (QR), or eighth rate (eighth rate, ER).
To check if. MDL when mode M is FR, QR, or ER
The P encoder proceeds to step 302. In step 302, MDLP encoder (depending on the value of M -FR, QR, or ER,) corresponding rate to apply to the residual index _{I R.} The time domain coding is conveniently a high precision, high rate coding in FR mode and CELP coding, but this time domain coding is the residual frame of the LP or its Instead it applies to audio frames. The frame is then sent (after further signal processing including digital-to-analog conversion and modulation). In one embodiment, the frame is an LP residual frame that represents the prediction error. In an alternative embodiment, the frame is an audio frame representing audio samples.

On the other hand, in step 300, when mode M is not FR, QR, or ER (ie, when mode M is half rate, HR), M
The DLP encoder proceeds to step 304. In step 304, spectral coding, preferably harmonic coding, is applied at a rate of one half to the LP residual, or alternatively to the speech signal. The MDLP encoder then proceeds to step 306. In step 306, the distortion measure D is obtained by decoding the coded speech and comparing it with the original input frame. The MDLP encoder then proceeds to step 308. In step 308, the strain measure D is compared with a predetermined threshold T. When the distortion measure D is greater than the threshold T, the corresponding quantized parameters are modulated and sent for a half rate spectrally coded frame. On the other hand, if the distortion measure D is less than or equal to the threshold T, then the MDLP encoder proceeds to step 310. In step 310, the decoded frame is recoded at full rate in this time domain. Conventional high rate and precision coding algorithms may be used, eg preferably CELP coding. The quantized parameters of the FR mode associated with the frame are then modulated and sent.

As shown in the flow chart of FIG. 5, a closed-loop multimode MDLP speech coder then processes and sends speech samples according to one embodiment.
Follow the set of steps. In step 400, the speech coder receives digital samples of the speech signal in consecutive frames. When a given frame is received,
The voice coder proceeds to step 402. In step 402, the speech coder detects the energy of the frame. Energy is a measure of the frame's speech activity. Speech detection is done by adding the squares of the amplitudes of the speech samples in digital form and comparing the energy produced with a threshold. In one embodiment, the threshold is adopted based on the level of change in background noise. An exemplary variable threshold voice activity detector is described in the aforementioned US Pat. No. 5,414,796. Some unvoiced speech is a very low energy sample and can be mistakenly coded as background noise. To prevent this from happening, spectral tilt of low energy samples is used to distinguish unvoiced speech from background noise, as described in the above-referenced US Pat. No. 5,414,796.

After detecting the energy of the frame, the speech coder proceeds to step 404. In step 404, the speech coder determines whether the detected frame energy is sufficient to classify the frame as to whether it contains speech information.
If the detected frame energy is below a predetermined threshold level, the speech coder proceeds to step 406. At step 406, the speech coder encodes the frame as background noise (ie, non-speech, or silence). In one embodiment,
A frame of background noise is the time domain coded at 1/8 rate, or 1 kilobit second. In step 404, if the energy of the detected frame is above a predetermined threshold level, the frame is classified as speech and the speech coder proceeds to step 408.

At step 408, the speech coder determines whether the frame is periodic. Various known methods of determining periodicity include, for example, the use of zero crossings and the use of a normalized autocorrelation function (NACF). In particular, the use of zero-crossing and NACF to detect periodicity is described in US Application No. 08 / 815,354 (Title of Invention: METHOD AND APPARATUS FOR PERFORMIN
G REDUCED RATE VARIABLE RATE VOCODING), which is March 1997.
Filed on 11th, assigned to the assignee of the present invention and is hereby fully incorporated by reference. In addition, the above-described method used to distinguish voiced speech from unvoiced speech is based on the Telecommunications Industry Association's industry interim standard (Telecommunicat
ion Industry Association Industry Interim Standards) TIA / EIA IS-127 and TIA / EIA IS-733. When it is determined in step 408 that the frame is not periodic, the speech coder proceeds to step 410. In step 410,
The voice coder encodes the frame as unvoiced speech. In one embodiment, unvoiced speech frames are in the time domain encoded at quarter rate, or 2 kilobit seconds. In step 408, when the frame is determined to be periodic, the speech coder proceeds to step 412.

In step 412, the speech coder uses a periodicity detection method known in the art, such as described in US patent application Ser. Determine if it is periodic. If it is determined that the frame is not sufficiently periodic, the speech coder proceeds to step 414. Step 414
Then, the frame is coded in the time domain as transition speech (ie, transition from unvoiced to voiced speech). In one embodiment, transition audio frames are coded in the time domain at full rate, ie 8 kilobit seconds.

If the speech coder determines in step 412 that the frame is sufficiently periodic, it proceeds to step 416. In step 416, the speech coder encodes the frame as voiced speech. In one embodiment, voiced speech frames are spectrally coded, especially at half rate, or 4 kilobit seconds.
Advantageously, the voiced speech frame is spectrally coded with a harmonic coder, as described below with reference to FIG. Instead, other spectral coders, for example, sinusoidal transmission coders or multi-band excitation coders (mults), as are known in the art.
It can be conveniently used as an iband excitation coder). The voice coder then proceeds to step 418. In step 418, the speech coder decodes the coded voiced speech frame. The voice coder then proceeds to step 420. In step 420, the decoded voiced speech frame is compared to the corresponding input speech sample of this frame to obtain a distortion measure of the synthesized speech to obtain a half rate voiced speech spectral code. Determine if the optimization model is operating within acceptable limits. The voice coder then proceeds to step 422.

In step 422, the speech coder includes the decoded voiced speech frames,
It is determined whether the error from the input voice frame corresponding to this frame is smaller than a predetermined threshold value. In one embodiment, this determination is made in the manner described separately with reference to FIG. If the coding distortion is below a predetermined threshold, the speech coder proceeds to step 426. In step 426, the speech coder sends the frame as voiced speech using the parameters of step 416. In step 422,
If the coding distortion is greater than or equal to the predetermined threshold, the speech coder proceeds to step 414 and encodes the frames of the digital format speech samples received in step 400 as transition speech at full rate in the time domain.

It should be noted that steps 400-410 include an open loop coding decision mode. On the other hand, steps 412-426 include a closed loop coding decision mode.

In one embodiment, as shown in FIG. 6, closed-loop multi-mode MDL
P's voice coder is an analog-to-digital converter.
ter, A / D) 500, the A / D 500 is connected to the frame buffer 502, and the frame buffer 502 is connected to the control processor 504. Energy calculator 506, voiced speech detector 508, background noise encoder 510, high rate time domain encoder 51
2, and a low rate spectrum encoder 514 is connected to the control processor 504. The spectrum decoder 516 is connected to the spectrum encoder 514, and the error calculator
518 is connected to spectrum decoder 516 and control processor 504. The threshold comparator 520 is connected to the error calculator 518 and the control processor 504. Buffer 522 is connected to spectrum encoder 514, spectrum decoder 516, and threshold comparator 520.

In the embodiment of FIG. 6, the components of the voice coder are conveniently configured in the voice coder as firmware or other software driven modules, the voice coder itself being in the DSP or ASIC. It is convenient to have. Those skilled in the art will appreciate that the components of the voice coder can be similarly configured in a number of other known ways. Control processor 504 is conveniently a microprocessor, but may be configured with a controller, state machine, or discrete logic.

In the multimode coder of FIG. 6, the audio signal is supplied to the A / D 500. A /
The D500 converts the analog signal into audio samples S (n) in digital form. The audio samples in digital form are provided to frame buffer 502. The control processor 504 obtains audio samples in digital form from the frame buffer 502,
Supply them to the energy calculator 506. Energy calculator 506 calculates the energy E of a voice sample according to the following formula:

[Equation 3]

The frame is 20 milliseconds long and the sampling rate is 8 kilohertz. The calculated energy E is sent to the control processor 504.

The control processor 504 sends the calculated voice energy to speech activi.
ty) threshold. When the calculated energy is less than the threshold of voice activity, the control processor 504 sends the voice samples in digital form to the frame buffer 5.
Send from 02 to background noise encoder 510. Background noise encoder 510 encodes the frame using the least number of bits needed to hold an estimate of the background noise.

When the calculated energy is above the threshold of voice activity, the control processor 50
4 directs audio samples in digital form from frame buffer 502 to voiced speech detector 508. The voiced speech detector 508 determines whether the periodicity of the speech frame allows efficient coding using low bit rate spectral coding. Methods for determining the level of periodicity within a speech frame are well known in the art and include, for example, a normalized autocorrelation function.
utocorrelation function (NACF) and the use of zero crossings. These and other methods are described in the above-referenced US patent application Ser. No. 08 / 815,354.

The voiced speech detector 508 provides a signal to the control processor 504 that indicates whether the speech frame contains speech with sufficient periodicity for the spectral encoder 514 to efficiently code it. When the voiced speech detector 508 determines that the speech frame lacks sufficient periodicity, the control processor 504 directs the speech samples in digital form to a high rate encoder 512, which encoder 512 has a predetermined maximum. Coding audio in the time domain at the data rate. In one embodiment, the predetermined maximum data rate is 8 kilobit seconds and the high rate encoder 512 is a CEL.
It is a P coder.

When the voiced speech detector 508 first determines that the speech signal is sufficiently periodic to be efficiently encoded by the spectral encoder 514, the control processor 50
4 directs audio samples in digital form from frame buffer 502 to spectrum encoder 514. An exemplary spectrum encoder is described in detail below with reference to FIG.

[0040]

[Equation 4]

[0041]

[Equation 5]

If the calculated MSE is within the acceptable range, the threshold comparator 520 provides a signal to the buffer 522 and the spectrally coded data is output from the speech coder. On the other hand, if the MSE is not within acceptable limits, the threshold comparator 520 sends a signal to the control processor 504, which directs samples in digital form from the frame buffer 502 to the high rate time domain encoder 512. . The time domain encoder 512 encodes frames at a predetermined maximum rate and buffers 522
The contents of are discarded.

In the embodiment of FIG. 6, the type of spectral coding employed is harmonic coding, which is described separately with reference to FIG. 7, but in an alternative embodiment, a sinusoidal transform. It may also be a type of spectral coding, such as V. coding or multi-band excitation coding. The use of multiband excitation coding is described in US Pat. No. 5,195,166, and the use of sinusoidal transform coding is described in, for example, US Pat. No. 4,865,068.

For transition frames and voiced frames where the phase distortion threshold is less than or equal to the periodicity parameter, the multi-mode coder of FIG. 6 is at full rate, ie 8 kbps, and is coded by CELP by a high rate time domain coder 512. It is convenient to adopt Alternatively, other known forms of high rate time domain coding may be used for such frames. Therefore, transition frames (and voiced frames that are not sufficiently periodic) are coded with high accuracy,
The waveforms at the input and output are properly matched and the phase information is properly preserved. 1
In one embodiment, the multi-mode coder, independent of the decision of the threshold comparator 520,
After processing a predetermined number of consecutive voiced frames whose threshold exceeds the measure of periodicity, each frame switches from half rate spectral coding to full rate CELP coding.

It should be noted that in connection with control processor 504, energy calculator 506 and voiced speech detector 508 include open-loop coding decisions. In contrast, in connection with control processor 504, spectral encoder 514, spectral decoder 51
6, error calculator 518, threshold comparator 520, and buffer 522 include closed-loop coding decisions.

In one embodiment described with reference to FIG. 7, spectral coding, preferably harmonic coding, is used to code a sufficiently periodic voiced frame at a low bit rate. . Spectral coders generally maintain time-evolution of speech spectral characteristics in a perceptually significant way by modeling and coding each speech frame in the frequency domain. Is defined as an algorithm that tries to. In the essential part of such an algorithm, (1) spectrum analysis or parameter estimation, (2) parameter quantization, (3) output speech waveform and decoded parameter synthesis are performed.
Therefore, it is aimed to retain the important properties of the short-term speech spectrum with a set of spectral parameters and to use the decoded spectral parameters to synthesize the output speech. Normally, the output speech is synthesized as a weighted sum of sinusoids. The sinusoidal amplitude, frequency, and phase are spectral parameters estimated during the analysis.

“Synthetic analysis” is a well-known technique in CELP coding, but this technique has not been used for spectral coding. The main reason synthesis analysis does not apply to spectral coders is that the loss of initial phase information causes the mean square energy of the synthesized speech, even if the speech model is working properly from a perceptual perspective. This is because the MSE) is high. Therefore, if we correctly generate the initial phase, we can directly compare the speech sample with the reconstructed speech,
Another advantage is that it can determine if the speech model correctly encodes a speech frame.

[0048] In coding of the spectrum, the audio frame output may be synthesized as _{follows: S [n] = S v} [n] + S uv [n], n = 1,2 ,. ． ． , N, where N is the number of samples per frame, and S _v and S _uv are voiced sound components and unvoiced sound components, respectively. Sinusoid sum synthesis process (sum-of-sinus
The oid synthesis process) produces a voiced component as shown in the following equation:

[Equation 6]

Amplitude, frequency, and phase parameters are estimated from the short-term spectrum of the input frame by the spectral analysis process. The unvoiced components are either generated together with the voiced part in a single sinusoidal sum synthesis or calculated separately by a dedicated unvoiced synthesis process and added back to S _v .

In the embodiment of FIG. 7, a particular type of spectral coder, called a harmonic coder, is used to spectrally code voiced frames that are sufficiently periodic at low bit rates. The harmonic coder characterizes the frame as a sinusoidal sum,
Analyze small segments of the frame. Each sinusoid in the sinusoid sum is
It has a frequency that is an integral multiple of the frame pitch F ₀ . In an alternative embodiment, a specific type of spectral coder other than a harmonic coder is used and the sinusoidal frequencies for each frame are derived from a set of real numbers from 0 to 2π. In the embodiment of FIG. 7, it is convenient for the amplitude and phase of each sinusoid in the sum to be selected so that the sum best matches the signal in one period, as illustrated by the graph of FIG. To do. Harmonic coders generally employ an external classification, where each input speech frame is displayed as voiced or unvoiced. In voiced frames, the sinusoidal frequencies are limited to harmonics of the estimated pitch (F ₀ ), ie f _k = kF ₀ . For unvoiced speech, short-term spectral peaks are used to determine sinusoids. Amplitude and phase are interpolated to mimic the evolution in the frame as shown in the following equation:

[Equation 7]

The parameters sent for each sinusoid are amplitude and frequency. The phase is not sent, but instead is modeled according to several known techniques including, for example, a quadratic phase model, or a conventional polynomial representation of the phase.

As shown in FIG. 7, the harmonic coder includes a pitch extractor 600, which is connected to windowing logic 602, which is a Discrete Fourier Transform, DFT), and harmonic analysis logic 604
Connected to. Pitch extractor 600, which receives speech samples S (n) as input, is also connected to DFT and harmonic analysis logic 604. The DFT and harmonic analysis logic 604 is connected to the residual encoder 606. Pitch extractor 600, DFT and harmonic analysis logic 604, and residual encoder 606 are each connected to a parameter quantizer 608. The parameter quantizer 608 is connected to the channel encoder 610, and the channel encoder 610 is connected to the transmitter 612. The transmitter 612 receives on a over-the-air interface by a standard radio-frequency (RF) interface, eg, code division multiple access (CDMA). Connected to machine 614. Receiver 614 is connected to channel decoder 616, which is connected to dequantizer 618. The dequantizer 618 is connected to the sinusoidal sum speech synthesizer 620. Further connected to the sinusoidal sum speech synthesizer 620 is a phase estimator 622, which receives as input the previous frame information. Sinusoid sum voice synthesizer 6
Twenty is configured to produce a synthesized speech output S _SYNTH (n).

Pitch extractor 600, windowing logic 602, DTF and harmonic analysis logic 604
, Residual encoder 606, parameter quantizer 608, channel encoder 610, channel decoder 616, dequantizer 618, sinusoidal sum speech synthesizer 620, and phase estimator 622 include, for example, firmware or software modules,
It can be configured in a variety of different ways familiar to those skilled in the art. Transmitter
612 and receiver 614 may be implemented with corresponding standard RF components known to those skilled in the art.

In the harmonic coder of FIG. 7, the input sample S (n) is received by the pitch extractor 600, which extracts the pitch frequency information F ₀ . The sample is then multiplied by the windowing logic 602 with the appropriate windowing function, allowing analysis of a small segment of the speech frame. Pitch extractor 60
Using the pitch information provided by 0, the DFT and harmonic analysis logic 604 computes the sample DFT to produce a composite spectral point from which the graph of FIG. 8 illustrates. As shown in FIG. 8, the amplitude A _I of the harmonic is extracted, and in FIG. 8, L indicates the total number of harmonics. The DFT is supplied to the residual encoder 606, and the residual encoder 606 is voicing information.
Extract V _c .

The V _c parameter indicates a point on the frequency axis, as shown in FIG. 8, and as V _c becomes higher, the spectrum shows the characteristics of an unvoiced voice signal and is no longer a harmonic. Should be noted. In contrast, below the point V _c , the spectrum is harmonic and characteristic of voiced speech.

The components of A _I , F ₀ , and V _c are supplied to the parameter quantizer 608, which quantizes the information. The quantized information is provided in the form of packets to the channel encoder 610, which quantizes the packets at a low bit rate, eg half rate, ie 4 kilobit seconds. The packet is provided to transmitter 612, which modulates the packet and
Send the generated signal to receiver 614 over the air. Receiver 614 receives the signal, demodulates it, and sends the encoded packets to channel decoder 616. The channel decoder 616 decodes the packet and supplies the decoded packet to the dequantizer 618. Dequantizer 618 dequantizes information. Information is provided to the sinusoidal sum speech synthesizer 620.

The sinusoidal sum speech synthesizer 620 is configured to synthesize a plurality of sinusoidal modeling models a short-term speech spectrum according to the above equation for S [n]. The frequency of the sinusoid f _k is a multiple or harmonic of the fundamental frequency F ₀ and is a frequency with pitch periodicity for quasi-periodic (ie, transitional) voiced speech segments.

In addition, the sinusoidal sum speech synthesizer 620 receives phase information from the phase estimator 622. Phase estimator 622 receives the information of the previous frame, that is, the parameters of A _I , F ₀ , and V _c for the previous frame. The phase estimator 622 also receives the reconstructed N samples of the previous frame, where N is the frame length (ie, N
Is the number of samples per frame). The phase estimator 622 determines the initial phase of the frame based on the information of the previous frame. The determination of the initial phase is provided to the sinusoidal sum speech synthesizer 620. Based on the information about the current frame and the initial phase calculation performed by the phase estimator 622 based on the past frame information, the sinusoidal sum speech synthesizer 620 produces speech frames as described above.

As already mentioned, the harmonic coder uses the information of the previous frame to synthesize, ie reconstruct, the speech frame by predicting a linear change in phase from frame to frame. . The synthesis model described above is generally called a quasi-phase model, and in such a synthesis model, the coefficient B ₃ (k) indicates that the initial phase of the current voiced frame is synthesized. There is. When determining the phase, conventional harmonic coders either set the initial phase to zero or generate the initial phase value randomly or using a pseudo-random generation method. To estimate the phase more accurately, the phase estimator 622 depends on whether the immediately preceding frame is a voiced speech frame (ie, a sufficiently periodic frame) or a transition speech frame. , Use one of two possible methods for determining the initial phase. If the previous frame was a voiced speech frame, the estimated final phase value of this frame is used as the initial phase value of the current frame. On the other hand, when the previous frame is classified as a transition frame, the initial phase value of the current frame is obtained from the spectrum of the previous frame, which is D at the decoder output of the previous frame.
Obtained by performing FT. Therefore, the phase estimator 622 can use the exact phase information already available (since the previous frame, which is a transition frame, was processed at full rate).

In one embodiment, the closed-loop multimode MDLP speech coder follows the speech processing steps shown in the flowchart of FIG. The speech coder codes the LP residual of each input speech frame by selecting the most appropriate coding mode. Certain modes encode the LP residual, ie the residual speech, in the time domain, while the other modes represent the LP residual, the residual speech, in the frequency domain. The mode set includes a full rate time domain for transition frames (T mode); a half rate frequency domain for voiced frames (
V mode); quarter rate time domain for unvoiced frames (U mode);
And there is a 1/8 rate time domain (N mode) for noise frames.

Those skilled in the art will appreciate that by following the steps shown in FIG. 9, the speech signal or the corresponding LP residue is coded. Noise, unvoiced sound,
The waveform characteristics of transitions and voiced speech can be referenced as a function of time in the graph of FIG. 10a. The residual waveform characteristics of LP of noise, unvoiced sound, transitions, and voiced sound can be referenced as a function of time in the graph of FIG. 10b.

In step 700, an open-loop mode decision is made for any one of the four modes (T, V, U, or N) and applied to the residual S (n) of the input speech. T
If the mode is applied, then in step 702 the residual speech is processed in T-mode, or full rate, in the time domain. If U-mode is applied, then in step 704 the U-mode, ie, the quarter-rate residual speech, is processed in the time domain. If N mode is applied, then in step 706 the residual speech is processed in the time domain at N mode, ie at a rate of 1/8. If V-mode is applied, then in step 708 the residual speech is processed in V-mode in the frequency domain, ie, at half rate.

In step 710, the speech coded in step 708 is decoded and compared with the residual S (n) of the input speech and a performance measure D is calculated. In step 712, the performance measure D is compared with a predetermined threshold T. When the performance measure D is greater than or equal to the threshold T,
At step 714, the residual spectrally coded speech at step 708 is allowed to be transmitted. On the other hand, if the performance measure D is less than the threshold T, then in step 716 the residual S (n) of the input speech is processed in T mode. In another embodiment, no performance measure is calculated and no threshold is specified. Instead, the next frame is processed in T mode after a predetermined number of speech residual frames have been processed in V mode.

In the decision step shown in FIG. 9, the high bitrate T-mode can be used only when needed to take advantage of the periodicity of voiced speech segments in the lower bitrate V-mode. On the other hand, when V-mode is not performed properly, it is convenient to switch to full rate to prevent quality degradation. Therefore, a very high voice quality approaching the full rate voice quality can be produced at an average rate considerably lower than the full rate. Furthermore, the target voice quality can be controlled by the selected performance measure and the selected threshold.

“Updating” to T-mode can improve the behavior of applying V-mode later by keeping the model phase tracking close to that of the input speech. If the V-mode performance is inadequate, the closed-loop performance test of steps 710 and 712 switches to T-mode and "refreshes" the initial phase values to restore the model phase tracking to the original input voice phase tracking. By approaching again, the performance of the next V-mode process can be improved. For example, as shown in the graphs of FIGS. 11a-c, the fifth frame from the start does not work properly in V-mode, as evidenced by the PSNR distortion measure used. As a result, in the absence of closed-loop decision and update, the modeled phase tracking deviates significantly from the original input speech phase tracking, and significantly degrades the PSNR, as shown in Figure 11c. further,
The performance of the next frame processed in V mode is degraded. However, under the closed loop decision, the fifth frame is switched to T mode processing, as shown in Figure 11a. The performance of the fifth frame is, as shown in FIG. 11b,
Updates provide a significant improvement, as evidenced by the improvement in PSNR. In addition, the performance of the next frame processed under V mode is also improved.

The decision step shown in FIG. 9 improves the V-mode representation quality by providing a very accurate initial phase estimate, and the generated V-mode synthesized speech residual signal is It is guaranteed to be exactly temporally aligned with the residual S (n) of the input speech of. The initial phase in the residual segment of the speech processed in the first V-mode is
It is obtained from the immediately preceding decoded frame in the following manner. At each harmonic, the initial phase is set equal to the estimated final phase of the previous frame when the previous frame was processed in V mode. For each harmonic, the initial phase is set equal to the phase of the actual harmonic of the previous frame when the previous frame was processed in T mode.
The actual harmonic phase of the previous frame is determined by taking the past decoded residual DFT using all previous frames. Instead, the actual harmonic phase of the previous frame is determined by taking the DFT of past decoded frames in a pitch-synchronized manner by processing the various pitch periods of the previous frame.

In this specification, a novel closed-loop multi-mode mixed domain linear prediction (mixed-
The domain coder (MDLP) voice coder is described. Those skilled in the art will appreciate that various exemplary logic blocks and algorithm steps described in connection with the embodiments disclosed herein may be implemented by digital signal processors.
, DSP), application specific integrated circuit,
ASIC), discrete gate or transistor logic, such as registers and FIFOs
It will be appreciated that it may be configured or executed with discrete hardware components such as, a processor executing a set of firmware instructions, or conventional programmable software modules and processors. The processor is conveniently a microprocessor, but may alternatively be a conventional processor, controller, microprocessor or state machine. The software modules may reside in RAM memory, flash memory, registers, or any other form of writable storage medium known in the art. Those skilled in the art will further appreciate that data, instructions, commands, information, signals, bits, signs, and chips referred to throughout the above description may include voltage, current, electromagnetic waves, magnetic fields or particles, ranges of light or optical fields. or particles), or a combination thereof, will be conveniently represented.

The preferred embodiments of the present invention have been shown and described herein. However,
One of ordinary skill in the art will appreciate that numerous modifications can be made to the embodiments described herein without departing from the spirit or scope of the invention. Accordingly, the invention is not limited except in accordance with the appended claims.

[Brief description of drawings]

[Figure 1]   Block diagram of communication channels terminated at each end by a voice coder.

[Fig. 2] Mixed-domain linear prediction (MDLP)
) A block diagram of an encoder that can be used in the voice coder.

FIG. 3 is a block diagram of a decoder that can be used in a multimode MDLP voice coder.

4 is a flow chart showing MDLP encoding steps performed by an MDLP encoder that may be used in the encoder of FIG.

[Figure 5]   The flowchart which shows a voice coding decision process.

[Figure 6]   FIG. 3 is a block diagram of a closed-loop multimode MDLP voice coder.

7 is a block diagram of a spectrum coder that may be used in the coder of FIG. 6 or the encoder of FIG.

[Figure 8]   Amplitude vs. frequency graph showing the sinusoidal amplitude of a harmonic coder.

FIG. 9 is a flowchart showing a mode decision process in a multimode MDLP voice coder.

10 is a graph of the amplitude of a speech signal versus time (FIG. 10a) and linear prediction (linear predict).
ion, LP) residual amplitude versus time (FIG. 10b).

FIG. 11 is a graph of rate / mode versus frame index under closed-loop coding decision (
Fig. 11a), the perceptual signa of the perception under closed-loop decision
l-to-noise ratio (PSNR) vs. frame index graph (Fig. 11b), both rate / mode and PSNR vs. frame index graph (Fig. 11c) in the absence of closed-loop coding decisions.

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) G10L 19/12 G10L 9/14 C H03M 7/36 D (81) Designated country EP (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT, SE), OA (BF, BJ, CF, CG, CI, CM, GA, GN, GW, ML, MR, NE, SN, TD, TG), AP (GH, GM, KE, LS, MW, SD, SL, SZ, TZ, UG, ZW), EA (AM, AZ, BY, KG) , KZ, MD, RU, TJ, TM), AE, AL, AM, AT, AU, AZ, BA, BB, BG, BR, BY, CA, CH, CN, CR, CU, C , DE, DK, DM, EE, ES, FI, GB, GD, GE, GH, GM, HR, HU, ID, IL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, LK, LR, LS, LT, LU, LV, MA, MD, MG, MK, MN, MW, MX, NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL , TJ, TM, TR, TT, TZ, UA, UG, UZ, VN, YU, ZA, ZW [Continued Summary] Calculated from the decoded speech frame information of the region-coded speech frame. Each speech frame coded in the frequency domain coding mode can be compared to the corresponding input speech frame to obtain a performance measure. When the performance measure is below a predetermined threshold, the input speech frame is coded in the time domain coding mode.

Claims (26)

[Claims] 1. A coder having at least one time domain coding mode and at least one frequency domain coding mode and a coding mode of the coder based on the frame content connected to the coder and processed by a speech processor. A multi-mode mixed domain audio processor including a closed loop mode selection device configured to select. 2. The audio processor of claim 1, wherein the coder encodes audio frames. 3. The speech processor of claim 1, wherein the coder encodes a linear prediction residual of the speech frame. 4. The at least one time domain coding mode comprises a coding mode for coding a frame at a first coding rate and the at least one frequency domain coding mode at a second coding rate. The audio processor of claim 1, including a coding mode for coding a frame, wherein the second coding rate is lower than the first coding rate. 5. The audio processor of claim 1, wherein the at least one frequency domain coding mode comprises a harmonic coding mode. 6. A comparison circuit coupled to the coder, wherein the uncoded frame is compared with the frame coded in at least one frequency domain coding mode and a performance measure is based on the comparison. Further including a comparing circuit for generating,
The speech of claim 1, wherein the coder applies at least one time domain coding mode only when the performance measure is below a predetermined threshold, otherwise the coder applies at least one frequency domain coding mode. Processor. 7. A coder applies at least one time domain coding mode to each frame immediately after a predetermined number of consecutively processed frames coded in at least one frequency domain coding mode. The audio processor of claim 1. 8. At least one frequency domain coding mode represents a short-term spectrum of each frame with a plurality of sinusoids having a set of parameters including frequency, phase, and amplitude, where the phase is a polynomial representation and an initial phase. Modeled by a value, the initial phase value is (1) the estimated final phase value of the previous frame when the previous frame was coded in at least one frequency domain coding mode, or (2) The speech processor according to claim 1, wherein when the previous frame is coded in at least one time domain coding mode, it is a phase value obtained from the short-term spectrum of the previous frame. 9. The speech processor according to claim 8, wherein the frequency of the sinusoid in each frame is an integral multiple of the pitch frequency of the frame. 10. The frequency of the sinusoid in each frame is 0 to 2π.
9. The speech processor of claim 8 derived from a set of real numbers of 11. A method of processing frames, the method comprising: applying an open loop coding mode selection process to each successive input frame, based on the audio content of the input frame, in a time domain coding mode, or If one of the frequency domain coding modes is selected, and if the speech content of the input frame indicates a steady state voiced speech, the steps of coding the input frame in the frequency domain and the speech content of the input frame are When presenting anything other than steady-state voiced speech, the steps of coding the input frame in the time domain and comparing the coded frame in the frequency domain with the input frame to obtain a performance measure , If the performance measure is lower than a predetermined threshold, then coding the input frame in the time domain. Method. 12. The method of claim 11, wherein the frame is a linear prediction residual frame. 13. The method of claim 11, wherein the frame is a voice frame. 14. The step of encoding in the time domain comprises encoding the frame at a first coding rate and the step of encoding in the frequency domain encodes the frame at a second coding rate. 13. The method of claim 11, wherein the second coding rate is lower than the first coding rate. 15. The method of claim 11, wherein the step of encoding in the frequency domain comprises encoding with harmonics. 16. The frequency domain encoding step represents a short-term spectrum of each frame with a plurality of sinusoids having a set of parameters including frequency, phase, and amplitude, where the phase is a polynomial representation and an initial phase value. And the initial phase value is (1) the estimated final phase value of the previous frame when the previous frame was coded in the frequency domain, or (2) the previous frame is in the time domain. 12. The method according to claim 11, which is a phase value obtained from the short-term spectrum of the previous frame when coded with. 17. The method of claim 16, wherein the sinusoidal frequency of each frame is an integer multiple of the pitch frequency of the frame. 18. The method of claim 16, wherein the sinusoidal frequency of each frame is obtained from a set of real numbers from 0 to 2π. 19. A multi-mode mixed domain speech processor, wherein an open loop coding mode selection process is applied to an input frame to provide a time domain coding mode based on the speech content of the input frame, or The means for selecting one of the frequency domain coding modes and the means for coding the input frame in the frequency domain when the speech content of the input frame indicates a steady state voiced speech When presenting anything other than steady-state voiced speech, there are means for coding the input frame in the time domain and means for comparing the coded frame in the frequency domain with the input frame to obtain a performance measure. , When the performance measure is below a predetermined threshold, including means for coding the input frame in the time domain Processor. 20. The speech processor of claim 19, wherein the frame is a linear prediction residual frame. 21. The audio processor of claim 19, wherein the input frame is an audio frame. 22. The means for encoding in the time domain comprises means for encoding a frame at a first coding rate and the means for encoding in the frequency domain encodes a frame at a second coding rate. 20. The audio processor of claim 19, including means for performing, the second coding rate being lower than the first coding rate. 23. The speech processor of claim 19, wherein the frequency domain encoding means comprises a harmonic coder. 24. The means for encoding in the frequency domain comprises means for representing a short-term spectrum of each frame with a plurality of sinusoids having a set of parameters including frequency, phase and amplitude, the phase being a polynomial representation and Modeled with an initial phase value, which is (1) the estimated final phase value of the immediately preceding frame when the immediately preceding frame was coded in the frequency domain, or (2) 20. A speech processor according to claim 19, which is a phase value obtained from the short-term spectrum of the immediately preceding frame when the immediately preceding frame was coded in the time domain. 25. The audio processor of claim 24, wherein the sinusoidal frequency of each frame is an integer multiple of the pitch frequency of the frame. 26. The audio processor of claim 24, wherein the sinusoidal frequency of each frame is derived from a set of real numbers from 0 to 2π.