EP2710589A1 - Unité de codage redondant pour codec audio - Google Patents

Unité de codage redondant pour codec audio

Info

Publication number
EP2710589A1
EP2710589A1 EP11723805.5A EP11723805A EP2710589A1 EP 2710589 A1 EP2710589 A1 EP 2710589A1 EP 11723805 A EP11723805 A EP 11723805A EP 2710589 A1 EP2710589 A1 EP 2710589A1
Authority
EP
European Patent Office
Prior art keywords
signal
frequency band
source model
residual signal
quantization indices
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP11723805.5A
Other languages
German (de)
English (en)
Inventor
Turaj ZAKIZADEH SHABESTARY
Jan Skoglund
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Google LLC
Original Assignee
Google LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Google LLC filed Critical Google LLC
Publication of EP2710589A1 publication Critical patent/EP2710589A1/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
    • G10L19/0204—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders using subband decomposition
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/005—Correction of errors induced by the transmission channel, if related to the coding algorithm
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/04—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using predictive techniques
    • G10L19/08—Determination or coding of the excitation function; Determination or coding of the long-term prediction parameters
    • G10L19/087—Determination or coding of the excitation function; Determination or coding of the long-term prediction parameters using mixed excitation models, e.g. MELP, MBE, split band LPC or HVXC

Definitions

  • the technical field relates to packet loss concealment in communication systems (such as Voice over IP, also referred to as VoIP), having an audio codec (coder/decoder).
  • Voice over IP also referred to as VoIP
  • audio codec coder/decoder
  • coder/decoder One such codec may be iSAC.
  • Real-time communication refers to communication where the delay between one user speaking and another user hearing the speech is so short that it is imperceptible or nearly imperceptible.
  • packet-switched networks such as the Internet
  • VoIP is one audio communication approach enabling real-time communication over packet-switched networks.
  • an audio signal is broken up into short time segments by an audio coder, and the time segments are transmitted individually as audio frames in packets.
  • the packets are received by the receiver, the audio frames are extracted, and the short time segments are reassembled by an audio decoder into the original audio signal, enabling the receiver to hear the transmitted audio signal.
  • BEC Backward Error Correction
  • FEC Forward Error Correction
  • a packet may contain audio data (audio frame) corresponding to a time period t2 and the immediately preceding time period tl .
  • the second packet may contain audio data corresponding to the time period t3 and the immediately preceding time period t2.
  • the third packet may contain audio data corresponding to time period t4 and the immediately preceding time period t3. If the second packet is lost, it is possible to recreate the full audio segment of tl, t2, and t3, from only the first packet and the third packet, because the third packet contains audio data corresponding to time period t2.
  • RFC 2198 Internet Engineering Task Force Request for Comments 2198
  • packet n contains the payload of frame n-1 (redundant payload), and the payload of frame n (main payload). Therefore, in this example, it is necessary to have twice the transmission rate to maintain the same data throughput as compared to when no redundant data is transmitted. In other word, the effective bandwidth available for communication of data is halved, because redundant data is being transmitted along with primary data.
  • RFC 2198 describes another approach where the redundant payload is encoded with a different encoder than the primary payload.
  • that approach requires two coders to be executed by the transmitter, and two decoders to be operated by the receiver.
  • a coder and decoder normally run continuously, with associated memory buffering of incoming/outgoing data, thus the approach is expensive in terms of processing and memory load, and is impractical for situations where the processing power and memory are expensive, or altogether unavailable, such as on mobile communication devices.
  • the present invention recognizes the problem posed by lost packets in real-time audio communication over packet switched networks, and provides a solution that avoids the disadvantages of the above examples.
  • a method of generating multiple payloads encoded at different coding rates for inclusion in a data packet transmitted across a packet data network, the data packet containing an encoded audio source signal may include separating the audio source signal into a first frequency band signal and a second frequency band signal, the first frequency band being lower than the second frequency band.
  • the method may also include extracting from the first frequency band signal a source model representing linear dependencies of the first frequency band signal, generating a residual signal by filtering the first frequency band signal with a filter having filter coefficients derived from the source model to remove short-term and long-term linear dependencies from the first frequency band signal, transforming the residual signal into a transform domain, scaling the transformed residual signal with a first scale factor, quantizing the scaled transformed residual signal, quantizing the source model to create quantization indices of the source model, entropy encoding the quantization indices of the scaled transformed residual signal and the quantization indices of the source model to generate a redundant bitstream and constructing the payloads from the redundant bitstream.
  • the method may also include extracting from the second frequency band signal a second source model representing linear dependencies of the second frequency band signal, generating a second residual signal by filtering the second frequency band signal with filter coefficients derived from the second source model to remove linear dependencies from the second frequency band signal, transforming the second residual signal into the transform domain, scaling the transformed second residual signal with a second scale factor, quantizing the scaled transformed second residual signal to create quantization indices of the transformed residual signal, quantizing the second source model to create quantization indices of the source model, entropy encoding the quantization indices of the scaled transformed second residual signal and the quantization indices of the second source model to generate a second redundant bitstream, and constructing the payloads from the second redundant bitstream.
  • the method may also include storing in a memory the transformed residual signal and the quantization indices of the source model prior to the scaling of the transformed residual signal, and extracting from the memory the stored transformed residual signal and the quantization indices of the source model when the redundant payload is to be encoded at a lower coding rate than a coding rate of a primary payload.
  • the step of separating the audio source signal into the first frequency band signal and the second frequency band signal may include dividing the first frequency band into a first sub-band and a second sub-band, the first sub-band being lower than the second sub-band.
  • the method may further include generating a data frame including an encoded segment of the audio source signal corresponding to a first time period, and an encoded segment of the audio source signal corresponding to a second time period different than the first time period, the second time period preceding the first time period.
  • the segment of the audio source signal corresponding to the first time period is encoded at a higher coding rate than the coding rate of the segment of audio source data corresponding to the second time period.
  • the first scale factor and the second scale factor are set independently of each other, based on the coding rate of the segment of audio source data corresponding to the second time period.
  • first scale factor and the second scale factor are each independently set to a value between 0.4 and 0.5, inclusive, to reach a suitable overall-quality vs concealment-quality trade off, at packet loss rates in the rage of 10% to 15%, inclusive.
  • the method may also include downsampling the second frequency band signal prior to the extracting of the second source model.
  • an encoding apparatus for encoding a source audio signal at different coding rates to generate multiple payloads included in data packets transmitted across a packet data network, includes a filter-bank configured to separate the audio source signal into a first frequency band signal and a second frequency band signal, the first frequency band being lower than the second frequency band.
  • the apparatus may also include a source model analysis unit configured to generate a source model representing linear dependencies of the first frequency band signal, an analysis filter having its filter coefficients derived from the source model and configured to filter the first frequency band signal to generate a residual signal, a domain transformer transforming the residual signal into a transform domain, a multiplier multiplying the transformed residual signal with a first scale factor, a quantizer quantizing the scaled transformed residual signal, and quantizing the source model to create associated quantization indices of the source model and the scaled transformed residual signal, and an entropy coder encoding the quantization indices of the scaled transformed residual signal and the quantization indices of the source model to generate a redundant bitstream.
  • a source model analysis unit configured to generate a source model representing linear dependencies of the first frequency band signal
  • an analysis filter having its filter coefficients derived from the source model and configured to filter the first frequency band signal to generate a residual signal
  • a domain transformer transforming the residual signal into a transform domain
  • a multiplier multiplying the
  • the apparatus may also include a second source model analysis unit configured to extract from the second frequency band signal a second source model representing linear dependencies of the second frequency band signal, a second analysis filter having filter coefficients derived from the second source model and configured to filter the second frequency band signal to generate a second residual signal, a second domain transformer transforming the second residual signal into the transform domain, a second multiplier multiplying the transformed second residual signal with a second scale factor, a second quantizer quantizing the scaled transformed second residual signal, and quantizing the second source model to create quantization indices of the second source model and quantization indices of the scaled transformed second residual signal, and a second entropy coder encoding the quantization indices of the scaled transformed second residual signal and the quantization indices of the second source model to generate a second redundant bitstream.
  • a second source model analysis unit configured to extract from the second frequency band signal a second source model representing linear dependencies of the second frequency band signal
  • a second analysis filter having filter coefficients derived from the second source model and
  • the apparatus may include a storage unit storing the transformed residual signal and the quantization indices of the source model prior to the multiplication by the multiplier, wherein the stored transformed residual signal and the quantization indices of the source model are extracted from the storage unit when the redundant payload is to be encoded at a lower coding rate than a coding rate of a primary payload.
  • the filter bank is further configured to divide the first frequency band into a first sub-band and a second sub-band, the first sub-band being lower than the second sub-band.
  • the apparatus may include a concatenation unit configured to generate a data frame including an encoded segment of the audio source signal corresponding to a first time period and an encoded segment of the audio source signal corresponding to a second time period different than the first time period, the second time period preceding the first time period.
  • the segment of the audio source signal corresponding to the first time period is encoded at a higher coding rate than the coding rate of the segment of audio source data corresponding to the second time period.
  • the first scale factor and the second scale factor are set independently of each other, based on the coding rate of the segment of audio source data corresponding to the second time period.
  • the first scale factor and the second scale factor are each independently set to a value between 0.4 and 0.5, inclusive, to reach a suitable overall-quality vs concealment-quality trade off, at packet loss rates in the rage of 10% to 15%, inclusive.
  • the apparatus may also include a downsampler configured to downsample the second frequency band signal prior to processing by the second source model analysis unit and the second analysis filter.
  • a computer readable tangible recording medium is encoded with instructions, wherein the instructions when executed by a processor cause the processor to perform a method of generating multiple payloads encoded at different coding rates for inclusion in a data packet transmitted across a packet data network, the method including separating the audio source signal into a first frequency band signal and a second frequency band signal, the first frequency band being lower than the second frequency band.
  • the method also includes extracting from the first frequency band signal a source model representing linear dependencies of the first frequency band signal, generating a residual signal by filtering the first frequency band signal with a filter having filter coefficients derived from the source model to remove short-term and long-term linear dependencies from the first frequency band signal, transforming the residual signal into a transform domain, scaling the transformed residual signal with a first scale factor, quantizing the scaled transformed residual signal, quantizing the source model to create quantization indices of the source model, entropy encoding the quantization indices of the scaled transformed residual signal and the quantization indices of the source model to generate a redundant bitstream, and constructing the payloads from the redundant bitstream.
  • method performed by the processor may further include extracting from the second frequency band signal a second source model representing linear dependencies of the second frequency band signal, generating a second residual signal by filtering the second frequency band signal with filter coefficients derived from the second source model to remove linear dependencies from the second frequency band signal, transforming the second residual signal into the transform domain, scaling the transformed second residual signal with a second scale factor, quantizing the scaled transformed second residual signal to create quantization indices of the transformed residual signal, quantizing the second source model to create quantization indices of the source model, entropy encoding the quantization indices of the scaled transformed second residual signal and the quantization indices of the second source model to generate a second redundant bitstream, and constructing the payloads from the second redundant bitstream.
  • FIG. 1 is a block diagram illustrating a communication system according to an embodiment of the present invention.
  • FIG. 2 illustrates an example of the communication system of FIG. 1 in greater detail.
  • FIG. 3 illustrates an example of a wideband encoder according to an embodiment of the present invention.
  • FIG. 4 illustrates an example of a wideband FEC processor according to an embodiment of the present invention.
  • FIG. 5 illustrates an example of a super-wideband encoder according to an embodiment of the present invention.
  • FIG. 6 illustrates an example of a super-wideband FEC processor according to an embodiment of the present invention.
  • FIG. 7 illustrates an example of a process flow of the encoding process according to an embodiment of the present invention.
  • FIG. 8 illustrates an example of a wideband decoder according to an embodiment of the present invention.
  • FIG. 9 illustrates an example of a super-wideband decoder according to an embodiment of the present invention.
  • FIG. 10 illustrates an example of a process flow of the decoding process according to an embodiment of the present invention.
  • FIG. 11 illustrates an example of a computing device configured to perform encoding and decoding according to an embodiment of the present invention.
  • Fig. 1 illustrates a communication system. Audio input is passed into one end of the system, and is ultimately output at the other end of the system. The communication can be concurrently bi-directional, as in a telephone conversation between two callers. The audio input can be generated by a user speaking, by a recording, or any other audio source. The audio input is supplied to encoding module 101, where it is encoded and transmitted to packet network 104.
  • Encoding module 101 encodes the audio input into multiple packets, which are transmitted over packet network/IP channel 104 to decoding module 109.
  • Packet network 104 can be any packet-switched network, whether using physical link connection and/or wireless link connections. Packet network 104 may also be a wireless communication network, and/or an optical link network. Packet network 104 conveys packets from encoding module 101 to decoding module 109. Some of the packets sent may get lost.
  • Decoding module 109 receives packets conveyed by network 104 and decodes the packets into audio data.
  • Fig. 2 illustrates additional details of the system of Fig. 1.
  • the audio input may be sampled at a sampling frequency of 32 kHz or 16 kHz, as illustrated in Fig. 2.
  • Audio sampled at 16 kHz corresponds to a bandwidth of 0-8kHz, and will be referred to as "wideband.”
  • Audio sampled at 32 kHz corresponds to a bandwidth of 0-16kHz. In this bandwidth, the frequency range 0-8kHz is referred as wideband, while the frequency range of 8- 16kHz will be referred as "super-wideband.”
  • other frequency ranges could be selected, and the specific ranges noted are not limiting, but merely exemplary.
  • fs 32kHz
  • Filter-bank 202 separates the incoming signal into the wideband signal and the super-wideband signal.
  • the wideband signal is encoded by the wideband encoder 102, while the super-wideband signal is encoded by super-wideband encoder 103.
  • the respective encoder After the wideband and the super-wideband signals are encoded, the respective encoder produce encoded bitstreams which are concatenated and transmitted via an IP channel such as packet switched network 104.
  • bitstream After transmission via the IP channel, the bitstream is received and separated into separate bitstreams for the wideband and the super-wideband signal, respectively.
  • the wideband bitstream is decoded by wideband decoder 106, while the super-wideband bitstream is decoded by super-wideband decoder 107.
  • the output signals are combined in the filter-bank 204.
  • FIG. 3 shows an example of an embodiment of wideband encoder 102. Audio input is received in filter-bank 202, where it is separated into a low band (0-4 kHz) and a high band (4-8 kHz), as illustrated in the figure.
  • Source model analysis 310 conducts source model analysis of the incoming audio signals and produces a corresponding source model for each of the low band and the high band signals.
  • the source model may be derived by performing linear prediction coding (LPC) analysis together with pitch analysis on the incoming signals.
  • LPC linear prediction coding
  • a given frame of audio is described as a quasi -time-invariant linear filter, production filter, excited by a residual signal.
  • the quasi-time-invariance is due to the fact that the production filter needs to be updated every 5 to 10 ms, therefore, within each sub-frame (5- 10ms) the filter is time-invariant.
  • the production filter captures the short-term and the long-term linear dependencies in the signal. Short-term dependencies may be modeled by LPC analysis, and the long-term dependencies may be modeled by pitch analysis.
  • LPC analysis describes spectral envelope of the signal in question and pitch analysis reveals fine structure in the frequency domain.
  • the source model is passed from source model analysis 310 to quantizer 334, where the source model is quantized.
  • the process of quantization maps continuous values of a variable to a set of discrete values.
  • the encoding of the source model output from source model analysis 310 is rate independent. Thus, for all transmission rates the source model as generated by source modeling 301 is the same. The effect of available transmission rates is discussed with regard to wideband FEC processor 350.
  • the output of the quantizer 334 is a quantized version of the source model, which is supplied as filter parameters of analysis filters 315 and 316, and quantization indices of the source model, which are supplied to wideband FEC processor 350.
  • the analysis filter 315 takes as input the low band (0-4kHz) signal, and derives the residual signal of the low band signal.
  • the analysis filter 316 takes as input the high band (4-8kHz) signal and derives the residual signal of the high band signal.
  • Analysis filter 315 filters the digital signal input based on the quantized values of the source model.
  • the analysis filter 315 can be implemented to perform the analysis in two steps - short-term analysis and long-term analysis.
  • the short-term analysis filter is an all-zero filter, which can be implemented as a lattice filter, where filter coefficients are given by LPC analysis.
  • the long-term analysis filter can be a pole-zero filter, with filter coefficients derived from the quantized pitch lag and gain (which are part of the source model). Long- term analysis filter is only applied to the low band signal as pitch structure very rarely extends beyond 4 kHz.
  • the analysis filter 315 removes short-term and long-term structure (as determined during source modeling) from the input signal, outputting a residual signal.
  • the residual signal is transformed from the time domain into an alternate domain, for example the frequency domain through Discrete Fourier Transform (DFT), by domain transformer 330.
  • the domain transformer 330 can be implemented as an FFT (Fast Fourier Transform).
  • Other transformations such as Modified Discrete Cosine Transform (MDCT) or Modified Lapped Transform (MLT) might be used instead of DFT.
  • MDCT Modified Discrete Cosine Transform
  • MMT Modified Lapped Transform
  • the output of the domain transformer 330 is the residual signal in the transform domain.
  • the output may be DFT coefficients of the residual signal.
  • the DFT of the residual signal output from domain transformer 330, along with quantization indices of the source model are supplied to wideband FEC processor 350, where they are stored in FEC cabinet 410. Additional details of FEC processor 350 are illustrated in Fig. 4.
  • the DFT of the residual model is also supplied to quantizer 335, where the residual signal is quantized and a set of control parameters are derived and quantized. Then, quantization indices of the residual signal and quantization indices of control parameters are supplied to entropy coder 340.
  • the control parameters define a cumulative distribution function (CDF) of residual signal.
  • the quantized control parameters can be defined as an Auto Regressive (AR) model for entropy coding of DFTs.
  • Quantization indices of the source model are also supplied to entropy coder 340.
  • the entropy coder 340 is progressive over the entire bit-stream, so to encode/decode index k, index k-1 had to be encoded/decoded.
  • the entropy coder 340 may be implemented as a range encoder.
  • the input to a range encoder is a sequence of indices with the associated CDF for each index. Indices are fed into the range encoder one by one. With each input, the state of the range encoder changes and when it reaches to a pre-defined state then a sequence of bits are generated and the state is modified accordingly. After the last index is inserted, depending on the state of the range encoder a sequence of bits is generated as a termination point.
  • the order in which quantization indices are fed into the entropy coding is as follows: frame-size, bandwidth information, pitch lag, pitch gain, LP-shape, LP- gain, and residual signal.
  • the entropy coding of DFT coefficients is not any different than coding any other coefficients.
  • the difference is in computation of CDFs.
  • any other coefficient has a fixed CDF, which is used for entropy coding/decoding.
  • An assumption is made that if DFT coefficients are properly normalized then there is a single CDF which describes the statistics of the normalized coefficients.
  • the normalization factor is the standard deviation of the coefficient in question.
  • Given a set of DFT coefficients the spectral envelope is considered as an estimate for the standard deviation of each coefficient.
  • Such an envelope is modeled by an Auto Regressive (AR) process.
  • the AR coefficients can be computed by LPC analysis over the given DFT coefficients.
  • the set of AR coefficients are referred by the "control parameters" which are mentioned previously.
  • U.S. Patent No. 7,756,350 describes additional aspects of using the AR process for selecting CDF's and is incorporated herein by reference in its entirety.
  • the output of entropy coder 340 is an entropy coded bitstream representing the wideband (0-8kHz) components of the incoming audio signal.
  • bitstream can then be concatenated with the bitstream generated by the super-wideband encoder to create packets for transmission over the IP channel.
  • FIG. 4 illustrates an example of the wideband FEC processor 350.
  • FEC cabinet 410 stores the unquantized DFT of the residual signal (previously supplied by domain transformer 330) along with quantization indices of the source model.
  • Wideband FEC processor 350 reads quantization indices of the source model previously stored in FEC cabinet 410 (e.g., quantization indices of pitch gain and pitch lag, quantization indices of LP-shape), and entropy codes them with an instance of entropy coder 340, which has been described previously.
  • Wideband FEC processor 350 also reads the unquantized DFT of the residual signal from the FEC cabinet 410, and multiplies the DFT with the FEC Scale in multiplier 415, and subsequently quantizes the result in quantizer 335.
  • Quantizing a signal introduces quantization errors.
  • the errors depend on the quantization step size. More steps represent more possible values (assuming that the quantizer has a support which is sufficiently large with respect to the range of the signal), thus the actual value of the input signal is likely to be closer to one of the available values, thus resulting in a smaller quantization error.
  • adjusting the quantization step size effects the quantization error. It is also possible to control the quantization error by maintaining a constant step size, and instead scaling the signal to be quantized. This is accomplished by the multiplication by the FEC scale.
  • the amplitude of the incoming signal is scaled with a large gain, more quantization steps are available to map to the actual values of the incoming signal (assuming that the quantizer has a support which is sufficiently large), again reducing the relative quantization error. If the incoming signal is scaled with a small gain, the result is higher quantization error, but also a lower encoding rate.
  • a higher encoding rate corresponds to finer quantization of the residual signal, while a lower encoding rate corresponds to coarser quantization. Therefore, the encoding rate is controlled by the value of the FEC scale, which is applied to the DFT of the residual signal. A higher value result is a finer quantization (relatively smaller step-size) and higher encoding rate.
  • the FEC scale is set to a value less than one, to decrease the coding rate.
  • Choosing the FEC scale is a compromise between concealment quality and the capacity which is consumed by redundant payloads. It is desirable to pick the FEC scale to provide good concealment quality in a situation that is likely to occur. Thus, it is possible to select the FEC scale as a function of the anticipated packet loss percentage, as long as the encoder and the decoder in two sides of the call agree on value of the packet loss. This might be communicated through in-band or out-of-band signaling. On the other hand, the FEC scale can also be pre-selected and hard-coded into the coder and decoder.
  • the signal model and the residual signals are independent of the encoding rate.
  • all complex operations of source modeling e.g., LPC analysis, pitch estimation
  • analysis filtering analysis filter is inverse of production filter
  • the residual signal is stored together with the source model and re-used to obtain payloads at different encoding rates by simply multiplying the residual signal with a scale factor prior to quantization to obtain a payload bitstream at a desired encoding rate.
  • Fig. 5 illustrates an example of an embodiment of the super-wideband encoder 103.
  • the super-wideband encoder 103 operates on the frequency band of 8-16 kHz. Optionally, this band may be downsampled to 8-12 kHz by down-sampler 501.
  • the super- wideband encoder 103 can have two modes. In the first mode only 8-12 kHz signal is encoded, while in the second mode the entire 8-16 kHz is encoded. The modes are chosen according to the available bit-rate. In both modes input and output are sampled at 32 kHz but only 0-12 kHz is encoded when the first mode is used. Thus, in the first mode output has bandwidth of 12 kHz sampled at 32 kHz.
  • the input signal is processed in source model analysis 510.
  • the source model analysis 510 produces a source model of the super-wideband signal, which is supplied to an instance of quantizer 534.
  • Quantizer 534 is similar to quantizer 334, as they both encode the source model using the same quantization technique, but with some differences.
  • a quantizer is designed according to the statistics of the input signal. Source models of wideband and super- wideband are statistically different, thus the quantizers are specifically adapted for the respective statistics of the input signals they quantize.
  • the output of quantizer 534 is the quantized value of the source model, which is supplied as filter parameters of analysis filter 516.
  • Analysis filter 516 filters the incoming digital signal input based on the quantized values of the source model and outputs a residual signal of the incoming digital signal.
  • Analysis filter 516 is similar to analysis filter 316, but uses lower order LPC and includes more subframes as compared to the analysis filter 316 used for the 4-8kHz band.
  • An instance of entropy coder 340 entropy codes the quantized source model, outputting an entropy coded bitstream of the source model, which is supplied to super- wideband FEC processor 550, and stored in FEC cabinet 610. Output of the entropy coder 340 together with the state of the entropy coder is passed to FEC processor 550. The entropy coded bitstream is also available for creating the payload for packets to be transmitted.
  • the residual signal output from analysis filter 516 is transformed in an instance of domain transformer 330, and the transformed residual signal is supplied to super-wideband FEC processor 550 (e.g., as unquantized DFT coefficients) and stored in FEC cabinet 610.
  • the DFT of the residual signal is then quantized by an instance of quantizer 335, and entropy coded by an instance of entropy coder 340.
  • the output of this instance of entropy coder 340 is an entropy-coded bitstream of the quantized residual signal, and the bitstream is available for creating payload for packets to be transmitted.
  • FIG. 6 illustrates an example of an embodiment of super-wideband FEC processor 550.
  • FEC cabinet 610 stores entropy coded bitstream of the source model and unquantized DFT coefficients of the residual signal.
  • the output of entropy coder 340 is stored in the FEC cabinet 610 together with state of the entropy coder.
  • the output of the entropy coder is the actual bit-stream which constitutes a payload. Therefore, in the super-wideband FEC processor 550 this bitstream will be directly used to constitute the first segment of the super-wideband bit-stream.
  • the state of the entropy coder stored in FEC cabinet 610 is used to initialize the entropy coder 340 of FEC processor 550 and then the process continues as depicted.
  • the super-wideband FEC processor 550 retrieves the entropy coded bit stream of the source model and outputs it along with a redundant bitstream representing the residual signal. To generate the redundant bitstream, the unquantized DFT coefficients of the residual signal is retrieved from FEC cabinet 610 and multiplied by the FEC scale, which has been previously described.
  • the FEC scale used in the super-wideband FEC processor 550 may be set to the same value as the FEC scale used in the wideband FEC processor 350, or may be set to any other appropriate value. For example, the wideband FEC scale may be set to 0.4 and the super- wideband FEC scale can be set to 0.5.
  • the scaled residual signal is quantized by an instance of quantizer 335, and then entropy coded by an instance of entropy coder 340. It can be appreciated that the scaling and subsequent quantization effectively controls the coding rate of the redundant bit-stream by simply varying the FEC scale.
  • Entropy coder 340 can be implemented as a range encoder with a state, which is modified upon encoding a given index.
  • the source model e.g., LP parameters
  • the source model is encoded only once, and stored in FEC cabinet 610.
  • the stored bit-stream constitutes the first segment of the redundant payload, and the entropy coder 340 of the FEC processor 550 is initialized to the state was stored from the entropy coder 340 of the super-wideband encoder 103.
  • both the wideband encoder 102 and super-wideband encoder 103 output bitstreams, which are concatenated and transmitted over an IP channel.
  • the bitstreams are assembled into packets, which can include redundant payload for the FEC scheme.
  • the redundant payload is created from the bitstreams provided by FEC processors 350 and 550.
  • Packets for the FEC scheme include encoded data at two different coding rates.
  • the primary rate corresponds to FEC scale equal to 1, while the encoding rate of the redundant data is some fraction (less than 1) of the primary coding rate, based on the value of the FEC scale set for the wideband FEC processor and the FEC scale set for the super- wideband FEC processor.
  • the FEC scale is known both by encoders 102, 103 and decoders 106, 107.
  • Packets are assembled so that each packet contains encoded data corresponding to a time segment of audio encoded at the primary coding rate (primary payload) and also encoded data corresponding to an earlier time segment of audio data encoded at the redundant rate (redundant payload). This allows FEC using redundant payloads, while reducing required network bandwidth, and keeping the processing overhead for generating redundant payloads to a minimum.
  • the coding rate is changed by changing the coarseness of quantization, which is a computationally inexpensive operation.
  • step S710 the incoming audio signal is analyzed to obtain the source model and the residual signal.
  • the details vary for the wideband portion and the super-wideband portion, and further for the low band and high band of the wideband portion.
  • the source model and the residual signal can be output as a bitstream for creating of the primary payload of data packets.
  • step S720 the representation of the source model and the residual signal are stored in respective FEC cabinets for wideband and super-wideband portions.
  • the encoding of the source model is independent of the coding rate, while the residual signal is scaled by the FEC scale to control the coding rate.
  • step S730 the previously stored representations of the source model and the residual signal are retrieved, and the residual signal is scaled by the FEC scale.
  • the scaled residual signal is then quantized in step S740, which effectively controls the coding rate.
  • the quantized residual signal is also entropy coded, and in step S750 the bitstream for forming the redundant payload of data packets is output.
  • bitstreams generated by the wideband encoder 102 and super-wideband encoder 103 are concatenated, and data packets are formed.
  • the data packets may contain a primary payload encoded at a primary rate, and a redundant payload encoded at a lower rate set by the FEC scale.
  • a packet may thus contain primary payload for time segment n and a redundant payload for time segment n-1. It is also possible to have additional redundant payloads for other time segments coded at various coding rates, but the example below is directed to one primary payload and one redundant payload.
  • the data packets are transmitted to the receiving side, where the bitstreams of encoded audio data are separated from the data packets and are separated for the wideband and the super-wideband portion.
  • the wideband portion of the bitstream is decoded by wideband decoder 106, which is illustrated in Fig. 8.
  • the super-wideband portion is decoded by super-wideband decoder 107, illustrated in Fig. 9.
  • Entropy decoder 822 of the wideband decoder 106 receives the bitstream, decodes it, and outputs quantization indices of the source model and the residual signal.
  • the source model is supplied to source model decoder 810, which includes de-quantizer 814.
  • the output of de-quantizer 814 represents the source model for each of the low band (0-4kHz) and the high band (4-8kHz), and provides filter coefficients for the synthesis filters 835 and 836.
  • the quantization indices of the residual signal output from entropy decoder 822 are provided to de-quantizer 815 of the spectrum decoder 820, which de-quantizes the residual signal (resulting in a DFT coefficients of the residual signal).
  • the DFT of the residual signal is divided by the same FEC scale factor in divider 818 as used when creating the redundant payload.
  • the FEC scale is known by both the encoder and decoder. In the case of primary payload that has not been scaled, the division is skipped (effectively dividing by one).
  • the scaled DFT of the residual signal is transformed back into the time domain by inverse domain transformer 830, and supplied to synthesis filters 835 and 836.
  • the inverse domain transformer 830 may be implemented as an IFFT (Inverse Fast Fourier Transform), and transforms the residual signal into the time domain.
  • IFFT Inverse Fast Fourier Transform
  • the processing for the low band and the high band can vary in the synthesis filters 835 and 836.
  • synthesis filter 835 may perform the job of synthesis filtering.
  • the synthesis filter 835 can be implemented in two steps. The first step is a pole-zero filter derived from pitch gain and lag, reconstructing long-term dependencies.
  • the second step is an all-pole filter derived from LPC parameters.
  • the synthesis filter 836 can be derived from LPC and is an all-pole filter. All-pole synthesis filters may be implemented as lattice filters.
  • the synthesis filter 835 is linear quasi-time-invariant filter, where filter coefficients are updated with the rate that LP parameters are updated.
  • Analysis filter 315 and synthesis filter 835 are inverse of each other (within the accuracy of the implementation), such that if there were no quantization of the DFT coefficients, the input to analysis filter 315 (shown in Fig. 3) would be the same as the output of synthesis filter 835 (shown in Fig. 8). The same holds for analysis filter 316 and synthesis filter 836.
  • the output of synthesis filters 835 is the audio signal for each of low band and high band, which is then combined in filter-bank 840 into the wideband signal (0-8kHz).
  • Filter-bank 840 is the inverse of filter-bank 302. Thus, if there were no quantization in the path from filter-bank 302 to filter-bank 840, then the reconstructed signal would be the same as the input (within the accuracy of the implementation).
  • the super-wideband decoder 107 operates in a similar fashion, but processes the entire signal (8-16 kHz, or 8- 12kHz) together, as illustrated in Fig. 9.
  • the source model decoder 910 includes de-quantizer 914.
  • the de-quantizer 914 differs from the de-quantizer 814 as their counterpart quantizers (334 and 534, respectively) are different.
  • a pair of quantizer and de-quantizer should match each other for the best reconstruction of the source. For instance, a pair of uniform scalar quantizer and dequantizer should have the same steps size.
  • the output of the synthesis filter 936 of the super-wideband decoder 107 is optionally upsampled by upsampler 920 (when 8-12 kHz as the bandwidth) is used.
  • Analysis filter 516 and synthesis filter 936 are inverse of each other (within the accuracy of the implementation), such that if there was no quantization of the DFT coefficients, the input to analysis filter 516 (shown in Fig. 5) would be the same as the output of synthesis filter 936 (shown in Fig. 9).
  • Fig. 10 illustrates an example of a high level processing flow of decoding a redundant payload for frame n corresponding to a time segment of audio data.
  • step S1010 a packet with a primary payload and a redundant payload is received.
  • step S1020 the redundant payload is extracted from the packet.
  • the encoded residual signal is separated from the encoded source model (e.g., LP shape parameters).
  • step S1030 the residual signal is entropy decoded and dequantized.
  • step S1040 the residual signal is divided by the FEC scale.
  • step S1050 the residual signal is transformed into the time domain by inverse domain transformer 830.
  • step S1060 the residual signal is filtered by the synthesis filter, based on the decoded source model, to recreate a digital representation of the transmitted signal.
  • step S1070 the digital representation is optionally upsampled.
  • step S1080 the signal is converted to an analog signal.
  • FIG. 11 is a block diagram illustrating an example of a computing device 1100 that is arranged for performing redundant coding and decoding in accordance with the present disclosure.
  • computing device 1100 typically includes one or more processors 1110 and system memory 1120.
  • a memory bus 1130 can be used for communicating between the processor 1110 and the system memory 1120.
  • processor 1110 can be of any type including but not limited to a microprocessor ( ⁇ ), a microcontroller ( ⁇ ), a digital signal processor (DSP), or any combination thereof.
  • Processor 1110 can include one more levels of caching, such as a level one cache 1111 and a level two cache 1112, a processor core 1113, and registers 1114.
  • the processor core 1113 can include an arithmetic logic unit (ALU), a floating-point unit (FPU), a digital signal processing core (DSP Core), or any combination thereof.
  • a memory controller 1115 can also be used with the processor 1110, or in some implementations the memory controller 1115 can be an internal part of the processor 1110.
  • system memory 1120 can be of any type including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.) or any combination thereof.
  • System memory 1120 typically includes an operating system 1121, one or more applications 1122, and program data 1124.
  • Application 1122 includes a coding and decoding algorithm with FEC support 1123 that is arranged to perform the coding and decoding as described in this disclosure.
  • Program Data 1124 includes service data 1125 that is useful for performing coding and decoding of audio signals, as will be further described below.
  • application 1122 can be arranged to operate with program data 1124 on an operating system 1121. This described basic configuration is illustrated in FIG. 11 by those components within dashed line 1101.
  • Computing device 1100 can have additional features or functionality, and additional interfaces to facilitate communications between the basic configuration 1101 and any required devices and interfaces.
  • a bus/interface controller 1140 can be used to facilitate communications between the basic configuration 1101 and one or more data storage devices 1150 via a storage interface bus 1141.
  • the data storage devices 1150 can be removable storage devices 1151, non-removable storage devices 1152, or a combination thereof.
  • removable storage and non-removable storage devices include magnetic disk devices such as flexible disk drives and hard-disk drives (HDD), optical disk drives such as compact disk (CD) drives or digital versatile disk (DVD) drives, solid state drives (SSD), and tape drives to name a few.
  • Example computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data.
  • System memory 1120, removable storage 1151 and non-removable storage 1152 are all examples of computer storage media.
  • Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device 1100. Any such computer storage media can be part of device 1100.
  • Computing device 1100 can also include an interface bus 1142 for facilitating communication from various interface devices (e.g., output interfaces, peripheral interfaces, and communication interfaces) to the basic configuration 1101 via the bus/interface controller 1140.
  • Example output devices 1160 include a graphics processing unit 1161 and an audio processing unit 1162, which can be configured to communicate to various external devices such as a display or speakers via one or more A/V ports 1163.
  • Example peripheral interfaces 1170 include a serial interface controller 1171 or a parallel interface controller 1172, which can be configured to communicate with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device, etc.) or other peripheral devices (e.g., printer, scanner, etc.) via one or more I/O ports 1173.
  • An example communication device 1180 includes a network controller 1181, which can be arranged to facilitate communications with one or more other computing devices 1190 over a network communication via one or more communication ports 1182.
  • the communication connection is one example of a communication media.
  • Communication media may typically be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media.
  • a "modulated data signal" can be a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
  • communication media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared (IR) and other wireless media.
  • RF radio frequency
  • IR infrared
  • the term computer readable media as used herein can include both storage media and communication media.
  • Computing device 1100 can be implemented as a portion of a small-form factor portable (or mobile) electronic device such as a cell phone, a personal data assistant (PDA), a personal media player device, a wireless web-watch device, a personal headset device, an application specific device, or a hybrid device that include any of the above functions.
  • a small-form factor portable (or mobile) electronic device such as a cell phone, a personal data assistant (PDA), a personal media player device, a wireless web-watch device, a personal headset device, an application specific device, or a hybrid device that include any of the above functions.
  • PDA personal data assistant
  • Computing device 1100 can also be implemented as a personal computer including both laptop computer and non-laptop computer configurations.
  • the implementer may opt for a mainly hardware and/or firmware vehicle; if flexibility is paramount, the implementer may opt for a mainly software implementation; or, yet again alternatively, the implementer may opt for some combination of hardware, software, and/or firmware.
  • a signal bearing medium examples include, but are not limited to, the following: a recordable type medium such as a floppy disk, a hard disk drive, a Compact Disc (CD), a Digital Video Disk (DVD), a digital tape, a computer memory, etc.; and a transmission type medium such as a digital and/or an analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communications link, a wireless communication link, etc.).
  • a typical data processing system generally includes one or more of a system unit housing, a video display device, a memory such as volatile and nonvolatile memory, processors such as microprocessors and digital signal processors, computational entities such as operating systems, drivers, graphical user interfaces, and applications programs, one or more interaction devices, such as a touch pad or screen, and/or control systems including feedback loops and control motors (e.g., feedback for sensing position and/or velocity; control motors for moving and/or adjusting components and/or quantities).
  • a typical data processing system may be implemented utilizing any suitable commercially available components, such as those typically found in data computing/communication and/or network computing/communication systems.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Abstract

L'invention concerne un procédé de génération de multiples charges utiles codées selon des taux de codage différents pour inclusion dans un paquet de données transmis sur un réseau de données en paquets, consistant à extraire d'un modèle source représentant une enveloppe spectrale de signal d'entrée et un signal résiduel par filtrage du signal d'entrée au moyen de coefficients de filtrage dérivés du modèle source pour éliminer les effets de tonalité provenant du premier signal de bande de fréquence. Le modèle source et le modèle résiduel sont stockés. Lorsque la charge utile doit être codée selon un faible taux de codage, le modèle résiduel est mis à l'échelle au moyen d'un facteur de mise à l'échelle et quantifié. Le modèle résiduel mis à l'échelle et le modèle source peuvent être codés par entropie pour générer un flux binaire redondant. Ledit flux binaire redondant est utilisé pour générer des charges utiles pour des paquets transmis sur un réseau.
EP11723805.5A 2011-05-20 2011-05-20 Unité de codage redondant pour codec audio Withdrawn EP2710589A1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2011/037336 WO2012161675A1 (fr) 2011-05-20 2011-05-20 Unité de codage redondant pour codec audio

Publications (1)

Publication Number Publication Date
EP2710589A1 true EP2710589A1 (fr) 2014-03-26

Family

ID=44626687

Family Applications (1)

Application Number Title Priority Date Filing Date
EP11723805.5A Withdrawn EP2710589A1 (fr) 2011-05-20 2011-05-20 Unité de codage redondant pour codec audio

Country Status (2)

Country Link
EP (1) EP2710589A1 (fr)
WO (1) WO2012161675A1 (fr)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA2877161C (fr) 2012-06-28 2020-01-21 Tom Backstrom Codage audio par prediction lineaire utilisant une estimation de distribution de probabilite amelioree
EP2830058A1 (fr) 2013-07-22 2015-01-28 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Codage audio en domaine de fréquence supportant la commutation de longueur de transformée
CN107369455B (zh) * 2014-03-21 2020-12-15 华为技术有限公司 语音频码流的解码方法及装置
CN112289327B (zh) * 2020-10-29 2024-06-14 北京百瑞互联技术股份有限公司 一种lc3音频编码器后置残差优化方法、装置和介质
CN113450808B (zh) * 2021-06-28 2024-03-15 杭州网易智企科技有限公司 音频码率确定方法、装置、存储介质及电子设备

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1997031367A1 (fr) * 1996-02-26 1997-08-28 At & T Corp. Vocodeur multi-niveau a codage par transformee des signaux predictifs residuels et quantification sur modeles auditifs
US7177804B2 (en) * 2005-05-31 2007-02-13 Microsoft Corporation Sub-band voice codec with multi-stage codebooks and redundant coding
US7756350B2 (en) 2006-11-13 2010-07-13 Global Ip Solutions, Inc. Lossless encoding and decoding of digital data

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
"3rd Generation Partnership Project; Technical Specification Group Service and System Aspects; Audio codec processing functions; Extended Adaptive Multi-Rate - Wideband (AMR-WB+) codec; Transcoding functions (Release 6)", 7 June 2005 (2005-06-07), XP050909272, Retrieved from the Internet <URL:http://www.3gpp.org/ftp/Specs/2014-12/Rel-6/26_series/> [retrieved on 20050607] *
MAKINEN J ET AL: "AMR-WB+: a New Audio Coding Standard for 3rd Generation Mobile Audio Services", 2005 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING - 18-23 MARCH 2005 - PHILADELPHIA, PA, USA, IEEE, PISCATAWAY, NJ, vol. 2, 18 March 2005 (2005-03-18), pages 1109 - 1112, XP010790838, ISBN: 978-0-7803-8874-1, DOI: 10.1109/ICASSP.2005.1415603 *
See also references of WO2012161675A1 *
SJOBERG M WESTERLUND ERICSSON A LAKANIEMI S WENGER NOKIA J: "RTP Payload Format for the Extended Adaptive Multi-Rate Wideband (AMR-WB+) Audio Codec; rfc4352.txt", RTP PAYLOAD FORMAT FOR THE EXTENDED ADAPTIVE MULTI-RATE WIDEBAND (AMR-WB+) AUDIO CODEC; RFC4352.TXT, INTERNET ENGINEERING TASK FORCE, IETF; STANDARD, INTERNET SOCIETY (ISOC) 4, RUE DES FALAISES CH- 1205 GENEVA, SWITZERLAND, 1 January 2006 (2006-01-01), XP015044785 *

Also Published As

Publication number Publication date
WO2012161675A1 (fr) 2012-11-29

Similar Documents

Publication Publication Date Title
US20230352032A1 (en) Audio encoder, audio decoder and related methods using two-channel processing within an intelligent gap filling framework
JP5186054B2 (ja) マルチステージコードブックおよび冗長コーディング技術フィールドを有するサブバンド音声コーデック
JP5165559B2 (ja) オーディオコーデックポストフィルタ
WO2008007873A1 (fr) Procédés et appareils de codage et décodage adaptatifs
WO2012158159A1 (fr) Dissimulation de perte de paquet pour un codec audio
EP2710589A1 (fr) Unité de codage redondant pour codec audio
JP7285830B2 (ja) Ｃｅｌｐコーデックにおいてサブフレーム間にビット配分を割り振るための方法およびデバイス
EP1872364A1 (fr) Codage et/ou decodage source
WO2009044346A1 (fr) Système et procédé pour combiner un codage de golomb adaptatif avec une quantification à débit fixe
Movassagh New approaches to fine-grain scalable audio coding
Oztoprak Advanced techniques for error robust audio and speech communications
Seto Scalable Speech Coding for IP Networks

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20131203

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

RIN1 Information on inventor provided before grant (corrected)

Inventor name: SKOGLUND, JAN

Inventor name: ZAKIZADEH SHABESTARY, TURAJ

DAX Request for extension of the european patent (deleted)
RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: GOOGLE LLC

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20180523

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: G10L0019080000

Ipc: G10L0019087000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 19/02 20130101ALI20190308BHEP

Ipc: G10L 19/005 20130101ALI20190308BHEP

Ipc: G10L 19/087 20130101AFI20190308BHEP

INTG Intention to grant announced

Effective date: 20190410

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20190821

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230519