EP4690187A1 - Method for performing packet loss concealment in complex filter bank domain - Google Patents
Method for performing packet loss concealment in complex filter bank domainInfo
- Publication number
- EP4690187A1 EP4690187A1 EP24713964.5A EP24713964A EP4690187A1 EP 4690187 A1 EP4690187 A1 EP 4690187A1 EP 24713964 A EP24713964 A EP 24713964A EP 4690187 A1 EP4690187 A1 EP 4690187A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- frame
- tile
- replacement
- values
- tile values
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/005—Correction of errors induced by the transmission channel, if related to the coding algorithm
Definitions
- codecs that operate in the frequency domain, whereby segments of time-domain audio signals are transformed into vector- valued samples, each sample being represented in the encoded bitstream by a set of time- frequency, TF tiles combined with some side information.
- Each TF tile is associated with a specific frequency band and at the decoding end the samples carrying TF tiles are reconstructed by decoding the bitstream after which the time-domain samples are regenerated by applying an inverse transformation on the TF tiles.
- time-frequency transformations and corresponding inverse transformations).
- Examples include the Discrete Fourier Transform (DFT), the Discrete Cosine Transform DCT, the Modified Discrete Cosine Transform (MDCT) or Sub-band techniques that represent the original time-domain signal by samples in a sub-band domain.
- DFT Discrete Fourier Transform
- DCT Discrete Cosine Transform
- MDCT Modified Discrete Cosine Transform
- Sub-band techniques that represent the original time-domain signal by samples in a sub-band domain.
- An example of a sub-band domain transformation is using Quadrature Mirror Filter (QMF) banks that have the property that each bandpass (or sub-band) signal may be critically sampled so as to still allow an inverse operation with perfect reconstruction of the original time signal.
- QMF Quadrature Mirror Filter
- the CQMF bank is designed to introduce a minimum delay in the forward direction (CQMF analysis) as well as in the backward direction (CQMF synthesis), which makes CQMF banks especially useful in conversational speech applications and applications benefiting from low latency, such as telephony or teleconferencing applications.
- CQMF analysis CQMF analysis
- CQMF synthesis backward direction
- a particular type of CQMF bank is used in the codec standardized for the 3GPP Immersive Voice and Audio Services (IVAS) codec.
- This CQMF bank is called the Complex Low-Delay Filter Bank (CLDFB) and, according to the CLDFB design, when CLDFB analysis is applied to a 48 kHz sampled audio signal, one TF tile (represented with one TF tile value) represents a frequency band of 400 Hz bandwidth and the CLDFB samples cover a time slot of 1.25 ms.
- CLDFB Complex Low-Delay Filter Bank
- Services like 3GPP IVAS are often deployed over error-prone radio channels between a transmitting device and a receiving unit.
- the TF tiles of multiple CQMF samples are called frames which are encoded as a block of data, wherein each frame contains CQMF samples representing a predetermined duration of audio content (e.g.20 ms).
- Coded frames are sent in a packetized batch of one or multiple coded frames, transmitted over the radio channel.
- Some error-detecting mechanisms such as cyclic redundancy check, CRC
- CRC cyclic redundancy check
- Some error-detecting mechanisms may be applied in the receiving unit to detect if a packet (carrying one or more frames) has been received without errors or if the packet is corrupt.
- a packet can also arrive too late to be used by the receiving unit.
- only the audio frames contained in packets received in time and without error can be used for audio decoding and extraction of the time audio signal.
- the receiving unit may use techniques to generate substitution frames.
- One technique for forming a substitution frame is to use a zero frame (i.e. a frame with each spectral coefficient equal to zero) as substitution signal.
- Another technique is to repeat the most recent frame that was received without error.
- the art of generating substitution frames is called frame loss concealment or packet loss concealment (PLC).
- PLC packet loss concealment
- a general objective of these techniques is that frame losses should become as inaudible as possible so that their impact on the audio quality at the receiver is minimized.
- the packet loss concealment may be based on the frame repetition technique and in cases that rely on parametric coding, coding parameters of the most recent good frame are repeated.
- a drawback with existing concealment techniques is that they offer less than desired quality, often introducing perceptible audio artifacts. Additionally, many of the techniques are computationally complex (e.g. in terms of computational time, memory requirements and number of computational operations), requiring powerful processing hardware and/or increasing the latency of the audio encoding-decoding chain. At the same time, audio decoding is frequently performed on very power-limited end-devices such as earbuds or AR- glasses meaning that many concealment techniques are directly unsuitable for achieving low- latency and high quality concealment in some applications.
- the present disclosure identifies an improved concealment technique that can recreate high quality substitution frames with reduced complexity operations.
- a method to conceal a lost or corrupt frame with audio content in a frame sequence comprises obtaining at least one first frame associated with the frame sequence, the at least one first frame being a frame of the frame sequence or a generated replacement frame and identifying whether a second frame, subsequent to the at least one first frame, is valid or invalid.
- the method comprises generating a set of Time-Frequency (TF) tile values for a replacement frame to replace the second frame by analyzing the at least one first frame to determine a degree of tonality and comparing the degree of tonality to a threshold value to determine if the degree of tonality exceeds the threshold value.
- the method comprises applying a sinusoidal extension process to generate the set of TF tile values for the replacement frame and when the degree of tonality does not exceed the threshold value, the method comprises applying a linear prediction process to generate the set of TF tile values for the replacement frame.
- the method further comprises outputting the replacement frame to replace the second frame.
- a frame that has been successfully received in full, and thus determined to be uncorrupted, at a time of decoding is identified as a valid frame.
- a frame that is determined to be corrupt or has not been received or is otherwise unavailable at a time of decoding (e.g., lost), is identified as an invalid frame.
- measures are taken to generate a replacement frame so as to conceal the invalid frame(s).
- the preceding first frame may be a valid frame, or a replacement frame for a preceding frame previously determined to be an invalid frame.
- TF tile values for a replacement frame are generated adaptively by adaptively selecting one of two processes for generating the TF tile values of the replacement frame.
- the selection of which out of the two processes, sinusoidal extension and linear prediction, that is to be used is based on the degree of tonality of a preceding (first) frame. Since sinusoidal extension is expected to generate more accurate TF tile values when the degree of tonality is higher sinusoidal extension is used when the degree of tonality exceeds the threshold and since linear prediction is expected to generate more accurate TF tile values when the degree of tonality is lower linear prediction is used when the degree of tonality is below the threshold.
- TF tile values which make for a more realistic and believable replacement frame (serving as a continuation of the preceding first frame) can be generated which makes for a higher quality PLC technique.
- PLC technique There are various methods for quantifying the quality of a PLC technique. For example, subjective listening tests may be performed where listeners score the quality of audio signals which have been generated through a simulation of an error-prone transmission system resulting in frame losses and where the PLC technique to be tested is used to conceal the perceptual effect of the frame losses. The test subjects may be asked to score the perceived clarity or quality of the audio. Using these tests, different PLC techniques can be compared on the basis of their scoring.
- a method to conceal a lost or corrupt frame in Complex Quadrature Mirror Filter, CQMF, domain comprising receiving at least one first frame of CQMF samples, wherein each CQMF sample spans one or more time-frequency (TF) tiles, and wherein each TF tile is associated with a frequency band and a complex TF tile value, respectively.
- CQMF Complex Quadrature Mirror Filter
- the method further comprises identifying whether a second frame of CQMF samples, subsequent to the at least one first frame, is valid or invalid and when the second frame is identified as invalid, the method comprises for at least one of the respective frequency bands of the one or more TF tiles of the CQMF samples of the at least one first frame determining a complex parameter based on at least two of the TF tile values in the at least one first frame and generating a replacement TF tile value by modifying at least one TF tile value of the first frame with the complex parameter.
- the method further comprises and forming a replacement frame for the second frame based on the replacement TF tile value and outputting the replacement frame.
- the complex parameter which is based on at least two TF tile values of at least one preceding frame, may be used to generate a set of TF tile values for a replacement frame. In some implementations, the same complex parameter is used iteratively to generate two or more (or all) TF tile values of the replacement frame.
- an apparatus comprising a processor and memory, the apparatus being configured to perform the method of the first or second aspect.
- a non- transitory media having software stored thereon, the software including instructions for controlling one or more devices to perform the method of any of the first or second aspect.
- a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of the first or second aspect.
- a receiving unit comprising a decoder configured to receive a bitstream.
- the bitstream comprising data packets wherein each data packet carries a frame comprising a plurality of Time-Frequency, TF, tile values, wherein the decoder is configured to decode the received data packets of the bitstream to obtain the TF tile values of a current frame of a current data packet.
- the receiving unit further comprises a frame replacement module configured to identify if a current frame is invalid and, in response to identifying that the current frame is invalid, generate a set of TF tile values of a replacement frame for replacing the current invalid frame.
- the frame replacement module comprises a tonality extractor configured to analyze at least one preceding frame, the at least preceding frame preceding the current frame, to determine a degree of tonality of the preceding frame, and an adaptive sinusoidal extension and linear prediction module configured to determine if the degree of tonality exceeds the threshold value and when the degree of tonality exceeds the threshold value, generate the set of TF tile values for the replacement frame with a sinusoidal extension process, and when the degree of tonality does not exceed the threshold value, generate the set of TF tile values for the replacement frame with a linear prediction process.
- the receiving unit further comprises a synthesis filter bank configured to receive the TF tile values of the replacement frame and convert the TF tile values of the replacement frame to a time domain audio segment.
- a synthesis filter bank configured to receive the TF tile values of the replacement frame and convert the TF tile values of the replacement frame to a time domain audio segment.
- FIG.1a is a block diagram illustrating an encoder and a decoder with a frame replacement module.
- FIG.1b is a block diagram illustrating a frame replacement module with adaptive selection of sinusoidal extension and linear prediction.
- FIG.2 is a diagram illustrating a frame comprising a plurality of samples.
- FIG.3 is a diagram illustrating a frame sequence wherein one frame is invalid and replaced with a generated replacement frame.
- FIG.4 is a flowchart illustrating processing in a frame replacement module according to some implementations.
- FIG.5 is a diagram illustrating a sinusoidal extension process.
- FIG.6a-d are diagrams illustrating a linear prediction process.
- FIG.7 is a diagram illustrating a linear prediction process using weighted combination with a substitute frame.
- FIG.8 is a diagram illustrating a crossfade from a generated replacement frame to a subsequent valid frame.
- FIG.9 illustrates a schematic block diagram of an example device or architecture that may be used to implement embodiments of the invention.
- DETAILED DESCRIPTION [040] In the following description, numerous details are set forth, such as systems, device configurations, timings, operations, and the like, in order to provide an understanding of one or more aspects of the present disclosure. It will be readily apparent to one skilled in the art that these specific details are merely examples and not intended to limit the scope of this application. [041] In this document, the terms “and”, “or” and “and/or” are used. Such terms are to be read as having an inclusive meaning.
- a and B may mean at least the following: “both A and B”, “at least both A and B”.
- a or B may mean at least the following: “at least A”, “at least B”, “both A and B”, “at least both A and B”.
- a and/or B may mean at least the following: “A and B”, “A or B”.
- the term “includes” and its variants are to be read as open-ended terms that mean “includes, but is not limited to.”
- the term “one example implementation” and “an example implementation” are to be read as “at least one example implementation.”
- the term “another implementation” is to be read as “at least one other implementation.”
- the terms “determined,” “determines,” or “determining” are to be read as obtaining, receiving, computing, calculating, estimating, predicting, or deriving.
- all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.
- the encoder 18 is configured to receive the TF samples of the TF representation and responsively encode the TF samples into a bitstream, which may be output for use by a receiving unit 12.
- the encoder 18 may form the bitstream by collecting a predetermined number of subsequent TF samples so as to form a frame of TF samples whereby the frames are coded into data packets that constitute the bitstream.
- the decoder 19 is configured to receive the data packets of the bitstream and responsively decode the data packets of the bitstream to retain the frames of TF samples.
- the frame replacement module 15 is configured to receive the frames of TF samples from the decoder 19 and, in the case of a frame of TF samples being invalid, the frame replacement module 15 generates a replacement frame of TF samples.
- the synthesis filter bank 14 is configured to receive the frames of TF samples (or replacement frames thereof) and for each TF sample generate time domain samples of the time domain audio segments.
- the synthesis filter bank 14 in the receiving unit 12 is configured to perform an inverse filtering operation relative to the analysis filter bank 13 of the sending unit 11.
- an encoding and decoding process employed by the system of FIG. 1a may be described as follows. Time-domain audio segments that are received by the sending unit 11 are processed by the analysis filter bank 13 to obtain TF samples in the TF domain.
- the TF samples are grouped into frames which are encoded into a bitstream by encoder 18 and subsequently output (e.g., stored, transmitted, conveyed, etc.).
- the bitstream is received by the receiving unit 12 and processed by the decoder 19 to obtain the frames of TF samples.
- the frames of TF samples are processed by the frame replacement module 15 to generate a replacement frame with TF samples, when needed.
- the replacement frame of TF samples is processed by the synthesis filter bank 14 to generate time domain samples of recovered time domain audio segments.
- the function of the frame replacement module 15 is to ensure that the adverse effects of frames with TF samples which are not received, corrupt or otherwise unavailable are mitigated by generating a replacement frame to replace frames which are not received, corrupt or otherwise unavailable.
- each analysis filter bank 13 is made up of multiple filters, where each filter has a passband defined over a specific range of frequencies.
- the time domain samples of the audio segment are multiplied by the filter characteristics (e.g., filter coefficients of the pass band or stop band, or interpolations thereof) to result in an output value for each time domain sample.
- the output values may subsequently be decimated.
- the optionally decimated output values form TF tile values that each occupy a TF tile of a TF sample.
- the output from the analysis filter bank 13 is in the form of QMF samples wherein each QMF sample represents a portion of audio content, and wherein each QMF sample may include multiple TF tiles, each representing a respective frequency band.
- each QMF filter may be defined as a bandpass filter with a passband that is centered about a unique center frequency.
- a first example bandpass filter may include two frequencies ( ⁇ ⁇ , ⁇ ⁇ ) defining a first bandwidth BW1 of a first passband, while an adjacent second bandpass filter may include two frequencies ( ⁇ ⁇ , defining a second bandwidth BW2 of a second passband.
- the example passband frequencies are defined such that ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ then the first and second passbands do not overlap.
- the frame replacement module 15 is employed to generate a set of TF tile values for a replacement frame, which replaces the one or more frames of the non-received or corrupt data packet(s).
- the frame replacement module 15 is depicted as a module which is separate from the decoder 19. However this is merely an example and the frame replacement module 15 may be an integral part of the decoder 19.
- FIG.2 shows schematically a frame 20 of QMF (e.g. CQMF) samples.
- an individual frequency band is represented by a row 23 wherein all TF tiles 21 of the same row contains a TF tile values associated with the same frequency band and the same corresponding QMF in the QMF bank used to extract the TF tile values.
- the frame replacement module is configured to operate using frames 20 like the one depicted in FIG.2. For example, the frame replacement module is configured to generate TF tile values for populating all the TF tiles 21 of a replacement frame (given one more preceding frames) for replacing a frame which has been lost or is otherwise unavailable.
- N 5, which results in a total of 25 TF tiles 21.
- the frame 20 shown in FIG.2 is merely an example and it is understood that the frame size may be determined almost arbitrarily depending on the implementation.
- the Complex Low-Delay Filter Bank (CLDFB) used in the 3GPP IVAS codec uses 60 filters, each with a bandwidth of 400 Hz and each sample 22 representing 1.25 ms.
- the CLDFB may be applied to a 48 kHz sampled audio signal whereby 60 filters of 400 Hz bandwidth capture the lower 24 kHz of the 48 kHz sampled audio signal.
- the frequency spectrum of a 48 kHz audio signal is symmetric around half of the sampling frequency (24 kHz, also referred to as the folding frequency) the frequency spectrum above 24 kHz may be reconstructed after decoding by mirroring the frequency spectrum of the lower frequencies around the 24 kHz.
- FIG.1b shows schematically a frame replacement module 15 arranged according to some implementations.
- the frame replacement module 15 comprises a tonality extractor 17 and an adaptive sinusoidal extension and linear prediction module 16.
- the frame replacement module 15 is configured to generate TF tile values for a replacement frame 20B’ given at least one preceding frame 20A which may be referred to as the first frame 20A.
- the frame replacement module 15 comprises an adaptive sinusoidal extension and linear prediction module 16 which generates the TF tile values for the replacement frame using 20B’ using either a sinusoidal extension process or a linear prediction process.
- the frame replacement module 15 further comprises a tonality extractor 17 configured to extract, for each frequency band of the preceding first frame 20A, a degree of tonality.
- the degree of tonality is provided to the adaptive sinusoidal extension and linear prediction module 16, which generates the TF tile values for the replacement frame using 20B’ using either a sinusoidal extension process or a linear prediction process, wherein the type of process employed is based on the degree of tonality of each frequency band.
- the preceding first frame 20A of QMF samples is input to the frame replacement module 15.
- the first frame 20A is provided to the tonality extractor 17 which extracts, for each frequency band ⁇ ⁇ , a degree of tonality ⁇ ⁇ .
- the tonality extractor 17 extracts individual degree of tonality values ⁇ ⁇ , ⁇ ⁇ , ⁇ ⁇ , ⁇ ⁇ for each of frequency bands ⁇ ⁇ , ⁇ ⁇ , ⁇ ⁇ , ⁇ ⁇ .
- the degree of tonality ⁇ ⁇ for each frequency band ⁇ ⁇ in the first frame 20A is provided to the adaptive sinusoidal extension and linear prediction module 16 alongside the first frame 20A.
- the adaptive sinusoidal extension and linear prediction module 16 determines whether to use a sinusoidal extension process or a linear prediction process for each frequency band ⁇ ⁇ based on the band-specific degree of tonality ⁇ ⁇ .
- the adaptive sinusoidal extension and linear prediction module 16 uses the sinusoidal extension process or the linear prediction process to generate TF tile values of the replacement frame 20B’ and outputs the replacement frame 20B’ with the generated TF tile values of bands ⁇ ⁇ with the selected type of processing.
- the frame replacement module 15 is according to some implementations configured to analyze at least one preceding first frame 20A and generate a set TF tile values that can be used to form a replacement frame 20B’ that replaces a subsequent second frame.
- the frame replacement module 15 may comprise an adaptive sinusoidal extension and linear prediction module 16 which generates the set of TF tile values using either a sinusoidal extension process or a linear prediction process.
- the type of process employed may be selected individually for each frequency band based on a degree of tonality of each frequency band of the first frame as determined by the tonality extractor 17 of the frame replacement module 15.
- the TF tile values of one or more frequency bands in the replacement frame 20B’ may be generated using sinusoidal extension and the TF tile values of one or more other frequency bands may be generated using linear prediction based on the degree of tonality of each individual frequency band.
- the frame replacement module 15 may further be configured to store one or more of the preceding first frames 20A so as to be able to access the TF tile values of these frames when a replacement frame 20B for a current second frame is to be generated.
- FIG.3 shows schematically a frame sequence of three frames 20A, 20B, 20C.
- the frame sequence comprises a first frame 20A, a second frame 20B temporally subsequent to the first frame 20A and a third frame 20C, temporally subsequent to the second frame 20B.
- Each frame comprises a plurality of QMF samples (columns), wherein each QMF sample comprises a plurality of TF tiles, each TF tile associated with a respective frequency band ⁇ ⁇ (rows).
- Each of the frames 20A, 20B, 20C of the frame sequence may be encoded by the encoder in the sending unit to a corresponding number of coded data packets (each data packet carrying one frame) wherein the data packets are transmitted to the receiving unit as a bitstream.
- coded data packets each data packet carrying one frame
- one or more data packets may be lost or corrupted on its way to the receiving unit whereby the frame included in these lost or corrupted data packets will not be available for the receiving unit.
- the second frame 20B may be a frame which is not available at the receiving unit (e.g.
- a method for forming a replacement frame will now be described. The method may be performed by the frame replacement module 15 described above.
- step S1 “obtaining a first frame”, a first frame 20A is obtained.
- the first frame 20A is associated with a bitstream received by the frame replacement module and/or the decoder.
- the first frame 20A may either be a replacement frame (e.g., a frame which has been previously generated by the frame replacement module in response to an original first frame being lost or corrupt) or an original first frame which was delivered to the frame replacement module and/or decoder.
- a replacement frame e.g., a frame which has been previously generated by the frame replacement module in response to an original first frame being lost or corrupt
- an original first frame which was delivered to the frame replacement module and/or decoder “076]
- the decoder and/or the frame replacement module checks if a second frame 20B, subsequent to the first frame 20A, is available and is not corrupt. A frame that is available for decoding and not corrupted is identified as a valid frame. If a frame is not available, is corrupted or otherwise unavailable, then the frame is identified as invalid.
- the frames (and/or the data packets carrying the frames) may be intended to arrive at the receiving unit as a consecutive sequence. However, if a frame (or data packet) is lost, delayed or arrives corrupted (i.e. is invalid) at the time the frame should be processed by the receiving unit (e.g. to maintain audio playback without interruption) this should be identified to allow a replacement frame to be generated. Therefore, at step S2a the receiving unit checks if a valid second frame is available so as to e.g. allow sufficient time to generate a replacement frame. [078] At step S2b, “Is second frame valid?”, the result from step S2a is used to identify if the second frame 20B is available or if it has been lost, is corrupt or otherwise unavailable.
- This identification is referred to as identifying whether the second frame 20B is a valid frame or an invalid frame. Accordingly, if the second frame 20B is not available at step S2a, or the second frame 20B is determined at step S2a to be corrupt, then the method determines at step S2b that the second frame is invalid. On the other hand, if the second frame 20B is available at step S2a and the second frame is not corrupt, then the method determines at step S2b that the second frame 20B is valid. [079]
- the decoder (and/or analogously the frame replacement module) may operate frame-synchronously, meaning that the decoder decodes a frame each ⁇ seconds, wherein T is a predetermined time interval.
- ⁇ is equal to the temporal duration of one frame, such as 20 ms.
- the frames 20A, 20B are conveyed as data packets using one or more data transmission protocols to the receiving unit.
- Example data transmission protocols include IP, UDP and RTP.
- the decoder of the receiving unit collects the data packets and may store them in a buffer.
- the data packets may arrive asynchronously, and some data packets may not arrive at all. For example, one or more data packets may not arrive due to buffer overflows caused by data traffic congestion along the transmission path. Additionally, some packages may arrive with an error, which is detected by a CRC mechanism.
- a packet which arrives too late so that it is unavailable when the decoder requires it in order to operate frame-synchronously is one example of an invalid data packet and the frame associated with this data packet is an example of an invalid frame.
- a data packet which does not arrive at the receiving unit at all is another example of an invalid data packet and the frame associated with this data packet is an example of an invalid frame.
- a data packet which arrives with an error detected by a CRC mechanism are is yet another example of an invalid data packet and the frame associated with this data packet is an example of an invalid frame.
- Identifying if a frame is valid or invalid may therefore comprise identifying if a BFI flag has been raised for the frame, whereby the frame is identified as invalid if the BFI flag is raised and invalid if there is no raised BFI flag. [081] Accordingly, it is possible for the decoder and/or frame replacement module to identify that a frame is invalid even if the frame is not received. In frame-synchronous operation, it may be assumed that a frame should have reached the decoder and/or frame replacement module of the receiving unit within a predetermined time interval.
- the beginning of the predetermined time interval may be measured from the last successful receipt of a frame or data packet. Therefore, a frame may be labeled (identified) as invalid when it is determined that a frame or data packet is not available after the expiration of the predetermined time interval after (successfully) receiving the preceding frame or data packet.
- a set of TF tile values for a replacement frame 20B’ are to be generated.
- the type of processing used to generate the TF tile values is selected individually for each frequency band based on the degree of tonality of the corresponding frequency band of the at least one preceding frame.
- the generated replacement frame may comprise a mix of frequency bands containing TF tile values generated with sinusoidal extension and frequency bands containing TF tile values generated with linear prediction.
- the same general processing may be applied for frames having a single frequency band.
- the method may proceed to step S3, “Obtaining tonality of first frame”, which comprises analyzing the first frame 20A to determine a degree of tonality for the first frame 20A.
- the degree of tonality is determined individually for each frequency band so as to yield one degree of tonality for each of a plurality of frequency bands, individually.
- Determining the degree of tonality of a frame e.g.
- the first frame 20A may involve analyzing the TF tile values of the frame (e.g. in each frequency band individually). For example, the first frame 20A may be temporarily stored and accessible, wherein determining the degree of tonality comprises analyzing the TF tile values of the first frame 20A. [086] In some implementations, determining the degree of tonality of a frame (e.g. the first frame 20A) comprises evaluating TF tile values of the at least one first frame 20A to identify differences in (e.g. complex) phase between different TF tile values of the frame and calculating a phase standard deviation based on the identified differences in phase between different TF tile values of the frame.
- differences in e.g. complex
- the frame replacement module 15 comprises a tonality extractor 17 which analyzes the first frame 20A to determine the degree of tonality for each frequency band individually.
- the degree of tonality of individual frequency bands is denoted ⁇ ⁇ and in the example shown in FIG.1b the degree of tonality ⁇ ⁇ , ⁇ ⁇ , ⁇ ⁇ , ⁇ ⁇ , for four frequency bands are provided by the tonality extractor 17 to the adaptive sinusoidal extension and linear prediction module 16.
- the method may then proceed to step S4, “Is tonality above threshold?”, which comprises determining, for each frequency band, if the degree of tonality ⁇ ⁇ , ⁇ ⁇ , ⁇ ⁇ , ⁇ ⁇ , exceeds a predetermined threshold.
- This step may be performed by the adaptive sinusoidal extension and linear prediction module 16.
- the predetermined threshold may be the same for each frequency band, or it is envisaged that different predetermined thresholds may be used.
- the predetermined threshold is a phase threshold. By comparing the phase standard deviation of a specific frequency band to the phase threshold it is possible to determine if a specific frequency band is tonal. When the phase standard deviation is below the phase threshold, the specific frequency band is identified as tonal. When the phase standard deviation is equal to or greater than the phase threshold, the specific frequency band is determined as noise-like.
- the phase threshold for the standard deviation may be determined by calculation, experimentation, measurements, and other such techniques. In one example, the ⁇ phase threshold for the standard deviation is about ⁇ .
- the predetermined threshold is a magnitude threshold. By comparing the magnitude standard deviation to the magnitude threshold it is possible to determine if a specific frequency band is tonal.
- the sinusoidal extension process may further comprise calculating extension TF tile.
- Extension TF tile values of a frame e.g. the first frame 20A
- QMF sample ⁇ ⁇ 1 of that frame e.g. the first frame 20A
- step S6a “Generating TF tile values with linear predictor”, which comprises generating the TF tile values for a replacement frame using a linear prediction process.
- individual TF tile values of frequency bands B1, B3 and B4 in the replacement frame 20B’ have been reconstructed using linear prediction (abbreviated L.P).
- the adaptive sinusoidal extension and linear prediction module 16 generates TF tile values for frequency bands with a comparatively high degree of tonality with sinusoidal extension, and generates TF tile values for frequency bands with a comparatively low degree of tonality with linear prediction.
- the linear prediction process may comprise calculating extension TF tile values with the linear prediction process based on at least two TF tile values in the first frame 20A.
- the method then proceeds to step S6b, “Forming weighted superposition”, which comprises, for each frequency band, forming a weighted superposition between the TF tile values generated with the linear prediction process and a substitute frame.
- the substitute frame may be a copy of the preceding first frame 20A.
- the weighted superposition is performed for a predetermined number of the temporally earliest TF tile values of the replacement frame, as will be described in further detail below.
- the frame structure of the replacement frame may be equal to that of the invalid second frame 20B.
- the sinusoidal extension process and the linear prediction process may be used to generate a subset (e.g. a proper or strict subset) of the TF tile values of the replacement frame or generate all TF tile values of the replacement frame.
- the step of generating TF tile values with sinusoidal extension or linear prediction at step S5 and S6a is done for each frequency band, individually.
- TF tile values of one or more frequency bands may be generated with sinusoidal extension, while the TF tile values of one or more other frequency bands of the same frame may be generated with linear prediction, based on the degree of tonality calculated for each frequency band at step S3.
- generating a set of TF tile values for a replacement frame may be performed individually for each of a plurality of frequency bands when the frame comprises TF tile values spanning more than one frequency band.
- the TF tile values are generated using either the sinusoidal extension process or the linear prediction process for each frequency band, individually.
- the method may proceed to optional step S7, “Pre-calculating extension TF tile values”, which comprises determining extension TF tile values using the selected TF tile value generation process.
- the extension tile values are tile values which are temporally later than the tile values of sample ⁇ ⁇ 1 of the replacement frame.
- the TF tile value generation processes may in principle operate continuously, generating an arbitrary number of extension samples after generating a set of (or optionally all) TF tile values of the replacement frame.
- the extension TF tile values may be stored or otherwise made available for a third frame 20C, which is subsequent to the second frame 20B.
- the extension TF tile values are then used to form a perceptually high quality transition into the third frame 20C when the third frame 20C is a valid frame. If the third frame 20C is another invalid frame, the extension TF tile values may be retained or regenerated as a replacement frame for the third frame 20C is generated. [102]
- the method may then proceed to step S8a, “Outputting replacement frame”, which comprises outputting the replacement frame as a replacement for the second frame.
- the replacement frame outputted at step S8a is still in the QMF (e.g. the CQMF) domain.
- step S8b “Performing synthesis filtering and outputting time domain frame”, which may comprise processing the replacement frame with the synthesis filter to form a time domain audio frame which is outputted.
- step S9 “Was first frame valid?”, which comprises determining if the first frame 20A was a valid frame or an invalid frame. Determining whether the first frame 20A was valid may comprise identifying if a BFI flag associated with the first frame 20A (or a data packet carrying the first frame 20A) was raised.
- the BFI flag value indicates if the frame was valid or invalid.
- the BFI flag may be temporally stored to allow identification of whether a preceding frame was valid or invalid.
- the method may proceed to step S10a, “Outputting second frame without modification”, which may comprise outputting the second frame without modification. That is, if both the first and second frame 20A, 20B are identified as valid frames, then it is not necessary to perform any frame replacement and all information is available to the decoder whereby the second frame 20B is output without modification.
- step S10b “Performing synthesis filtering and outputting time domain frame”, which may comprise processing the replacement frame with the synthesis filter to form a time domain audio frame which is outputted.
- step S11 “Generating crossfaded TF tile values”, which comprises obtaining an extension set of TF tile values of the first frame 20A and generating one or more crossfaded TF tile values for the second frame 20B.
- Generating the crossfaded TF tile values may comprise crossfading the extension TF tile values with the TF tile values of the second frame in a crossfade region. It is noted that at step S11 the first frame 20A is invalid (and has been replaced with a replacement frame), while the second frame 20B is valid. Hereby, a perceptually high quality transition from the replacement frame to the second frame 20B is to be achieved.
- the extension set of TF tile values for the first frame 20A may be generated with the sinusoidal extension process or the linear predictor process, which may already be available since the extension TF tile values were generated when the first frame 20A was replaced with a replacement frame. [106] Step S11 may also be repeated for each frequency band, individually.
- the extension TF tile value for one frequency band may be generated with sinusoidal extension, while the extension TF tile values for another frequency band may be generated with linear prediction, depending on the determined degree of tonality of the frequency bands of the preceding frames.
- the crossfade region may comprise a predetermined number ⁇ of the temporally earliest samples of the second frame 20B.
- the crossfading TF tile values are formed as a crossfade weighted sum of a TF tile value of the second frame 20B with a TF tile value of the extension set of TF tile values from the first frame 20A. The two summands of the crossfade weighted sum are weighted with a respective crossfade weighing factor.
- the crossfade weighting factor for the TF tile values of the second frame 20B is temporally increasing between subsequent QMF samples.
- the crossfade weighting factor for the TF tile values of the extension set of TF tile values is temporally decreasing between subsequent QMF samples in a complementary manner.
- the crossfading weighting factor may be inversely proportional to the sample index ⁇ . This results in each crossfaded TF tile value comprising a reduced number of the extension TF tile values and an increased number of the TF tile values of the second frame 20B as the sample index ⁇ increases.
- the crossfade weighting factor may change linearly or non-linearly (e.g., exponentially) with the sample index to control the transition from generated TF tile values to the TF tile values of the second frame 20B.
- the method may then proceed to step S12a, “Outputting replacement frame with crossfaded TF tile values”, which comprises outputting a replacement frame for replacing the second frame 20B wherein the replacement frame comprises the crossfaded TF tile values.
- the replacement frame may comprise the crossfaded TF tile values in the crossfading region, while in a trailing region, which is temporally later than the crossfading region, the replacement frame comprises the unmodified TF tiles values of the second frame 20B.
- each frame comprises at least two temporally separated TF tile values.
- each frame comprises complex TF tile values.
- each frame has been generated with a CQMF bank (such as the CLDFB).
- the frames 20A, 20B, 20C of FIG.3 are complex frames generated with a CQMF bank.
- the frames 20A, 20B, 20C carry TF tiles 21, wherein each TF tile comprises a complex TF tile value representing a frequency band and time duration of the time domain audio signal.
- TF tiles 21 wherein each TF tile comprises a complex TF tile value representing a frequency band and time duration of the time domain audio signal.
- a stream of the frames 20A, 20B, 20C is available for decoding in the intended order.
- a given frame is identified as invalid (e.g. has been lost or corrupted) there will in general be at least one frame preceding the given frame which has been received without error.
- the sinusoidal extension process and the linear prediction processes may operate recursively and generate TF tile values for replacement of an arbitrary number of invalid frames, as will be described below.
- the second frame 20B is an invalid frame (for which a replacement frame is to be generated) and that the preceding first frame 20A is available or at least a replacement frame for the first frame 20A is available.
- the third frame 20C which is subsequent to the second frame 20B, may be identified as a valid frame or an invalid frame as will be discussed below.
- the two TF tile value generation processes i.e. the sinusoidal extension process and the linear prediction process
- the two processes may be used together in sequence and/or in parallel for different frequency bands.
- an adaptive frame replacement technique is used, where one of the sinusoidal extension process and the linear prediction process is used for each invalid frame, or for each frequency band of each invalid frame, based on the degree of tonality of a preceding frame (or a replacement frame thereof), or each frequency band of the preceding frame.
- the second frame 20B is an invalid frame and a set of TF tile values (e.g. all TF tile values of a replacement frame 20B’ having the same frame structure of the second frame 20B) are to be generated to form a replacement frame 20B’.
- both the sinusoidal extension process and the linear prediction process is performed by determining a complex parameter ⁇ based on at least two TF tile values of the first frame 20A.
- the TF tiles 21 in the first frame 20A whose TF tile values are used to determine the complex parameter ⁇ may be of the same frequency band ⁇ ⁇ as the set of TF tile values that are to be generated. For example, when a complex TF tile value in the second frame 20B belonging to a TF tile 21 of frequency band ⁇ ⁇ is to be generated the complex parameter ⁇ is determined from TF tile values of frequency band ⁇ ⁇ in the first frame 20A.
- the complex parameter ⁇ may describe a ratio between the two TF tile values of the first frame 20A.
- the complex parameter ⁇ may describe a difference between two parameters derived from the two TF tile values.
- the complex parameter ⁇ describes the difference between the phase of two TF tile values.
- the complex parameter ⁇ allows generation of a TF tile value in the second frame 20B by extrapolating from a TF tile value of the first frame 11 using the complex parameter ⁇ .
- the TF tile value from which the extrapolation is made may be one of the TF tile values used to determine the complex parameter ⁇ or a different TF tile value.
- the TF tile value used for extrapolation is the temporally latest TF tile value of the first frame 20A.
- the complex parameter ⁇ may be one or more of a phase parameter (e.g.
- the sinusoidal extension process and the linear prediction process may also be performed (with some limitations) in the real valued QMF domain wherein the complex parameter ⁇ is replaced with a real valued parameter that describes e.g. a ratio between two TF tile values of the first frame 20A.
- the sinusoidal extension and linear prediction process will now be described in detail. In this description it is assumed that each technique process in a specific frequency band ⁇ ⁇ but it is understood that the processes may operate in an analogous manner in a plurality of frequency bands.
- the sinusoidal extension process for generating TF tile values of a replacement frame 20B’ based on at least one first frame 20A is illustrated schematically.
- the sinusoidal extension process is performed by the frame replacement module to generate TF tile values of one or more frequency bands of a replacement frame.
- the first frame 20A is a valid frame or a replacement frame for an invalid preceding first frame.
- the subsequent second frame is an invalid frame wherein the TF tile values of the replacement frame 20B’ for the second frame should be generated.
- the first frame 20A features a plurality of frequency bands ⁇ ⁇ , comprising a specific frequency band ⁇ ⁇ , and CQMF samples ⁇ ranging from 0 to ⁇ ⁇ 1.
- the second frame and the replacement frame 20B’ have the same general structure as the first frame 20A. Since the second frame is invalid a set of TF tile values of a replacement frame 20B’ are to be generated.
- the sinusoidal extension process assumes that for the specific frequency band ⁇ ⁇ there is a sinusoidal signal component in the first frame 20A.
- a sinusoidal signal component ⁇ ( ⁇ ) can be modelled as wherein ⁇ is a magnitude parameter, ⁇ ⁇ is the frequency of the sinusoidal signal component, ⁇ ⁇ is the sampling frequency, ⁇ ⁇ is a phase offset at the start of the first frame 20A and ⁇ denotes the QMF sample index.
- generation of a QMF domain representation of a replacement frame 20B’ may be performed based the assumption that the same sinusoidal signal component will continue also into the second frame. That is, the replacement frame 20B’ for the second frame can be generated under the assumption that the sinusoidal signal component ⁇ continues into the second frame (and therefore exists in the replacement frame 20B’).
- the extension of the sinusoidal signal component ⁇ into the replacement frame 20B’ for the frame is denoted ⁇ ( ⁇ ) where ⁇ in this context denotes the QMF sample index of the replacement frame 20B’ of the second frame.
- the sinusoidal extension of a QMF domain representation of a frame into the replacement frame 20B’ may be determined as wherein ⁇ ⁇ denotes the phase of the sinusoidal component in the last time domain sample ⁇ ⁇ 1 of the first frame 20A and ⁇ is a phase parameter describing the phase difference between subsequent time domain samples in the first frame 20A i.e. a by-sample phase evolution parameter.
- each TF tile 21 comprises a complex TF tile value, and not a real number as is the case for real valued QMF bank domains or for time domain audio signals.
- a complex sinusoidal signal component may be identified for each frequency band individually.
- the extension of the complex sinusoidal signal component into the replacement frame 20B’ may be expressed as wherein ⁇ ⁇ is a complex valued end parameter having a magnitude equal to the sinusoidal magnitude ⁇ of above and a phase which corresponds to of above.
- the complex sinusoidal signal component described in equation 6 has a by-sample phase evolution which can be described in its entirely by a single complex evolution parameter ⁇
- phase evolution from one CQMF sample to the next is assumed to be constant, meaning the phase of any TF tile value in a replacement frame 20B’ for the invalid second frame may be determined using the complex evolution parameter ⁇ .
- the TF tile values ⁇ ( ⁇ ) of the replacement frame 20B’ for the second frame may be generated by multiplying the end parameter ⁇ ⁇ with ⁇ raised to the ( ⁇ + 1)-th power.
- equation 9 which generates a subsequent TF tile value based on a preceding TF tile value
- a set of TF tile values for a replacement frame 20B’ for the second frame may be generated based on ⁇ ⁇ and/or ⁇ (including ⁇ ). The most accurate sinusoidal extension is achieved if both ⁇ ⁇ and ⁇ are known but it is envisaged that either one is sufficient to achieve TF tile value generation.
- a default value for ⁇ may be used.
- ⁇ ⁇ is obtained by setting ⁇ ⁇ to the center frequency of the specific frequency band. For example, for a critically sampled frequency band of 400 Hz bandwidth, a suitable choice of ⁇ ⁇ would be 200Hz. While this may not correspond to the sinusoidal component of the frequency band this may be a sufficient approximation. On the other hand, if only ⁇ is known a default value of ⁇ ⁇ may be used. While this may not correspond to the TF tile value of the last CQMF sample of the first frame a retention of the frequency of the sinusoidal component as described by ⁇ may be sufficient to form an acceptable replacement frame.
- a vector with ⁇ elements is determined, each element corresponding to an index ⁇ ranging from 0 to ⁇ ⁇ 1 for the replacement frame 20B’.
- Each of the N elements in vector are proportional (e.g. equal to) to ⁇ ⁇ whereby the set of TF tile values for the replacement frame 20B’ may be generated by multiplying a TF tile value (e.g. the end parameter a ⁇ ) of the first frame 20A with the vector.
- the phase parameter ⁇ can be determined by various methods.
- phase parameter ⁇ determined as the difference in phase between the two TF tile values.
- the phase of an individual TF tile value ⁇ ( ⁇ ) may be determined by taking the arcus tangent of the ratio between the imaginary part of the TF tile value ⁇ ( ⁇ ) and the real part of the TF tile value ⁇ ( ⁇ ) . Since implementations of the arcus tangent function (e.g. atan2) returns the phase modulo 2 ⁇ the phase may be unwrapped to yield the unwrapped phase ⁇ ( ⁇ ).
- phase parameter ⁇ may be determined as the mean phase difference between TF tile values of two (e.g. consecutive) CQMF samples in a phase context window ⁇ ⁇ .
- the phase context window ⁇ ⁇ may span a plurality of consecutive CQMF samples of the first frame 20A.
- the phase context window ⁇ ⁇ spans all CQMF samples of the first frame 20A or a subset (e.g. a proper or strict subset) of the CQMF samples of the first frame 20A.
- the phase context window spans CQMF samples of more than one frame, such as at least two first frames which are subsequent to each other but preceding the second frame which is to be replaced with the replacement frame 20B’.
- the TF tile value generation processes defined herein are not limited to analysis of just the directly preceding first frame, and a plurality of preceding first frames (being valid frames or replacement frames for invalid frames) may be analyzed to reduce statistical fluctuations. [143] Since the CQMF samples that are closer to the temporal end of first frame 20A (i.e.
- the mean phase parameter ⁇ ⁇ may be calculated by summing the (unwrapped) phase difference between TF tile values of consecutive CQMF samples and dividing by the number of CQMF samples in the phase context window ⁇ ⁇ . That is, if e.g. the phase context window ⁇ ⁇ is the entire first valid frame 20A with ⁇ CQMF samples the mean phase parameter ⁇ ⁇ may be determined as where ⁇ ranges from 0 to ⁇ ⁇ 2.
- each TF tile value is available in polar coordinates (i.e. each TF tile value is represented with a magnitude and an unwrapped phase value) or each TF tile value is converted into polar coordinates.
- ⁇ ( ⁇ ) representing the magnitude of the TF tile value
- ⁇ ( ⁇ ) representing the unwrapped phase of the TF tile value.
- multiple TF tile values in the first frame 20A are analyzed to determine an expected value of ⁇ ( ⁇ ⁇ 1) which mitigates any statistical deviations wherein the expected value of ⁇ ( ⁇ ⁇ 1) is used as ⁇ ⁇ .
- a linear predictor (as will be described below) may be used to predict a value of ⁇ ( ⁇ ⁇ 1) based on one or more preceding TF tile values in the first frame 20A.
- the expected value of ⁇ ( ⁇ ⁇ 1) is identified as the predicted value of ⁇ ( ⁇ ⁇ 1) using the linear predictor and the observed value of ⁇ ( ⁇ ⁇ 1) of the first frame 20A.
- the expected value of ⁇ ( ⁇ ⁇ 1) may be determined based on the observed value of ⁇ ( ⁇ ⁇ 1).
- the expected value of ⁇ ( ⁇ ⁇ 1) may be determined by a smoothing operation which removes statistical fluctuations (e.g. noise).
- a smoothing operation may e.g. be a regression based on CQMF sample ⁇ ( ⁇ ⁇ 1) and one or more preceding CQMF samples.
- a useful further improvement of the sinusoidal extension process described above is to consider the case of decaying or increasing magnitudes.
- the magnitude parameter ⁇ determines the magnitude parameter ⁇ in equations 10 and 11 above, namely determining the mean magnitude ratio between subsequent TF tile values in a magnitude context window ⁇ ⁇ .
- the magnitude context window ⁇ ⁇ may be of the same or different size as the phase context window ⁇ ⁇ .
- ⁇ can also be calculated as geometric mean (rather than arithmetic mean) across the magnitude context window, which yields [157] Similar to the phase parameter ⁇ it is understood that the mean magnitude parameter may be replaced with the median or mode magnitude parameter determined in the magnitude context window. [158] In some implementations, a magnitude parameter ⁇ that exceeds one may be undesired since this leads to an exponential increase in magnitude which may be problematic for the resulting audio quality and/or lead to overloading effects. Hereby, an upper limit of 1 for the magnitude parameter ⁇ may be implemented such that the magnitude parameter ⁇ is equal to or smaller than 1. In practice, this limitation allows the magnitude of the sinusoidal extension into the replacement frame 20B of the second frame 20B to be sustained or decreasing, but not increasing.
- the sinusoidal extension process extends sinusoidal signal components of the first frame 20A to generate the TF tile values of a replacement frame 20B’ for the second frame 20B.
- the frequency spectrum of a purely sinusoidal signal component will be in the form of a single spectral peak located at the frequency ⁇ ⁇ of the sinusoidal signal component. That is, the extended sinusoidal signal component is a single tone of a single frequency which is a very narrowband signal. In theory, a single frequency peak may be described as having zero bandwidth. In some implementations, a narrow band (single frequency) generated signal is desired, especially if the signal represented by the first frame 20A also is narrowband and has a high degree of tonality.
- the generated TF tile values of the replacement frame 20B’ for the second frame 20B will then be a plausible and accurate extension of the tonal signal in the first frame 20A.
- the signal of the first frame 20A has a low degree of tonality and features a non-zero bandwidth.
- many types of audio signals will feature a spectral maximum at a specific frequency while being surrounded by additional frequency components forming a non-zero bandwidth.
- the sinusoidal extension process may be modified to implement one or both of a phase perturbation parameter ⁇ ( ⁇ ) and a magnitude perturbation parameter ⁇ ( ⁇ ).
- phase perturbation parameter ⁇ ( ⁇ ) and the magnitude perturbation parameter ⁇ ( ⁇ ) are modify the phase parameter ⁇ and magnitude parameter ⁇ determined by the sinusoidal extension process so as to generate a non-zero bandwidth extension ⁇ ( ⁇ ) that more closely resembles the non-zero bandwidth of the signal represented by the TF tile values ⁇ ( ⁇ ) of the first frame 20A.
- the phase perturbation parameter and the magnitude perturbation parameter ⁇ ( ⁇ ) may be described as an additive perturbation noise that may be determined individually for each generated TF tile value or drawn from a random or pseudorandom sequence.
- phase perturbation parameter ⁇ ( ⁇ ) may be determined and used to perturb the phase parameter ⁇ to form a perturbed phase parameter ⁇ ⁇ ( ⁇ ) that is used to generate the TF tile values for the replacement frame 20B’ instead of the phase parameter.
- the phase perturbation parameter ⁇ ( ⁇ ) may be determined for each CQMF sample ⁇ for the replacement frame 20B’ by drawing a random parameter value from a distribution having a predetermined standard deviation ⁇ ⁇ .
- the predetermined standard deviation ⁇ ⁇ may be based on the standard deviation of the by-sample phase difference ⁇ ( ⁇ ) of the valid frame 20A for the full first frame 20A or in a context window spanning at least a portion of the at least one first frame 20A (that may be equal to or different from the phase context window).
- the predetermined standard deviation ⁇ ⁇ may be equal to or proportional to the standard deviation of the by-sample phase difference ⁇ ( ⁇ ) for all CQMF samples ⁇ of the first frame 20A (or of a context window thereof).
- the phase perturbation parameter ⁇ ( ⁇ ) for one TF tile value to be generated is obtained as the difference between a by-sample phase difference ⁇ ( ⁇ ) and the mean by-sample phase difference ⁇ ⁇ from the first frame 20A (or a context window thereof).
- the difference between the by-sample phase difference ⁇ ( ⁇ ) of another pair of TF tile values and the mean by-sample phase difference ⁇ ⁇ in the first frame 20A is calculated.
- phase perturbation parameters ⁇ ( ⁇ ) for the generation of TF tile values in the replacement frame 20B’ of the second frame that inherently will feature the same standard deviation without operation of a random number generator.
- the difference between a by-sample phase difference ⁇ ( ⁇ ) and the mean by-sample phase difference ⁇ ⁇ from the valid frame 20A (or a context window thereof) is scaled prior to being used for generating the TF tile values of the replacement frame 20B’ for the second frame.
- phase perturbation parameter ⁇ ( ⁇ ) may be determined for a subset of the TF tile value pairs in the first frame 20A and this set of phase perturbation parameters ⁇ ( ⁇ ) may then be repeated (e.g. cycled) for the TF tile values to be generated for the replacement frame 20B’ for the second frame.
- a magnitude perturbation parameter ⁇ ( ⁇ ) may be determined for each TF tile value to be generated and used to perturb the magnitude parameter ⁇ to form a perturbed phase parameter ⁇ ⁇ ( ⁇ ) that is used generate TF tile values instead of the original magnitude parameter ⁇ .
- the magnitude perturbation parameter ⁇ ( ⁇ ) may be determined for each CQMF sample of the second frame by drawing a random parameter value from a distribution having a predetermined standard deviation ⁇ ⁇ .
- the predetermined standard deviation ⁇ ⁇ may be based on the standard deviation of the by-sample magnitude ratio ⁇ ( ⁇ ) of the first frame 20A or in a context window of the first frame 20A (that may be equal to or different from the magnitude context window ⁇ ⁇ ).
- the predetermined standard deviation ⁇ ⁇ may be equal to or proportional to the standard deviation of the by-sample magnitude ratio ⁇ ( ⁇ ) of the first frame 20A (or of a context window thereof).
- a magnitude perturbation parameter ⁇ ( ⁇ ) may be determined by calculating a difference between the relative magnitude change of two neighboring CQMF samples in the valid frame 20A and the mean magnitude parameter ⁇ ⁇ .
- additional magnitude perturbation parameters ⁇ ( ⁇ ) may be determined by calculating the relative magnitude change ⁇ ( ⁇ ) of different pairs and the mean magnitude parameter ⁇ ⁇ .
- the calculation the of magnitude perturbation parameter ⁇ ( ⁇ ) is performed in the logarithmic domain and in the logarithmic domain the magnitude perturbation parameters ⁇ ( ⁇ ) may be determined analogously to the phase perturbation parameters ⁇ ( ⁇ ) as described above.
- Figs.6a-d illustrate schematically the linear predictor process for generating a set of TF tile values for a replacement frame 20B’ of the second frame.
- the linear predictor process involves defining a linear predictor with complex predictor coefficients ⁇ ⁇ that recursively generates TF tile values of one or more frequency bands of a replacement frame.
- the linear predictor process is performed by the frame replacement module to generate TF tile values of one or more frequency bands of a replacement frame.
- a preceding first frame 20A (or a generated replacement frame therefore) is used to generate TF tile values of a replacement frame 20B’ for a subsequent invalid frame.
- the linear prediction process is a process which is alternative to the sinusoidal extension process.
- the TF tiles of the replacement frame 20B’ may be generated by the frame replacement module using either the sinusoidal extension process or the sinusoidal extension process.
- the TF tile values of a specific frequency band ⁇ ⁇ of the first frame 20A are analyzed.
- the TF tile values of at least two CQMF samples of the first frame are analyzed.
- the TF tile values are analyzed to determine a linear predictor that generates a predicted TF tile value ⁇ ( ⁇ ) based on one or more preceding TF tile values ⁇ ( ⁇ ⁇ ⁇ ), ⁇ > 0.
- a linear predictor of the first order generates a predicted TF tile value ⁇ ( ⁇ ) based on one preceding TF tile value, e.g. ⁇ ( ⁇ ⁇ 1)
- a linear predictor of the second order generates a predicted TF tile value ⁇ ( ⁇ ) based on two preceding TF tile values, e.g. ⁇ ( ⁇ ⁇ 1) and ⁇ ( ⁇ ⁇ 2)
- a linear predictor of the third order generates a predicted TF tile value ⁇ ( ⁇ ) based on three preceding TF tile values and so on for the fourth and higher order linear predictors.
- the linear predictor is of the first, second or third order to keep the computational complexity low.
- the linear predictor since the linear predictor according to some implementations operates in a single specific frequency band of limited bandwidth (e.g. around 400 Hz for the CLDFB, or at least less than 1 kHz) the order of the linear predictor can be low, e.g. only of the first or second order.
- the linear predictor may be defined so as to minimize a prediction error for all TF tile values in the specific frequency band ⁇ ⁇ in the first frame 20A (or a context window thereof).
- the prediction error is the expected value of the mean squared error between a predicted TF tile value ⁇ ( ⁇ ) and the TF tile value ⁇ ( ⁇ ) of the first frame 20A for all TF tile values in the first frame 20A (or in a predictor context window ⁇ ⁇ of the first frame 20A or in a predictor context window ⁇ ⁇ spanning multiple frames).
- the prediction error may be expressed as E [( ⁇ ⁇ ⁇ ) ⁇ ] .
- This type of linear predictor generates a TF tile value based on at least one preceding, and immediately adjacent TF tile value.
- the linear predictor comprises as many predictor parameters ⁇ ⁇ as the predictor order. That is, a first order linear predictor comprises only the parameter ⁇ ⁇ whereas a second order linear predictor comprise predictor parameters ⁇ ⁇ and ⁇ ⁇ .
- the one or more predictor coefficients ⁇ ⁇ are calculated according to Levinson-Durbin based on an autocorrelation sequence ⁇ ⁇ ( ⁇ ) of the TF tile values of the first frame 20A.
- the single complex predictor coefficient ⁇ ⁇ may then be extracted simply as [181]
- the linear predictor is used to predict the tile values of the second frame 20B using, as a starting condition, one or more TF tile values of the first frame 20A.
- FIG.6a and 6b illustrate schematically how a linear predictor of the third order from equation 21 has been defined to generate accurate predictions of the TF tile values in the first frame 20A.
- the third order predictor generates a prediction of the TF tile value in the specific band ⁇ ⁇ for CQMF sample ⁇ ⁇ 2 using the three complex predictor parameters ⁇ ⁇ , ⁇ ⁇ , ⁇ ⁇ operating on the TF tile values of preceding CQMF samples ⁇ ⁇ 3, ⁇ ⁇ 4, ⁇ ⁇ 5.
- the third order predictor may generate a prediction of a TF tile value in the specific band ⁇ ⁇ for CQMF sample ⁇ ⁇ 1 using the three complex predictor parameters ⁇ ⁇ , ⁇ ⁇ , ⁇ ⁇ operating on the TF tile values of preceding CQMF samples ⁇ ⁇ 2, ⁇ ⁇ 3, ⁇ ⁇ 4 as illustrated in FIG.6b.
- FIG.6a and 6b illustrate the task the linear predictor has been configured to perform (namely to generate accurate predictions of the TF tile values first frame 20A) the same predictor is used to generate the TF tile values of the specific frequency band ⁇ ⁇ in the replacement frame 20B’ of the second frame.
- the linear predictor may continue to operate through the TF tile values of the replacement frame 20B’ for the second frame 20B to generate all TF tile values of the replacement frame 20B’, in the specific frequency band ⁇ ⁇ .
- the reason for this behavior is that the poles of the inference filter defined by the linear predictor are often inside the unit circle whereas only poles located exactly on the unit circle would allow generating TF tile values of sustained magnitude.
- a replacement frame 20B’ with decaying TF tile values may result in a decoded time domain audio signal which is perceived as unsteady.
- the calculated complex predictor parameter ⁇ ⁇ may be modified by setting its magnitude to one, or at least a predetermined value close to one if
- the modified complex predictor parameter ⁇ ⁇ thus obtained may then be used by the linear predictor instead of the calculated complex predictor parameter ⁇ ⁇ when generating TF tile values for a replacement frame for the second frame.
- the complex predictor parameter ⁇ ⁇ of a first order predictor can be used to determine the phase evolution parameter ⁇ (as the phase of the complex predictor parameter ⁇ ⁇ ) and/or the magnitude evolution parameter ⁇ (as the magnitude of the complex predictor parameter ⁇ ⁇ ).
- the complex predictor parameter ⁇ ⁇ therefore introduces another method for determining the phase evolution parameter ⁇ and/or the magnitude evolution parameter ⁇ for the sinusoidal extension process by first determining the complex predictor parameter ⁇ ⁇ and then calculating ⁇ and/or ⁇ based on ⁇ ⁇ .
- the opposite also applies, meaning that ⁇ ⁇ can be calculated from ⁇ (or ⁇ and ⁇ ).
- FIG.7 illustrates a first frame 20A, a generated a replacement frame 20B’ and a substitution frame 20A’.
- the TF tile values of the first frame 20A are used to generate TF tile values for the replacement frame 20B’ using the sinusoidal extension process or linear prediction process in each frequency band. Additionally, the TF tile values of the replacement frame 20B’ are further modified by weighted addition of a corresponding TF tile value of the substitution frame 20A’. [192] In FIG.7 the TF tile values of frequency band ⁇ ⁇ in the replacement frame 20B’ are generated using a linear prediction process as described above. The generated TF tile values in the replacement frame 20B’ are further modified as by addition of a corresponding TF tile value in a substitute frame 20A’. For example, the substitute frame 20A’ is a copy of the first frame 20A.
- the TF tile values of the replacement frame 20B’ for the second frame 20B may be formed as a weighted sum of generated TF tile values (using sinusoidal extension or linear prediction) and a substitution frame 20A’.
- the substitution frame 20A’ is equal to the first frame 20A or a circularly shifted version thereof.
- each CQMF sample ⁇ ( ⁇ ) in the replacement frame 20B’ for the second frame 20B may be generated as wherein ⁇ ⁇ ( ⁇ ) is the corresponding CQMF sample (i.e. of the same frequency band and index ⁇ ) of the substitution frame 20A’.
- the substitution frame 20A’ is a circularly shifted version of the first frame 20A.
- FIG.8 illustrates a replacement frame 20B’ which has been generated to replace a second frame, a subsequent third valid frame 20C and a replacement frame 20C’ which has been generated to replace the third valid frame 20C.
- Generation of a replacement frame may be performed for all invalid frames of a frame sequence. Additionally, generation of replacement frames may also be performed for valid frames during certain circumstances. For example, when a replacement frame 20B’ has been generated to replace a second invalid frame and the subsequent third frame 20C is a valid frame, a replacement frame 20C’ for the third frame 20C may be generated and used as a replacement frame 20C’ for the third frame 20C.
- the purpose of replacing a valid frame, which follows an invalid frame, with a replacement frame may be to smoothly transition from a replacement frame (which is an approximation of an original frame) into a valid frame.
- the TF tile values ⁇ ( ⁇ ) of the replacement frame 20B’ have been generated using the sinusoidal extension process and/or the linear prediction process as described above.
- the third frame 20C is a valid frame that is subsequent to the replacement frame 20B’ for the second frame. To provide a high quality transition from the replacement frame 20B’ to the third frame 20C various techniques may be used to avoid signal discontinuities which would be disruptive for listeners.
- a crossfading technique is applied to form the replacement frame 20C’ for the third frame to transition from the replacement frame 20B’ of the second frame to TF tile values the third frame 20C.
- the crossfading technique involves generating extension TF tile values of the replacement frame 20B’ for the second frame that go beyond the ⁇ ⁇ 1 CQMF sample of the replacement frame 20B’ for the second frame and, using these extension CQMF samples that go beyond the ⁇ ⁇ 1 CQMF sample, the replacement frame 20C’ for the third frame 20C is achieved with a crossfading into TF tile values the third frame 20C over a crossfade region.
- the ⁇ temporally earliest TF tile values of the replacement frame 20C’ for the third frame 20C are hereby obtained as a crossfade of generated extension TF tile values ⁇ ( ⁇ + ⁇ ) that go beyond CQMF sample ⁇ ⁇ 1 of the replacement frame 20B’ of the second frame and the TF tile value ⁇ ⁇ ( ⁇ ) of the valid third frame 20C.
- the crossfading interpolation parameter ⁇ ⁇ may be adjusted so as to decrease with increasing ⁇ . This results in a replacement frame 20C’ with a gradual boosting of the first ⁇ TF tile values of the third frame 20C and a matching gradual attenuation of the generated extension TF tile values ⁇ ( ⁇ + ⁇ ) .
- the TF tile values ⁇ are set to that of the third frame 20C.
- the TF tile values of the third frame 20C that are temporally later than the crossfade region may be referred as the TF tile values of a trailing region which are equal to the corresponding TF tile values of the third frame 20C.
- the crossfading interpolation parameter ⁇ ⁇ may be adjusted over the ⁇ extension TF tile values to achieve a desired crossfading profile it is further possible to make use of a decaying linear prediction parameter ⁇ ⁇ or a sinusoidal extension with a magnitude parameter ⁇ which is smaller than 1.
- the crossfading duration will not be governed by a predetermined number ⁇ of CQMF samples but by the decay rate of the linear prediction parameter ⁇ ⁇ or the magnitude evolution parameter ⁇ .
- the linear predictor process or the sinusoidal extension process is used to generate the extension TF tile values that go beyond CQMF sample ⁇ ⁇ 1 of the second frame 20B whereby these extension TF tile values are crossfaded with the TF tile values of the third frame 20C with a crossfade weighted sum which emphasizes less and less of the extension TF tile values and wherein the crossfade weighting factor corresponds to the decay of the magnitude of the prediction parameter ⁇ ⁇ or the magnitude parameter ⁇ .
- ⁇ ⁇ is replaced with the magnitude evolution parameter ⁇ if the sinusoidal extension process is used instead of the linear predictor process.
- the length of the crossfade region will therefore depend on the magnitude of the prediction parameter ⁇ ⁇ or the magnitude parameter ⁇ wherein a ⁇ ⁇ or ⁇ which is close to one will result in a long crossfade region and a ⁇ ⁇ or ⁇ which is closer to zero will result in a shorter crossfade region.
- the replacement frame 20B’ for the third frame 20C is generated according to equation 26 for the ⁇ first CQMF samples or until
- updated extension a first invalid frame is replaced with a replacement frame in accordance with the above using a preceding valid frame.
- the replacement frame for the first invalid frame is treated like a valid frame and the processed described above are repeated by generating a replacement frame for the second invalid frame based on the (generated) TF tile values of replacement frame for the first invalid frame.
- phase evolution parameter ⁇ and/or magnitude evolution parameter ⁇ are calculated based on the replacement frame for the first invalid frame and/or the complex prediction coefficients ⁇ ⁇ are calculated from the replacement frame for the first invalid frame.
- phase evolution parameter ⁇ and/or the magnitude evolution parameter ⁇ used to perform sinusoidal extension to generate the TF tile values of the replacement frame for the second invalid frame may differ from the ⁇ and/or ⁇ that were used to perform sinusoidal extension to generate the TF tile values for the replacement frame for the first invalid frame.
- phase or magnitude perturbation parameters which may be updated accordingly.
- the one or more complex prediction coefficients ⁇ ⁇ used to generate the TF tile values of the replacement frame for the second invalid frame will differ from the one or more complex prediction coefficients ⁇ ⁇ that were used to generate the TF tile values for the replacement frame for the first invalid frame.
- the linear prediction or sinusoidal extension process continues from the replacement frame of the first invalid frame into the replacement frame for the second invalid frame without updating the parameters ⁇ , ⁇ and ⁇ ⁇ .
- the sinusoidal extension process or the linear prediction process may in principle continue continuously and generate more TF tile values beyond CQMF sample ⁇ ⁇ 1 a frame.
- the linear predictor process or sinusoidal extension process continues to generate TF tile values beyond the ⁇ CQMF samples constituting the replacement frame for the first invalid frame whereby the next ⁇ CQMF samples are used as the replacement frame for the second invalid frame.
- the process may continue to generate the TF tile values of CQMF samples forming a third, fourth and fifth replacement frame for the corresponding invalid frames as well, with each batch of ⁇ TF tile values generated with the sinusoidal extension process or linear prediction process constituting a new replacement frame.
- the complex prediction coefficients ⁇ ⁇ or the phase evolution parameter ⁇ and/or the magnitude evolution parameter ⁇ will therefore be retained throughout the generation of TF tile values of all replacement frames.
- FIG.9 shows a schematic block diagram of an example electronic device or architecture 200 (e.g., an apparatus 200) suitable for implementing example embodiments of the present disclosure.
- Architecture 200 includes but is not limited to servers and client devices, systems, modules and methods as described in reference to Figs.1a, 1b and 3.
- the architecture 200 includes central processing unit (CPU) 201 which is capable of performing various processes in accordance with a program stored in, for example, read only memory (ROM) 202 or a program loaded from, for example, storage unit 208 to random access memory (RAM) 203.
- CPU central processing unit
- ROM read only memory
- RAM random access memory
- the CPU 201 may be, for example, an electronic processor 201, which may include one or more processor cores, and in some examples the processor 201 may be multiple processors.
- RAM 203 the data required when CPU 201 performs the various processes is also stored, as required.
- CPU 201, ROM 202 and RAM 203 are connected to one another via bus 204.
- I/O interface 205 is also connected to bus 204.
- I/O interface 205 The following components are connected to I/O interface 205: input unit 206, that may include a keyboard, a mouse, or the like; output unit 207 that may include a display such as a liquid crystal display (LCD) and one or more speakers; storage unit 208 including a hard disk, or another suitable storage device; and communication unit 209 which may include a network interface card such as a network card (e.g., wired or wireless).
- input unit 206 includes one or more microphones in different positions (depending on the host device) enabling capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).
- output unit 207 include systems with various number of speakers.
- Output unit 207 (depending on the capabilities of the host device) can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats).
- communication unit 209 is configured to communicate with other devices (e.g., via a network).
- Drive 210 is also connected to I/O interface 205, as required.
- Removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a flash drive or another suitable removable medium is mounted on drive 210, so that a computer program read therefrom is installed into storage unit 208, as required.
- apparatus 200 is described as including the above-described components, in real applications, it is possible to add, remove, and/or replace some of these components and all these modifications or alteration all fall within the scope of the present disclosure.
- the processes described above may be implemented as computer software programs or on a computer-readable storage medium.
- embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program including program code for performing methods.
- the computer program may be downloaded and mounted from the network via the communication unit 209, and/or installed from the removable medium 211, as shown in FIG.9.
- control circuitry e.g., CPU 201 in combination with other components of FIG.9
- the control circuitry may be performing the actions described in this disclosure.
- Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, a processor and/or other computing device(s), which may include control circuitry.
- a machine-readable medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
- the machine-readable medium may be a machine- readable signal medium or a machine-readable storage medium.
- a machine-readable medium may be non-transitory and may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.
- machine-readable storage medium More specific examples of the machine-readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
- Computer program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages.
- These computer program codes may be provided to one or more processors of a general-purpose computer, special purpose computer, or other programmable data processing apparatus that has control circuitry, such that the program codes, when executed by one or more processors of the computer or other programmable data processing apparatus, cause the functions/operations specified in the flowcharts and/or block diagrams to be implemented.
- the program code may execute entirely on a computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server or distributed over one or more remote computers and/or servers.
- Embodiments described herein perform generation of TF tile values for one or more replacement frames for replacing one or more frames identified as invalid and/or for replacing one or more valid frame when the preceding frame was valid.
- the methods and processes described may be implemented in an iterative manner wherein the methods and processes are repeated for each frame in a sequence of frames to generate replacement frames, or refrain from generating replacement frames, based on the validity of each frame in the frame sequence.
Landscapes
- Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US202363457320P | 2023-04-05 | 2023-04-05 | |
| PCT/EP2024/057822 WO2024208612A1 (en) | 2023-04-05 | 2024-03-22 | Method for performing packet loss concealment in complex filter bank domain |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| EP4690187A1 true EP4690187A1 (en) | 2026-02-11 |
Family
ID=90468691
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP24713964.5A Pending EP4690187A1 (en) | 2023-04-05 | 2024-03-22 | Method for performing packet loss concealment in complex filter bank domain |
Country Status (5)
| Country | Link |
|---|---|
| EP (1) | EP4690187A1 (en) |
| KR (1) | KR20260018797A (en) |
| CN (1) | CN121444166A (en) |
| AU (1) | AU2024251474A1 (en) |
| WO (1) | WO2024208612A1 (en) |
-
2024
- 2024-03-22 KR KR1020257037023A patent/KR20260018797A/en active Pending
- 2024-03-22 CN CN202480031776.1A patent/CN121444166A/en active Pending
- 2024-03-22 EP EP24713964.5A patent/EP4690187A1/en active Pending
- 2024-03-22 WO PCT/EP2024/057822 patent/WO2024208612A1/en not_active Ceased
- 2024-03-22 AU AU2024251474A patent/AU2024251474A1/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| KR20260018797A (en) | 2026-02-09 |
| AU2024251474A1 (en) | 2025-10-23 |
| CN121444166A (en) | 2026-01-30 |
| WO2024208612A1 (en) | 2024-10-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7004773B2 (en) | Packet loss compensation device and packet loss compensation method, as well as voice processing system | |
| JP6728416B2 (en) | Method for parametric multi-channel encoding | |
| JP5165559B2 (en) | Audio codec post filter | |
| RU2432625C2 (en) | Synthesis of lost blocks of digital audio signal with pitch period correction | |
| US8738385B2 (en) | Pitch-based pre-filtering and post-filtering for compression of audio signals | |
| TWI480857B (en) | Audio codec using noise synthesis during inactive phases | |
| US10049674B2 (en) | Method and apparatus for evaluating voice quality | |
| EP2502230B1 (en) | Improved excitation signal bandwidth extension | |
| US12148434B2 (en) | Audio frame loss concealment | |
| KR102510685B1 (en) | Improved frequency band extension in an audio signal decoder | |
| EP1526507A1 (en) | Method for packet loss and/or frame erasure concealment in a voice communication system | |
| US20090180531A1 (en) | codec with plc capabilities | |
| CN114550732A (en) | Coding and decoding method and related device for high-frequency audio signal | |
| US10249317B2 (en) | Estimating noise of an audio signal in a LOG2-domain | |
| US8073687B2 (en) | Audio regeneration method | |
| US20250124934A1 (en) | Multi-lag format for audio coding | |
| JP2017515155A (en) | Improved frame erasure correction using speech information | |
| AU2024251474A1 (en) | Method for performing packet loss concealment in complex filter bank domain | |
| US20250191598A1 (en) | High frequency reconstruction using neural network system | |
| WO2024017800A1 (en) | Neural network based signal processing | |
| EP4139919B1 (en) | Low cost adaptation of bass post-filter | |
| HK40120551A (en) | Multi-lag format for audio coding | |
| HK40070387B (en) | Method for encoding and decoding high-frequency audio signal, and related apparatus | |
| HK40070387A (en) | Method for encoding and decoding high-frequency audio signal, and related apparatus | |
| US20120143614A1 (en) | Encoding apparatus, encoding method, decoding apparatus, decoding method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
| PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
| 17P | Request for examination filed |
Effective date: 20251030 |
|
| AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
| P01 | Opt-out of the competence of the unified patent court (upc) registered |
Free format text: CASE NUMBER: UPC_APP_0012056_4690187/2026 Effective date: 20260408 |