EP3319087A1 - Filling of non-coded sub-vectors in transform coded audio signals - Google Patents
Filling of non-coded sub-vectors in transform coded audio signals Download PDFInfo
- Publication number
- EP3319087A1 EP3319087A1 EP17208522.7A EP17208522A EP3319087A1 EP 3319087 A1 EP3319087 A1 EP 3319087A1 EP 17208522 A EP17208522 A EP 17208522A EP 3319087 A1 EP3319087 A1 EP 3319087A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- vectors
- residual sub
- sub
- virtual codebook
- coded
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 239000013598 vector Substances 0.000 title claims abstract description 169
- 230000005236 sound signal Effects 0.000 title claims abstract description 15
- 238000000034 method Methods 0.000 claims description 9
- 230000006835 compression Effects 0.000 claims description 4
- 238000007906 compression Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 25
- 238000001228 spectrum Methods 0.000 description 23
- 238000005516 engineering process Methods 0.000 description 13
- 239000000945 filler Substances 0.000 description 13
- 238000013139 quantization Methods 0.000 description 12
- 230000007704 transition Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000003595 spectral effect Effects 0.000 description 2
- 230000008685 targeting Effects 0.000 description 2
- 230000002238 attenuated effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008450 motivation Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
- G10L19/0212—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders using orthogonal transformation
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
- G10L19/028—Noise substitution, i.e. substituting non-tonal spectral components by noisy source
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
- G10L19/032—Quantisation or dequantisation of spectral components
- G10L19/038—Vector quantisation, e.g. TwinVQ audio
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/038—Speech enhancement, e.g. noise reduction or echo cancellation using band spreading techniques
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L2019/0001—Codebooks
- G10L2019/0007—Codebook element generation
Definitions
- the present technology relates to coding of audio signals, and especially to filling of non-coded sub-vectors in transform coded audio signals.
- FIG. 1 A typical encoder/decoder system based on transform coding is illustrated in Fig. 1 .
- a drawback of the conventional noise-fill scheme e.g. as in [1], is that it in step H creates audible distortion in the reconstructed audio signal, when used with the FPC scheme.
- a general object is an improved filling of non-coded residual sub-vectors of a transform coded audio signal.
- Another object is generation of virtual codebooks used to fill the non-coded residual sub-vectors.
- a first aspect of the present technology involves an apparatus for filling non-coded residual sub-vectors of a transform coded audio signal.
- the apparatus comprises:
- a second aspect of the present technology involves an audio decoder comprising the apparatus in accordance with the first aspect.
- a third aspect of the present technology involves a user equipment comprising the audio decoder in accordance with the second aspect.
- a fourth aspect of the present technology involves a method for filling non-coded residual sub-vectors of a transform coded audio signal.
- the method comprises:
- An advantage of the present spectrum filling technology is a perceptual improvement of decoded audio signals compared to conventional noise filling.
- Fig. 1 is a block diagram illustrating a typical transform based audio coding/decoding system.
- An input signal x ( n ) is forwarded to a frequency transformer, for example an MDCT transformer 10, where short audio frames (20-40 ms) are transformed into a frequency domain.
- the resulting frequency domain signal X(k) is divided into multiple bands (sub-vectors SV1, SV2, ...), as illustrated in Fig. 2 .
- the width of the bands increases towards higher frequencies [1].
- the energy of each band is determined in an envelope calculator and quantizer 12. This gives an approximation of the spectrum envelope, as illustrated in Fig. 3 .
- Each sub-vector is normalized into a residual sub-vector in a sub-vector normalizer 14 by scaling with the inverse of the corresponding quantized envelope value (gain).
- a bit allocator 16 assigns bits for quantization of different residual sub-vectors based on envelope energies. Due to a limited bit-budget, some of the sub-vectors are not assigned any bits. This is illustrated in Fig. 4 , where sub-vectors corresponding to envelope gains below a threshold TH are not assigned any bits. Residual sub-vectors are quantized in a sub-vector quantizer 18 according to the assigned bits. Residual quantization can, for example, be performed with the Factorial Pulse Coding (FPC) scheme [2]. Residual sub-vector quantization indices and envelope quantization indices are then transmitted to the decoder over a multiplexer (MUX) 20.
- FPC Factorial Pulse Coding
- the received bit stream is de-multiplexed into residual sub-vector quantization indices and envelope quantization indices in a demultiplexer (DEMUX) 22.
- the residual sub-vector quantization indices are dequantized into residual sub-vectors in a sub-vector dequantizer 24, and the envelope quantization indices are dequantized into envelope gains in an envelope dequantizer 26.
- a bit allocator 28 uses the envelope gains to control the residual sub-vector dequantization.
- Residual sub-vectors with zero bits assigned have not been coded at the encoder, and are instead noise-filled by a noise filler 30 at the decoder. This is achieved by creating a Virtual Codebook (VC) from coded sub-vectors by concatenating the perceptually relevant coefficients of the decoded spectrum ([1] section 8.4.1). Thus, the VC creates content in the non-coded residual sub-vectors.
- VC Virtual Codebook
- the MDCT vector x ⁇ ( n ) is then reconstructed by up-scaling residual sub-vectors with corresponding envelope gains in an envelope shaper 32, and transforming the resulting frequency domain vector X ⁇ ( k ) in an inverse MDCT transformer 34.
- a drawback of the conventional noise-fill scheme described above is that It creates audible distortion in the reconstructed audio signal, when used with the FPC scheme.
- the main reason is that some of the coded vectors may be too sparse, which creates energy mismatch problems in the noise-filled bands. Additionally some of the coded vectors may contain too much structure (color), which leads to perceptual degradations when the noise-fill is performed at high frequencies.
- This step guarantees that there will be no excessive structure (such as periodicity at high-frequencies) in the noise-filled regions.
- the specific form of compressed residual Y ( k ) allows a low complexity in the following steps.
- the value of T may be used to control the amount of compression. This embodiment is also useful for signals that have been coded by an encoder that quantizes symmetrically around 0 but does not include the actual value 0.
- a virtual codebook VC1 is formed by concatenating the remaining or surviving sub-vectors, as illustrated in Fig. 8 . Since the length of the sub-vectors is a multiple of M, the criterion (3) may be used also for longer sub-vectors. In this case the parts that do not fulfill the criterion are rejected.
- a compressed sub-vector is considered “populated” if it contains more that 20-30% of non-zero components.
- a second virtual codebook VC2 is created from the obtained virtual codebook VC1.
- This second virtual codebook VC2 is even more "populated” and is used to fill frequencies above 4.8 kHz (other transition frequencies are of course also possible; typically the transition frequency is between 4 and 6 kHz).
- Fig. 9A-B This combining or merging step is illustrated in Fig. 9A-B . It is noted that the same pair of coefficients Y ( k ), Y ( N - k ) is used twice in the merging process, once in the lower half ( Fig. 9A ) and once in the upper half ( Fig. 9B ).
- Non-coded sub-vectors may be filled by cyclically stepping through the respective virtual codebook, VC1 or VC2 depending on whether the sub-vector to be filled is below or above the transition frequency, and copying the required number of codebook coefficients to the empty sub-vector.
- the codebooks are short and there are many sub-vectors to be filled, the same coefficients will be reused for filling more than one sub-vector.
- An energy adjustment of the filled sub-vectors is preferably performed on a sub-vector basis. It accounts for the fact that after the spectrum filling the residual sub-vectors may not have the expected unit RMS energy.
- a motivation for the perceptual attenuation is that the noise-fill operation often results in significantly different statistics of the residual vector and it is desirable to attenuate such "inaccurate" regions.
- energy adjustment of a particular sub-vector can be adapted to the type of neighboring sub-vectors: If the neighboring regions are coded at high-bitrate, attenuation of the current sub-vector is more aggressive (alpha goes towards zero). If the neighboring regions are coded at a low-bitrate or noise-filled, attenuation of the current sub-vector is limited (alpha goes towards one). This scheme prevents attenuation of large continuous spectral regions, which might lead to audible loudness loss. At the same time if the spectral region to be attenuated is narrow, even a very strong attenuation will not affect the overall loudness.
- the described technology provides improved noise-filling. Perceptual improvements have been measured by means of listening tests. These tests indicate that the spectrum fill procedure described above was preferred by listeners in 83% of the tests while the conventional noise fill procedure was preferred in 17% of the tests.
- Fig. 10 is a block diagram illustrating an example embodiment of a low frequency virtual codebook generator 60.
- Residual sub-vectors are forwarded to a sub-vector compressor 42, which is configured to compress actually coded residual sub-vectors (i.e. sub-vectors that have actually been allocated bits for coding), for example in accordance with equation (1).
- the compressed sub-vectors are forwarded to a sub-vector rejecter 44, which is configured to reject compressed residual sub-vectors that do not fulfill a predetermined sparseness criterion, for example criterion (3).
- the remaining compressed sub-vectors are collected in a sub-vector collector 46, which is configured to concatenate them to form the low frequency virtual codebook VC1.
- Fig. 11 is a block diagram illustrating an example embodiment of a high frequency virtual codebook generator 70.
- Residual sub-vectors are forwarded to a sub-vector compressor 42, which is configured to compress actually coded residual sub-vectors (i.e. sub-vectors that have actually been allocated bits for coding), for example in accordance with equation (1).
- the compressed sub-vectors are forwarded to a sub-vector rejecter 44, which is configured to reject compressed residual sub-vectors that do not fulfill a predetermined sparseness criterion, for example criterion (3).
- the remaining compressed sub-vectors are collected in a sub-vector collector 46, which is configured to concatenate them to form the low frequency virtual codebook VC1.
- the high frequency virtual codebook generator 70 includes the same elements as the low frequency virtual codebook generator 60.
- Coefficients from the low frequency virtual codebook VC1 are forwarded to a coefficient combiner 48, which is configured to combine pairs of coefficients to form the high frequency virtual codebook VC2, for example in accordance with equation (5).
- Fig. 12 is a block diagram illustrating an example embodiment of a spectrum filler 40.
- Residual sub-vectors are forwarded to a sub-vector compressor 42, which is configured to compress actually coded residual sub-vectors (i.e. sub-vectors that have actually been allocated bits for coding), for example in accordance with equation (1).
- the compressed sub-vectors are forwarded to a sub-vector rejecter 44, which is configured to reject compressed residual sub-vectors that do not fulfill a predetermined sparseness criterion, for example criterion (3).
- the remaining compressed sub-vectors are collected in a sub-vector collector 46, which is configured to concatenate them to form a first (low frequency) virtual codebook VC1.
- Coefficients from the first virtual codebook VC1 are forwarded to a coefficient combiner 48, which is configured to combine pairs of coefficients to form a second (high frequency) virtual codebook VC2, for example in accordance with equation (5).
- the spectrum filler 40 includes the same elements as the high frequency virtual codebook generator 70.
- the residual sub-vectors are also forwarded to a sub-vector filler 50, which is configured to fill non-coded residual sub-vectors below a predetermined frequency with coefficients from the first virtual codebook VC1, and to fill non-coded residual sub-vectors above the predetermined frequency with coefficients from the second virtual codebook.
- the spectrum filler 40 also includes an energy adjuster 52 configured to adjust the energy of filled non-coded residual sub-vectors to obtain a perceptual attenuation, as described above.
- Fig. 13 is a block diagram illustrating an example embodiment of a decoder 300 including a spectrum filler 40.
- the general structure of the decoder 300 is the same as of the decoder in Fig. 1 , but with the noise filler 30 replaced by the spectrum filler 40.
- Fig. 14 is a flow chart illustrating low frequency virtual codebook generation.
- Step S1 compresses actually coded residual sub-vectors, for example in accordance with equation (1).
- Step S2 rejects compressed residual sub-vectors that are too sparse, i.e. compressed residual sub-vectors that do not fulfill a predetermined sparseness criterion, for example criterion (3).
- Step S3 concatenates the remaining compressed residual sub-vectors to form the virtual codebook VC1.
- Fig. 15 is a flow chart illustrating high frequency virtual codebook generation.
- Step S1 compresses actually coded residual sub-vectors, for example in accordance with equation (1).
- Step S2 rejects compressed residual sub-vectors that are too sparse, i.e. compressed residual sub-vectors that do not fulfill a predetermined sparseness criterion, such as criterion (3).
- Step S3 concatenates the remaining compressed residual sub-vectors to form a first virtual codebook VC1.
- the high frequency virtual codebook generation includes the same steps as the low frequency virtual codebook generation.
- Step S4 combines pairs of coefficients of the first virtual codebook VC1, for example in accordance with equation (5), thereby forming the high frequency virtual codebook VC2.
- Fig. 16 is a flow chart illustrating spectrum filling.
- Step S1 compresses actually coded residual sub-vectors, for example in accordance with equation (1).
- Step S2 rejects compressed residual sub-vectors that are too sparse, i.e. compressed residual sub-vectors that do not fulfill a predetermined sparseness criterion, such as criterion (3).
- Step S3 concatenates the remaining compressed residual sub-vectors to form a first virtual codebook VC1.
- Step S4 combines pairs of coefficients of the first virtual codebook VC1, for example in accordance with equation (5), to form a second virtual codebook VC2.
- the spectrum filling includes the same steps as the high frequency virtual codebook generation.
- Step S5 fills non-coded residual sub-vectors below a predetermined frequency with coefficients from the first virtual codebook VC1.
- Step S6 fills non-coded residual sub-vectors above a predetermined frequency with coefficients from the second virtual codebook VC2.
- Optional step S7 adjusts the energy of filled non-coded residual sub-vectors to obtain a perceptual attenuation, as described above.
- Fig. 17 is a block diagram illustrating an example embodiment of a low frequency virtual codebook generator 60.
- This embodiment is based on a processor 110, for example a micro processor, which executes a software component 120 for compressing actually coded residual sub-vectors, a software component 130 for rejecting compressed residual sub-vectors that are too sparse, and a software component 140 for concatenating the remaining compressed residual sub-vectors to form the virtual codebook VC1.
- These software components are stored in memory 150.
- the processor 110 communicates with the memory over a system bus.
- the residual sub-vectors are received by an input/output (I/O) controller 160 controlling an I/O bus, to which the processor 110 and the memory 150 are connected.
- I/O input/output
- the residual sub-vectors received by the I/O controller 160 are stored in the memory 150, where they are processed by the software components.
- Software component 120 may implement the functionality of block 42 in the embodiment described with reference to Fig. 10 above.
- Software component 130 may implement the functionality of block 44 in the embodiment described with reference to Fig. 10 above.
- Software component 140 may implement the functionality of block 46 in the embodiment described with reference to Fig. 10 above.
- the virtual codebook VC1 obtained from software component 140 is outputted from the memory 150 by the I/O controller 160 over the I/O bus or is stored in memory 150.
- Fig. 18 is a block diagram illustrating an example embodiment of a high frequency virtual codebook generator 70.
- This embodiment is based on a processor 110, for example a micro processor, which executes a software component 120 for compressing actually coded residual sub-vectors, a software component 130 for rejecting compressed residual sub-vectors that are too sparse, a software component 140 for concatenating the remaining compressed residual sub-vectors to form low frequency virtual codebook VC1, and a software component 170 for combining coefficient pairs from the codebook VC1 to form the high frequency virtual codebook VC2.
- These software components are stored in memory 150.
- the processor 110 communicates with the memory over a system bus.
- the residual sub-vectors are received by an input/output (I/O) controller 160 controlling an I/O bus, to which the processor 110 and the memory 150 are connected.
- the residual sub-vectors received by the I/O controller 160 are stored in the memory 150, where they are processed by the software components.
- Software component 120 may implement the functionality of block 42 in the embodiment described with reference to Fig. 11 above.
- Software component 130 may implement the functionality of block 44 in the embodiments described with reference to Fig. 11 above.
- Software component 140 may implement the functionality of block 46 in the embodiment described with reference to Fig. 11 above.
- Software component 170 may implement the functionality of block 48 in the embodiment described with reference to Fig. 11 above.
- the virtual codebook VC1 obtained from software component 140 is preferably stored in memory 150 for this purpose.
- the virtual codebook VC2 obtained from software component 170 is outputted from the memory 150 by the I/O controller 160 over the I/O bus or is stored in memory 150.
- Fig. 19 is a block diagram illustrating an example embodiment of a spectrum filler 40.
- This embodiment is based on a processor 110, for example a micro processor, which executes a software component 180 for generating a low frequency virtual codebook VC1, a software component 190 for generating a high frequency virtual codebook VC2, a software component 200 for filling non-coded residual sub-vectors below a predetermined frequency from the virtual codebook VC1, and a software component 210 for filling non-coded residual sub-vectors above a predetermined frequency from the virtual codebook VC2.
- These software components are stored in memory 150.
- the processor 110 communicates with the memory over a system bus.
- the residual sub-vectors are received by an input/output (I/O) controller 160 controlling an I/O bus, to which the processor 110 and the memory 150 are connected.
- the residual sub-vectors received by the I/O controller 160 are stored in the memory 150, where they are processed by the software components.
- Software component 180 may implement the functionality of blocks 42-46 in the embodiment described with reference to Fig. 12 above.
- Software component 190 may implement the functionality of block 48 in the embodiments described with reference to Fig. 12 above.
- Software components 200, 210 may implement the functionality of block 50 in the embodiment described with reference to Fig. 12 above.
- the virtual codebooks VC1, VC2 obtained from software components 180 and 190 are preferably stored in memory 150 for this purpose.
- the filled residual sub-vectors obtained from software components 200, 201 are outputted from the memory 150 by the I/O controller 160 over the I/O bus or are stored in memory 150.
- An audio decoder which can be used in a mobile device (e.g. mobile phone, laptop) or a stationary PC.
- UE User Equipment
- An audio decoder with the proposed spectrum fill scheme may be used in real-time communication scenarios (targeting primarily speech) or streaming scenarios (targeting primarily music).
- Fig. 20 illustrates an embodiment of a user equipment in accordance with the present technology. It includes a decoder 300 provided with a spectrum filler 40 in accordance with the present technology. This embodiment illustrates a radio terminal, but other network nodes are also feasible. For example, if voice over IP (Internet Protocol) is used in the network, the user equipment may comprise a computer.
- IP Internet Protocol
- an antenna 302 receives an encoded audio signal.
- a radio unit 304 transforms this signal into audio parameters, which are forwarded to the decoder 300 for generating a digital audio signal, as described with reference to the various embodiments above.
- the digital audio signal is then D/A converted and amplified in a unit 306 and finally forwarded to a loudspeaker 308.
Abstract
Description
- The present technology relates to coding of audio signals, and especially to filling of non-coded sub-vectors in transform coded audio signals.
- A typical encoder/decoder system based on transform coding is illustrated in
Fig. 1 . - Major steps in transform coding are:
- A. Transform a short audio frame (20-40 ms) to a frequency domain, e.g., through the Modified Discrete Cosine Transform (MDCT).
- B. Split the MDCT vector X(k) into multiple bands (sub-vectors SV1, SV2, ...), as illustrated in
Fig. 2 . Typically the width of the bands increases towards higher frequencies [1]. - C. Calculate the energy in each band. This gives an approximation of the spectrum envelope, as illustrated in
Fig. 3 . - D. The spectrum envelope is quantized, and the quantization indices are transmitted to the decoder.
- E. A residual vector is obtained by scaling the MDCT vector with the envelope gains, e.g., the residual vector is formed by the MDCT sub-vectors (SV1,SV2, ...) scaled to unit Root-Mean-Square (RMS) energy.
- F. Bits for quantization of different residual sub-vectors are assigned based on envelope energies. Due to a limited bit-budget, some of the sub-vectors are not assigned any bits. This is illustrated in
Fig. 4 , where sub-vectors corresponding to envelope gains below a threshold TH are not assigned any bits. - G. Residual sub-vectors are quantized according to the assigned bits, and quantization indices are transmitted to the decoder. Residual quantization can, for example, be performed with the Factorial Pulse Coding (FPC) scheme [2].
- H. Residual sub-vectors with zero bits assigned are not coded, but instead noise-filled at the decoder. This is achieved by creating a Virtual Codebook (VC) from coded sub-vectors by concatenating the perceptually relevant coefficients of the decoded spectrum. The VC creates content in the non-coded residual sub-vectors.
- I. At the decoder, the MDCT vector is reconstructed by up-scaling residual sub-vectors with corresponding envelope gains, and the inverse MDCT is used to reconstruct the time-domain audio frame.
- A drawback of the conventional noise-fill scheme, e.g. as in [1], is that it in step H creates audible distortion in the reconstructed audio signal, when used with the FPC scheme.
- A general object is an improved filling of non-coded residual sub-vectors of a transform coded audio signal.
- Another object is generation of virtual codebooks used to fill the non-coded residual sub-vectors.
- These objects are achieved in accordance with the attached claims.
- A first aspect of the present technology involves an apparatus for filling non-coded residual sub-vectors of a transform coded audio signal. The apparatus comprises:
- Means for compressing coded residual sub-vectors.
- Means for rejecting compressed residual sub-vectors that do not fulfill a predetermined criterion.
- Means for concatenating the remaining quantized residual sub-vectors to form the first virtual codebook.
- Means for combining pairs of coefficients of the first virtual codebook to form a second virtual codebook.
- Means for filling non-coded residual sub-vectors below a predetermined frequency with coefficients from the first virtual codebook, and to fill non-coded residual sub-vectors above the predetermined frequency with coefficients from the second virtual codebook.
- A second aspect of the present technology involves an audio decoder comprising the apparatus in accordance with the first aspect.
- A third aspect of the present technology involves a user equipment comprising the audio decoder in accordance with the second aspect.
- A fourth aspect of the present technology involves a method for filling non-coded residual sub-vectors of a transform coded audio signal. The method comprises:
- Compressing coded residual sub-vectors.
- Rejecting compressed residual sub-vectors that do not fulfill a predetermined criterion.
- Concatenating the remaining compressed residual sub-vectors to form a first virtual codebook.
- Combining pairs of coefficients of the first virtual codebook to form a second virtual codebook.
- Filling non-coded residual sub-vectors below a predetermined frequency with coefficients from the first virtual codebook.
- Filling non-coded residual sub-vectors above the predetermined frequency with coefficients from the second virtual codebook.
- An advantage of the present spectrum filling technology is a perceptual improvement of decoded audio signals compared to conventional noise filling.
- The present technology, together with further objects and advantages thereof, may best be understood by making reference to the following description taken together with the accompanying drawings, in which:
-
Fig. 1 is a block diagram illustrating a typical transform based audio coding/ decoding system; -
Fig. 2 is a diagram illustrating the structure of an MDCT vector; -
Fig. 3 is a diagram illustrating the energy distribution in the sub-vectors of an MDCT vector; -
Fig. 4 is a diagram illustrating the use of the spectrum envelope for bit allocation; -
Fig. 5 is a diagram illustrating a coded residual; -
Fig. 6 is a diagram illustrating compression of a coded residual; -
Fig. 7 is a diagram illustrating rejection of coded residual sub-vectors; -
Fig. 8 is a diagram illustrating concatenation of surviving residual sub-vectors to form a first virtual codebook; -
Fig. 9A-B are diagrams illustrating combining of coefficients from the first virtual codebook to form a second virtual codebook; -
Fig. 10 is a block diagram illustrating an example embodiment of a low frequency virtual codebook generator; -
Fig. 11 is a block diagram illustrating an example embodiment of a high frequency virtual codebook generator; -
Fig. 12 is a block diagram illustrating an example embodiment of a spectrum filler; -
Fig. 13 is a block diagram illustrating an example embodiment of a decoder including a spectrum filler; -
Fig. 14 is a flow chart illustrating low frequency virtual codebook generation; -
Fig. 15 is a flow chart illustrating high frequency virtual codebook generation; -
Fig. 16 is a flow chart illustrating spectrum filling; -
Fig. 17 is a block diagram illustrating an example embodiment of a low frequency virtual codebook generator; -
Fig. 18 is a block diagram illustrating an example embodiment of a high frequency virtual codebook generator; -
Fig. 19 is a block diagram illustrating an example embodiment of a spectrum filler; and -
Fig. 20 is a block diagram illustrating an example embodiment of a user equipment. - Before the present technology is described in more detail, transform based coding/decoding will be briefly described with reference to
Fig. 1-7 . -
Fig. 1 is a block diagram illustrating a typical transform based audio coding/decoding system. An input signal x(n) is forwarded to a frequency transformer, for example anMDCT transformer 10, where short audio frames (20-40 ms) are transformed into a frequency domain. The resulting frequency domain signal X(k) is divided into multiple bands (sub-vectors SV1, SV2, ...), as illustrated inFig. 2 . Typically the width of the bands increases towards higher frequencies [1]. The energy of each band is determined in an envelope calculator andquantizer 12. This gives an approximation of the spectrum envelope, as illustrated inFig. 3 . Each sub-vector is normalized into a residual sub-vector in asub-vector normalizer 14 by scaling with the inverse of the corresponding quantized envelope value (gain). - A
bit allocator 16 assigns bits for quantization of different residual sub-vectors based on envelope energies. Due to a limited bit-budget, some of the sub-vectors are not assigned any bits. This is illustrated inFig. 4 , where sub-vectors corresponding to envelope gains below a threshold TH are not assigned any bits. Residual sub-vectors are quantized in asub-vector quantizer 18 according to the assigned bits. Residual quantization can, for example, be performed with the Factorial Pulse Coding (FPC) scheme [2]. Residual sub-vector quantization indices and envelope quantization indices are then transmitted to the decoder over a multiplexer (MUX) 20. - At the decoder the received bit stream is de-multiplexed into residual sub-vector quantization indices and envelope quantization indices in a demultiplexer (DEMUX) 22. The residual sub-vector quantization indices are dequantized into residual sub-vectors in a
sub-vector dequantizer 24, and the envelope quantization indices are dequantized into envelope gains in anenvelope dequantizer 26. A bit allocator 28 uses the envelope gains to control the residual sub-vector dequantization. - Residual sub-vectors with zero bits assigned have not been coded at the encoder, and are instead noise-filled by a
noise filler 30 at the decoder. This is achieved by creating a Virtual Codebook (VC) from coded sub-vectors by concatenating the perceptually relevant coefficients of the decoded spectrum ([1] section 8.4.1). Thus, the VC creates content in the non-coded residual sub-vectors. - At the decoder, the MDCT vector x̂(n) is then reconstructed by up-scaling residual sub-vectors with corresponding envelope gains in an
envelope shaper 32, and transforming the resulting frequency domain vector X̂(k) in aninverse MDCT transformer 34. - A drawback of the conventional noise-fill scheme described above is that It creates audible distortion in the reconstructed audio signal, when used with the FPC scheme. The main reason is that some of the coded vectors may be too sparse, which creates energy mismatch problems in the noise-filled bands. Additionally some of the coded vectors may contain too much structure (color), which leads to perceptual degradations when the noise-fill is performed at high frequencies.
- The following description will focus on an embodiment of an improved procedure for virtual codebook generation in step H above.
- A coded residual X̂(k), illustrated in
Fig. 5 , is compressed or quantized according to:Fig. 6 . This step guarantees that there will be no excessive structure (such as periodicity at high-frequencies) in the noise-filled regions. In addition the specific form of compressed residual Y(k) allows a low complexity in the following steps. - As an alternative the coded residual X̂(k) may be compressed or quantized according to:
actual value 0. - The virtual codebook is built only from "populated" M-dimensional sub-vectors. If a coded residual sub-vector does not fulfill the criterion:
Fig. 7 , where sub-vector SV3 is rejected, since it has 7 zeros. A virtual codebook VC1 is formed by concatenating the remaining or surviving sub-vectors, as illustrated inFig. 8 . Since the length of the sub-vectors is a multiple of M, the criterion (3) may be used also for longer sub-vectors. In this case the parts that do not fulfill the criterion are rejected. - In general a compressed sub-vector is considered "populated" if it contains more that 20-30% of non-zero components. In the example above with M=8 the criterion is "more than 25% of non-zero components".
- A second virtual codebook VC2 is created from the obtained virtual codebook VC1. This second virtual codebook VC2 is even more "populated" and is used to fill frequencies above 4.8 kHz (other transition frequencies are of course also possible; typically the transition frequency is between 4 and 6 kHz). The second virtual codebook VC2 is formed in accordance with:
- This combining or merging step is illustrated in
Fig. 9A-B . It is noted that the same pair of coefficients Y(k), Y(N - k) is used twice in the merging process, once in the lower half (Fig. 9A ) and once in the upper half (Fig. 9B ). - Non-coded sub-vectors may be filled by cyclically stepping through the respective virtual codebook, VC1 or VC2 depending on whether the sub-vector to be filled is below or above the transition frequency, and copying the required number of codebook coefficients to the empty sub-vector. Thus, if the codebooks are short and there are many sub-vectors to be filled, the same coefficients will be reused for filling more than one sub-vector.
- An energy adjustment of the filled sub-vectors is preferably performed on a sub-vector basis. It accounts for the fact that after the spectrum filling the residual sub-vectors may not have the expected unit RMS energy. The adjustment may be performed in accordance with:
- In a more advanced scheme energy adjustment of a particular sub-vector can be adapted to the type of neighboring sub-vectors: If the neighboring regions are coded at high-bitrate, attenuation of the current sub-vector is more aggressive (alpha goes towards zero). If the neighboring regions are coded at a low-bitrate or noise-filled, attenuation of the current sub-vector is limited (alpha goes towards one). This scheme prevents attenuation of large continuous spectral regions, which might lead to audible loudness loss. At the same time if the spectral region to be attenuated is narrow, even a very strong attenuation will not affect the overall loudness.
- The described technology provides improved noise-filling. Perceptual improvements have been measured by means of listening tests. These tests indicate that the spectrum fill procedure described above was preferred by listeners in 83% of the tests while the conventional noise fill procedure was preferred in 17% of the tests.
-
Fig. 10 is a block diagram illustrating an example embodiment of a low frequencyvirtual codebook generator 60. Residual sub-vectors are forwarded to asub-vector compressor 42, which is configured to compress actually coded residual sub-vectors (i.e. sub-vectors that have actually been allocated bits for coding), for example in accordance with equation (1). The compressed sub-vectors are forwarded to asub-vector rejecter 44, which is configured to reject compressed residual sub-vectors that do not fulfill a predetermined sparseness criterion, for example criterion (3). The remaining compressed sub-vectors are collected in asub-vector collector 46, which is configured to concatenate them to form the low frequency virtual codebook VC1. -
Fig. 11 is a block diagram illustrating an example embodiment of a high frequencyvirtual codebook generator 70. Residual sub-vectors are forwarded to asub-vector compressor 42, which is configured to compress actually coded residual sub-vectors (i.e. sub-vectors that have actually been allocated bits for coding), for example in accordance with equation (1). The compressed sub-vectors are forwarded to asub-vector rejecter 44, which is configured to reject compressed residual sub-vectors that do not fulfill a predetermined sparseness criterion, for example criterion (3). The remaining compressed sub-vectors are collected in asub-vector collector 46, which is configured to concatenate them to form the low frequency virtual codebook VC1. Thus, up to this point the high frequencyvirtual codebook generator 70 includes the same elements as the low frequencyvirtual codebook generator 60. Coefficients from the low frequency virtual codebook VC1 are forwarded to acoefficient combiner 48, which is configured to combine pairs of coefficients to form the high frequency virtual codebook VC2, for example in accordance with equation (5). -
Fig. 12 is a block diagram illustrating an example embodiment of aspectrum filler 40. Residual sub-vectors are forwarded to asub-vector compressor 42, which is configured to compress actually coded residual sub-vectors (i.e. sub-vectors that have actually been allocated bits for coding), for example in accordance with equation (1). The compressed sub-vectors are forwarded to asub-vector rejecter 44, which is configured to reject compressed residual sub-vectors that do not fulfill a predetermined sparseness criterion, for example criterion (3). The remaining compressed sub-vectors are collected in asub-vector collector 46, which is configured to concatenate them to form a first (low frequency) virtual codebook VC1. Coefficients from the first virtual codebook VC1 are forwarded to acoefficient combiner 48, which is configured to combine pairs of coefficients to form a second (high frequency) virtual codebook VC2, for example in accordance with equation (5). Thus, up to this point thespectrum filler 40 includes the same elements as the high frequencyvirtual codebook generator 70. The residual sub-vectors are also forwarded to asub-vector filler 50, which is configured to fill non-coded residual sub-vectors below a predetermined frequency with coefficients from the first virtual codebook VC1, and to fill non-coded residual sub-vectors above the predetermined frequency with coefficients from the second virtual codebook. In a preferred embodiment thespectrum filler 40 also includes anenergy adjuster 52 configured to adjust the energy of filled non-coded residual sub-vectors to obtain a perceptual attenuation, as described above. -
Fig. 13 is a block diagram illustrating an example embodiment of adecoder 300 including aspectrum filler 40. The general structure of thedecoder 300 is the same as of the decoder inFig. 1 , but with thenoise filler 30 replaced by thespectrum filler 40. -
Fig. 14 is a flow chart illustrating low frequency virtual codebook generation. Step S1 compresses actually coded residual sub-vectors, for example in accordance with equation (1). Step S2 rejects compressed residual sub-vectors that are too sparse, i.e. compressed residual sub-vectors that do not fulfill a predetermined sparseness criterion, for example criterion (3). Step S3 concatenates the remaining compressed residual sub-vectors to form the virtual codebook VC1. -
Fig. 15 is a flow chart illustrating high frequency virtual codebook generation. Step S1 compresses actually coded residual sub-vectors, for example in accordance with equation (1). Step S2 rejects compressed residual sub-vectors that are too sparse, i.e. compressed residual sub-vectors that do not fulfill a predetermined sparseness criterion, such as criterion (3). Step S3 concatenates the remaining compressed residual sub-vectors to form a first virtual codebook VC1. Thus, up to this point the high frequency virtual codebook generation includes the same steps as the low frequency virtual codebook generation. Step S4 combines pairs of coefficients of the first virtual codebook VC1, for example in accordance with equation (5), thereby forming the high frequency virtual codebook VC2. -
Fig. 16 is a flow chart illustrating spectrum filling. Step S1 compresses actually coded residual sub-vectors, for example in accordance with equation (1). Step S2 rejects compressed residual sub-vectors that are too sparse, i.e. compressed residual sub-vectors that do not fulfill a predetermined sparseness criterion, such as criterion (3). Step S3 concatenates the remaining compressed residual sub-vectors to form a first virtual codebook VC1. Step S4 combines pairs of coefficients of the first virtual codebook VC1, for example in accordance with equation (5), to form a second virtual codebook VC2. Thus, up to this point the spectrum filling includes the same steps as the high frequency virtual codebook generation. Step S5 fills non-coded residual sub-vectors below a predetermined frequency with coefficients from the first virtual codebook VC1. Step S6 fills non-coded residual sub-vectors above a predetermined frequency with coefficients from the second virtual codebook VC2. Optional step S7 adjusts the energy of filled non-coded residual sub-vectors to obtain a perceptual attenuation, as described above. -
Fig. 17 is a block diagram illustrating an example embodiment of a low frequencyvirtual codebook generator 60. This embodiment is based on aprocessor 110, for example a micro processor, which executes asoftware component 120 for compressing actually coded residual sub-vectors, asoftware component 130 for rejecting compressed residual sub-vectors that are too sparse, and asoftware component 140 for concatenating the remaining compressed residual sub-vectors to form the virtual codebook VC1. These software components are stored inmemory 150. Theprocessor 110 communicates with the memory over a system bus. The residual sub-vectors are received by an input/output (I/O)controller 160 controlling an I/O bus, to which theprocessor 110 and thememory 150 are connected. In this embodiment the residual sub-vectors received by the I/O controller 160 are stored in thememory 150, where they are processed by the software components.Software component 120 may implement the functionality ofblock 42 in the embodiment described with reference toFig. 10 above.Software component 130 may implement the functionality ofblock 44 in the embodiment described with reference toFig. 10 above.Software component 140 may implement the functionality ofblock 46 in the embodiment described with reference toFig. 10 above. The virtual codebook VC1 obtained fromsoftware component 140 is outputted from thememory 150 by the I/O controller 160 over the I/O bus or is stored inmemory 150. -
Fig. 18 is a block diagram illustrating an example embodiment of a high frequencyvirtual codebook generator 70. This embodiment is based on aprocessor 110, for example a micro processor, which executes asoftware component 120 for compressing actually coded residual sub-vectors, asoftware component 130 for rejecting compressed residual sub-vectors that are too sparse, asoftware component 140 for concatenating the remaining compressed residual sub-vectors to form low frequency virtual codebook VC1, and asoftware component 170 for combining coefficient pairs from the codebook VC1 to form the high frequency virtual codebook VC2. These software components are stored inmemory 150. Theprocessor 110 communicates with the memory over a system bus. The residual sub-vectors are received by an input/output (I/O)controller 160 controlling an I/O bus, to which theprocessor 110 and thememory 150 are connected. In this embodiment the residual sub-vectors received by the I/O controller 160 are stored in thememory 150, where they are processed by the software components.Software component 120 may implement the functionality ofblock 42 in the embodiment described with reference toFig. 11 above.Software component 130 may implement the functionality ofblock 44 in the embodiments described with reference toFig. 11 above.Software component 140 may implement the functionality ofblock 46 in the embodiment described with reference toFig. 11 above.Software component 170 may implement the functionality ofblock 48 in the embodiment described with reference toFig. 11 above. The virtual codebook VC1 obtained fromsoftware component 140 is preferably stored inmemory 150 for this purpose. The virtual codebook VC2 obtained fromsoftware component 170 is outputted from thememory 150 by the I/O controller 160 over the I/O bus or is stored inmemory 150. -
Fig. 19 is a block diagram illustrating an example embodiment of aspectrum filler 40. This embodiment is based on aprocessor 110, for example a micro processor, which executes asoftware component 180 for generating a low frequency virtual codebook VC1, asoftware component 190 for generating a high frequency virtual codebook VC2, asoftware component 200 for filling non-coded residual sub-vectors below a predetermined frequency from the virtual codebook VC1, and asoftware component 210 for filling non-coded residual sub-vectors above a predetermined frequency from the virtual codebook VC2. These software components are stored inmemory 150. Theprocessor 110 communicates with the memory over a system bus. The residual sub-vectors are received by an input/output (I/O)controller 160 controlling an I/O bus, to which theprocessor 110 and thememory 150 are connected. In this embodiment the residual sub-vectors received by the I/O controller 160 are stored in thememory 150, where they are processed by the software components.Software component 180 may implement the functionality of blocks 42-46 in the embodiment described with reference toFig. 12 above.Software component 190 may implement the functionality ofblock 48 in the embodiments described with reference toFig. 12 above.Software components block 50 in the embodiment described with reference toFig. 12 above. The virtual codebooks VC1, VC2 obtained fromsoftware components memory 150 for this purpose. The filled residual sub-vectors obtained fromsoftware components 200, 201 are outputted from thememory 150 by the I/O controller 160 over the I/O bus or are stored inmemory 150. - The technology described above is intended to be used in an audio decoder, which can be used in a mobile device (e.g. mobile phone, laptop) or a stationary PC. Here the term User Equipment (UE) will be used as a generic name for such devices. An audio decoder with the proposed spectrum fill scheme may be used in real-time communication scenarios (targeting primarily speech) or streaming scenarios (targeting primarily music).
-
Fig. 20 illustrates an embodiment of a user equipment in accordance with the present technology. It includes adecoder 300 provided with aspectrum filler 40 in accordance with the present technology. This embodiment illustrates a radio terminal, but other network nodes are also feasible. For example, if voice over IP (Internet Protocol) is used in the network, the user equipment may comprise a computer. - In the user equipment in
Fig. 20 anantenna 302 receives an encoded audio signal. Aradio unit 304 transforms this signal into audio parameters, which are forwarded to thedecoder 300 for generating a digital audio signal, as described with reference to the various embodiments above. The digital audio signal is then D/A converted and amplified in aunit 306 and finally forwarded to aloudspeaker 308. - It will be understood by those skilled in the art that various modifications and changes may be made to the present technology without departure from the scope thereof, which is defined by the appended claims.
-
- [1] ITU-T Rec. G.719, "Low-complexity full-band audio coding for high-quality conversational applications," 2008, Sections 8.4.1, 8.4.3.
- [2] Mittal, J. Ashley, E. Cruz-Zeno, "Low Complexity Factorial Pulse Coding of MDCT Coefficients using Approximation of Combinatorial Functions," ICASSP 2007
-
- FPC Factorial Pulse Coding
- MDCTModified Discrete Cosine Transform
- RMS Root-Mean-Square
- UE User Equipment
- VC Virtual Codebook
Claims (9)
- An apparatus (40) for filling non-coded residual sub-vectors of a transform coded audio signal, the apparatus comprising:means (42) for compressing coded residual sub-vectors;means (44) for rejecting compressed residual sub-vectors that do not fulfill a predetermined criterion;means (46) for concatenating the remaining compressed residual sub-vectors to form a first virtual codebook (VC1);means (48) for combining pairs of coefficients of the first virtual codebook (VC1) to form a second virtual codebook (VC2);means (50) for filling non-coded residual sub-vectors below a predetermined frequency with coefficients from the first virtual codebook (VC1), and for filling non-coded residual sub-vectors above the predetermined frequency with coefficients from the second virtual codebook (VC2), characterized in that the means (42) for compressing is configured to compress components X̂(k) of the coded residual sub-vectors in accordance with:where Y(k) are the components of the compressed residual sub-vectors and T is a small positive number that controls the amount of compression.
- The apparatus according to claim 1, wherein the means (44) for rejecting is configured to reject compressed residual sub-vectors having less than a predetermined percentage of non-zero components.
- The apparatus according to any one of claims 1 to 3, wherein the means (48) for combining is configured to combine pairs of coefficients Y(k) of the first virtual codebook (VC1) in accordance with:
- The apparatus according to any one of claims 1 to 4, further comprising means (52) for adjusting the energy of filled non-coded residual sub-vectors to obtain a perceptual attenuation.
- An audio decoder (300) comprising the apparatus (40) according to any one of claims 1 to 5.
- A user equipment (UE) comprising the audio decoder according to claim 6.
- A method for filling non-coded residual sub-vectors of a transform coded audio signal, the method comprising:compressing (S1) coded residual sub-vectors;rejecting (S2) compressed residual sub-vectors that do not fulfill a predetermined criterion;concatenating (S3) the remaining compressed residual sub-vectors to form a first virtual codebook (VC1);combining (S4) pairs of coefficients of the first virtual codebook (VC1) to form a second virtual codebook (VC2);filling (S5) non-coded residual sub-vectors below a predetermined frequency with coefficients from the first virtual codebook (VC1);where Y(k) are the components of the compressed residual sub-vectors and T is a small positive number that controls the amount of compression.
- A computer program comprising computer readable code units which when run on an apparatus causes the apparatus to perform the method according to claim 8.
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201161451363P | 2011-03-10 | 2011-03-10 | |
EP15183624.4A EP2975611B1 (en) | 2011-03-10 | 2011-09-14 | Filling of non-coded sub-vectors in transform coded audio signals |
EP11860593.0A EP2684190B1 (en) | 2011-03-10 | 2011-09-14 | Filling of non-coded sub-vectors in transform coded audio signals |
PCT/SE2011/051110 WO2012121638A1 (en) | 2011-03-10 | 2011-09-14 | Filing of non-coded sub-vectors in transform coded audio signals |
Related Parent Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP11860593.0A Division EP2684190B1 (en) | 2011-03-10 | 2011-09-14 | Filling of non-coded sub-vectors in transform coded audio signals |
EP15183624.4A Division EP2975611B1 (en) | 2011-03-10 | 2011-09-14 | Filling of non-coded sub-vectors in transform coded audio signals |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3319087A1 true EP3319087A1 (en) | 2018-05-09 |
EP3319087B1 EP3319087B1 (en) | 2019-08-21 |
Family
ID=46798435
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP17208522.7A Active EP3319087B1 (en) | 2011-03-10 | 2011-09-14 | Filling of non-coded sub-vectors in transform coded audio signals |
EP11860593.0A Active EP2684190B1 (en) | 2011-03-10 | 2011-09-14 | Filling of non-coded sub-vectors in transform coded audio signals |
EP15183624.4A Active EP2975611B1 (en) | 2011-03-10 | 2011-09-14 | Filling of non-coded sub-vectors in transform coded audio signals |
Family Applications After (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP11860593.0A Active EP2684190B1 (en) | 2011-03-10 | 2011-09-14 | Filling of non-coded sub-vectors in transform coded audio signals |
EP15183624.4A Active EP2975611B1 (en) | 2011-03-10 | 2011-09-14 | Filling of non-coded sub-vectors in transform coded audio signals |
Country Status (11)
Country | Link |
---|---|
US (6) | US9424856B2 (en) |
EP (3) | EP3319087B1 (en) |
CN (1) | CN103503063B (en) |
AU (1) | AU2011361945B2 (en) |
DK (3) | DK3319087T3 (en) |
ES (3) | ES2559040T3 (en) |
HU (2) | HUE026874T2 (en) |
NO (1) | NO2753696T3 (en) |
PL (1) | PL2684190T3 (en) |
PT (2) | PT3319087T (en) |
WO (1) | WO2012121638A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2578603A (en) * | 2018-10-31 | 2020-05-20 | Nokia Technologies Oy | Determination of spatial audio parameter encoding and associated decoding |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
SG194945A1 (en) | 2011-05-13 | 2013-12-30 | Samsung Electronics Co Ltd | Bit allocating, audio encoding and decoding |
MX350162B (en) | 2011-06-30 | 2017-08-29 | Samsung Electronics Co Ltd | Apparatus and method for generating bandwidth extension signal. |
KR20130032980A (en) * | 2011-09-26 | 2013-04-03 | 한국전자통신연구원 | Coding apparatus and method using residual bits |
JP5945626B2 (en) * | 2012-03-29 | 2016-07-05 | テレフオンアクチーボラゲット エルエム エリクソン(パブル) | Bandwidth expansion of harmonic audio signals |
PL3471093T3 (en) * | 2013-01-29 | 2021-04-06 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Noise filling in perceptual transform audio coding |
EP2980792A1 (en) | 2014-07-28 | 2016-02-03 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Apparatus and method for generating an enhanced signal using independent noise-filling |
EP3413308A1 (en) * | 2017-06-07 | 2018-12-12 | Nokia Technologies Oy | Efficient storage of multiple structured codebooks |
DK3776547T3 (en) | 2018-04-05 | 2021-09-13 | Ericsson Telefon Ab L M | Support for generating comfort clothing |
RU2757860C1 (en) * | 2021-04-09 | 2021-10-21 | Общество с ограниченной ответственностью "Специальный Технологический Центр" | Method for automatically assessing the quality of speech signals with low-rate coding |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030233234A1 (en) * | 2002-06-17 | 2003-12-18 | Truman Michael Mead | Audio coding system using spectral hole filling |
US20100241437A1 (en) * | 2007-08-27 | 2010-09-23 | Telefonaktiebolaget Lm Ericsson (Publ) | Method and device for noise filling |
Family Cites Families (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0451199A (en) * | 1990-06-18 | 1992-02-19 | Fujitsu Ltd | Sound encoding/decoding system |
CA2206652A1 (en) * | 1996-06-04 | 1997-12-04 | Claude Laflamme | Baud-rate-independent asvd transmission built around g.729 speech-coding standard |
US6714907B2 (en) * | 1998-08-24 | 2004-03-30 | Mindspeed Technologies, Inc. | Codebook structure and search for speech coding |
US6173257B1 (en) * | 1998-08-24 | 2001-01-09 | Conexant Systems, Inc | Completed fixed codebook for speech encoder |
US6456964B2 (en) * | 1998-12-21 | 2002-09-24 | Qualcomm, Incorporated | Encoding of periodic speech using prototype waveforms |
US6691084B2 (en) * | 1998-12-21 | 2004-02-10 | Qualcomm Incorporated | Multiple mode variable rate speech coding |
US6952671B1 (en) | 1999-10-04 | 2005-10-04 | Xvd Corporation | Vector quantization with a non-structured codebook for audio compression |
US6944350B2 (en) * | 1999-12-17 | 2005-09-13 | Utah State University | Method for image coding by rate-distortion adaptive zerotree-based residual vector quantization and system for effecting same |
US6909749B2 (en) * | 2002-07-15 | 2005-06-21 | Pts Corporation | Hierarchical segment-based motion vector encoding and decoding |
US8064520B2 (en) * | 2003-09-07 | 2011-11-22 | Microsoft Corporation | Advanced bi-directional predictive coding of interlaced video |
WO2006106508A2 (en) * | 2005-04-04 | 2006-10-12 | Technion Research & Development Foundation Ltd. | System and method for designing of dictionaries for sparse representation |
US20090299738A1 (en) | 2006-03-31 | 2009-12-03 | Matsushita Electric Industrial Co., Ltd. | Vector quantizing device, vector dequantizing device, vector quantizing method, and vector dequantizing method |
WO2007132750A1 (en) | 2006-05-12 | 2007-11-22 | Panasonic Corporation | Lsp vector quantization device, lsp vector inverse-quantization device, and their methods |
US7822289B2 (en) * | 2006-07-25 | 2010-10-26 | Microsoft Corporation | Locally adapted hierarchical basis preconditioning |
JP5006404B2 (en) * | 2006-12-05 | 2012-08-22 | ホアウェイ・テクノロジーズ・カンパニー・リミテッド | Vector quantizer |
WO2009049895A1 (en) * | 2007-10-17 | 2009-04-23 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Audio coding using downmix |
CN101911185B (en) | 2008-01-16 | 2013-04-03 | 松下电器产业株式会社 | Vector quantizer, vector inverse quantizer, and methods thereof |
US8619918B2 (en) * | 2008-09-25 | 2013-12-31 | Nec Laboratories America, Inc. | Sparse channel estimation for MIMO OFDM systems |
US8320489B2 (en) * | 2009-02-20 | 2012-11-27 | Wisconsin Alumni Research Foundation | Determining channel coefficients in a multipath channel |
-
2011
- 2011-09-14 ES ES11860593.0T patent/ES2559040T3/en active Active
- 2011-09-14 PT PT172085227T patent/PT3319087T/en unknown
- 2011-09-14 EP EP17208522.7A patent/EP3319087B1/en active Active
- 2011-09-14 EP EP11860593.0A patent/EP2684190B1/en active Active
- 2011-09-14 DK DK17208522T patent/DK3319087T3/en active
- 2011-09-14 DK DK11860593.0T patent/DK2684190T3/en active
- 2011-09-14 EP EP15183624.4A patent/EP2975611B1/en active Active
- 2011-09-14 HU HUE11860593A patent/HUE026874T2/en unknown
- 2011-09-14 DK DK15183624.4T patent/DK2975611T3/en active
- 2011-09-14 ES ES15183624.4T patent/ES2664090T3/en active Active
- 2011-09-14 PL PL11860593T patent/PL2684190T3/en unknown
- 2011-09-14 PT PT118605930T patent/PT2684190E/en unknown
- 2011-09-14 HU HUE15183624A patent/HUE037111T2/en unknown
- 2011-09-14 US US14/003,820 patent/US9424856B2/en active Active
- 2011-09-14 AU AU2011361945A patent/AU2011361945B2/en active Active
- 2011-09-14 ES ES17208522T patent/ES2758370T3/en active Active
- 2011-09-14 CN CN201180070735.6A patent/CN103503063B/en active Active
- 2011-09-14 WO PCT/SE2011/051110 patent/WO2012121638A1/en active Application Filing
-
2012
- 2012-09-06 NO NO12758827A patent/NO2753696T3/no unknown
-
2016
- 2016-07-14 US US15/210,505 patent/US9966082B2/en active Active
-
2018
- 2018-03-30 US US15/941,566 patent/US20180226081A1/en not_active Abandoned
-
2021
- 2021-05-28 US US17/333,400 patent/US11551702B2/en active Active
-
2022
- 2022-12-12 US US18/079,088 patent/US11756560B2/en active Active
-
2023
- 2023-08-04 US US18/365,322 patent/US20230410822A1/en active Pending
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030233234A1 (en) * | 2002-06-17 | 2003-12-18 | Truman Michael Mead | Audio coding system using spectral hole filling |
US20100241437A1 (en) * | 2007-08-27 | 2010-09-23 | Telefonaktiebolaget Lm Ericsson (Publ) | Method and device for noise filling |
Non-Patent Citations (2)
Title |
---|
MITTAL; J. ASHLEY; E. CRUZ-ZENO: "Low Complexity Factorial Pulse Coding of MDCT Coefficients using Approximation of Combinatorial Functions", ICASSP, 2007 |
SANJEEV MEHROTRA ET AL: "Hybrid low bitrate audio coding using adaptive gain shape vector quantization", MULTIMEDIA SIGNAL PROCESSING, 2008 IEEE 10TH WORKSHOP ON, IEEE, PISCATAWAY, NJ, USA, 8 October 2008 (2008-10-08), pages 927 - 932, XP031356759, ISBN: 978-1-4244-2294-4 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2578603A (en) * | 2018-10-31 | 2020-05-20 | Nokia Technologies Oy | Determination of spatial audio parameter encoding and associated decoding |
Also Published As
Publication number | Publication date |
---|---|
PT3319087T (en) | 2019-10-09 |
US20130346087A1 (en) | 2013-12-26 |
CN103503063B (en) | 2015-12-09 |
EP2684190B1 (en) | 2015-11-18 |
US20230410822A1 (en) | 2023-12-21 |
ES2559040T3 (en) | 2016-02-10 |
AU2011361945B2 (en) | 2016-06-23 |
US9424856B2 (en) | 2016-08-23 |
ES2664090T3 (en) | 2018-04-18 |
CN103503063A (en) | 2014-01-08 |
EP2684190A1 (en) | 2014-01-15 |
HUE037111T2 (en) | 2018-08-28 |
US20160322058A1 (en) | 2016-11-03 |
DK2975611T3 (en) | 2018-04-03 |
US20230106557A1 (en) | 2023-04-06 |
US9966082B2 (en) | 2018-05-08 |
EP2975611A1 (en) | 2016-01-20 |
DK2684190T3 (en) | 2016-02-22 |
US11756560B2 (en) | 2023-09-12 |
PL2684190T3 (en) | 2016-04-29 |
ES2758370T3 (en) | 2020-05-05 |
HUE026874T2 (en) | 2016-07-28 |
EP2684190A4 (en) | 2014-08-13 |
US20180226081A1 (en) | 2018-08-09 |
US20210287685A1 (en) | 2021-09-16 |
WO2012121638A1 (en) | 2012-09-13 |
NO2753696T3 (en) | 2018-04-21 |
EP2975611B1 (en) | 2018-01-10 |
PT2684190E (en) | 2016-02-23 |
EP3319087B1 (en) | 2019-08-21 |
US11551702B2 (en) | 2023-01-10 |
DK3319087T3 (en) | 2019-11-04 |
AU2011361945A1 (en) | 2013-09-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11551702B2 (en) | Filling of non-coded sub-vectors in transform coded audio signals | |
US10714113B2 (en) | Audio decoding device, audio coding device, audio decoding method, audio coding method, audio decoding program, and audio coding program | |
JP5539203B2 (en) | Improved transform coding of speech and audio signals | |
EP2650876B1 (en) | Methods, apparatuses and system for encoding and decoding signal | |
EP2791937B1 (en) | Generation of a high band extension of a bandwidth extended audio signal | |
US9881625B2 (en) | Device and method for execution of huffman coding | |
EP1806737A1 (en) | Sound encoder and sound encoding method | |
EP3413307B1 (en) | Audio signal coding apparatus, audio signal decoding device, and methods thereof | |
KR20130107257A (en) | Method and apparatus for encoding and decoding high frequency for bandwidth extension | |
US8892428B2 (en) | Encoding apparatus, decoding apparatus, encoding method, and decoding method for adjusting a spectrum amplitude | |
EP3220390A1 (en) | Transform encoding/decoding of harmonic audio signals | |
CN105448298A (en) | Filling of non-coded sub-vectors in transform coded audio signals |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED |
|
AC | Divisional application: reference to earlier application |
Ref document number: 2975611 Country of ref document: EP Kind code of ref document: P Ref document number: 2684190 Country of ref document: EP Kind code of ref document: P |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20181105 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G10L 19/00 20130101ALN20190517BHEP Ipc: G10L 21/038 20130101ALI20190517BHEP Ipc: G10L 19/02 20130101ALN20190517BHEP Ipc: G10L 19/028 20130101AFI20190517BHEP |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
INTG | Intention to grant announced |
Effective date: 20190618 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AC | Divisional application: reference to earlier application |
Ref document number: 2684190 Country of ref document: EP Kind code of ref document: P Ref document number: 2975611 Country of ref document: EP Kind code of ref document: P |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602011061547 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1170658 Country of ref document: AT Kind code of ref document: T Effective date: 20190915 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: SE Ref legal event code: TRGR |
|
REG | Reference to a national code |
Ref country code: PT Ref legal event code: SC4A Ref document number: 3319087 Country of ref document: PT Date of ref document: 20191009 Kind code of ref document: T Free format text: AVAILABILITY OF NATIONAL TRANSLATION Effective date: 20190920 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: FP |
|
REG | Reference to a national code |
Ref country code: DK Ref legal event code: T3 Effective date: 20191031 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
REG | Reference to a national code |
Ref country code: NO Ref legal event code: T2 Effective date: 20190821 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191121 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191221 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20191122 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1170658 Country of ref document: AT Kind code of ref document: T Effective date: 20190821 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 |
|
REG | Reference to a national code |
Ref country code: ES Ref legal event code: FG2A Ref document number: 2758370 Country of ref document: ES Kind code of ref document: T3 Effective date: 20200505 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200224 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602011061547 Country of ref document: DE |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG2D | Information on lapse in contracting state deleted |
Ref country code: IS |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190930 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190930 Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190914 |
|
26N | No opposition filed |
Effective date: 20200603 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20110914 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190821 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230523 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: TR Payment date: 20230824 Year of fee payment: 13 Ref country code: NO Payment date: 20230927 Year of fee payment: 13 Ref country code: NL Payment date: 20230926 Year of fee payment: 13 Ref country code: IT Payment date: 20230921 Year of fee payment: 13 Ref country code: IE Payment date: 20230927 Year of fee payment: 13 Ref country code: GB Payment date: 20230927 Year of fee payment: 13 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: SE Payment date: 20230927 Year of fee payment: 13 Ref country code: PT Payment date: 20230831 Year of fee payment: 13 Ref country code: FR Payment date: 20230925 Year of fee payment: 13 Ref country code: DK Payment date: 20230927 Year of fee payment: 13 Ref country code: DE Payment date: 20230927 Year of fee payment: 13 Ref country code: BE Payment date: 20230927 Year of fee payment: 13 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: ES Payment date: 20231002 Year of fee payment: 13 |