EP2320416A1 - Spectral smoothing device, encoding device, decoding device, communication terminal device, base station device, and spectral smoothing method - Google Patents
Spectral smoothing device, encoding device, decoding device, communication terminal device, base station device, and spectral smoothing method Download PDFInfo
- Publication number
- EP2320416A1 EP2320416A1 EP09804758A EP09804758A EP2320416A1 EP 2320416 A1 EP2320416 A1 EP 2320416A1 EP 09804758 A EP09804758 A EP 09804758A EP 09804758 A EP09804758 A EP 09804758A EP 2320416 A1 EP2320416 A1 EP 2320416A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- section
- spectrum
- subband
- linear transformation
- subbands
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000009499 grossing Methods 0.000 title claims abstract description 109
- 238000000034 method Methods 0.000 title claims description 27
- 238000004891 communication Methods 0.000 title claims description 12
- 230000003595 spectral effect Effects 0.000 title abstract description 17
- 238000001228 spectrum Methods 0.000 claims abstract description 222
- 238000004364 calculation method Methods 0.000 claims abstract description 36
- 230000009466 transformation Effects 0.000 claims description 130
- 238000012545 processing Methods 0.000 abstract description 128
- 230000005236 sound signal Effects 0.000 abstract description 5
- 238000006243 chemical reaction Methods 0.000 abstract 4
- 238000001914 filtration Methods 0.000 description 36
- 238000005070 sampling Methods 0.000 description 20
- 238000010586 diagram Methods 0.000 description 15
- 239000000872 buffer Substances 0.000 description 9
- 230000005540 biological transmission Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 4
- 230000015556 catabolic process Effects 0.000 description 3
- 238000006731 degradation reaction Methods 0.000 description 3
- 230000010354 integration Effects 0.000 description 3
- 238000011426 transformation method Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000010295 mobile communication Methods 0.000 description 2
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- NRNCYVBFPDDJNE-UHFFFAOYSA-N pemoline Chemical compound O1C(N)=NC(=O)C1C1=CC=CC=C1 NRNCYVBFPDDJNE-UHFFFAOYSA-N 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000003245 working effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/002—Dynamic bit allocation
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
- G10L19/0204—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders using subband decomposition
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/04—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using predictive techniques
- G10L19/16—Vocoder architecture
- G10L19/18—Vocoders using multiple modes
- G10L19/24—Variable rate codecs, e.g. for generating different qualities using a scalable representation such as hierarchical encoding or layered encoding
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
- G10L19/0212—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders using orthogonal transformation
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
- G10L19/032—Quantisation or dequantisation of spectral components
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
Definitions
- the present invention relates to a spectrum smoothing apparatus, a coding apparatus, a decoding apparatus, a communication terminal apparatus, a base station apparatus and a spectrum smoothing method smoothing spectrum of speech signals.
- Patent literature 1 introduces transformation methods such as power transform and logarithmic transform as examples of non-linear processing.
- the spectrum smoothing apparatus employs a configuration to include: a time-frequency transformation section that performs a time-frequency transformation of an input signal and generates a frequency component; a subband dividing section that divides the frequency component into a plurality of subbands; a representative value calculating section that calculates a representative value of each divided subband by calculating an arithmetic mean and by using a multiplication calculation using a calculation result of the arithmetic mean; a non-linear transformation section that performs a non-linear transformation of representative values of the subbands; and a smoothing section that smoothes the representative values subjected to the non-linear transformation in the frequency domain.
- the spectrum smoothing method includes: a time-frequency transformation step of performing a time-frequency transformation of an input signal and generates a frequency component; a subband division step of dividing the frequency component into a plurality of subbands; a representative value calculation step of calculating a representative value of each divided subband by calculating an arithmetic mean and by using a multiplication calculation using a calculation result of the arithmetic mean; a non-linear transformation step of performing a non-linear transformation of representative values of the subbands; and a smoothing step of smoothing the representative values subjected to the non-linear transformation in the frequency domain.
- FIG.1 shows spectrum diagrams for explaining an overview of the spectrum smoothing method according to the present embodiment.
- FIG.1A shows a spectrum of an input signal.
- an input signal spectrum is divided into a plurality of subbands.
- FIG.1B shows how an input signal spectrum is divided into a plurality of subbands.
- the spectrum diagram of FIG.1 is for explaining an overview of the present invention, and the present invention is by no means limited to the number of subbands shown in the drawing.
- a representative value of each subband is calculated.
- samples in a subband are further divided into a plurality of subgroups.
- an arithmetic mean of absolute spectrum values is calculated per subgroup.
- a geometric mean of the arithmetic mean values of individual subgroups is calculated per subband.
- This geometric mean value is not an accurate geometric mean value yet, and, at this point, a value that is obtained by simply multiplying individual groups' arithmetic mean values may be calculated, and an accurate geometric mean value may be found after non-linear transformation (described later).
- the above processing is to reduce the amount of calculation processing, and it is equally possible to find an accurate geometric mean value at this point.
- FIG.1C shows representative values of individual subbands over an input signal spectrum shown with dotted lines.
- FIG.1C shows accurate geometric mean values as representative values, instead of values obtained by simply multiplying arithmetic mean values of individual subgroups.
- non-linear transformation for example, logarithmic transform
- smoothing processing is performed in the frequency domain.
- inverse non-linear transformation for example, inverse logarithmic transform
- FIG.1D shows a smoothed spectrum of each subband over an input signal spectrum shown with dotted lines.
- the spectrum smoothing apparatus smoothes an input spectrum, and outputs the spectrum after the smoothing (hereinafter “smoothed spectrum”) as an output signal.
- the spectrum smoothing apparatus divides an input signal every N samples (where N is a natural number), and performs smoothing processing per frame using N samples as one frame.
- FIG.2 shows a principal-part configuration of spectrum smoothing apparatus 100 according to the present embodiment.
- Spectrum smoothing apparatus 100 shown in FIG.2 is primarily formed with time-frequency transformation processing section 101, subband dividing section 102, representative value calculating section 103, non-linear transformation section 104, smoothing section 105 and inverse non-linear transformation section 106.
- Time-frequency transformation processing section 101 applies a fast Fourier transform (FFT) to input signal x n and finds a frequency component spectrum S1(k) (hereinafter "input spectrum”).
- FFT fast Fourier transform
- time-frequency transformation processing section 101 outputs input spectrum S1(k) to subband dividing section 102.
- Subband dividing section 102 divides input spectrum S1(k) received as input from time-frequency transformation processing section 101, into P subbands (where P is an integer equal to or greater than 2). Now, a case will be described below where subband dividing section 102 divides input spectrum S1(k) such that each subband contains the same number of samples. The number of samples may vary between subbands. Subband dividing section 102 outputs the spectrums divided per subband (hereinafter "subband spectrums"), to representative value calculating section 103.
- subband spectrums hereinafter “subband spectrums"
- Representative value calculating section 103 calculates a representative value for each subband of an input spectrum divided into subbands, received as input from subband dividing section 102, and outputs the representative value calculated per subband, to non-linear transformation section 104. The processing in representative value calculating section 103 will be described in detail later.
- FIG.3 shows an inner configuration of representative value calculating section 103.
- Representative value calculating section 103 shown in FIG.3 has arithmetic mean calculating section 201, and geometric mean calculating section 202.
- subband dividing section 102 outputs a subband spectrum to arithmetic mean calculating section 201.
- Arithmetic mean calculating section 201 divides each subband of the subband spectrum received as input into Q subgroups of subgroup 0, subgroup Q-1, etc. (where Q is an integer equal to or greater than 2). Now, a case will be described below where Q subgroups are each formed with R samples (R is an integer equal to or greater than 2). Although a case will be described below where Q subgroups are all formed with R samples, the number of samples may vary between subgroups.
- FIG.4 shows a sample configuration of subbands and subgroups.
- FIG.4 shows, as an example, a case where the number of samples to constitute one subband is eight, the number of subgroups Q to constitute one subband is two and the number of samples R in one subgroup is four.
- arithmetic mean calculating section 201 calculates an arithmetic mean of the absolute values of the spectrums (FFT coefficients) contained in each subgroup, using equation 1.
- AVE1 q is an arithmetic mean of the absolute values of the spectrums contained in subgroup q
- BS q is the index of the leading sample in subgroup q.
- P is the number of subbands.
- Equation 5 represents smoothing filtering processing, and, in this equation 5, MA_LEN is the order of smoothing filtering and W i is the smoothing filter weight.
- subband index p is at the top or near the last, spectrums are smoothed using equation 6 and equation 7 taking into account the boundary conditrions.
- smoothing section 105 performs smoothing based on simple moving average, as smoothing processing by smoothing filtering processing, as described above (when W i is 1 for all i's, smoothing is performed based on moving average).
- smoothing filtering processing as described above (when W i is 1 for all i's, smoothing is performed based on moving average).
- window function weight
- Hanning window or other window functions may be used.
- Inverse non-linear transformation section 106 outputs the smoothed spectrum values of all samples as a processing result of spectrum smoothing apparatus 100.
- subband dividing section 102 divides an input spectrum into a plurality of subbands
- representative value calculating section 103 calculates representative value per subband using an arithmetic mean or geometric mean
- non-linear transformation section 104 performs non-linear transformation having a characteristic of emphasizing greater values to each representative value
- smoothing section 105 smoothes representative values subjected to non-linear transformation per subband in the frequency domain.
- all samples of a spectrum are divided into a plurality of subbands, and, for each subband, a representative value is found by combining an arithmetic mean with multiplication calculation or geometric mean, and then smoothing is performed after the representative value is subjected to non-linear transformation, so that it is possible to maintain good speech quality and reduce the amount of calculation processing substantially.
- the present invention employs a configuration for calculating representative values of subbands by combining arithmetic means and geometric means of samples in subbands, so that it is possible to prevent speech quality degradation that can occur due to the variation of the scale of sample values in a subband when average values in the linear domain are used simply as representative values of subbands.
- the fast Fourier transform has been explained as an example of time-frequency transformation processing with the present embodiment
- the present invention is by no means limited to this, and other time-frequency transformation methods besides the fast Fourier transform (FFT) are equally applicable.
- the modified discrete cosine transform MDCT
- the fast Fourier transform FFT
- the present invention is applicable to configurations using the modified discrete cosine transform (MDCT) and other time-frequency transformation methods in a time-frequency transformation processing section.
- the present invention is not necessarily limited to the above configuration.
- smoothing section 105 is able to acquire a representative value having been subjected to non-linear transformation, per subband.
- the calculation of equation 4 in non-linear transformation section 104 may be omitted.
- the present invention is by no means limited to this and is equally applicable to a case where, for example, the number of samples to constitute a subgroup is one, that is, a case where a geometric mean value of all samples in a subband is used as a representative value of the subband without calculating an arithmetic mean value of each subgroup.
- non-linear transformation section 104 performs inverse logarithmic transformation as inverse non-linear transformation processing and inverse non-linear transformation section 106 performs inverse logarithmic transformation as inverse non-linear transformation processing
- this is by no means limiting, and it is equally possible to use power transform and others and perform inverse processing of non-linear transformation as inverse non-linear transformation processing.
- calculation of a radical root can be replaced by simple division (multiplication) by multiplying the reciprocal of the number of subgroups Q using equation 4, the fact that non-linear transformation section 104 performs logarithmic transform as non-linear transformation, should be credited for the reduction of the amount of calculation.
- the sampling frequency of an input signal is 32 kHz and one frame is 20 msec long, that is, if an input signal is comprised of 640 samples
- the present invention is by no means limited to this setting and is equally applicable to cases where different values are applied.
- the spectrum smoothing apparatus and spectrum smoothing method according to the present invention are applicable to any and all of spectrum smoothing devices or components that perform smoothing in the spectral domain, including speech coding apparatus and speech coding method, speech decoding apparatus and speech decoding method, and speech recognition apparatus and speech recognition method.
- the present invention is by no means limited to this, and is equally applicable to configurations where subgroups are divided such that a subgroup on the lower band side has a smaller number of samples and a subgroup on the higher band side has a larger number of samples.
- weighted moving average has been described as an example of smoothing processing with the present embodiment
- the present invention is by no means limited to this and is equally applicable to various smoothing processing.
- a moving average filter not the same between the left and the right and increase the number of taps in the higher band.
- the present invention is applicable to cases using a moving average filter that is asymmetrical between the left and the right and has a greater number of taps on the higher band side.
- FIG.5 is a block diagram showing a configuration of a communication system having a coding apparatus and decoding apparatus according to embodiment 2.
- the communication system has a coding apparatus and decoding apparatus that are mutually communicable via a transmission channel.
- the coding apparatus and decoding apparatus are usually mounted in a base station apparatus and communication terminal apparatus for use.
- Coding apparatus 301 divides an input signal every N samples (where N is a natural number) and performs coding on a per frame basis using N samples as one frame.
- n is the (n+1)-th signal component in the input signal divided every N samples.
- Input information having been subjected to coding (coded information) is transmitted to decoding apparatus 303 via transmission channel 302.
- Decoding apparatus 303 receives the coded information transmitted from coding apparatus 301 via transmission channel 302, and, by decoding this, acquires an output signal.
- FIG.6 is a block diagram showing an inner principal-part configuration of coding apparatus 301. If input signal sampling frequency is SR input , down-sampling processing section 311 down-samples the input signal sampling frequency from SR input to SR base (SR base ⁇ SR input ), and outputs input signal after down-sampling to first layer coding section 312 as a down-sampled input signal.
- First layer coding section 312 generates first layer coded information by encoding the down-sampled input signal received as input from down-sampling processing section 311, using a speech coding method of a CELP (Code Excited Linear Prediction) scheme, and outputs the generated first layer coded information to first layer decoding section 313 and coded information integrating section 317.
- CELP Code Excited Linear Prediction
- First layer decoding section 313 generates a first layer decoded signal by decoding the first layer coded information received as input from first layer coding section 312, using, for example, a CELP speech decoding method, and outputs the generated first layer decoded signal to up-sampling processing section 314.
- Up-sampling processing section 314 up-samples the sampling frequency of the input signal received as input from first layer decoding section 313 from SR base to SR input , and outputs the first layer decoded signal after up-sampling to time-frequency transformation processing section 315 as an up-sampled first layer decoded signal.
- Delay section 318 gives a delay of a predetermined length, to the input signal. This delay is to correct the time delay in down-sampling processing section 311, first layer coding section 312, first layer decoding section 313, and up-sampling processing section 314.
- MDCT modified discrete cosine transform
- time-frequency transformation processing section 315 initializes buf1 n and buf2 n using the initial value "0" according to equation 9 and equation 10 below.
- time-frequency transformation processing section 315 performs an MDCT of input signal x n and up-sampled first layer decoded signal y n , and finds MDCT coefficient S2(k) of the input signal (hereinafter “input spectrum”) and MDCT coefficient S1(k) of up-sampled first layer decoded signal y n (hereinafter “first layer decoded spectrum”).
- Time-frequency transformation processing section 315 finds x n ', which is a vector combining input signal x n and buffer buf1 n from equation 13 below. Time-frequency transformation processing section 315 also finds y n ' which is a vector combining up-sampled first layer decoded signal y n and buffer buf2 n .
- time-frequency transformation processing section 315 updates buffer buf1 n and buf2 n using equation 15 and equation 16.
- time-frequency transformation processing section 315 outputs input spectrum S2(k) and first layer decoded spectrum S1(k) to second layer coding section 316.
- Second layer coding section 316 generates second layer coded information using input spectrum S2(k) and first layer decoded spectrum S1(k) received as input from time-frequency transformation processing section 315, and outputs the generated second layer coded information to coded information integrating section 317.
- the details of second layer coding section 316 will be described later.
- Coded information integrating section 317 integrates the first layer coded information received as input from first layer coding section 312 and the second layer coded information received as input from second layer coding section 316, and, if necessary, attaches a transmission error correction code to the integrated information source code, and outputs the result to transmission channel 302 as coded information.
- Second layer coding section 316 has band dividing section 360, spectrum smoothing section 361, filter state setting section 362, filtering section 363, search section 364, pitch coefficient setting section 365, gain coding section 366 and multiplexing section 367, and these sections perform the following operations.
- FIG.8 shows an internal configuration of spectrum smoothing section 361.
- Spectrum smoothing section 361 is primarily configured with subband dividing section 102, representative value calculating section 103, non-linear transformation section 104, smoothing section 105, and inverse non-linear transformation section 106. These components are the same as the components described with embodiment 1 and will be assigned the same reference numerals without explanations.
- Filtering section 363 outputs estimated spectrum S2 p '(k) of subband SB p to search section 364. The details of filtering processing in filtering section 363 will be described later.
- the number of multiple taps may be any value (integer) equal to or greater than 1.
- This degree of similarity is calculated by, for example, correlation calculation.
- Processing in filtering section 363, search section 364 and pitch coefficient setting section 365 constitute closed-loop search processing per subband, and, in every closed loop, search section 364 calculates the degree of similarity with respect to each pitch coefficient by variously modifying pitch coefficient T received as input from pitch coefficient setting section 365 into filtering section 363.
- search section 364 finds optimal pitch coefficient T p ' to maximize the degree of similarity (in the range of Tmin ⁇ Tmax), and outputs P optimal pitch coefficients to multiplexing section 367.
- pitch coefficient setting section 365 performs closed-loop search processing corresponding to first subband SB 0 with filtering section 363 and search section 364, modifies pitch coefficient T gradually in a predetermined search range between Tmin and Tmax and sends outputs to filtering section 363 sequentially.
- BL j is the minimum frequency of the (j+1)-th subband
- BH j is the maximum frequency of the (j+1)-th subband.
- gain coding section 366 calculates the amount of variation, V j , of the spectral power of estimated spectrum S2'(k) per subband, with respect to input spectrum S2(k), using equation 19 below.
- gain coding section 366 encodes amount of variation V j , and outputs an index corresponding to coded amount of variation VQ j to multiplexing section 367.
- the transfer function F(z) of the filter used in filtering section 363 is represented by equation 20 below.
- T is a pitch coefficient provided from pitch coefficient setting section 365
- ⁇ i is a filter coefficient stored inside in advance.
- FIG.10 is a flowchart showing the steps of processing for searching for optimal pitch coefficient T p ' for subband SB p in search section 364.
- search section 364 initializes the minimum degree of similarity, D min , which is a variable for saving the minimum value of the degree of similarity, to "+ ⁇ " (ST 110).
- M' is the number of samples upon calculating the degree of similarity D, and may assume arbitrary values equal to or smaller than the bandwidth of each subband.
- S2 p '(k) is not present in equation 22 but is represented using BS p and S2'(k).
- search section 364 determines whether or not the calculated degree of similarity, D, is smaller than the minimum degree of similarity, D min (ST 130). If degree of similarity D calculated in ST 120 is smaller than minimum degree of similarity D min ("YES" in ST 130), search section 364 substitutes degree of similarity D in minimum degree of similarity D min (ST 140). On the other hand, if degree of similarity D calculated in ST 120 is equal to or greater than minimum degree of similarity D min ("NO" in ST 130), search section 364 determines whether or not processing in the search range has finished. That is to say, search section 364 determines whether or not the degree of similarity has been calculated with respect to all pitch coefficients in the search range in ST 120 according to equation 22 above (ST 150).
- Search section 364 returns to ST 120 again when the processing has not finished over the search range ("NO" in ST 150). Then, search section 364 calculates the degree of similarity according to equation 22, for different pitch coefficients from the case of calculating the degree of similarity according to equation 22 in earlier ST 120. On the other hand, when processing is finished over the search range ("YES" in ST 150), search section 364 outputs pitch coefficient T corresponding to the minimum degree of similarity, to multiplexing section 367, as optimal pitch coefficient T p ' (ST 160).
- FIG.11 is a block diagram showing an internal principal-part configuration of decoding apparatus 303.
- coded information demultiplexing section 331 demultiplexs between first layer coded information and second layer coded information in coded information received as input, outputs the first layer coded information to first layer decoding section 332, and outputs the second layer coded information to second layer decoding section 335.
- First layer decoding section 332 decodes the first layer coded information received as input from coded information demultiplexing section 331, and outputs the generated first layer decoded signal to up-sampling processing section 333.
- the operations of first layer decoding section 332 are the same as in first layer decoding section 313 shown in FIG.6 and will not be explained in detail.
- Up-sampling processing section 333 performs processing of up-sampling the sampling frequency from SR base to SR input with respect to the first layer decoded signal received as input from first layer decoding section 332, and outputs the resulting up-sampled first layer decoded signal to time-frequency transformation processing section 334.
- Time-frequency transformation processing section 334 applies orthogonal transformation processing (MDCT) to the up-sampled first layer decoded signal received as input from up-sampling processing section 333, and outputs the MDCT coefficient S1(k) (hereinafter "first layer decoded spectrum") of the resulting up-sampled first layer decoded signal to second layer decoding section 335.
- MDCT orthogonal transformation processing
- Second layer decoding section 335 generates a second layer decoded signal including higher band components using first layer decoded spectrum S1(k) received as input from time-frequency transformation processing section 334 and second layer coded information received as input from coded information demultiplexing section 331, and outputs this as an output signal.
- FIG. 12 is a block diagram showing an internal principal-part configuration of second layer decoding section 335 shown in FIG.11 .
- the processing in spectrum smoothing section 352 is the same as the processing in spectrum smoothing section 361 in second layer coding section 316 and therefore will not be described here.
- the configuration and operations of filter state setting section 353 are the same as filter state setting section 362 shown in FIG.7 and will not be described in detail here.
- Gain decoding section 355 decodes the index of coded variation amount VQ j received as input from demultiplexing section 351, and finds amount of variation VQ j which is a quantized value of amount of variation V j .
- S ⁇ 3 k S ⁇ 2 ⁇ ⁇ k ⁇ VQ j BL l ⁇ k ⁇ BH j , for all j
- S ⁇ 3 k S ⁇ 1 k 0 ⁇ k ⁇ FL
- Time-frequency transformation processing section 357 performs orthogonal transformation of decoded spectrum S3(k) received as input from spectrum adjusting section 356 into a time domain signal, and outputs the resulting second layer decoded signal as an output signal.
- adequate processing such as windowing or overlap addition is performed to prevent discontinuities from being produced between frames.
- Time-frequency transformation processing section 357 has buffer buf'(k) inside and initializes buffer buf'(k) as shown with equation 25 below.
- time-frequency transformation processing section 357 finds second layer decoded signal y n " using second layer decoded spectrum S3(k) received as input from spectrum adjusting section 356.
- Z4(k) is a vector combining decoded spectrum S3(k) and buffer buf'(k) as shown by equation 27 below.
- time-frequency transformation processing section 357 updates buffer buf'(k) according to equation 28 below.
- time-frequency transformation processing section 357 outputs decoded signal y n " as an output signal.
- the present invention is by no means limited to this and is equally applicable to a configuration for performing smoothing processing for a lower band spectrum of an input signal, estimating a higher band spectrum from a smoothed input spectrum and then coding the higher band spectrum.
- the present invention is equally applicable to cases where a signal processing program is recorded or written in a computer-readable recording medium such as a CD and DVD and operated, and provides the same working effects and advantages as with the present embodiment.
- each function block employed in the above descriptions of embodiments may typically be implemented as an LSI constituted by an integrated circuit. These may be individual chips or partially or totally contained on a single chip. "LSI” is adopted here but this may also be referred to as “IC,” “system LSI,” “super LSI,” or “ultra LSI” depending on differing extents of integration.
- circuit integration is not limited to LSPs, and implementation using dedicated circuitry or general purpose processors is also possible.
- FPGA Field Programmable Gate Array
- reconfigurable processor where connections and settings of circuit cells in an LSI can be regenerated is also possible.
- the spectrum smoothing apparatus, coding apparatus, decoding apparatus, communication terminal apparatus, base station apparatus and spectrum smoothing method according to the present invention make possible smoothing in the frequency domain by a small of amount and are therefore applicable to, for example, packet communication systems, mobile communication systems and so forth.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Quality & Reliability (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
Description
- The present invention relates to a spectrum smoothing apparatus, a coding apparatus, a decoding apparatus, a communication terminal apparatus, a base station apparatus and a spectrum smoothing method smoothing spectrum of speech signals.
- When speech/audio signals are transmitted in a packet communication system typified by Internet communication and a mobile communication system, a compression/coding technique is often used to improve the transmission rate of speech/audio signals. Furthermore, in recent years, in addition to a demand for simply encoding speech/audio signals at low bit rates, there is an increasing demand for a technique to encode speech/audio signals in high quality.
- To meet this demand, studies are underway to develop various techniques to perform orthogonal transformation (i.e. time-frequency transformation) of a speech signal to extract frequency components (i.e. spectrum) of the speech signal and apply various processing such as linear transformation and non-linear transformation to the calculated spectrum to improve the quality of the decoded signal (see, for example, patent literature 1). According to the method disclosed in
patent literature 1, first, a frequency spectrum contained in a speech signal of a certain time length is analyzed, and then non-linear transformation processing to emphasize greater spectrum power values is applied to the analyzed spectrum. Next, linear smoothing processing for the spectrum subjected to non-linear transformation processing, is performed in the frequency domain. After this, inverse non-linear transformation processing is performed to cancel non-linear transformation characteristics, and, furthermore, inverse smoothing processing is performed to cancel smoothing characteristics, so that noise components included in the speech signal over the entire band are suppressed. Thus, with the method disclosed inpatent literature 1, all samples of a spectrum acquired from a speech signal are subjected to non-linear transformation processing and then the spectrum is smoothed, so that the speech signal is acquired in good quality.Patent literature 1 introduces transformation methods such as power transform and logarithmic transform as examples of non-linear processing. -
-
PTL 1
Japanese Patent Application Laid-Open No.2002-244695 - PTL 2
WO 2007/037361 -
- NPL 1
Yuichiro TAKAMIZAWA, Toshiyuki NOMURA and Masao IKEKAWA, "High-Quality and Processor-Efficient Implementation of and MPEG-2 AAC Encoder", IEICE TRANS. INF. &SYST., VOL.E86-D, No.3 MARCH 2003 - However, with the method disclosed in
patent literature 1, non-linear transformation processing needs to be performed for all samples of a spectrum acquired from a speech signal, and therefore there is a problem that the amount of calculation processing is enormous. Furthermore, if only part of samples of a spectrum are extracted to reduce the amount of calculation processing, sufficiently high speech quality cannot be always achieved by simply performing spectrum smoothing after non-linear transformation. - Based upon a configuration for performing non-linear transformation of a spectrum value calculated from a speech signal and then smoothing the spectrum, it is an object of the present invention to provide a spectrum smoothing apparatus, a coding apparatus, a decoding apparatus, a communication terminal apparatus, a base station apparatus and a spectrum smoothing method, whereby good speech quality is maintained and the amount of calculation processing can be reduced substantially.
- The spectrum smoothing apparatus according to the present invention employs a configuration to include: a time-frequency transformation section that performs a time-frequency transformation of an input signal and generates a frequency component; a subband dividing section that divides the frequency component into a plurality of subbands; a representative value calculating section that calculates a representative value of each divided subband by calculating an arithmetic mean and by using a multiplication calculation using a calculation result of the arithmetic mean; a non-linear transformation section that performs a non-linear transformation of representative values of the subbands; and a smoothing section that smoothes the representative values subjected to the non-linear transformation in the frequency domain.
- The spectrum smoothing method according to the present invention includes: a time-frequency transformation step of performing a time-frequency transformation of an input signal and generates a frequency component; a subband division step of dividing the frequency component into a plurality of subbands; a representative value calculation step of calculating a representative value of each divided subband by calculating an arithmetic mean and by using a multiplication calculation using a calculation result of the arithmetic mean; a non-linear transformation step of performing a non-linear transformation of representative values of the subbands; and a smoothing step of smoothing the representative values subjected to the non-linear transformation in the frequency domain.
- With the present invention, it is possible to maintain good speech quality and reduce the amount of calculation processing substantially.
-
-
FIG.1 provides spectrum overviews showing an overview of processing according toembodiment 1 of the present invention; -
FIG.2 is a block diagram showing a principal-part configuration of a spectrum smoothing apparatus according toembodiment 1; -
FIG.3 is a block diagram showing a principal-part configuration of a representative value calculating section according toembodiment 1; -
FIG.4 is an overview showing a configuration of subbands and subgroups of an input signal according toembodiment 1; -
FIG.5 is a block diagram showing a configuration of a communication system having a coding apparatus and decoding apparatus according to embodiment 2 of the present invention; -
FIG.6 is a block diagram showing an inner principal-part of the coding apparatus according to embodiment 2 shown inFIG.5 ; -
FIG.7 is a block diagram showing an inner principal-part configuration of the second layer coding section according to embodiment 2 shown inFIG.6 ; -
FIG.8 is a block diagram showing a principal-part configuration of the spectrum smoothing apparatus according to embodiment 2 shown inFIG.7 ; -
FIG.9 shows a diagram for explaining the details of the filtering processing in the filtering section according to embodiment 2 shwon inFIG.7 ; -
FIG.10 is a flowchart for explaining the steps of processing for searching for optimal pitch coefficient Tp' with respect to subband SBp in the search section according to embodiment 2 shwon inFIG.7 ; -
FIG.11 is a block diagram showing an inner principal-part configuration of the decoding apparatus according to embodiment 2 shown inFIG.5 ; and -
FIG.12 is a block diagram showing an inner principal-part configuration of the second layer decoding section according to embodiment 2 shown inFIG.11 . - Embodiments of the present invention will be described in detail with reference to the accompanying drawings.
- First, an overview of the spectrum smoothing method according to an embodiment of the present invention will be described using
FIG.1. FIG.1 shows spectrum diagrams for explaining an overview of the spectrum smoothing method according to the present embodiment. -
FIG.1A shows a spectrum of an input signal. With the present embodiment, first, an input signal spectrum is divided into a plurality of subbands.FIG.1B shows how an input signal spectrum is divided into a plurality of subbands. The spectrum diagram ofFIG.1 is for explaining an overview of the present invention, and the present invention is by no means limited to the number of subbands shown in the drawing. - Next, a representative value of each subband is calculated. To be more specific, samples in a subband are further divided into a plurality of subgroups. Then, an arithmetic mean of absolute spectrum values is calculated per subgroup.
- Next, a geometric mean of the arithmetic mean values of individual subgroups is calculated per subband. This geometric mean value is not an accurate geometric mean value yet, and, at this point, a value that is obtained by simply multiplying individual groups' arithmetic mean values may be calculated, and an accurate geometric mean value may be found after non-linear transformation (described later). The above processing is to reduce the amount of calculation processing, and it is equally possible to find an accurate geometric mean value at this point.
- A geometric mean value found this way may be used as a representative value of each subband.
FIG.1C shows representative values of individual subbands over an input signal spectrum shown with dotted lines. For ease of explanation,FIG.1C shows accurate geometric mean values as representative values, instead of values obtained by simply multiplying arithmetic mean values of individual subgroups. - Next, referring to each subband's representative value, non-linear transformation (for example, logarithmic transform) is performed for a spectrum of an input signal such that greater spectrum power values are emphasized, and then smoothing processing is performed in the frequency domain. Afterward, inverse non-linear transformation (for example, inverse logarithmic transform) is performed, and a smoothed spectrum is calculated in each subband.
FIG.1D shows a smoothed spectrum of each subband over an input signal spectrum shown with dotted lines. - By means of this processing, it is possible to perform spectrum smoothing in the logarithmic domain while reducing speech quality degradation and reducing the amount of calculation processing substantially. Now, a configuration of a spectrum smoothing apparatus providing the above advantage, according to an embodiment of the present invention, will be described.
- The spectrum smoothing apparatus according to the present embodiment smoothes an input spectrum, and outputs the spectrum after the smoothing (hereinafter "smoothed spectrum") as an output signal. To be more specific, the spectrum smoothing apparatus divides an input signal every N samples (where N is a natural number), and performs smoothing processing per frame using N samples as one frame. Here, an input signal that is subject to smoothing processing is represented as "xn" (n=0, ..., N-1).
-
FIG.2 shows a principal-part configuration ofspectrum smoothing apparatus 100 according to the present embodiment. -
Spectrum smoothing apparatus 100 shown inFIG.2 is primarily formed with time-frequencytransformation processing section 101, subband dividingsection 102, representativevalue calculating section 103,non-linear transformation section 104, smoothingsection 105 and inversenon-linear transformation section 106. - Time-frequency
transformation processing section 101 applies a fast Fourier transform (FFT) to input signal xn and finds a frequency component spectrum S1(k) (hereinafter "input spectrum"). - Then, time-frequency
transformation processing section 101 outputs input spectrum S1(k) tosubband dividing section 102. -
Subband dividing section 102 divides input spectrum S1(k) received as input from time-frequencytransformation processing section 101, into P subbands (where P is an integer equal to or greater than 2). Now, a case will be described below wheresubband dividing section 102 divides input spectrum S1(k) such that each subband contains the same number of samples. The number of samples may vary between subbands.Subband dividing section 102 outputs the spectrums divided per subband (hereinafter "subband spectrums"), to representativevalue calculating section 103. - Representative
value calculating section 103 calculates a representative value for each subband of an input spectrum divided into subbands, received as input fromsubband dividing section 102, and outputs the representative value calculated per subband, tonon-linear transformation section 104. The processing in representativevalue calculating section 103 will be described in detail later. -
FIG.3 shows an inner configuration of representativevalue calculating section 103. Representativevalue calculating section 103 shown inFIG.3 has arithmeticmean calculating section 201, and geometricmean calculating section 202. - First, subband dividing
section 102 outputs a subband spectrum to arithmeticmean calculating section 201. - Arithmetic
mean calculating section 201 divides each subband of the subband spectrum received as input into Q subgroups ofsubgroup 0, subgroup Q-1, etc. (where Q is an integer equal to or greater than 2). Now, a case will be described below where Q subgroups are each formed with R samples (R is an integer equal to or greater than 2). Although a case will be described below where Q subgroups are all formed with R samples, the number of samples may vary between subgroups. -
FIG.4 shows a sample configuration of subbands and subgroups.FIG.4 shows, as an example, a case where the number of samples to constitute one subband is eight, the number of subgroups Q to constitute one subband is two and the number of samples R in one subgroup is four. - Next, for each of the Q subgroups, arithmetic
mean calculating section 201 calculates an arithmetic mean of the absolute values of the spectrums (FFT coefficients) contained in each subgroup, usingequation 1.equation 1, AVE1q is an arithmetic mean of the absolute values of the spectrums contained in subgroup q, and BSq is the index of the leading sample in subgroup q. - Next, arithmetic
mean calculating section 201 outputs arithmetic mean value spectrums calculated per subband, AVE1q (q=0∼Q-1) (subband arithmetic mean value spectrums), to geometricmean calculating section 202. - Geometric
mean calculating section 202 multiplies arithmetic mean value spectrums AVE1q (q=0∼Q-1) of all subbands received as input from arithmeticmean calculating section 201, as shown in equation 2, and calculates a representative spectrum, AVE2p (p=0∼P-1), for each subband. - Next, geometric
mean calculating section 202 outputs calculated subband representative value spectrums AVE2p (p=0∼P-1) tonon-linear transformation section 104. -
Non-linear transformation section 104 applies non-linear transformation having a characteristic of emphasizing greater representative values, to subband representative value spectrums AVE2p, received as input from geometricmean calculating section 202, using equation 3, and calculates first subband logarithmic representative value spectrums, AVE3p (p=0∼P-1). A case will be described here where logarithmic transform is performed as non-linear transformation processing. -
- Although in the processing of equation 2 in geometric
mean calculating section 202 subband arithmetic mean value spectrums AVE1p of individual subbands are simply multiplied, in the processing of equation 4 innon-linear transformation section 104, a geometric mean is calculated. With the present embodiment, transformation into the logarithmic domain is performed using equation 3, and then multiplication by the reciprocal of the number of subgroups, Q, is performed using equation 4. By this means, radical root calculation, which involves a large amount of calculation, can be replaced by simple division. Furthermore, when the number of subgroups, Q, is a constant, the radical root calculation can be replaced by simple multiplication, by calculating the reciprocal of Q in advance, so that the amount of calculation can be reduced further. - Next,
non-linear transformation section 104 outputs second subband logarithmic representative value spectrums AVE4p (p=0∼P-1) calculated using equation 4, to smoothingsection 105. - Referring back to
FIG.2 again, smoothingsection 105 smoothes second subband logarithmic representative value spectrums AVE4p (p=0∼P-1) received as input fromnon-linear transformation section 104, in the frequency domain, using equation 5, and calculates logarithmic smoothed spectrums AVE5p (p=0∼P-1). - Equation 5 represents smoothing filtering processing, and, in this equation 5, MA_LEN is the order of smoothing filtering and Wi is the smoothing filter weight.
- Furthermore, in equation 5 provides a method of calculating a logarithmic smoothed spectrum when subband index p is p>=(MA_LEN-1)/2 and p<=P-1-(MA_LEN-1)/2. When subband index p is at the top or near the last, spectrums are smoothed using equation 6 and equation 7 taking into account the boundary conditrions.
- Furthermore, smoothing
section 105 performs smoothing based on simple moving average, as smoothing processing by smoothing filtering processing, as described above (when Wi is 1 for all i's, smoothing is performed based on moving average). For the window function (weight), Hanning window or other window functions may be used. - Next, smoothing
section 105 outputs calculated smoothed spectrums AVE5p (p=0∼P-1) to inversenon-linear transformation section 106. - Inverse
non-linear transformation section 106 performs inverse logarithmic transformation as inverse non-linear transformation for logarithmic smoothed spectrums AVE5p (p=0∼P-1) received as input from smoothingsection 105. Inversenon-linear transformation section 106 performs inverse logarithmic transformation for logarithmic smoothed spectrums AVE5p (p=0∼P-1) using equation 8, and calculates smoothed spectrum AVE6p (p=0∼P-1). - Furthermore, inverse
non-linear transformation section 106 calculates a smoothed spectrum of all samples using the values of samples in each subband as the values of linear domain smoothed spectrum AVE6p (p=0∼P-1). - Inverse
non-linear transformation section 106 outputs the smoothed spectrum values of all samples as a processing result ofspectrum smoothing apparatus 100. - The spectrum smoothing apparatus and spectrum smoothing method according to the present invention have been described.
- As described above, with the present embodiment, subband dividing
section 102 divides an input spectrum into a plurality of subbands, representativevalue calculating section 103 calculates representative value per subband using an arithmetic mean or geometric mean,non-linear transformation section 104 performs non-linear transformation having a characteristic of emphasizing greater values to each representative value, and smoothingsection 105 smoothes representative values subjected to non-linear transformation per subband in the frequency domain. - Thus, all samples of a spectrum are divided into a plurality of subbands, and, for each subband, a representative value is found by combining an arithmetic mean with multiplication calculation or geometric mean, and then smoothing is performed after the representative value is subjected to non-linear transformation, so that it is possible to maintain good speech quality and reduce the amount of calculation processing substantially.
- As described above, the present invention employs a configuration for calculating representative values of subbands by combining arithmetic means and geometric means of samples in subbands, so that it is possible to prevent speech quality degradation that can occur due to the variation of the scale of sample values in a subband when average values in the linear domain are used simply as representative values of subbands.
- Although the fast Fourier transform (FFT) has been explained as an example of time-frequency transformation processing with the present embodiment, the present invention is by no means limited to this, and other time-frequency transformation methods besides the fast Fourier transform (FFT) are equally applicable. For example, according to
patent literature 1, upon calculation of perceptual masking values (seeFIG.2 ), the modified discrete cosine transform (MDCT), not the fast Fourier transform (FFT), is used to calculate frequency components (spectrum). Thus, the present invention is applicable to configurations using the modified discrete cosine transform (MDCT) and other time-frequency transformation methods in a time-frequency transformation processing section. - In the configuration described above, geometric
mean calculating section 202 multiplies an arithmetic mean value spectrum AVE1q (q=0∼Q-1), and does not calculate radical roots. That is to say, strictly speaking, geometricmean calculating section 202 does not calculate geometric mean values, because, as explained above, innon-linear transformation section 104, transformation into the logarithmic domain is performed using equation 3 as non-linear transformation processing and then multiplication by the reciprocal of the number of subgroups Q is performed using equation 4, so that it is possible to replace radical root calculation by simple division (multiplication) and consequently reduce the amount of calculation. - Consequently, the present invention is not necessarily limited to the above configuration. The present invention is equally applicable to, for example, a configuration for multiplying, in geometric
mean calculating section 202, arithmetic mean value spectrums AVE1q (q=0∼Q-1) by the values of arithmetic mean value spectrums per subband, and then calculating a radical root of the number of subgroups and outputting the calculated radical root tonon-linear transformation section 104 as subband representative value spectrums AVE2p (p=0∼P-1). Either way, smoothingsection 105 is able to acquire a representative value having been subjected to non-linear transformation, per subband. In this case, the calculation of equation 4 innon-linear transformation section 104 may be omitted. - A case has been described above with the present embodiment where a representative value of each subband is calculated by, first, calculating an arithmetic mean value of a subgroup, and next finding a geometric mean value of the arithmetic mean values of all subgroups in a subband. However, the present invention is by no means limited to this and is equally applicable to a case where, for example, the number of samples to constitute a subgroup is one, that is, a case where a geometric mean value of all samples in a subband is used as a representative value of the subband without calculating an arithmetic mean value of each subgroup. In this configuration again, as described above, rather than calculating an accurate geometric mean value, it is possible to calculate a geometric mean value in the logarithmic domain by performing non-linear transformation and then performing multiplication by the reciprocal of the number of subgroups.
- In the above description, all samples in a subband have the same spectrum value in inverse
non-linear transformation section 106. However, the present invention is by no means limited to this, and it is equally possible to provide an inverse smoothing processing section after inversenon-linear transformation section 106 so that the inverse smoothing processing section may assign weight to samples in each subband and perform inverse smoothing processing. This inverse smoothing processing needs not be completely opposite to smoothingsection 105. - Although a case has been described with the above description where
non-linear transformation section 104 performs inverse logarithmic transformation as inverse non-linear transformation processing and inversenon-linear transformation section 106 performs inverse logarithmic transformation as inverse non-linear transformation processing, this is by no means limiting, and it is equally possible to use power transform and others and perform inverse processing of non-linear transformation as inverse non-linear transformation processing. However, given that calculation of a radical root can be replaced by simple division (multiplication) by multiplying the reciprocal of the number of subgroups Q using equation 4, the fact thatnon-linear transformation section 104 performs logarithmic transform as non-linear transformation, should be credited for the reduction of the amount of calculation. Consequently, if processing that is different from logarithmic transform is performed as non-linear transformation processing, it is then equally possible to calculate a representative value per subband by calculating a geometric mean value of arithmetic mean values of subgroups and apply non-linear processing to the representative values. - Furthermore, as for the number of subbands and the number of subgroups, if, for example, the sampling frequency of an input signal is 32 kHz and one frame is 20 msec long, that is, if an input signal is comprised of 640 samples, it is possible to, for example, set the number of subbands to eighty, the number of subgroups to two, the number of samples per subgroup to four, and the order of smoothing filtering to seven, for example. The present invention is by no means limited to this setting and is equally applicable to cases where different values are applied.
- The spectrum smoothing apparatus and spectrum smoothing method according to the present invention are applicable to any and all of spectrum smoothing devices or components that perform smoothing in the spectral domain, including speech coding apparatus and speech coding method, speech decoding apparatus and speech decoding method, and speech recognition apparatus and speech recognition method. For example, although, with the bandwidth enhancement technique disclosed in patent literature 2, processing for calculating a spectral envelope from LPCs (Linear Predictive Coefficients), and, based on this calculated spectral envelope, removing the spectral envelope from the lower band spectrum, is used to calculate parameters for generating a higher band spectrum, it is equally possible to use a smoothed spectrum calculated by applying the spectrum smoothing method according to the present invention to a lower band spectrum instead of the spectral envelope used in spectral envelope removing processing in patent literature 2.
- Furthermore, although a configuration has been explained with the present embodiment where an input spectrum S1(k) is divided into P subbands (where P is an integer equal to or greater than 2) all having the same number of samples, the present invention is by no means limited to this and is equally applicable to a configuration in which the number of samples varies between subbands. Fro example, a configuration is possible in which subbands are divided such that a subband on the lower band side has a smaller number of samples and a subband on the higher band side has a greater number of samples. Generally speaking, in human perception, frequency resolution decreases in the higher band side, so that more efficient spectrum smoothing is made possible with the above configuration. The same applies to subgroups to constitute each subband. Although a case has been described above with the present embodiment where Q subgroups are all formed with R samples, the present invention is by no means limited to this, and is equally applicable to configurations where subgroups are divided such that a subgroup on the lower band side has a smaller number of samples and a subgroup on the higher band side has a larger number of samples.
- Although weighted moving average has been described as an example of smoothing processing with the present embodiment, the present invention is by no means limited to this and is equally applicable to various smoothing processing. For example, as described above, in a configuration in which the number of samples varies between subbands (that is, the number of samples increases in the higher band), it is possible to make the number of taps in a moving average filter not the same between the left and the right and increase the number of taps in the higher band. When the number of samples increases in subbands in the higher band, it is possible to perform perceptually more adequate smoothing processing by using a moving average filter having a small number of taps in the higher band side. The present invention is applicable to cases using a moving average filter that is asymmetrical between the left and the right and has a greater number of taps on the higher band side.
- A configuration will be described now with the present embodiment where the spectrum smoothing processing explained with
embodiment 1 is used in preparatory processing upon band enhancement coding disclosed in patent literature 2. -
FIG.5 is a block diagram showing a configuration of a communication system having a coding apparatus and decoding apparatus according to embodiment 2. InFIG.5 , the communication system has a coding apparatus and decoding apparatus that are mutually communicable via a transmission channel. The coding apparatus and decoding apparatus are usually mounted in a base station apparatus and communication terminal apparatus for use. -
Coding apparatus 301 divides an input signal every N samples (where N is a natural number) and performs coding on a per frame basis using N samples as one frame. The input signal to be subject to coding is represented as xn (n=0, ..., N-1). n is the (n+1)-th signal component in the input signal divided every N samples. Input information having been subjected to coding (coded information) is transmitted todecoding apparatus 303 viatransmission channel 302. -
Decoding apparatus 303 receives the coded information transmitted fromcoding apparatus 301 viatransmission channel 302, and, by decoding this, acquires an output signal. -
FIG.6 is a block diagram showing an inner principal-part configuration ofcoding apparatus 301. If input signal sampling frequency is SRinput, down-sampling processing section 311 down-samples the input signal sampling frequency from SRinput to SRbase (SRbase<SRinput), and outputs input signal after down-sampling to firstlayer coding section 312 as a down-sampled input signal. - First
layer coding section 312 generates first layer coded information by encoding the down-sampled input signal received as input from down-sampling processing section 311, using a speech coding method of a CELP (Code Excited Linear Prediction) scheme, and outputs the generated first layer coded information to firstlayer decoding section 313 and codedinformation integrating section 317. - First
layer decoding section 313 generates a first layer decoded signal by decoding the first layer coded information received as input from firstlayer coding section 312, using, for example, a CELP speech decoding method, and outputs the generated first layer decoded signal to up-sampling processing section 314. - Up-
sampling processing section 314 up-samples the sampling frequency of the input signal received as input from firstlayer decoding section 313 from SRbase to SRinput, and outputs the first layer decoded signal after up-sampling to time-frequencytransformation processing section 315 as an up-sampled first layer decoded signal. -
Delay section 318 gives a delay of a predetermined length, to the input signal. This delay is to correct the time delay in down-sampling processing section 311, firstlayer coding section 312, firstlayer decoding section 313, and up-sampling processing section 314. - Time-frequency
transformation processing section 315 has buffer buf1n and buf2n (n=0,...,N-1) inside, and applies a modified discrete cosine transform (MDCT) to input signal xn and up-sampled first layer decoded signal yn received as input from up-sampling processing section 314. - Next, the orthogonal transformation processing in time-frequency
transformation processing section 315 will be described as to its calculation step and data output to internal buffers. -
- Next, time-frequency
transformation processing section 315 performs an MDCT of input signal xn and up-sampled first layer decoded signal yn, and finds MDCT coefficient S2(k) of the input signal (hereinafter "input spectrum") and MDCT coefficient S1(k) of up-sampled first layer decoded signal yn (hereinafter "first layer decoded spectrum"). - K is the index of each sample in a frame. Time-frequency
transformation processing section 315 finds xn', which is a vector combining input signal xn and buffer buf1n from equation 13 below. Time-frequencytransformation processing section 315 also finds yn' which is a vector combining up-sampled first layer decoded signal yn and buffer buf2n. -
- Then, time-frequency
transformation processing section 315 outputs input spectrum S2(k) and first layer decoded spectrum S1(k) to secondlayer coding section 316. - Second
layer coding section 316 generates second layer coded information using input spectrum S2(k) and first layer decoded spectrum S1(k) received as input from time-frequencytransformation processing section 315, and outputs the generated second layer coded information to codedinformation integrating section 317. The details of secondlayer coding section 316 will be described later. - Coded
information integrating section 317 integrates the first layer coded information received as input from firstlayer coding section 312 and the second layer coded information received as input from secondlayer coding section 316, and, if necessary, attaches a transmission error correction code to the integrated information source code, and outputs the result totransmission channel 302 as coded information. - Next, the inner principal-part configuration of second
layer coding section 316 shown inFIG.6 will be described usingFIG.7 . - Second
layer coding section 316 hasband dividing section 360,spectrum smoothing section 361, filterstate setting section 362, filteringsection 363,search section 364, pitchcoefficient setting section 365, gaincoding section 366 andmultiplexing section 367, and these sections perform the following operations. -
Band dividing section 360 divides the higher band part (FL<=k<FH) of input spectrum S2(k) received as input from time-frequencytransformation processing section 315 into P subbands SBp (p=0, 1, ... , P-1). Then,band dividing section 360 outputs bandwidth BWp (p=0, 1, ... , P-1) and leading index BSp (p=0, 1, ... , P-1) (FL<=BSp<FH) of each divided subband tofiltering section 363,search section 364 andmultiplexing section 367 as band division information. The part in input spectrum S2(k) corresponding to subband SBp will be referred to as subband spectrum S2p(k) (BSp<=k<BSp+BWp). -
Spectrum smoothing section 361 applies smoothing processing to first layer decoded spectrum S1(k) (0<=k<FL) received as input from time-frequencytransformation processing section 315, outputs smoothed first layer decoded spectrum S1'(k) (0<=k<FL) after smoothing processing, to filterstate setting section 362. -
FIG.8 shows an internal configuration ofspectrum smoothing section 361.Spectrum smoothing section 361 is primarily configured withsubband dividing section 102, representativevalue calculating section 103,non-linear transformation section 104, smoothingsection 105, and inversenon-linear transformation section 106. These components are the same as the components described withembodiment 1 and will be assigned the same reference numerals without explanations. - Filter
state setting section 362 sets smoothed first layer decoded spectrum S1'(k) (0<=k<FL) received as input fromspectrum smoothing section 361 as the internal filter state to use insubsequent filtering section 363. Smoothed first layer decoded spectrum S1'(k) is accommodated as the internal filter state (filter state) in the 0<=k<FL band of spectrum S(k) over the entire frequency range infiltering section 363. -
Filtering section 363, having a multi-tap pitch filter, filters the first layer decoded spectrum based on the filter state set in filterstate setting section 362, the pitch coefficient received as input from pitchcoefficient setting section 365 and band division information received as input fromband dividing section 360, and calculates estimated spectrum S2p'(k) (BSp<=k<BSp+BWp) (p=0, 1, ..., P-1) of each subband SBp (p=0, 1, ..., P-1) (hereinafter "subband SBp estimated spectrum").Filtering section 363 outputs estimated spectrum S2p'(k) of subband SBp to searchsection 364. The details of filtering processing infiltering section 363 will be described later. The number of multiple taps may be any value (integer) equal to or greater than 1. - Based on band division information received as input from
band dividing section 360,search section 364 calculates the degree of similarity between estimated spectrum S2p'(k) of subband SBp received as input from filteringsection 363, and each subband spectrum S2p(k) in the higher band (FL<=k<FH) of input spectrum S2(k) received as input from time-frequencytransformation processing section 315.
This degree of similarity is calculated by, for example, correlation calculation. Processing infiltering section 363,search section 364 and pitchcoefficient setting section 365 constitute closed-loop search processing per subband, and, in every closed loop,search section 364 calculates the degree of similarity with respect to each pitch coefficient by variously modifying pitch coefficient T received as input from pitchcoefficient setting section 365 intofiltering section 363. In each subband's closed loop, or, for example, in a closed loop corresponding to subband SBp,search section 364 finds optimal pitch coefficient Tp' to maximize the degree of similarity (in the range of Tmin∼Tmax), and outputs P optimal pitch coefficients to multiplexingsection 367.Search section 364 calculates part of the band of first layer decoded spectrum to resemble each subband SBp using each optimal pitch coefficient Tp'. Then,search section 364 outputs estimated spectrum S2p'(k) corresponding to each optimal pitch coefficient Tp' (p=0, 1, ..., P-1), to gaincoding section 366. The details of search processing for optimal pitch confident Tp' (p=0, 1, ..., P-1) insearch section 364 will be described later. - Based on control by
search section 364, when pitchcoefficient setting section 365 performs closed-loop search processing corresponding to first subband SB0 withfiltering section 363 andsearch section 364, modifies pitch coefficient T gradually in a predetermined search range between Tmin and Tmax and sends outputs tofiltering section 363 sequentially. -
Gain coding section 366 calculates gain information with respect to higher band part (FL<=k<FH) of input spectrum S2(k) received as input from time-frequencytransformation processing section 315. To be more specific,gain coding section 366 divides frequency band FL<=k<FH into J subbands, and finds spectral power of input spectrum S2(k) per subband. In this case, spectral power Bj of the (j+1)-th subband is represented by equation 17 below. - In equation 17, BLj is the minimum frequency of the (j+1)-th subband, and BHj is the maximum frequency of the (j+1)-th subband.
Gain coding section 366 forms estimated spectrum S2'(k) of the higher band of input spectrum by connecting estimated spectrum S2p'(k) (p=0, 1,..., P-1) of each subband received as input fromsearch section 364 continue in the frequency domain. Then, gaincoding section 366 calculates spectral power B'j of estimated spectrum S2'(k) per subband, as in the case of calculating the spectral power of input spectrum S2(k), using equation 18 below. Next, gaincoding section 366 calculates the amount of variation, Vj, of the spectral power of estimated spectrum S2'(k) per subband, with respect to input spectrum S2(k), using equation 19 below. - Then, gain
coding section 366 encodes amount of variation Vj, and outputs an index corresponding to coded amount of variation VQj to multiplexingsection 367. - Multiplexing
section 367 multiplexes band division information received as input fromband dividing section 360, optimal pitch coefficient Tp' for each subband SBp (p=0, 1, ..., P-1) received as input fromsearch section 364, and an index of variation amount VQj received as input fromgain coding section 366, as second layer coded information, and outputs that second layer coded information to codedinformation integrating section 317. It is equally possible to input Tp' and the index of VQj directly in codedinformation integrating section 317, and multiplex these with first layer coded information in codedinformation integrating section 317. - The details of filtering processing in
filtering section 363 shown inFIG.7 will be described in detail usingFIG.9 . - Using the filter state received as input from filter
state setting section 362, pitch coefficient T received as input from pitchcoefficient setting section 365, and band division information received as input fromband dividing section 360, filteringsection 363 generates an estimated spectrum in band BSp<=k<BSp+BWp (p=0, 1, ..., P-1) of subband SBp (p=0, 1, ..., P-1). The transfer function F(z) of the filter used infiltering section 363 is represented by equation 20 below. -
- In equation 20, T is a pitch coefficient provided from pitch
coefficient setting section 365, and βi is a filter coefficient stored inside in advance. For example, when the number of taps is three, filter coefficient candidates include (β-1, β0, β1)=(0.1, 0.8, 0.1), for example. Other values such as (β-1, β0, β1)=(0.2, 0.6, 0.2), (0.3, 0.4, 0.3) are also applicable. Values (β-1, β0, β1)=(0.0, 1.0, 0.0) are also applicable, and, in this case, part of theband 0<=k<FL of first layer decoded spectrum is not modified in shape and copied as is in the band of BSp<=k<BSp+BWp. M=1 in equation 20. M is an indicator related to the number of taps. - Smoothed first layer decoded spectrum S1'(k) is accommodated in the 0<=k<FL band of spectrum S(k) of the entire frequency band in
filtering section 363 as the internal filter state (filter state). - In the BSp<=k<BSp+BWp band of S(k), estimated spectrum S2p'(k) of subband SBp is accommodated by filtering processing of the following steps. Basically, in S2p'(k), spectrum S(k-T) having a frequency T lower than this k, is substituted. To improve the smoothness of a spectrum, in practice, spectrum βi·S(k-T+i) given by multiplying nearby spectrum S(k-T+i) that is i apart from spectrum S(k-T) by predetermined filter coefficient βi is found with respect to all i's, and a spectrum adding the spectrums of all i's is substituted in S2p'(k). This processing is represented by equation 21 below.
- Estimated spectrum S2p'(k) in BSp<=k<BSp+BWp is calculated by performing the above calculation in order from the lowest frequency and changing k in the range of BSp<=k<BSp+BWp.
- The above filtering processing is performed by zero-clearing S(k) in the range BSp<=k<BSp+BWp every time pitch coefficient T is provided from pitch
coefficient setting section 365.
That is to say, S(k) is calculated every time pitch coefficient T changes and outputted to searchsection 364. -
FIG.10 is a flowchart showing the steps of processing for searching for optimal pitch coefficient Tp' for subband SBp insearch section 364.Search section 364 searches for optimal pitch coefficient Tp' (p=0, 1, ..., P-1) in each subband SBp (p=0, 1, ..., P-1) by repeating the steps shown inFIG.10 . - First,
search section 364 initializes the minimum degree of similarity, Dmin, which is a variable for saving the minimum value of the degree of similarity, to "+∞" (ST 110). Next, following equation 22 below, at a given pitch coefficient,search section 364 calculates the degree of similarity, D, between the higher band part (FL<=k<FH) of input spectrum S2(k) and estimated spectrum S2p'(k) (ST 120). - In equation 22, M' is the number of samples upon calculating the degree of similarity D, and may assume arbitrary values equal to or smaller than the bandwidth of each subband. S2p'(k) is not present in equation 22 but is represented using BSp and S2'(k).
- Next,
search section 364 determines whether or not the calculated degree of similarity, D, is smaller than the minimum degree of similarity, Dmin (ST 130). If degree of similarity D calculated in ST 120 is smaller than minimum degree of similarity Dmin ("YES" in ST 130),search section 364 substitutes degree of similarity D in minimum degree of similarity Dmin (ST 140). On the other hand, if degree of similarity D calculated in ST 120 is equal to or greater than minimum degree of similarity Dmin ("NO" in ST 130),search section 364 determines whether or not processing in the search range has finished. That is to say,search section 364 determines whether or not the degree of similarity has been calculated with respect to all pitch coefficients in the search range in ST 120 according to equation 22 above (ST 150).Search section 364 returns to ST 120 again when the processing has not finished over the search range ("NO" in ST 150). Then,search section 364 calculates the degree of similarity according to equation 22, for different pitch coefficients from the case of calculating the degree of similarity according to equation 22 in earlier ST 120. On the other hand, when processing is finished over the search range ("YES" in ST 150),search section 364 outputs pitch coefficient T corresponding to the minimum degree of similarity, to multiplexingsection 367, as optimal pitch coefficient Tp' (ST 160). - Next,
decoding apparatus 303 shown inFIG.5 will be described. -
FIG.11 is a block diagram showing an internal principal-part configuration ofdecoding apparatus 303. - In
FIG.11 , codedinformation demultiplexing section 331 demultiplexs between first layer coded information and second layer coded information in coded information received as input, outputs the first layer coded information to firstlayer decoding section 332, and outputs the second layer coded information to secondlayer decoding section 335. - First
layer decoding section 332 decodes the first layer coded information received as input from codedinformation demultiplexing section 331, and outputs the generated first layer decoded signal to up-sampling processing section 333. The operations of firstlayer decoding section 332 are the same as in firstlayer decoding section 313 shown inFIG.6 and will not be explained in detail. - Up-
sampling processing section 333 performs processing of up-sampling the sampling frequency from SRbase to SRinput with respect to the first layer decoded signal received as input from firstlayer decoding section 332, and outputs the resulting up-sampled first layer decoded signal to time-frequencytransformation processing section 334. - Time-frequency
transformation processing section 334 applies orthogonal transformation processing (MDCT) to the up-sampled first layer decoded signal received as input from up-sampling processing section 333, and outputs the MDCT coefficient S1(k) (hereinafter "first layer decoded spectrum") of the resulting up-sampled first layer decoded signal to secondlayer decoding section 335. The operations of time-frequencytransformation processing section 334 are the same as the processing in time-frequencytransformation processing section 315 for an up-sampled first layer decoded signal shown inFIG.6 , and will not be described in detail. - Second
layer decoding section 335 generates a second layer decoded signal including higher band components using first layer decoded spectrum S1(k) received as input from time-frequencytransformation processing section 334 and second layer coded information received as input from codedinformation demultiplexing section 331, and outputs this as an output signal. -
FIG. 12 is a block diagram showing an internal principal-part configuration of secondlayer decoding section 335 shown inFIG.11 . -
Demultiplexing section 351 demultiplexes the second layer coded information received as input from codedinformation demultiplexing section 331 into band division information including bandwidth BWp (p=0, 1, ..., P-1) and leading index BSp (p=0, 1, ..., P-1) (FL<=BSp<FH) of each subband, optimal pitch coefficient Tp' (p=0, 1, ..., P-1), which is information related to filtering, and the index of coded amount of variation VQj (j=0, 1, ..., J-1), which is information related to gain. Furthermore,demultiplexing section 351 outputs band division information and optimal pitch coefficient Tp' (p=0, 1, ..., P-1) tofiltering section 354, and outputs the index of coded amount of variation VQj (j=0, 1, ..., J-1) to gaindecoding section 355. If in codedinformation demultiplexing section 331 band division information Tp' (p=0, 1, ..., P-1) and VQj (j=0, 1,..., J-1) index are demultiplexed,demultiplexing section 351 is not necessary. -
Spectrum smoothing section 352 applies smoothing processing to first layer decoded spectrum S1(k) (0<=k<FL) received as input from time-frequencytransformation processing section 334, and outputs smoothed first layer decoded spectrum S1'(k) (0<=k<FL) to filterstate setting section 353. The processing inspectrum smoothing section 352 is the same as the processing inspectrum smoothing section 361 in secondlayer coding section 316 and therefore will not be described here. - Filter
state setting section 353 sets smoothed first layer decoded spectrum S1'(k) (0<=k<FL) received as input fromspectrum smoothing section 352 as the filter state to use infiltering section 354. Calling the spectrum of the entire 0<=k<FH frequency band "S(k)" infiltering section 354 for convenience, smoothed first layer decoded spectrum S1'(k) is accommodated in the 0<=k<FL band of S(k) as the internal filter state (filter state). The configuration and operations of filterstate setting section 353 are the same as filterstate setting section 362 shown inFIG.7 and will not be described in detail here. -
Filtering section 354 has a multi-tap pitch filter (having at least two taps).Filtering section 354 filters smoothed first layer decoded spectrum S1'(k) based on band division information received as input fromdemultiplexing section 351, the filter state set in filterstate setting section 353, pitch coefficient Tp' (p=0, 1, ..., P-1) received as input fromdemultiplexing section 351, and a filter coefficient stored inside in advance, and calculates estimated spectrum S2p'(k) (BSp<=k<BSp+BWp) (p=0, 1,..., P-1) of each subband SBp (p=0, 1,..., P-1) shown in equation 21 above.Filtering section 354 also uses the filter function represented by equation 20. The filtering processing and filter function in this case are represented as in equation 20 and equation 21 except that T is replaced by Tp'. -
Gain decoding section 355 decodes the index of coded variation amount VQj received as input fromdemultiplexing section 351, and finds amount of variation VQj which is a quantized value of amount of variation Vj. -
Spectrum adjusting section 356 finds estimated spectrum S2'(k) of an input spectrum by connecting estimated spectrum S2p'(k) (BSp<=k<BSp+BWp) (p=0, 1, ..., P-1) of each subband received as input from filteringsection 354 in the frequency domain. According to equation 23 below,spectrum adjusting section 356 furthermore multiplies estimated spectrum S2'(k) by amount of variation VQj of each subband received as input fromgain decoding section 355. By this means,spectrum adjusting section 356 adjust the spectral shape in the FL<=k<FH frequency band of estimated spectrum S2'(k), generates decoded spectrum S3(k) and outputs decoded spectrum S3(k) to time-frequencytransformation processing section 357. - Next, according to equation 24,
spectrum adjusting section 356 substitutes first layer decoded spectrum S1(k) (0<=k<FL), received as input from time-frequencytransformation processing section 334, in the low band (0<=k<FL) of decoded spectrum S3(k).
The lower band part (0<=k<FL) of decoded spectrum S3(k) is formed with first layer decoded spectrum S1(k) and the higher band part (FL<=k<FH) of decoded spectrum S3(k) is formed with estimated spectrum S2'(k) after the spectral shape adjustment. - Time-frequency
transformation processing section 357 performs orthogonal transformation of decoded spectrum S3(k) received as input fromspectrum adjusting section 356 into a time domain signal, and outputs the resulting second layer decoded signal as an output signal. Here, if necessary, adequate processing such as windowing or overlap addition is performed to prevent discontinuities from being produced between frames. - The processing in time-frequency
transformation processing section 357 will be described in detail. -
-
-
-
- Next, time-frequency
transformation processing section 357 outputs decoded signal yn" as an output signal. - Thus, according to the present embodiment, in coding/decoding for performing bandwidth enhancement using a lower band spectrum and estimating a higher band spectrum, smoothing processing to combine an arithmetic mean and geometric mean is performed for a lower band spectrum as preparatory processing. By this means, it is possible to reduce the amount of calculation without causing quality degradation of a decoded signal.
- Furthermore, although a configuration has been explained above with the present embodiment where, upon bandwidth enhancement coding, a lower band decoded spectrum obtained by means of decoding is subjected to smoothing processing and a higher band spectrum is estimated using a smoothed lower band decoded spectrum and coded, the present invention is by no means limited to this and is equally applicable to a configuration for performing smoothing processing for a lower band spectrum of an input signal, estimating a higher band spectrum from a smoothed input spectrum and then coding the higher band spectrum.
- The spectrum smoothing apparatus and spectrum smoothing method according to the present invention are by no means limited to the above embodiments and can be implemented in various modifications. For example, embodiments may be combined in various ways.
- The present invention is equally applicable to cases where a signal processing program is recorded or written in a computer-readable recording medium such as a CD and DVD and operated, and provides the same working effects and advantages as with the present embodiment.
- Although example cases have been described above with the above embodiments where the present invention is implemented with hardware, the present invention can be implemented with software as well.
- Furthermore, each function block employed in the above descriptions of embodiments may typically be implemented as an LSI constituted by an integrated circuit. These may be individual chips or partially or totally contained on a single chip. "LSI" is adopted here but this may also be referred to as "IC," "system LSI," "super LSI," or "ultra LSI" depending on differing extents of integration.
- Further, the method of circuit integration is not limited to LSPs, and implementation using dedicated circuitry or general purpose processors is also possible. After LSI manufacture, utilization of an FPGA (Field Programmable Gate Array) or a reconfigurable processor where connections and settings of circuit cells in an LSI can be regenerated is also possible.
- Further, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. Application of biotechnology is also possible.
- The disclosures of Japanese Patent Application No.
2008-205645, filed on August 8, 2008 2009-096222, filed on April 10, 2009 - The spectrum smoothing apparatus, coding apparatus, decoding apparatus, communication terminal apparatus, base station apparatus and spectrum smoothing method according to the present invention make possible smoothing in the frequency domain by a small of amount and are therefore applicable to, for example, packet communication systems, mobile communication systems and so forth.
- Explanation of Reference Numerals
- 100
- Spectrum smoothing apparatus
- 101, 315, 334, 357
- Time-frequency transformation processing section
- 102
- Subband dividing section
- 103
- Representative value calculating section
- 104
- Non-linear transformation section
- 105
- Smoothing section
- 106
- Inverse non-linear transformation section
- 201
- Arithmetic mean calculating section
- 202
- Geometric mean calculating section
- 301
- Coding apparatus
- 302
- Transmission channel
- 303
- Decoding apparatus
- 311
- Down-sampling processing section
- 312
- First layer coding section
- 313,
- 332 First layer decoding section
- 314,
- 333 Up-sampling processing section
- 316
- Second layer coding section
- 317
- Coded information integrating section
- 318
- Delay section
- 331
- Coded information demultiplexing section
- 335
- Second layer decoding section
- 351
- Demultiplexing section
- 352,
- 361 Spectrum smoothing section
- 353,
- 362 Filter state setting section
- 354,
- 363 Filtering section
- 355
- Gain coding section
- 356
- Spectrum adjusting section
- 360
- Band dividing section
- 364
- Search section
- 365
- Pitch coefficient setting section
- 366
- Gain coding section
- 367
- Multiplexing section
Claims (12)
- A spectrum smoothing apparatus comprising:a time-frequency transformation section that performs a time-frequency transformation of an input signal and generates a frequency component;a subband dividing section that divides the frequency component into a plurality of subbands;a representative value calculating section that calculates a representative value of each divided subband by calculating an arithmetic mean and by using a multiplication calculation using a calculation result of the arithmetic mean;a non-linear transformation section that performs a non-linear transformation of representative values of the subbands; anda smoothing section that smoothes the representative values subjected to the non-linear transformation in the frequency domain.
- The spectrum smoothing apparatus according to claim 1, further comprising an inverse non-linear transformation section that performs an inverse non-linear transformation of an opposite characteristic to the non-linear transformation, for the smoothed representative values.
- The spectrum smoothing apparatus according to claim 1, wherein the non-linear transformation section performs the non-linear transformation having a characteristic of emphasizing a greater value, for the representative values.
- The spectrum smoothing apparatus according to claim 1, wherein the non-linear transformation section performs a logarithmic transform as the non-linear transformation.
- The spectrum smoothing apparatus according to claim 1, wherein the representative value calculating section calculates the representative values of the subbands by estimating a geometric mean using a result of the multiplication calculation.
- The spectrum smoothing apparatus according to claim 1, wherein the representative value calculating section calculates the representative values of the subbands by dividing each subband into a plurality of subgroups, calculating the arithmetic mean value per subgroup, and calculating the geometric mean value using a result of the multiplication calculation using the arithmetic mean values of the subgroups.
- The spectrum smoothing apparatus according to claim 1, wherein:the representative value calculating section calculates the representative values of each subband by dividing each subband into a plurality of subgroups, calculating an arithmetic mean value of each subgroup, and calculates a value obtained by multiplying arithmetic means values of the subgroups as a representative value of each subband; andthe non-linear transformation section calculates an intermediate value of each subband by performing the non-linear transformation of the representative value of each subband and calculates a value obtained by multiplying an intermediate in each subband by a reciprocal of a number of subgroups in each subband as a representative value subjected to the non-linear transformation.
- A coding apparatus comprising:a first coding section that generates first coded information by encoding a lower band part of an input signal at or below a predetermined frequency;a decoding section that generates a decoded signal by decoding the first coded information; anda second coding section that generates second coded information by dividing a higher band part of the input signal above the predetermined frequency into a plurality of subbands and estimating the plurality of subbands from the input signal or the decoded signal,wherein the second coding section comprises a spectrum smoothing apparatus according to one of claims 1 to 7 that receives as input and smoothes the decoded signal, and estimates the plurality of subbands fro the input signal or the smoothed decoded signal.
- A decoding apparatus comprising:a receiving section that receives first coded information and second coded information, the first coded information being obtained by encoding a lower band part of a coding side input signal at or below a predetermined frequency, and the second coded information being generated by dividing a higher band part of the coding side input signal above the predetermined frequency into a plurality of subbands and estimating the plurality of subbands from a first decoded signal obtained by decoding the coding side input signal or the first coded information;a first decoding section that decodes the first coded information and generates a second decoded signal; anda second decoding section that generates a third decoded signal by estimating a higher band part of the coding side input signal using the second coded information,wherein the second decoding section comprises the spectrum smoothing apparatus of one of claims 1 to 7 that receives as input and smoothes the second decoded signal andestimates the higher band part of the coding side input signal from the smoothed second decoded signal.
- A communication terminal apparatus comprising the spectrum smoothing apparatus of one of claims 1 to 7.
- A base station apparatus comprising the spectrum smoothing apparatus of one of claims 1 to 7.
- A spectrum smoothing method comprising:a time-frequency transformation step of performing a time-frequency transformation of an input signal and generates a frequency component;a subband division step of dividing the frequency component into a plurality of subbands;a representative value calculation step of calculating a representative value of each divided subband by calculating an arithmetic mean and by using a multiplication calculation using a calculation result of the arithmetic mean;a non-linear transformation step of performing a non-linear transformation of representative values of the subbands; anda smoothing step of smoothing the representative values subjected to the non-linear transformation in the frequency domain.
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2008205645 | 2008-08-08 | ||
JP2009096222 | 2009-04-10 | ||
PCT/JP2009/003799 WO2010016271A1 (en) | 2008-08-08 | 2009-08-07 | Spectral smoothing device, encoding device, decoding device, communication terminal device, base station device, and spectral smoothing method |
Publications (3)
Publication Number | Publication Date |
---|---|
EP2320416A1 true EP2320416A1 (en) | 2011-05-11 |
EP2320416A4 EP2320416A4 (en) | 2012-08-22 |
EP2320416B1 EP2320416B1 (en) | 2014-03-05 |
Family
ID=41663498
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP09804758.2A Active EP2320416B1 (en) | 2008-08-08 | 2009-08-07 | Spectral smoothing device, encoding device, decoding device, communication terminal device, base station device, and spectral smoothing method |
Country Status (11)
Country | Link |
---|---|
US (1) | US8731909B2 (en) |
EP (1) | EP2320416B1 (en) |
JP (1) | JP5419876B2 (en) |
KR (1) | KR101576318B1 (en) |
CN (1) | CN102099855B (en) |
BR (1) | BRPI0917953B1 (en) |
DK (1) | DK2320416T3 (en) |
ES (1) | ES2452300T3 (en) |
MX (1) | MX2011001253A (en) |
RU (1) | RU2510536C9 (en) |
WO (1) | WO2010016271A1 (en) |
Families Citing this family (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5754899B2 (en) | 2009-10-07 | 2015-07-29 | ソニー株式会社 | Decoding apparatus and method, and program |
JP5609737B2 (en) | 2010-04-13 | 2014-10-22 | ソニー株式会社 | Signal processing apparatus and method, encoding apparatus and method, decoding apparatus and method, and program |
JP5850216B2 (en) | 2010-04-13 | 2016-02-03 | ソニー株式会社 | Signal processing apparatus and method, encoding apparatus and method, decoding apparatus and method, and program |
CN104575517B (en) * | 2010-07-19 | 2018-06-01 | 杜比国际公司 | Audio Signal Processing during high-frequency reconstruction |
US12002476B2 (en) | 2010-07-19 | 2024-06-04 | Dolby International Ab | Processing of audio signals during high frequency reconstruction |
JP6075743B2 (en) | 2010-08-03 | 2017-02-08 | ソニー株式会社 | Signal processing apparatus and method, and program |
JP5707842B2 (en) | 2010-10-15 | 2015-04-30 | ソニー株式会社 | Encoding apparatus and method, decoding apparatus and method, and program |
EP2720222A1 (en) | 2012-10-10 | 2014-04-16 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Apparatus and method for efficient synthesis of sinusoids and sweeps by employing spectral patterns |
US9319790B2 (en) * | 2012-12-26 | 2016-04-19 | Dts Llc | Systems and methods of frequency response correction for consumer electronic devices |
EP3048609A4 (en) | 2013-09-19 | 2017-05-03 | Sony Corporation | Encoding device and method, decoding device and method, and program |
KR102513009B1 (en) | 2013-12-27 | 2023-03-22 | 소니그룹주식회사 | Decoding device, method, and program |
US20160379661A1 (en) * | 2015-06-26 | 2016-12-29 | Intel IP Corporation | Noise reduction for electronic devices |
US10043527B1 (en) * | 2015-07-17 | 2018-08-07 | Digimarc Corporation | Human auditory system modeling with masking energy adaptation |
WO2018225412A1 (en) * | 2017-06-07 | 2018-12-13 | 日本電信電話株式会社 | Encoding device, decoding device, smoothing device, reverse-smoothing device, methods therefor, and program |
JP6439843B2 (en) * | 2017-09-14 | 2018-12-19 | ソニー株式会社 | Signal processing apparatus and method, and program |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020049584A1 (en) * | 2000-10-20 | 2002-04-25 | Stefan Bruhn | Perceptually improved encoding of acoustic signals |
JP2002244695A (en) * | 2001-02-22 | 2002-08-30 | Nippon Telegr & Teleph Corp <Ntt> | Method, device, and program for improving voice spectrum, and storage medium stored with the program |
US20040013276A1 (en) * | 2002-03-22 | 2004-01-22 | Ellis Richard Thompson | Analog audio signal enhancement system using a noise suppression algorithm |
US20070136053A1 (en) * | 2005-12-09 | 2007-06-14 | Acoustic Technologies, Inc. | Music detector for echo cancellation and noise reduction |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH046450A (en) * | 1990-04-24 | 1992-01-10 | Sumitomo Light Metal Ind Ltd | Method for determining quantity of welded metal on al alloy material |
JPH0522151A (en) * | 1991-07-09 | 1993-01-29 | Toshiba Corp | Band divided encoding system |
DE4212339A1 (en) * | 1991-08-12 | 1993-02-18 | Standard Elektrik Lorenz Ag | CODING PROCESS FOR AUDIO SIGNALS WITH 32 KBIT / S |
US5495552A (en) * | 1992-04-20 | 1996-02-27 | Mitsubishi Denki Kabushiki Kaisha | Methods of efficiently recording an audio signal in semiconductor memory |
JP3087814B2 (en) * | 1994-03-17 | 2000-09-11 | 日本電信電話株式会社 | Acoustic signal conversion encoding device and decoding device |
JP4274614B2 (en) * | 1999-03-09 | 2009-06-10 | パナソニック株式会社 | Audio signal decoding method |
DE10105339B4 (en) * | 2001-02-05 | 2004-05-13 | november Aktiengesellschaft Gesellschaft für Molekulare Medizin | Counterfeit-proof marking method, counterfeit-proof marking and kit |
JP3976169B2 (en) * | 2001-09-27 | 2007-09-12 | 株式会社ケンウッド | Audio signal processing apparatus, audio signal processing method and program |
JP3926726B2 (en) | 2001-11-14 | 2007-06-06 | 松下電器産業株式会社 | Encoding device and decoding device |
US7447631B2 (en) * | 2002-06-17 | 2008-11-04 | Dolby Laboratories Licensing Corporation | Audio coding system using spectral hole filling |
JP3881932B2 (en) * | 2002-06-07 | 2007-02-14 | 株式会社ケンウッド | Audio signal interpolation apparatus, audio signal interpolation method and program |
JP4161628B2 (en) * | 2002-07-19 | 2008-10-08 | 日本電気株式会社 | Echo suppression method and apparatus |
US7277550B1 (en) * | 2003-06-24 | 2007-10-02 | Creative Technology Ltd. | Enhancing audio signals by nonlinear spectral operations |
CN1322488C (en) * | 2004-04-14 | 2007-06-20 | 华为技术有限公司 | Method for strengthening sound |
JP4810422B2 (en) * | 2004-05-14 | 2011-11-09 | パナソニック株式会社 | Encoding device, decoding device, and methods thereof |
KR100634506B1 (en) | 2004-06-25 | 2006-10-16 | 삼성전자주식회사 | Low bitrate decoding/encoding method and apparatus |
US8396717B2 (en) | 2005-09-30 | 2013-03-12 | Panasonic Corporation | Speech encoding apparatus and speech encoding method |
EP1928115A1 (en) * | 2006-11-30 | 2008-06-04 | Nokia Siemens Networks Gmbh & Co. Kg | Adaptive modulation and coding in a SC-FDMA system |
JP2008205645A (en) | 2007-02-16 | 2008-09-04 | Mitsubishi Electric Corp | Antenna device |
JP2009096222A (en) | 2007-10-12 | 2009-05-07 | Komatsu Ltd | Construction machine |
-
2009
- 2009-08-07 JP JP2010523772A patent/JP5419876B2/en active Active
- 2009-08-07 MX MX2011001253A patent/MX2011001253A/en active IP Right Grant
- 2009-08-07 DK DK09804758.2T patent/DK2320416T3/en active
- 2009-08-07 ES ES09804758.2T patent/ES2452300T3/en active Active
- 2009-08-07 CN CN2009801283823A patent/CN102099855B/en active Active
- 2009-08-07 KR KR1020117002822A patent/KR101576318B1/en active IP Right Grant
- 2009-08-07 WO PCT/JP2009/003799 patent/WO2010016271A1/en active Application Filing
- 2009-08-07 EP EP09804758.2A patent/EP2320416B1/en active Active
- 2009-08-07 RU RU2011104350/08A patent/RU2510536C9/en active
- 2009-08-07 US US13/057,454 patent/US8731909B2/en active Active
- 2009-08-07 BR BRPI0917953-4A patent/BRPI0917953B1/en active IP Right Grant
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020049584A1 (en) * | 2000-10-20 | 2002-04-25 | Stefan Bruhn | Perceptually improved encoding of acoustic signals |
JP2002244695A (en) * | 2001-02-22 | 2002-08-30 | Nippon Telegr & Teleph Corp <Ntt> | Method, device, and program for improving voice spectrum, and storage medium stored with the program |
US20040013276A1 (en) * | 2002-03-22 | 2004-01-22 | Ellis Richard Thompson | Analog audio signal enhancement system using a noise suppression algorithm |
US20070136053A1 (en) * | 2005-12-09 | 2007-06-14 | Acoustic Technologies, Inc. | Music detector for echo cancellation and noise reduction |
Non-Patent Citations (3)
Title |
---|
"Draft new Recommendation ITU-T G.718 Amendment 2 (ex G.718-SWB)", ITU-T DRAFTS ; STUDY PERIOD 2009-2012, INTERNATIONAL TELECOMMUNICATION UNION, GENEVA ; CH, vol. Study Group 16, 4 November 2009 (2009-11-04), pages 1-57, XP017450632, [retrieved on 2009-11-04] * |
OSHIKIRI ET AL: "An 8-32 kbit/s scalable wideband coder extended with MDCT-based bandwidth extension on top of a 6.8 kbit/s narrowband CELP coder", 1 January 2007 (2007-01-01), INTERSPEECH. 8TH ANNUAL CONFERENCE OF THE INTERNATIONAL SPEECH COMMUNICATION ASSOCIATION, AUGUST 27-31, 2007, ANTWERP, BELGIUM,, PAGE(S) 465 - 468, XP009161109, ISBN: 978-1-60560-316-2 * page 465, right-hand column, last paragraph - page 467, right-hand column, paragraph 1 * * |
See also references of WO2010016271A1 * |
Also Published As
Publication number | Publication date |
---|---|
CN102099855B (en) | 2012-09-26 |
BRPI0917953B1 (en) | 2020-03-24 |
DK2320416T3 (en) | 2014-05-26 |
EP2320416A4 (en) | 2012-08-22 |
BRPI0917953A2 (en) | 2015-11-10 |
RU2510536C9 (en) | 2015-09-10 |
ES2452300T3 (en) | 2014-03-31 |
JPWO2010016271A1 (en) | 2012-01-19 |
RU2510536C2 (en) | 2014-03-27 |
US8731909B2 (en) | 2014-05-20 |
EP2320416B1 (en) | 2014-03-05 |
JP5419876B2 (en) | 2014-02-19 |
RU2011104350A (en) | 2012-09-20 |
KR20110049789A (en) | 2011-05-12 |
US20110137643A1 (en) | 2011-06-09 |
MX2011001253A (en) | 2011-03-21 |
CN102099855A (en) | 2011-06-15 |
KR101576318B1 (en) | 2015-12-09 |
WO2010016271A1 (en) | 2010-02-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP2320416B1 (en) | Spectral smoothing device, encoding device, decoding device, communication terminal device, base station device, and spectral smoothing method | |
EP3288034B1 (en) | Decoding device, and method thereof | |
EP2402940B1 (en) | Encoder, decoder, and method therefor | |
EP3336843B1 (en) | Speech coding method and speech coding apparatus | |
EP2752849B1 (en) | Encoder and encoding method | |
EP2224432B1 (en) | Encoder, decoder, and encoding method | |
EP2239731B1 (en) | Encoding device, decoding device, and method thereof | |
EP1926083A1 (en) | Audio encoding device and audio encoding method | |
US20100280833A1 (en) | Encoding device, decoding device, and method thereof | |
EP2584561B1 (en) | Decoding device, encoding device, and methods for same | |
EP1892702A1 (en) | Post filter, decoder, and post filtering method | |
KR20180002907A (en) | Improved frequency band extension in an audio signal decoder |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20110128 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: AL BA RS |
|
DAX | Request for extension of the european patent (deleted) | ||
A4 | Supplementary search report drawn up and despatched |
Effective date: 20120724 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G10L 19/14 20060101ALI20120718BHEP Ipc: G10L 19/02 20060101AFI20120718BHEP Ipc: G10L 21/02 20060101ALI20120718BHEP Ipc: G10L 11/00 20060101ALI20120718BHEP |
|
17Q | First examination report despatched |
Effective date: 20130306 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G10L 19/032 20130101ALN20130904BHEP Ipc: G10L 19/02 20130101AFI20130904BHEP Ipc: G10L 21/02 20130101ALI20130904BHEP Ipc: G10L 19/24 20130101ALI20130904BHEP |
|
INTG | Intention to grant announced |
Effective date: 20130925 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 655345 Country of ref document: AT Kind code of ref document: T Effective date: 20140315 |
|
REG | Reference to a national code |
Ref country code: ES Ref legal event code: FG2A Ref document number: 2452300 Country of ref document: ES Kind code of ref document: T3 Effective date: 20140331 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602009022287 Country of ref document: DE Effective date: 20140417 |
|
REG | Reference to a national code |
Ref country code: DK Ref legal event code: T3 Effective date: 20140519 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602009022287 Country of ref document: DE Representative=s name: GRUENECKER, KINKELDEY, STOCKMAIR & SCHWANHAEUS, DE |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 655345 Country of ref document: AT Kind code of ref document: T Effective date: 20140305 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: VDEP Effective date: 20140305 Ref country code: GB Ref legal event code: 732E Free format text: REGISTERED BETWEEN 20140619 AND 20140625 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140605 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602009022287 Country of ref document: DE Representative=s name: GRUENECKER, KINKELDEY, STOCKMAIR & SCHWANHAEUS, DE Effective date: 20140711 Ref country code: DE Ref legal event code: R081 Ref document number: 602009022287 Country of ref document: DE Owner name: PANASONIC INTELLECTUAL PROPERTY CORPORATION OF, US Free format text: FORMER OWNER: PANASONIC CORP., KADOMA-SHI, OSAKA, JP Effective date: 20140711 Ref country code: DE Ref legal event code: R081 Ref document number: 602009022287 Country of ref document: DE Owner name: PANASONIC INTELLECTUAL PROPERTY CORPORATION OF, US Free format text: FORMER OWNER: PANASONIC CORPORATION, KADOMA-SHI, OSAKA, JP Effective date: 20140711 Ref country code: DE Ref legal event code: R082 Ref document number: 602009022287 Country of ref document: DE Representative=s name: GRUENECKER PATENT- UND RECHTSANWAELTE PARTG MB, DE Effective date: 20140711 Ref country code: DE Ref legal event code: R081 Ref document number: 602009022287 Country of ref document: DE Owner name: FRAUNHOFER-GESELLSCHAFT ZUR FOERDERUNG DER ANG, DE Free format text: FORMER OWNER: PANASONIC CORPORATION, KADOMA-SHI, OSAKA, JP Effective date: 20140711 Ref country code: DE Ref legal event code: R082 Ref document number: 602009022287 Country of ref document: DE Representative=s name: SCHOPPE, ZIMMERMANN, STOECKELER, ZINKLER, SCHE, DE Effective date: 20140711 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: TP Owner name: PANASONIC INTELLECTUAL PROPERTY CORPORATION OF, US Effective date: 20140722 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 Ref country code: BE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140605 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140705 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602009022287 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140707 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
26N | No opposition filed |
Effective date: 20141208 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602009022287 Country of ref document: DE Effective date: 20141208 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 Ref country code: LU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140807 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
REG | Reference to a national code |
Ref country code: ES Ref legal event code: PC2A Owner name: PANASONIC INTELLECTUAL PROPERTY CORPORATION OF AME Effective date: 20150409 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20140831 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20140831 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20140807 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140606 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20090807 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 8 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 9 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602009022287 Country of ref document: DE Representative=s name: SCHOPPE, ZIMMERMANN, STOECKELER, ZINKLER, SCHE, DE Ref country code: DE Ref legal event code: R081 Ref document number: 602009022287 Country of ref document: DE Owner name: FRAUNHOFER-GESELLSCHAFT ZUR FOERDERUNG DER ANG, DE Free format text: FORMER OWNER: PANASONIC INTELLECTUAL PROPERTY CORPORATION OF AMERICA, TORRANCE, CALIF., US |
|
REG | Reference to a national code |
Ref country code: ES Ref legal event code: PC2A Owner name: FRAUNHOFER-GESELLSCHAFT ZUR FOERDERUNG DER ANGEWAN Effective date: 20180403 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140305 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 10 |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: 732E Free format text: REGISTERED BETWEEN 20181115 AND 20181130 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230512 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: TR Payment date: 20230801 Year of fee payment: 15 Ref country code: IT Payment date: 20230831 Year of fee payment: 15 Ref country code: GB Payment date: 20230824 Year of fee payment: 15 Ref country code: FI Payment date: 20230823 Year of fee payment: 15 Ref country code: ES Payment date: 20230918 Year of fee payment: 15 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20230822 Year of fee payment: 15 Ref country code: DK Payment date: 20230823 Year of fee payment: 15 Ref country code: DE Payment date: 20230822 Year of fee payment: 15 |