KR20040013729A - Audio signal time-scale modification method using variable length synthesis and reduced cross-correlation computations - Google Patents

Abstract

PURPOSE: A method of modifying time scale of an audio signal using variable length synthesis and correlation computation reduction technique is provided to reduce the quantity of calculations for finding the maximum value of cross-correlation. CONSTITUTION: An analysis window composed of the first number of audio samples of an input stream is decided(S16). Similarity between the first audio samples of the analysis window and the second audio samples of an output signal is calculated using the third and fourth audio sample blocks composed of audio samples down-selected from the first and second audio samples at a predetermined ratio. The similarity is calculated whenever the analysis window is shifted within a predetermined search range. A shift value of the analysis window when the maximum value among the calculated similarity values is provided is obtained(S18).

Description

Audio signal time-scale modification method using variable length synthesis and reduced cross-correlation computations}

The present invention relates to a time-scale modification (TSM) technique of an audio signal. More particularly, the present invention relates to a time-scale modification (TSM) technique of the audio signal. The present invention relates to a method of performing time-scale correction while minimizing distortion of pitch information of an original audio signal.

In order to make the playback speed of the audio signal faster or slower than the normal speed, it is necessary to modify the time scale of the audio signal. The time scale correction method of an audio signal is classified into a frequency domain processing method and a time domain processing method. Since the frequency domain processing method uses fast fourier transformation (FFT), it is difficult to implement it and the calculation amount is too large. Therefore, the frequency domain processing method is generally evaluated as an unsuitable method for applications requiring real time processing. On the other hand, from the viewpoint of sound quality and calculation amount, the time domain processing method is evaluated as a method that can provide excellent sound quality with a relatively small amount of operations, and in particular, an application field where time scale correction needs to be processed in real time. It is evaluated in a way that can be applied effectively.

The basic concept of time domain processing was introduced under the name overlap-add (OLA). This method divides the input audio signal into a number of partially overlapped frames, and then modifies the distance (i.e., time) between adjacent frames according to a desired time scale and adds them to the output audio signal. In this case, the overlapping part of the output audio signal in the frame to be added is synthesized by applying a weighting function, and the remaining part is added as it is. Since the introduction of this OLA, various improved methods have been introduced, focusing on the direction of reducing the amount of computation and the direction to make the sound quality of the TSM processed audio signal more similar to that of the input audio signal. Representative examples include synchronized overlap and add (SOLA) and waveform similarity based overlap and add (WSOLA).

SOLA modifies the timescale of the input signal in two phases: analysis and synthesis. The analysis phase divides the input signal into multiple overlapping windows, similar to OLA, with each window having a fixed length N and a fixed analysis length. As far as Sa will space each other. In the synthesizing step, the windows secured in the analysis step are rearranged at the synthesizing length Ss interval, where each window partially overlaps the output signal which is the synthesis of the previous windows. When each window is superimposed on the output signal, each window is aligned at the point of similarity, or cross-correlation, with the output signal to reduce the discontinuity of the signal caused by the difference between the analysis and synthesis lengths. This sorting nesting is called synchronized nesting. Synthesis of overlapping sections and simple addition of non-overlapping sections are treated the same as OLA. This SOLA modifies the timescale in such a way as to retain the original pitch information as much as possible, which greatly improves the sound quality of the timescale-modified output signal compared to the OLA. However, because SOLA changes the alignment point Km while retrieving the maximum similarity point, and the overlap interval changes each time, the new cross-correlation needs to be recalculated, which makes the cross-correlation calculation very complicated and the amount of computation required. There are disadvantages, which make them unsuitable for applications requiring real-time processing. In addition, the length ratio between the input signal and the output signal whose time scale is corrected does not provide a method for accurately matching the desired time scale value.

WSOLA finds the maximum correlation point based on the waveform similarity of adjacent frames to ensure sufficient signal continuity at the waveform segment boundaries, especially for all neighboring samples of the associated sample index, where the time-scale modified composite signal And local maximum similarity. Although WSOL produces less frequency distortion and less computation than SOLA, it requires a short time Fourier transform (STFT) to determine the similarity of waveforms. The problem is that it is not easy to apply to applications that require processing.

One of the important issues to consider in the time scale correction of audio signals is the reduction of computation. If the amount of computation is large, the time scale correction cannot be processed in real time, and the application range is greatly limited. TSM methods processed in most time domains, such as SOLA, WSOLA, and their modified methods, combine spectral characteristics of an audio signal when a specific window (or frame) of the input signal is superimposed and summed on the output signal. Or cross-correlation between the window and the time-scale-corrected signal (hereafter referred to as the 'output signal') to match the pitch periods as closely as possible to the original signal (hereinafter referred to as the 'input signal'). Find the point that gives the maximum value of -correlation) and synthesize at that point. However, finding the maximum point of cross-correlation leads to many calculations. In the case of time-scale correction according to the conventional method, approximately 95% or more of the load on the processor (or CPU) is generated in the process of finding the maximum value point of the cross-correlation.

In addition, the cross-correlation-related computations increase exponentially as the sampling rate of the input signal to be timescale corrected increases. This is because the double loop operation must be performed to calculate the maximum value of the cross-correlation, and the calculation amount of each loop is proportional to the sampling rate of the input signal. That is, one of the double loops is a first loop for multiplying all samples belonging to each of a specific section (overlapping section) and an output signal of a specific section (overlapping section) of the input signal, and the other is the first loop. The second loop performs a multiplication operation of the loop repeatedly while shifting the analysis window one sample over the search range. The amount of computation in each loop is proportional to the sampling rate of the input signal. Since these two loops are executed in a double loop scheme, the amount of computation increases exponentially in proportion to the sampling rate of the input signal. Therefore, even WSOLA, which is known to be smaller than SOLA or other methods, requires a lot of operations, and it can be applied to a relatively low performance embedded processor that can be applied to a personal computer (PC) employing a high performance CPU. Difficult to apply

For a 20ms packet (segment) of an audio signal sampled at 8KHz, the existing TSM requires 24,000 multiplications and 24,000 additions. On a 773MHz Intel Pentium III chip, it takes nearly 0.35ms to perform a TSM operation on a 20ms packet of 8KHz sample rate audio. For an audio signal with a 44.1KHz sampling rate, it takes approximately 10.64ms to perform a TSM operation on a 20ms packet with a 773MHz Intel Pentium III chip. Therefore, CPU performance should be above 389 MHz, which means that the entire 389 MHz CPU processing capacity should be allocated to TSM. Audio signals for DVDs with a sampling rate of 96KHz are not possible to perform TSM operations in real time, even on 733 MHz Intel Pentium III processors. This is because it takes about 50.4 ms per 20 ms packet (segment) to process the above signal with TSM. In order to process an audio signal with a sampling rate of 44.1KHz with a typical SOLA or WSOLA, a minimum 389MHz total processing capacity of the embedded processor must be allocated to the TSM. Even in the case of an audio signal of 16 KHz sampling rate, at least 51 MHz processing capacity should be allocated for TSM processing. The peak performance of commercially available embedded processors has not yet exceeded 200MHz. In some systems employing embedded processors, the load imposed on the embedded processor is not only for TSM processing but also for processing various other services. Therefore, considering this point, it may be practically impossible to process a TSM in real time using an embedded processor with a built-in program according to the existing TSM method for an audio signal having a sampling rate higher than 8 KHz. Can be.

In particular, the recent trend is that the sampling rate of the audio signal is gradually increasing to reflect the demand for higher quality sound. The WAV format used in personal computers has recently used a sampling frequency of 44.1Khz, and the MPEG mono type is also made with the same sampling frequency. Furthermore, DVDs use a sampling rate of 96 kHz, or twice that of 192 kHz. In order to process TSM for such a high sampling rate audio signal in real time, it is necessary to reduce the amount of computation within the range of processing power that the processor can allocate. Existing known TSM methods do not provide a solution.

If the amount of computation can be significantly reduced, a part of the surplus capacity of the processor obtained therefrom can be diverted to the processing for improving the sound quality, so that sound quality better than the existing TSM method may be obtained. Since the human voice does not have a wide frequency bandwidth, even in the conventional TSM method of synthesizing on the basis of the maximum correlation, the distortion of the pitch information of the original sound does not occur significantly. However, in the case of music, since the frequency bandwidth is relatively wide, the output signal obtained by the conventional TSM method has a greater degree of distortion and noise mixing of pitch information. Therefore, in the case of music signals, additional processing for improving sound quality is required.

In view of the above, the present invention significantly reduces the amount of computation for finding the maximum value of cross-correlation in processing a time scale correction of an audio signal in a time domain, thereby reducing TSM for an audio signal having a high sampling rate. It is a first object to provide a method for processing in real time.

In addition, when the analysis window defined in the input signal is overlaid on the output signal, the present invention further considers an evaluation index called a coefficient of correlation in addition to the correlation between the overlapping position of the analysis window and the overlapping interval between the analysis window and the output signal. It is a second object to provide a method by which the pitch information of the output signal is kept closer to that of the input signal.

1A illustrates a method of determining a maximum cross-correlation point between an analysis window and an output signal while shifting an analysis window of an input signal according to the TSM method (Reduced computations and variable synthesis based TSM: RCVS-TSM) of the present invention. Drawing.

FIG. 1B is a view for explaining a method of determining and combining overlapping sections between an analysis window and an output signal of an input signal according to the RCVS-TSM method of the present invention.

2 is a view for explaining a method of calculating the cross-correlation for the overlap section between the analysis window and the output signal of the input signal according to the RCVS-TSM method of the present invention.

3A shows a method of determining an analysis window from an input signal when a desired time scale is 2 (α = 2).

FIG. 3B illustrates a method of synthesizing an output signal by superimposing the analysis windows defined in FIG. 3A by applying a predetermined maximum cross-correlation and overlapping intervals.

4A shows a method of determining an analysis window from an input signal when a desired time scale is 0.5 (α = 0.5).

4B illustrates a method of synthesizing an output signal by overlapping and adding the analysis windows defined in FIG. 4A by applying a predetermined maximum cross-correlation and overlapping intervals.

5 is a flowchart illustrating the overall execution of the RCVS-TSM method according to the present invention.

FIG. 6 is a flowchart showing the detailed procedure of step S18 (setting the maximum value of the cross-correlation diagram and the shift value of the analysis window at that time) of the flowchart shown in FIG. 5.

FIG. 7 is a flowchart illustrating a detailed procedure of step S20 (setting an overlap section based on a correlation coefficient between an analysis window and an output signal) of the flowchart illustrated in FIG. 5.

8 is a block diagram of an apparatus with resources needed to implement the method of the present invention.

According to the present invention for achieving the first object, in the time scale correction method of an audio signal to form an input signal consisting of an input stream of audio samples as an output signal modified to a desired time scale, Determining an analysis window consisting of a first predetermined number of audio samples; The similarity between the Nov first audio samples of the analysis window and the Nov second audio samples of the output signal is down-selected from each of the first and second audio samples by a predetermined ratio. Calculating using the third and fourth audio sample blocks, wherein the similarity calculation is repeated each time the analysis window is shifted within a predetermined search range; And calculating a shift value Km of the analysis window when a maximum value is provided among the calculated similarity values.

The method includes N + Nm-Nov based on an optimal overlapping section Nm when the shift value Km and the coefficient of correlation between the analysis window and the output signal are equal to or greater than a predetermined reference value or provide a maximum value. Preferably, the method further comprises the step of determining an additional frame (where N is a value obtained by subtracting the similarity search range Kmax between the analysis window and the output signal from the first predetermined number).

Furthermore, the method overlaps and adds Nm audio samples from the front of the additional frame and Nm audio samples at the end of the output signal by applying a weighting function to overlap-add block. Forming); And substituting the overlap-summing block instead of the Nm audio samples at the end of the output signal, and adding the remaining audio samples of the additional frame to the end of the overlap-summing block as they are. .

Meanwhile, according to the present invention for achieving the second object, in the time scale correction method of an audio signal, the input signal consisting of an input stream of audio samples is formed into an output signal modified to a desired time scale. Determining an analysis window consisting of N + Kmax audio samples, wherein N and Kmax are constant; By shifting the analysis window within a predetermined search range, the maximum value of the similarity between the Nov audio samples and the Nov audio samples from the end of the output signal and the Nov value are changed to various values. Calculating a coefficient of correlation; In the front of the analysis window, N + Nm-Nov audio samples are defined as add frames from the Km + Nov-Nm audio samples, where Km is the analysis window when the maximum value of the similarity is provided. Is a shift value, and Nm is an optimal overlap period when a coefficient of correlation between the analysis window and the output signal is equal to or greater than a predetermined reference value or provides a maximum value, and N is the first predetermined number. Defining a value obtained by subtracting the similarity search range Kmax between the analysis window and the output signal; The overlapping-summing block is formed by overlapping the optimal overlapping section Nm audio samples from the front of the additional frame and the optimal overlapping section Nm audio samples at the end of the output signal by applying a weighting function. forming an overlap-add block); And replacing the overlap-summing block at the end of the output signal instead of the Nm audio samples at the optimal overlap period, and simply adding the remaining audio samples of the additional frame to the end of the overlap-summing block. A time scale correction method of an audio signal is provided.

In the method of the present invention for achieving the first object or the second object, the audio samples constituting the third and fourth audio sample blocks have a sample index of M ₁ (wherein M ₁ is larger than 2). Natural number). The first predetermined number is N + Kmax (where N and Kmax are constants), the search range is Kmax audio sample intervals, and the shift of the analysis window is M ₂ (where M ₂ is 2 or more). Audio samples of natural numbers). In addition, the similarity is preferably determined by the calculation of cross-correlation.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The input signal represents the original audio signal subject to TSM processing, and the output signal represents the audio signal obtained by TSM processing. The input signal is a signal represented by a stream of sample signals obtained by sampling and quantizing analog audio signals.

The various processes described below are implemented by implementing the RCVS-TSM algorithm into a program and executing the engine program in sequence by the processor. Therefore, the apparatus for implementing the present invention basically shows a nonvolatile memory 84, such as a ROM for storing an engine program, an input signal by reading the engine program and executing each instruction in turn. Processor 80 for performing RCVS-TSM processing, and an input buffer memory 82a for temporarily storing input signals before RCVS-TSM processing, and an output buffer memory 82b for temporarily storing output signals after RCVS-TSM processing Memory 82 resources, which provide data processing space of the processor 80, are required. In addition, the present invention can set the desired time scale α, i.e., the speed value of the audio signal, in order to accept the time scale α value designated by the user and perform RCVS-TSM processing according to the value. There is also a need for means for reading the value and reflecting it to the RCVS-TSM processing by the processor 80 from that point on, i.e., user input means 86 such as an input keypad or remote control. Furthermore, the input signal providing unit 88 which provides the input signal targeted for RCVS-TSM processing to the input buffer 82a for RCVS-TSM processing and the output signal obtained as a result of the RCVS-TSM processing are transferred from the output buffer 82b. It is also necessary to provide an audio reproducing unit 90 or the like for processing for reproduction.

These resources may exist as separate chips, as in the case of personal computers, or may be integrated into one or several chips. Therefore, unless otherwise specified, these resources need to be understood to function together to perform the RCVS-TSM processing described below. The processor 80 may be implemented with, for example, a digital signal processor (DSP), a microcomputer or a central processing unit (CPU), or the like, or may be an audio chip, an audio / video chip, an MPEG chip, a DVD chip, or the like made for a specific purpose. have.

The RCVS-TSM method of the present invention largely consists of a process called analysis and synthesis. Referring to FIG. 3A or 4A, the input signal is divided into successive analysis windows Wm (where m = 1, 2, 3,) composed of N + Kmax samples, and each analysis window Wm is RCVS-TSM. As a unit of analysis for The starting point of each analysis window Wm is the mSa th sample of the input signal. Therefore, Sa means the starting point interval of consecutive analysis windows (hereinafter referred to as 'analysis interval'). Here, m represents a frame index and N represents a sample number of one reference frame F ₀ . The analysis window Wm shifts backwards by a constant sample interval M _{2 for} each shift period along the output signal to find a point that provides the maximum cross-correlation between the analysis window and the output signal. The shift of the analysis window Wm is within the Kmax sample range. Kmax represents a search range for analyzing a position providing a maximum value of the number of samples shifting the analysis window, that is, a maximum value of cross-correlation. Km represents the distance that the analysis window Wm shifts to the point at which the maximum value of the cross-correlation is provided, i.e., the number of shift samples, whose magnitude cannot always exceed Kmax.

1A and 1B, when synthesizing the analysis window Wm to the output signal, the synthesis interval Ss becomes a reference. The synthetic interval Ss has a relationship of Ss = αSa with the analysis interval Sa. The composite interval Ss is a fixed value. Therefore, given the desired value of time scale α, the analysis interval Sa is determined by the above relation. The synthesis interval Ss is the starting point of the shift for finding the maximum value of the correlation between the analysis window Wm and the output signal. That is, the calculation for calculating the maximum value of cross-correlation is started by placing the front of the analysis window Wm at the mSs th sample point of the output signal. The synthesis sums each analysis window Wm overlapping a part of the output signal at the position of the maximum cross-correlation found in the analysis step. At this time, while the overlap section is changed to various sizes, the correlation coefficient between the analysis window Wm and the output signal is calculated and the overlap section corresponding to the case where the predetermined condition is satisfied is determined as the overlap section Nm to be actually applied. In order to facilitate the implementation of the program, it is preferable to change the overlapping section in such a manner that it starts at the maximum Nov section and decreases by a constant ratio or interval. When the overlap section Nm is determined, the additional frame 20 is determined in the analysis window Wm. Then, Nm samples 45 from the end of the output signal and the Nm samples 35 at the beginning of the additional frame 20 are summed by applying a weighting function to form a new composite block 40, and the existing sample of the output signal. Substituting the output signal in place of (45), and then synthesized by simply adding the remaining samples of the additional frame (20).

The overall execution procedure of the RCVS-TSM algorithm of the present invention based on this basic concept is schematically illustrated in the flowchart of FIG. Execution of the RCVS-TSM method begins by first identifying information necessary for reproduction by referring to the information recorded in the header of the input signal, and performing various initializations necessary for RCVS-TSM processing (step S10). Since the basic information about the audio input signal such as the sampling rate and the sampling size is recorded in the header of the input signal, the sampling rate information can also be used to reduce the amount of computation occurring in finding the maximum cross-correlation point. Details thereof will be described later. In the initialization phase, the following variables are initialized. First, values of appropriate magnitudes are given to N, Nov, Ss, Kmax and the like. In particular, the larger the value of Kmax, the better the sound quality of the TSM-signaled signal. However, the higher the value, the more saturated the signal is. Therefore, it is desirable to appropriately select the value of Kmax near the point where the improvement of sound quality is gentle. Given the value of the desired timescale α, determining the value of Sa using the relation Sa = Ss / α is also an issue that needs to be addressed in the initialization phase.

After initialization, the first thing to do is to copy the first N samples of the input signal as they are. The copied N samples constitute the first frame F ₀ of the output signal (step S12). Since the processing for one frame is performed, the value of the frame index m is set to 1 (step S14).

TSM processing is performed on the input signal while repeatedly executing the loop composed of steps S16 to S28 until the end of the input signal is increased while increasing the value of the frame index m by one. Each time through this loop, the output signal is increased by one frame. In this respect, this loop can be called a frame loop. The first three steps S16, S18 and S20 of the frame loop are related to the 'analysis' step mentioned above, and the following three steps S22, S24 and S26 are related to the 'synthesis' step. This will be described in detail.

First, as the first procedure of the 'analysis' step, as described above, the samples constituting the analysis window Wm from the input signal is determined (step S16). The input signal consists of a sample stream of the audio signal, which is divided into a number of consecutive analysis windows. The mth analysis window Wm is composed of N + Kmax samples starting with the mSath sample and treated as one analysis unit. The analysis window Wm is to be played back when the value of the given time scale α is larger than 1 (slower than the normal playback speed, which is equivalent to FIG. 3A) or when it is smaller than 1 (faster than the normal playback speed). 4A corresponds to this) or always consists of the same number N + Kmax samples.

The second procedure of the 'Analysis' step shifts the analysis window Wm along the output signal within the Kmax sample range while providing the maximum value of the cross-correlation and the maximum value for the Nov section of the analysis window Wm and the output signal. It is to calculate the shift value Km of the analysis window Wm, respectively (step S18). The RCVS-TSM algorithm of the present invention introduces a scheme that can greatly reduce the amount of computation at this stage.

The first way to reduce the amount of computation is to reduce the number of samples involved in the calculation of cross-correlation. The second way to reduce the amount of computation is to increase the shift interval M ₂ in the analysis window Wm. Both of these methods can be applied or only one of them can be applied, and it is natural to see the greatest reduction in throughput when both are applied.

The cross-correlation calculation is made for the sample of the analysis window Wm and the sample of the output signal that spans the overlap section 17 having an area that can contain Nov samples. Since the output signal is fixed and shifts only the analysis window Wm, the output signal participating in the cross-correlation calculation is always constant at the last Nov sample intervals (15), while the analysis window Wm participating in the cross-correlation calculation is one. Each shift shifts the Nov section to the right by M ₂ samples. That is, at the beginning, it is performed for the Nov sample sections 10 at the front of the analysis window Wm and the Nov sample sections 15 at the end of the output signal (see FIG. 1A). Then, the analysis window Wm is shifted to the left by M ₂ samples, and again cross-correlation between the Nov sample sections of the analysis window Wm and the Nov sample sections of the output signal spanning the overlap section 17. The calculation is performed. This process is repeatedly performed until the shift value M ₂ of the analysis window Wm leaves the search section Kmax.

The present invention calculates the cross-correlation only for the selected samples by selecting only some of them rather than all samples of the analysis window Wm and the output signal included in the overlap section 17 in calculating the cross-correlation. Take In FIG. 2, when the overlap section 17 is composed of ten samples, the samples participating in the cross-correlation are three skipped samples, that is, three samples selected from the analysis window Wm 50 (x _m0 , x _m4 , x _m8 ) and the cross-correlation for the three samples (y _m0 , y _m4 , y _m8 ) selected in the output signal 55 are calculated. The ratio of the amount of calculation to be reduced may be appropriately determined in consideration of the performance of the processor to which the RCVS-TSM engine of the present invention is applied and the sampling rate of the input signal. Since the excerpt calculation method reduces the number of samples involved in the correlation calculation, the point where the maximum value of the correlation is provided in comparison with the conventional TSM method that calculates the correlation for all samples in the overlapping interval. May be low, but that can be neglected.

By calculating the cross-correlation using samples selected _one by one for every two or more constant sample intervals M ₁ , not only the actual waveform pattern of the input signal can be maintained almost as it is, but also the error range of the maximum possible correlation point is M _1/2 . Without exceeding the sample, the noise resulting from this degree of error in human hearing capacity is completely unrecognizable and can be ignored.

In addition, if one of the natural numbers larger than the shift value M ₂ of the analysis window Wm is also applied, a computational amount reduction effect can be similarly obtained. The separation interval M ₁ and the shift value M ₂ are not necessarily the same.

The separation interval M ₁ and the shift value M ₂ may be arbitrarily designated when the RCVS-TSM engine program is written, but may be determined in the following manner. One method calculates the value of one of the two integer values closest to the actual sampling rate of the input signal divided by the reference sampling rate of the predetermined size, and the calculated value is separated from the interval M ₁ and / or the shift value M. ₂ is used. It is preferable that the reference sampling rate is set as low as possible on the premise of satisfying a condition capable of properly transmitting the information of the audio signal. If set to a high value, it is contrary to the intention to reduce the load on the processor. In consideration of commercially available processor performance, it is judged that there is no problem when the reference sampling rate is set to, for example, 8Khz. However, the size of the reference sampling rate may be increased later as the performance of the processor is upgraded. In this way, regardless of what sampling rate the input signal has, the processor can always be adjusted to bear the optimal amount of computation it can afford.

Alternatively, the corresponding value of the separation interval M ₁ and / or the shift value M ₂ is determined in advance for each of the existing types of sampling rates of the audio signal, and the corresponding value mapped to the sampling rate obtained from the header information of the input signal is determined. It is a method of applying the separation interval M ₁ and / or shift value M ₂ .

The above algorithm reduces the TSM method for searching the optimal overlapping point of the analysis window based on cross-correlation, such as SOLA or WSOLA, and the basic concept is common to the modified or improved TSM methods for improving sound quality or reducing the throughput. It can also be widely applied.

6 is a detailed embodiment of the step S18, assuming that the case of applying both of the above methods to reduce the number of samples participating in the cross-correlation calculation and increase the shift interval of the analysis window Wm to the actual program This is a flowchart of the implementation.

To calculate the point Km where the maximum value of the cross-correlation that can be superimposed and summed up the analysis window Wm with the highest continuity is obtained, the maximum value of the cross-correlation K and the variable K representing the amount of shift of the analysis window Wm is provided. The value variable C (m, Km) is initialized to 0 (step S40). Further, the variable Denom representing the denominator for standardizing the size of the cross-correlation variable Corr and the variable j representing the sample index in the overlap section 17 are set to 0, respectively (steps S42 and S44).

Then, increase the sample index variable j by the interval M1 every cycle (where M1 is a natural number greater than 2) (step S48) until the value exceeds Nov-1 (step S50). Calculate the correlation coefficient Corr and the standardized variable Denom using the two equations (step S46).

Corr = Corr + x (mSa + j)-y (mSs + K + j) (1)

Denom = Denom + x (mSa + j)-x (mSa + j) (2)

When the above two equations are completed for the entire overlap section (17), the value of Corr is divided by the value of Denom to obtain the standardized cross-correlation degree c (m, K). C (m, K) is compared with c (m, Km), which is the largest value among the cross-correlation values calculated up to the previous time, and the larger of the two is the maximum cross-correlation c (m, Km) Decide on) This process (steps S42 to S52) is repeatedly performed on the changed value of K while increasing the size of the variable K by the shift interval M _{2, which} is a natural number greater than _2, while the value does not exceed Kmax-1. When the size of the variable K becomes larger than Kmax-1, that is, when the analysis window Wm shifts the search range Kmax altogether, the Km value that provides the maximum cross-correlation c (m, Km) at that time is immediately performed in step S18. This is the result to be obtained, and it is a shift value that should be applied when 'synthesizing' the output signal of the analysis window Wm.

Looking at the flow chart of Figure 6 it can be seen that the double loop is executed. If we do not apply the above computational reduction techniques, we can easily assume that a huge amount of computation must be performed to execute this double loop.

On the other hand, the RCVS-TSM method of the present invention does not distort the pitch information of the input signal after obtaining the shift value Km of the analysis window Wm when the maximum value of the cross-correlation with respect to the analysis window Wm is provided as above. A procedure of determining an optimal overlap section Nm for obtaining the best sound quality is performed (step S20).

When calculating the optimum shift value Km of the analysis window Wm, the overlap section 17 between the analysis window Wm and the output signal was applied with a constant length Nov. However, if the overlap section is Nov, it cannot be said that the analysis window Wm always has the best alignment with the output signal. The optimum shift value Km is only a relatively optimal value for the overlapping section of 'specific size Nov', and is not an 'absolutely' optimal value even when the overlapping section is changed. This can be confirmed by experimenting with various sound source types.

Rock Music, 2x slower (correlation base value: 70%) Nov (i) Nov (1) Nov (2) Nov (3) Nov (4) Nov (5) 5 msec 4 msec 3 msec 2 msec 1 msec Correlation Coefficient (Rxy-m) Rxy_1 100.00 100.00 100.00 100.00 100.00 Rxy_2 37.17 39.84 54.24 84.25 66.10 Rxy_3 92.80 92.80 92.80 92.80 92.80 Rxy_4 100.00 100.00 100.00 100.00 100.00 Rxy_5 -89.94 -84.63 -65.88 -44.29 7.61 Rxy_6 100.00 100.00 100.00 100.00 100.00 Rxy_7 -58.39 -33.17 22.60 -15.21 -25.79 Rxy_8 100.00 100.00 100.00 100.00 100.00 Rxy_9 52.50 32.36 32.53 24.64 4.62 Rxy_10 71.81 71.81 71.81 71.81 71.81 Rxy_11 100.00 100.00 100.00 100.00 100.00 Rxy_12 25.28 18.93 26.89 32.38 11.15 Rxy_13 39.13 41.48 -6.70 -8.43 -24.28 Rxy_14 100.00 100.00 100.00 100.00 100.00 Rxy_15 73.21 73.21 73.21 73.21 73.21 Rxy_16 84.84 84.84 84.84 84.84 84.84 Rxy_17 90.85 90.85 90.85 90.85 90.85 Rxy_18 100.00 100.00 100.00 100.00 100.00 Rxy_19 76.79 76.79 76.79 76.79 76.79 Rxy_20 90.18 90.18 90.18 90.18 90.18 Rxy_21 73.41 73.41 73.41 73.41 73.41 Rxy_22 88.59 88.59 88.59 88.59 88.59 Rxy_23 58.80 51.22 31.33 55.57 57.96 Sum 1,507.01 1,508.50 1,537.48 1,571.40 1,539.85 Average 65.52 65.59 66.85 68.32 66.95

Table 1 changes the overlapping interval to various values, i.e., 5 msec, 4 msec, 3 msec, 2 msec, 1 msec, etc., for an arbitrary section (23 analysis windows) of rock music having a relatively wide frequency bandwidth. The results of calculating the coefficient of correlation Rxy between the sample x of the analysis window Wm and the sample y of the output signal for each overlapping section are summarized.

The correlation coefficient Rxy was obtained using the following equation.

Rxy = [(Σxy) / (nσ _x σ _y )] 100% (3)

Here, n denotes the number of samples of each variable x and y participating in the correlation coefficient calculation, and σ _x and σ _y are the variance values of the variables x and y, respectively.

The correlation coefficient may vary from 100 [%] to +100 [%]. In general, if the value of the correlation coefficient between two variables x and y is negative, the value of one variable (y) decreases (or increases) when the value of one variable (x) increases (or decreases). It has a so-called negative relationship. Also, if the value of the correlation coefficient between the two variables x and y is positive, the two variables have the same so-called positive relationship, but if the value is in the range of 0% to 30%, the two variables The correlation between x and y is considered weak, and if it is 30% to 70%, the correlation can be regarded as moderate. If it is in the range of 70% to 100%, the correlation is high. Is evaluated. Therefore, when the analysis window is overlaid and summed on the output signal by applying the value of the overlapping interval where the correlation coefficient Rxy between the analysis window and the output signal cannot provide more than 70%, the sound quality is degraded, such as distortion of pitch information.

As can be seen from Table 1, it can be seen, for example, that the correlation coefficient is not always maximized when the value of the overlapping section is 5 msec. For example, in the case of the second analysis window, the maximum correlation coefficient value of 84.25 is provided when the overlapping interval value is 2 msec. Therefore, the second analysis window can have the best alignment or signal continuity when overlapping with the output signal by overlapping the interval by 2msec, thereby minimizing distortion of the pitch information.

7 shows the detailed procedure of the procedure (step S20) for determining the value of the optimum overlapping section Nm for each analysis window based on this concept. Suppose that the size of the overlapping section is 5msec, 4msec, 3msec, 2msec, 1msec and calculates the correlation coefficient while changing the values in order. While increasing the index value i of the candidate overlapping section Nov (i) by 1 from 0 to 4 (steps S60, S66, S68), the correlation coefficient Rxy_m (i) for each overlapping section was calculated using Equation (3). (Step S62), it is checked whether the magnitude of the calculated correlation coefficient Rxy_m (i) is greater than or equal to the reference value Ref (step S64).

Although the reference value is preferably 70%, but not necessarily limited thereto, the reference value may be set to a higher value (e.g., 80%) in consideration of further improvement in sound quality, and a lower value (e.g., more in consideration of suppression of increased throughput). 60%). Table 1 above applies 70% of the reference value (Ref) of the correlation coefficient. If the calculated correlation coefficient Rxy_m (i) has a value of 70% or more, the value of the overlapping section Nov (i) at that time is adopted as the optimal overlapping section Nm to be applied when the analysis window Wm is superimposed on the output signal (S72). step). If the calculated correlation coefficient Rxy (i) does not exceed 70%, i is increased by one (step S66), and then the correlation coefficient Rxy_m (i) is calculated again for the next overlapping section Nov (i). Check if it exceeds 70%.

In Table 1, the correlation coefficients Rxy_2 (0), Rxy_5 (0), Rxy_7 (0) for the 2nd, 5th, 7th, 9th, 10th, 12th, 13th and 23rd analysis windows when the overlapping section is 5 msec. It is less than 70%. For each of these analysis windows, the overlap section is changed to 4 msec, and the correlation coefficient Rxy_m (1) (where m is 2, 5, 7, 9, 10, 12, 13, 23) is calculated again. If the value of the calculated correlation coefficient is greater than 70%, the optimal overlap section Nm for the analysis window can be set to 4 msec. If the calculated correlation coefficient is still less than 70%, the overlap section Nov (2) is again set to 3 msec for the analysis window. Change and recalculate the correlation coefficient Rxy_m (2). In this way, the correlation coefficient is calculated. Even if the correlation coefficient is calculated until the overlapping interval is 1 msec, if the value of the correlation coefficient does not exceed 70%, the calculated correlation coefficients Rxy_m (i) (i is The overlapping section providing the maximum value among 0, 1, and 4) is determined as the overlapping section Nm to be actually applied. In Table 1, for example, the optimal overlap section N ₄ for the fifth analysis window will be determined to be 1 msec.

As described above, the RCVS-TSM method of the present invention variably determines the optimal overlapping section Nm, but an increase in the amount of computation for finding the optimal overlapping section Nm occurs, but it is a big problem to apply the above described calculation method to the correlation coefficient calculation. Cannot be. The concept of optimal overlapping interval Nm shows that the effect of sound quality improvement is not so great in the case of speech with narrow frequency band compared to the case where this concept is not applied. However, when the concept of the optimal overlap section Nm is applied to music with a wide frequency band, a noticeable sound quality improvement effect can be obtained as compared with the case where the concept of Nm is not applied.

Returning to the flowchart of FIG. 5, when the value of the optimum overlapping section Nm is determined, the additional frame Fm 20 to be superimposed and summed as the output signal in the corresponding analysis window Wm is determined using the optimal shifting value Km determined in step S18 (S22). step). Samples in the (Km + Nov-Nm) to (N + Nm-Nov) intervals in the analysis window Wm become the additional frame Fm (20).

When the additional frame Fm 20 is determined, the additional frame Fm 20 is overlaid with the output signal (step S24). Nm samples at the beginning of the additional frame Fm 20 and Nm samples at the end of the output signal are superimposed and synthesized. At this time, a weighted function is applied to each of the Nm samples on both sides of the analysis window and the output signal, and synthesized to form an overlap-added block 40 (step S24). The reason for applying the weighting function is to minimize the discontinuity of the signal in the overlapping section by naturally connecting from the end of the output signal to the beginning of the analysis window. As a representative example of the weighting function, it may be a linear ramp function g (j) as follows, but an exponential function or other appropriate function may be selected.

g (j) = 0, j <0; (4-1)

g (j) = j / Nm, 0 ≦ j ≦ Nm; (4-2)

g (j) = 1, j> Nm (4-3)

At the end of the output signal, the Nm samples 45 are replaced by the overlapped summation block 40 newly synthesized above. The remaining samples except for the Nm samples 35 in front of the additional frame Fm 20 are added to the end of the overlapped sum block 40 as they are (S26). The remaining samples of the analysis window Wm, that is, the Km + Nov-Nm samples 30 at the front and the Kmax-Km samples 25 at the rear, are discarded. FIG. 3B shows the case where the desired time scale α is 2, that is, when the length of the input signal is doubled, when the length of the input signal is doubled. The lengths of the overlapping sections 60a, 60b, 60c, and frames Fm (where m = 0, 1, 2, 3,) added to the output signal are variable every frame period. However, the total length of the output signal obtained every frame period increases in proportion to the specified time scale α value exactly.

The above characteristics are the same even when the length of the input signal is reduced. That is, for example, when the value of the time scale α is 0.5, FIG. 4A shows that a plurality of consecutive analysis windows Wm are divided according to the analysis window classification method described above, and FIG. 4B shows the divided analysis windows from above. The additional frame Fm is synthesized by applying the optimal shift Km and the overlapping interval Nm determined through the analysis and synthesis process. Even if the length of the input signal is reduced, the length of the added frames Fm (where m = 0, 1, 2, 3,) is variable, but the total length of the output signal obtained every frame period is determined by the specified time scale α value. Decreases in exact proportions.

Therefore, the characteristic that the length of the output signal obtained in each frame period is increased or decreased precisely in proportion to the designated time scale α value, the RCVS-TSM method of the present invention is perfectly synchronized with the video signal at any time scale when applied to multimedia playback It ensures that they can match. For example, during DVD playback, if you change the playback speed slower or faster, you can change the playback speed of the audio at exactly the same rate as the video playback rate. Can be.

As described above, when one analysis window is subjected to 'analysis' and 'synthesis', one TSM-processed frame is added to the output signal. At this time, the value of the frame index m is increased by 1 in order to be able to add another TSM-processed frame to the output signal through 'analysis' and 'synthesis' for the next analysis window (step S28). In order to check whether there is an input signal to be processed further, it is checked whether the end of the input signal has been met (step S30). If the end of the input signal is not met, proceed to the analysis and synthesis procedure for the next analysis window again. By repeatedly executing such a frame loop to the end of the input signal, a TSM output signal can be obtained at a desired time scale.

The input signal transmitted from the input signal providing unit 88 to the input buffer 82a is processed by the processor 80 according to the RCVS-TSM method described above to be an output signal. The output signal is sequentially written to the output buffer 82b by a predetermined unit, and then transferred to the audio reproducing unit 90 according to a predetermined output schedule and reproduced in real time.

The RCVS-TSM method of the present invention is a TSM method capable of substantially reducing the amount of computation compared to the existing TSM methods while maintaining almost the same sound quality of the original audio signal level. When performing TSM processing of 192KHz sampling rate using general SOLA or WSOLA algorithm, the processor's processing performance is about 7,366MHz. Therefore, a lot of time must pass before the appearance of a CPU, especially an embedded processor, that satisfies this level, and until then, it is impossible to process TSM in real time with such a high sampling rate. However, using the RCVS-TSM method of the present invention, since only 28 Mips (approximately 28 KHz) of processing capacity is required to perform TSM processing of an audio signal having a 192KHz sampling rate in real time, a CPU, especially an embe Even on a deed processor, the TSM can process the high sampling rate audio signal in real time.

In addition, the present invention provides better results than the conventional TSM method in terms of sound quality improvement. Since the overlap length between the analysis window of the input signal and the output signal is fixed to the optimum length like the method of the present invention rather than fixed to a constant value, the degree of discontinuity of the signal can be minimized, as a result. Distortion of the pitch information caused by time scale correction can be minimized.

The RCVS-TSM method of the present invention may be written as a program and included as a part of a multimedia player for a personal computer, and may require a function of reproducing digital audio signals such as a DVD player, a digital VTR, an MP3 player, a set-top box, and the like. It may be embedded in a dedicated chip of a playback device and applied to enhance the audio playback function of the device.

Although the present invention has been described above according to an embodiment of the present invention, it will be apparent to those skilled in the art that various modifications may be made without departing from the spirit of the present invention.

Claims (20)

In the time scale correction method of an audio signal for forming an input signal consisting of an input stream of audio samples as an output signal modified to a desired time scale, Determining an analysis window consisting of a first predetermined number of audio samples in the input stream; The similarity between the Nov first audio samples of the analysis window and the Nov second audio samples of the output signal is down-selected from each of the first and second audio samples by a predetermined ratio. Calculating using the third and fourth audio sample blocks, wherein the similarity calculation is repeated each time the analysis window is shifted within a predetermined search range; And And obtaining a shift value Km of the analysis window when a maximum value is provided among the calculated similarity values. The N + Nm method according to claim 1, wherein the shift value Km and the coefficient of correlation between the analysis window and the output signal are equal to or greater than a predetermined reference value or based on an optimum overlapping period Nm when a maximum value is provided. -Nov (wherein N is a value obtained by subtracting the similarity search range Kmax between the analysis window and the output signal from the first predetermined number), and determining an additional frame as an add frame. How to correct time scale of audio signal. 3. An overlap-add block according to claim 2, wherein Nm audio samples from the front of the additional frame and Nm audio samples at the end of the output signal are overlapped and summed under the application of a weighting function. forming a block); And And substituting the overlap-summing block instead of the Nm audio samples at the end of the output signal, and adding the remaining audio samples of the additional frame as they are at the end of the overlap-summing block. Time scale correction method of audio signal. The time period of the audio signal according to claim 1, wherein the audio samples constituting the third and fourth audio sample blocks have a sample index difference of M ₁ (wherein M ₁ is a natural number greater than 2). Scale correction method. The method of claim 1, wherein the first predetermined number is N + Kmax (where N and Kmax are constants), the search range is Kmax audio sample intervals, and the shift of the analysis window is M ₂ (where M is ₂ is a method of regularly adjusting audio samples of two or more natural numbers). The method of claim 1, wherein the audio samples constituting the third and fourth audio sample blocks have a sample index difference of M ₁ (wherein M ₁ is a natural number greater than 2), and the first predetermined number is N. + Kmax (where, N and Kmax is constant), and wherein the search range is Kmax audio sample period, shifted the analysis window crossing the audio samples of the M ₂ items (where, M ₂ is a natural number of 2 or more) regularly A time scale correction method of an audio signal, characterized in that the jumping method. The method according to any one of claims 4 to 6, wherein the sample index interval M ₁ of the audio samples constituting the third and fourth audio sample blocks M ₁ and / or the shift interval M ₂ of the analysis window is the input signal. The method of modifying the time scale of an audio signal according to claim 1, wherein the actual sampling rate is determined by one of two integer values closest to the value obtained by dividing the actual sampling rate by a reference sampling rate of a predetermined size. The method according to any one of claims 4 to 6, wherein the sample index interval M ₁ of the audio samples constituting the third and fourth audio sample blocks and / or the shift interval M ₂ of the analysis window are equal to each other. And setting a corresponding value in advance for each type of sampling rate and applying the corresponding value mapped to the sampling rate determined from the header information of the input signal as the value of M ₁ and / or M ₂ . A time scale correction method of an audio signal, characterized by the above-mentioned. 2. The method of claim 1, further comprising the step of accepting, by the input means, a value α specified by the user as the desired time scale, wherein the length ratio of the output signal to the input signal is equal to the value of α. A time scale correction method of an audio signal. 8. The method of claim 7, wherein the first audio sample of the m-th analysis window is the mSa-th audio sample at the front of the input stream, and the Nov value is reduced at a predetermined rate with N-Ss as a maximum value. Ss is a fixed value, and Sa is determined by Ss = αSa. The method of claim 1, wherein the similarity is determined by calculating cross-correlation. In the time scale correction method of an audio signal for forming an input signal consisting of an input stream of audio samples as an output signal modified to a desired time scale, Determining an analysis window consisting of N + Kmax audio samples in the input stream, wherein N and Kmax are constants; While shifting the analysis window within a predetermined search range, the maximum value of the similarity between the Nov audio samples and the Nov audio samples from the end of the output signal and the Nov value are changed to various values. Calculating a coefficient of correlation; In the front of the analysis window, N + Nm-Nov audio samples are defined as add frames from the Km + Nov-Nm audio samples, where Km is the analysis window when the maximum value of the similarity is provided. Is a shift value, and Nm is an optimal overlap period when a coefficient of correlation between the analysis window and the output signal is equal to or greater than a predetermined reference value or provides a maximum value, and N is the first predetermined number. Defining a value obtained by subtracting the similarity search range Kmax between the analysis window and the output signal; The overlapping-summing block is formed by overlapping the optimal overlapping section Nm audio samples from the front of the additional frame and the optimal overlapping section Nm audio samples at the end of the output signal by applying a weighting function. forming an overlap-add block); And Substituting the overlap-summing block for Nm audio samples at the end of the output signal in place of the optimal overlapping section, and simply adding the remaining audio samples of the additional frame to the end of the overlap-summing block. A time scale correction method of an audio signal, characterized in that. 13. The method of claim 12, further comprising the step of accepting a value α specified by the user through the input means as the desired time scale, wherein the length ratio of the output signal to the input signal is equal to the value of α. A time scale correction method of an audio signal. 13. The method of claim 12, wherein the first audio sample of the m-th analysis window is the mSa-th audio sample at the front of the input stream, and the Nov value is reduced at a predetermined rate with N-Ss as a maximum value. Ss is a fixed value, and Sa is determined by Ss = αSa. 13. The method of claim 12, wherein the reference value with respect to the correlation coefficient has a value of 0.7 or more. The audio sample of claim 12, wherein the audio sample participating in the calculation of the similarity and the correlation coefficient is selected from signals belonging to the Nov section of each of the analysis window and the output signal, and adjacent audio samples have a sample index of M _1. , M ₁ is a natural number greater than 2). 13. The method of claim 12, shifted the analysis window is M ₂ gae every shift is accomplished by skipping the audio samples (where, M ₂ is a natural number equal to or greater than 2) on a regular basis, the sum of the shifting audio samples is Kmax audio A time scale correction method of an audio signal, characterized by not exceeding a search range composed of samples. The audio sample of claim 12, wherein the audio sample participating in the calculation of the similarity and the correlation coefficient is selected from signals belonging to the Nov section of each of the analysis window and the output signal, and adjacent audio samples have a sample index of M _1. , M ₁ has a difference of as much as a natural number greater than 2), shifts the analysis window is made in such a way beats regularly over to the audio samples of the M ₂ items (where, M ₂ is two or more natural number) for every shift, shifting And a sum of the number of audio samples thus obtained does not exceed a search range consisting of Kmax audio samples. 13. The method of claim 12, wherein the variables M ₁ and / or M ₂ are determined as one of two integer values closest to a value obtained by dividing an actual sampling rate of the input signal by a reference sampling rate of a predetermined size. A time scale correction method of an audio signal to be performed. 13. The time of an audio signal according to claim 12, wherein the similarity between the Nov audio samples of the analysis window and the Nov audio samples of the output signal is determined using a cross correlation or the correlation coefficient. Scale correction method.