EP2026331B1

EP2026331B1 - Digital audio processing

Info

Publication number: EP2026331B1
Application number: EP20080162264
Authority: EP
Inventors: Jeff Butters; Tim Addy
Original assignee: Snell Ltd
Current assignee: Snell Advanced Media Ltd
Priority date: 2007-08-13
Filing date: 2008-08-12
Publication date: 2010-07-28
Anticipated expiration: 2028-08-12
Also published as: US8825186B2; DE602008001927D1; US20090048696A1; EP2026331A1; GB0715758D0; GB2451828A

Description

The invention concerns digital audio processing and in particular the detection of periods where samples can be deleted or repeated unobtrusively so as to change the average sample-rate or to provide time delay modification.
It is often necessary to modify the sample rate of a digital audio signal or to change the number of samples that represent a particular time segment of an audio signal in a file or digital store. Such modifications are used to synchronise the sample rates of independently-sourced digital signals, or to modify the size of a file or data set. A temporary increase or decrease in the sampling frequency can also be used as a means of delaying or advancing the timing of audio information.
Where there are no limitations on processing time or complexity, interpolation of new samples having the required sampling frequency and phase by taking weighted sums of existing samples is often the preferred method. However, a known alternative method, which has the advantage of not introducing significant delay in real-time processing applications, is the deletion or repetition of samples at points chosen to minimise the audibility of the deletion or repetition. An effective method for identifying such points is thus an important feature of audio sample rate converters operating by sample deletion and repetition.
Reference is directed to WO 02/32054 which discloses that a heuristic resampling technique may be employed to inaudibly insert or remover a specific number of samples, with groups of multiple consecutive samples being examined throughout the buffer.
The inventors have appreciated that points for unobtrusive sample repetition or deletion can be determined from evaluation of the differences between the values of succeeding audio samples.
The invention consists, in one aspect, of a method and apparatus for changing the number of samples representing a time segment of an audio signal by deleting or repeating existing samples at positions in the said time segment dependant on the values of said audio samples wherein differences between succeeding sample values are evaluated and compared with a threshold value and samples are deleted or repeated where two or more consecutive differences are less than the said threshold value.
The said threshold value is reduced when samples are deleted or repeated and the said threshold value is increased when a difference between succeeding samples exceeds the said threshold value.
In some embodiments stereo audio is processed and respective left and right differences between succeeding sample values are evaluated and samples are repeated or deleted where both the left difference and the right difference are below the said threshold at two or more consecutive sample points.
Suitably, a sample is deleted or repeated in response to a defined number of consecutive inter-sample differences being less than the threshold value.
In one embodiment sample deletion or repetition is inhibited during a set number of sample periods following a point of sample deletion or repetition.
Suitably, the said set number of sample periods during which sample deletion or repetition is inhibited is adjusted in dependence on a cumulative measure of inter-sample difference values.
Advantageously, the said set number of sample periods during which sample deletion or repetition is inhibited is reduced in response to an increase in the said threshold value.
In an alternative embodiment the said set number of sample periods during which sample deletion or repetition is inhibited is adjusted in dependence upon a measure of audio frequency.
The said measure of audio frequency may be a count of zero crossings of the audio signal.
In a further embodiment the said set number of sample periods during which sample deletion or repetition is inhibited is adjusted in dependence upon a measure of absolute audio amplitude.
An example of the invention will now be described with reference to the drawings in which:

Figure 1: shows a block diagram of a system for controlling the addition or deletion of audio samples according to an exemplary embodiment.
Figure 2: shows a flow diagram illustrating a control process for audio sample deletion or repetition according to an exemplary embodiment.

A system for identifying candidate digital audio samples for nearly-inaudible deletion or repetition according to an exemplary embodiment will now be described with reference to Figure 1. The description relates to a real-time process on a stream of sample values; however, the skilled person will appreciate that the method is equally applicable to non-real-time processing.
An input stream of audio samples (1) is applied to a subtractor (2) and a single-sample store (3). The subtractor (2) determines the magnitude of the value-difference between the current sample and the previous sample from the store (3). This value-difference magnitude between a pair of consecutive samples (separated in time by one period of the sampling clock) is input to a comparator (4) which compares it with a threshold value (5). The threshold value (5), which is greater than zero, is determined by a silence-threshold block (6) which will be described in detail below.
If the audio is silent, successive samples will have the same value and so the value-difference magnitude will be zero, and the comparator (4) will determine that it is below the threshold value (5). If the audio is nearly-silent the difference is also likely to be below the threshold value, but the outcome will depend on the frequency spectrum and the amplitude of the audio; lower-frequency, lower-amplitude audio will be more likely to give a difference-value below a given threshold value.
The output from the comparator (4) is passed to a control logic block (7) which generates commands (8) to delete (drop) or repeat the current sample in dependence upon a sample-rate control input (9). The control input (9) is derived from an external process (not illustrated) which determines whether the number of samples needs to be increased or decreased. If the external process determines that fewer samples are required, the control signal (9) causes the control logic block (7) to issue sample deletion commands (8) at points in the audio where the deletion is unlikely to be audible. Similarly, if the external process determines that additional samples are required, the control signal (9) causes the control logic block (7) to issue sample repetition commands (8) at points of low audibility. If no change to the number of samples is required, the control signal (9) causes the control logic (7) to prevent the output of sample repetition or deletion commands (8).
In some embodiments the control logic block (7) only outputs a sample deletion or sample repetition command if a fixed, small number (say 4) consecutive value-difference magnitudes have been less than the threshold (5); and/or, after a command has been issued, no further commands are issued until a further fixed number (say 1,020) of succeeding input samples have been received. These numbers are chosen to achieve the necessary maximum range of adjustment of the sample rate and to minimise the audibility of the process. The commands (8) are passed to an external process (not illustrated) which can either delete or repeat the current sample so as to obtain a modified sample stream or set of samples.
The silence-threshold block (6) determines the threshold value (5) so as to ensure that samples resulting in a value-difference less than the threshold value can be deleted or repeated with minimal subjective distortion to the audio. This is done by a process of continuous adjustment of the threshold in response to the sequence of audio samples to be processed. The threshold is lowered by a fixed decrement value whenever a sample deletion or repetition command is issued; and, the threshold is raised by a fixed increment value whenever the current value-difference magnitude is greater than the current threshold.
The level at which the threshold stabilises for a given audio signal of constant amplitude will depend on the ratio of the rate of threshold increment to the rate of threshold decrement. The fixed decrement value and the fixed increment value can be chosen so as to achieve automatically a threshold value for which sample deletion or repetition is substantially inaudible for a wide range of audio material. Suitable maximum and minimum limits to the variation of the threshold value can also be applied and the increment and decrement functions inhibited when necessary to keep the threshold value within these limits. The maximum and minimum limits can be set at the levels at which the audibility of sample deletion or repetition is just acceptable for the loudest and quietest signals respectively.
The process of threshold increment and decrement can continue during periods where, according to the control signal (9), no sample modification is required. This will ensure that sample modification can be re-started quickly without waiting for the threshold level to adapt to the characteristics of the audio signal.
It is possible to extend the technique to a two channel stereo audio system having left and right audio channels where samples are simultaneously deleted or repeated in both channels. Respective inter-sample difference magnitudes are determined for both audio channels and compared with a single threshold. In the system of Figure 1, the comparator (4) can be replaced by a comparator which gives an output when both inter-sample differences are below the threshold. The operation of the system is otherwise unchanged: a fixed consecutive number of "below threshold" outputs from the comparator is required before a sample delete or repeat command is issued; the threshold is decremented when a command is issued; and, the threshold is incremented when either of the two current inter-sample differences is above the current threshold.
Either the single-channel or the stereo system can be improved by providing for the rate of sample modification (i.e. deletion or repetition) to be adjusted in dependence on the dynamics of the audio. If the amplitude of the audio is decaying after a loud peak, there is a short time period after the peak (of the order of a few tens of milliseconds) during which distortion of the audio caused by sample deletion or repetition will be subjectively masked. If the amplitude continues to decay, sample modification becomes more noticeable and can be perceived as a pitch change.
In the system of Figure 1, or the stereo version described above, the delay between a sample modification and the next permitted sample modification can be adjusted so that the delay is reduced for a short period after a peak in the audio level, and then increased to the original delay value after a fixed number of samples have been processed. Figure 2 shows a flow-chart of a system in which the threshold level of the system of Figure 1 is used to identify points in the audio at which the rate of sample modification can be adjusted.
Referring to Figure 2, in step (21) an audio sample is evaluated as in the system of Figure 1 so as to determine whether or not it should be modified (i.e. deleted or repeated). If the sample is not modified, either because the inter-sample difference is above the threshold, or because a recent previous sample has been modified, the decision step (22) passes control to decision step (23). If the difference between the current sample and its predecessor is greater than the threshold, the threshold is incremented (as described previously with reference to Figure 1) in step (24) and the next audio sample is made available for processing in step (25). However, if the current inter-sample difference magnitude is less than the threshold at decision step (23), control is passed to step (25) without any change to the threshold value.
If the current audio sample was selected for modification in step (21), decision step (22) passes control to step (26) in which the current threshold value is stored. A further decision step (27) determines whether the current threshold value is above the threshold value stored on the previous occasion that a sample was modified. If the threshold is higher, the audio level is likely to be reducing after a peak (during which no sample modifications have occurred) and a faster rate of sample modification is likely to be acceptable. In this case, step (28) causes the delay between sample modifications to be reduced by changing the mode of operation of step (21) to a "Fast Modification" mode. This mode is maintained until a fixed number of samples have been processed in step (21).
After step (28), or a decision that the threshold is lower than at the time of the previous sample modification at step (27), the threshold value is decremented in step (29), as described with reference to Figure (1), and control is passed to step (25) to commence the processing of the next audio sample.
In this way the Fast Modification mode is entered as the audio level begins to fall. In the Fast Modification mode the minimum permitted interval between sample modifications is set at a low value for a fixed number of samples. For example, the normal delay between modifications could be 1,024 sample periods and in the Fast Modification Mode the minimum permitted interval could be set to 32 sample periods until about 20 ms of audio has been processed; the minimum period can then be doubled after each succeeding 20 ms period until the "normal" delay of 1,024 samples is reached, or the Fast Modification mode is re-started at step (28).
When the audio is very quiet the process of Figure 2 need not be applied, and the time delay between sample modifications can be maintained at a low value. This condition can be detected by noting that the threshold is at its minimum permitted value. Once the threshold rises, the process of Figure 2 can again be used to control the delay between modifications.
The system of Figure 2 is equally applicable to stereo processing.
Dropping or repeating samples has the effect of changing the pitch of tones. If tones can be recognised and the pitch measured, it is then possible to determine a permitted rate of sample modification that causes a barely perceptible pitch change for that tone and to set the minimum permitted delay to that value. If the absolute amplitude of the audio can also be measured then the permitted rate of sample modification can be adjusted knowing the effect of loudness on the perceptibility of pitch changes. The system of the invention can thus be modified by the addition of frequency or amplitude measuring processes which control the time between sample deletion or repetition. Simple frequency measurement by detecting and counting zero crossings of the audio signal, or signals for stereo, can be used.
The application of the invention to stereo audio has been described; however, it may also be extended to other multi-channel audio formats such as "surround sound" or "5.1 channel" systems. In one method a multi-channel audio programme is "mixed down" to a mono or stereo version, which is analysed as described above, and samples are deleted or repeated simultaneously in all the channels of the multi-channel programme in response to the analysis of the mixed-down audio.
Another method is to compare inter-sample difference values from more than two channels with the threshold; and, to adjust the threshold and simultaneously delete or repeat samples in all channels in dependence on the results of the comparisons.
The invention has been described by way of examples and other variations are possible within the scope of the following claims. It should be noted that this invention can be to hardware and/or software arrangements as will be apparent to a skilled person.

Claims

A method of changing the number of samples representing a time segment of an audio signal by deleting or repeating existing samples at positions in the said time segment dependant on the values of said audio samples, wherein differences between succeeding sample values are evaluated and compared with a threshold value and samples are deleted or repeated where two or more consecutive differences are less than the said threshold value, in which the threshold value is reduced when samples are deleted or repeated, and in which the threshold value is increased when a difference between succeeding samples exceeds the said threshold value.
A method according to claim 1 in which stereo audio is processed and respective left and right differences between succeeding sample values are evaluated and samples are repeated or deleted where both the left difference and the right difference are below the said threshold at two or more consecutive sample points.
A method according to any one of the preceding claims in which a sample is deleted or repeated in response to a defined number of consecutive inter-sample differences less than the threshold value.
A method according to any of the preceding claims in which sample deletion or repetition is inhibited during a set number of sample periods following a point of sample deletion or repetition.
A method according to Claim 4 in which the said set number of sample periods during which sample deletion or repetition is inhibited is adjusted in dependence on a cumulative measure of inter-sample difference values.
A method according to Claim 5 in which the said set number of sample periods during which sample deletion or repetition is inhibited is reduced in response to an increase in the said threshold value.
A method according to Claim 5 or Claim 6 in which the said set number of sample periods during which sample deletion or repetition is inhibited is adjusted in dependence upon a measure of audio frequency.
A method according to Claim 7 in which the said measure of audio frequency is a count of zero crossings of the audio signal.
A method according to any of Claims 5 to 8 in which the said set number of sample periods during which sample deletion or repetition is inhibited is adjusted in dependence upon a measure of absolute audio amplitude.
A computer program product comprising code adapted to implement a method in accordance with any one of claims 1-9.
Apparatus for changing the number of samples representing a time segment of an audio signal, configured and adapted to perform a method in accordance with any one of claims 1-9.