EP2774391A1

EP2774391A1 - Audio scene rendering by aligning series of time-varying feature data

Info

Publication number: EP2774391A1
Application number: EP11875048.8A
Authority: EP
Inventors: Juha Petteri OJANPERÄ
Original assignee: Nokia Oyj
Current assignee: Nokia Technologies Oy
Priority date: 2011-10-31
Filing date: 2011-10-31
Publication date: 2014-09-10
Also published as: WO2013064860A1; EP2774391A4

Abstract

Apparatus comprises means for receiving basis vectors relating to each of at least three series of time-varying feature data; means for performing multiple correlations, each correlation being for a pair of said at least three series, the correlations linking together each of the at least three series; and means for aligning each of the at least three series of time-varying feature data.

Description

AUDIO SCENE RENDERING BY ALIGNING SERIES OF TIME-VARYING FEATURE DATA

Field

This relates to aligning series of time-varying feature data.

Background

It is known to distribute devices around an audio space and use them to record an audio scene. Captured signals are transmitted and stored at a rendering location, from where an end user can select a listening point based on their preference from the reconstructed audio space.

This type of system presents numerous technical challenges.

Summary

A first aspect of the invention provides a method comprising:

receiving basis vectors relating to each of at least three series of time-varying feature data;

performing multiple correlations, each correlation being for a pair of said at least three series, the correlations linking together each of the at least three series; and

aligning each of the at least three series of time-varying feature data.

Receiving basis vectors relating to each of at least three series of time-varying feature data may comprise receiving basis vectors relating to each of said at least three series of time-varying feature data at at least first and second different resolutions, and wherein performing multiple correlations may comprise performing a first correlation for a first pair of said at least three series at the first resolution and also performing a second correlation for the first pair of said at least three series at the second resolution. Here, performing a correlation for the first pair of said at least three series at at least the two different resolutions may comprise performing the first correlation for the first pair of said at least three series at a first resolution, determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion, and performing a second correlation for the first pair of said at least three series at a second resolution, higher than the first resolution, only if the metric of the degree of correlation does not meet a predetermined criterion. Determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion may comprise calculating a histogram metric from the first correlation and comparing the histogram metric to a threshold.

Performing the second correlation for the first pair of said at least three series at the second resolution may comprise performing correlation within a calculation window that frames an alignment value of the result of the first correlation.

Performing multiple correlations may comprise performing the first correlation for the first pair of said at least three series at the first resolution and also performing the second correlation for the first pair of said at least three series at the second resolution without conducting any determination as to whether a metric of degree of correlation meets a predetermined criterion between performing the first and second correlations. This method may comprise assigning a correlated value to correlation of the first pair only if degrees of correlation of both the first and second correlations each meet a respective predetermined criterion.

The method may comprise receiving basis vectors relating to each of the at least three series of time-varying feature data also at a third resolution, different to the first and second resolutions, and wherein performing multiple correlations for the first pair may comprise performing a third correlation at the third resolution.

The method may comprise selecting a series of time-varying data as a reference series and calculating an alignment value for each of the other series of time-varying data that is indicative of a delay between the respective series and the reference series.

A second aspect of the invention provides apparatus comprising:

means for receiving basis vectors relating to each of at least three series of time-varying feature data; means for performing multiple correlations, each correlation being for a p of said at least three series, the correlations linking together each of the at least three series; and

means for aligning each of the at least three series of time-varying feature data.

The means for receiving basis vectors relating to each of at least three series of time-varying feature data may comprise means for receiving basis vectors relating to each of said at least three series of time-varying feature data at at least first and second different resolutions, and wherein the means for performing multiple correlations may comprise means for performing a first correlation for a first pair of said at least three series at the first resolution and also means for performing a second correlation for the first pair of said at least three series at the second resolution.

The means for performing a correlation for the first pair of said at least three series at at least the two different resolutions may comprise means for performing the first correlation for the first pair of said at least three series at a first resolution, means for determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion, and means for performing a second correlation for the first pair of said at least three series at a second resolution, higher than the first resolution, only if the metric of the degree of correlation does not meet a predetermined criterion. The means for determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion may comprise means for calculating a histogram metric from the first correlation and means for comparing the histogram metric to a threshold.

The means for performing the second correlation for the first pair of said at least three series at the second resolution may comprise means for performing correlation within a calculation window that frames an alignment value of the result of the first correlation.

The means for performing multiple correlations may comprise means for

performing the first correlation for the first pair of said at least three series at the first resolution and also means for performing the second correlation for the first pair of said at least three series at the second resolution without conducting any determination as to whether a metric of degree of correlation meets a

predetermined criterion between performing the first and second correlations.

The apparatus may comprise means for assigning a correlated value to correlation of the first pair only if degrees of correlation of both the first and second correlations each meet a respective predetermined criterion. The apparatus may comprise means for receiving basis vectors relating to each of the at least three series of time-varying feature data also at a third resolution, different to the first and second resolutions, and wherein the means for performing multiple correlations for the first pair may comprise means for performing a third correlation at the third resolution.

The apparatus may comprise means for selecting a series of time-varying data as a reference series and means for calculating an alignment value for each of the other series of time-varying data that is indicative of a delay between the respective series and the reference series.

The apparatus may comprise means for calculating the basis vectors relating the at least three series of time-varying feature data.

The apparatus may include at least one server.

The apparatus may include at least one server and a data-to-audio transducing device. The apparatus may include at least one server, plural audio-to-data transducing devices and a data-to-audio transducing device.

The apparatus may be a system comprising at least one server and plural audio-to- data transducing devices.

A third aspect of the invention provides a computer program comprising instructions that when executed by a computer apparatus control it to perform a as recited method above.

A fourth aspect of the invention provides a non-transitory computer-readable storage medium having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to perform a method comprising:

aligning each of the at least three series of time-varying feature data.

The computer-readable code, when executed by computing apparatus, may cause the computing apparatus to perform receiving basis vectors relating to each of at least three series of time-varying feature data by receiving basis vectors relating to each of said at least three series of time-varying feature data at at least first and second different resolutions, and to perform performing multiple correlations by performing a first correlation for a first pair of said at least three series at the first resolution and also performing a second correlation for the first pair of said at least three series at the second resolution.

The computer-readable code, when executed by computing apparatus, may cause the computing apparatus to perform performing a correlation for the first pair of said at least three series at at least the two different resolutions by performing the first correlation for the first pair of said at least three series at a first resolution, determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion, and performing a second correlation for the first pair of said at least three series at a second resolution, higher than the first resolution, only if the metric of the degree of correlation does not meet a predetermined criterion.

The computer-readable code, when executed by computing apparatus, may cause the computing apparatus to perform determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion by calculating a histogram metric from the first correlation and comparing the histogram metric to a threshold.

The computer-readable code, when executed by computing apparatus, may cause the computing apparatus to perform performing the second correlation for the first pair of said at least three series at the second resolution by performing correlation within a calculation window that frames an alignment value of the result of the first correlation.

The computer-readable code, when executed by computing apparatus, may cause the computing apparatus to perform performing multiple correlations by

performing the first correlation for the first pair of said at least three series at the first resolution and also performing the second correlation for the first pair of said at least three series at the second resolution without conducting any determination as to whether a metric of degree of correlation meets a predetermined criterion between performing the first and second correlations.

The computer-readable code, when executed by computing apparatus, may cause the computing apparatus to perform a method additionally comprising assigning a correlated value to correlation of the first pair only if degrees of correlation of both the first and second correlations each meet a respective predetermined criterion.

The computer-readable code, when executed by computing apparatus, may cause the computing apparatus to perform a method additionally comprising receiving basis vectors relating to each of the at least three series of time-varying feature data also at a third resolution, different to the first and second resolutions, and to perform performing multiple correlations for the first pair by performing a third correlation at the third resolution.

The computer-readable code, when executed by computing apparatus, may cause the computing apparatus to perform a method additionally comprising selecting a series of time-varying data as a reference series and calculating an alignment value for each of the other series of time-varying data that is indicative of a delay between the respective series and the reference series.

A fifth aspect of the invention provides apparatus, the apparatus having at least one processor and at least one memory having computer-readable code stored thereon which when executed controls the at least one processor:

to receive basis vectors relating to each of at least three series of time- varying feature data;

to perform multiple correlations, each correlation being for a pair of said at least three series, the correlations linking together each of the at least three series; and

to align each of the at least three series of time-varying feature data.

The computer-readable code when executed may control the at least one processor to receive basis vectors relating to each of at least three series of time-varying feature data by receiving basis vectors relating to each of said at least three series of time-varying feature data at at least first and second different resolutions, and to perform performing multiple correlations by performing a first correlation for a first pair of said at least three series at the first resolution and also performing a second correlation for the first pair of said at least three series at the second resolution.

The computer-readable code when executed may control the at least one processor to perform a correlation for the first pair of said at least three series at at least the two different resolutions by performing the first correlation for the first pair of said at least three series at a first resolution, determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion, and by performing a second correlation for the first pair of said at least three series at a second resolution, higher than the first resolution, only if the metric of the degree of correlation does not meet a predetermined criterion.

The computer-readable code when executed may control the at least one processor to determine whether a metric of the degree of correlation of the first correlation meets a predetermined criterion by calculating a histogram metric from the first correlation and comparing the histogram metric to a threshold.

The computer-readable code when executed may control the at least one processor to perform performing the second correlation for the first pair of said at least three series at the second resolution by performing correlation within a calculation window that frames an alignment value of the result of the first correlation.

The computer-readable code when executed may control the at least one processor to perform performing multiple correlations by performing the first correlation for the first pair of said at least three series at the first resolution and also to perform the second correlation for the first pair of said at least three series at the second resolution without conducting any determination as to whether a metric of degree of correlation meets a predetermined criterion between performing the first and second correlations. The computer-readable code when executed may control the at least one processor to assign a correlated value to correlation of the first pair only if degrees of correlation of both the first and second correlations each meet a respective predetermined criterion.

The computer-readable code when executed may control the at least one processor to receive basis vectors relating to each of the at least three series of time-varying feature data also at a third resolution, different to the first and second resolutions, and to perform performing multiple correlations for the first pair by performing a third correlation at the third resolution. The computer-readable code when executed may control the at least one processor to select a series of time-varying data as a reference series and calculate an alignment value for each of the other series of time-varying data that is indicative of a delay between the respective series and the reference series.

Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings.

Brief Description of the Drawings

Figure 1 shows audio scene with N capturing devices;

Figure 2 is a block diagram of an end-to-end system embodying aspects of the invention; Figure 3 shows details of some aspects of the Figure 2 system;

Figure 4 shows a high level flowchart illustrating equation of aspects of embodiments of the invention.

Figure 5 shows a flowchart of calculating a basis vector from Figure 4; and

Figure 6 shows a flowchart for aligning multi-user content from Figure 4.

Detailed Description of Embodiments

Figures 1 and 2 illustrate a system in which embodiments of the invention can be implemented. A system 10 consists of N devices 11 that are arbitrarily positioned within the audio space to record an audio scene. In these Figures, there are shown four areas of audio activity 12. The captured signals are then transmitted (or alternatively stored for later consumption) so an end user can select a listening point 13 based on his/her preference from a reconstructed audio space. A rendering part then provides one or more

downmixed signals from the multiple recordings that correspond to the selected listening point. In Figure 1 , microphones of the devices 11 are shown to have a directional beam, but embodiments of the invention use microphones having any form of suitable beam. Furthermore, the microphones do not necessarily employ a similar beam, but microphones with different beams may be used. The downmixed signal(s) may be a mono, stereo, binaural signal or may consist of more than two channels, for instance four or six channels.

In an end-to-end system context, the framework operates as follows. Each recording device 11 records the audio scene and uploads/upstreams (either in real-time or non realtime) the recorded content to an audio server 14 via a channel 15. The upload/upstream process provides also positioning information about where the audio is being recorded and the recording direction/ orientation. A recording device 11 may record one or more audio signals. If a recording device 11 records (and provides) more than one signal, the direction/ orientation of these signals may be different. The position information may be obtained, for example, using GPS coordinates, Cell-ID or A-GPS. Recording

direction/ orientation may be obtained, for example, using compass, accelerometer or gyroscope information.

Ideally, there are many users/ devices 11 recording an audio scene at different positions but in close proximity. The server 14 receives each uploaded signal and keeps track of the positions and the associated directions/ orientations.

Initially, the audio scene server 14 may provide high level coordinates, which correspond to locations where user uploaded/up streamed content is available for listening, to an end user device 17. These high level coordinates may be provided, for example, as a map to the end user device 17 for selection of the listening position. The end user device 17 or e.g. an application used by the end user device is responsible for determining the listening position and sending this information to the audio scene server 14. Finally, the audio scene server 14 transmits the downmixed signal corresponding to the specified location to the end user device 17. Alternatively, the audio server 14 may provide a selected set of downmixed signals that correspond to listening point and the end user device 17 selects the downmixed signal to which he/ she wants to listen. Furthermore, a media format encapsulating the signals or a set of signals may be formed and transmitted to the end user devices 17. The downmixed signals here can be audio only content or content where audio is accompanied with video content.

Embodiments of this specification relates to immersive person-to-person communication including also video and possibly synthetic content. Maturing 3D audio-visual rendering and capture technology facilitates a new dimension of natural communication. An 'all-3D' experience is created that brings a rich experience to users and brings opportunity to new businesses through novel product categories. To be able jointly to utilize the multi-user recorded content for various content processing methods, such as audio mixing from multiple users 11 and view switching from one user 11 to the other, the content between different users must employ common time. The local device clocks of different users would normally need to be at least within a few tens of milliseconds before content from multiple users can be jointly processed. If the clocks of different user devices 11 (and, hence, the timestamp of the creation time of the content itself) are not in synchronization, content processing methods fail (as they produce poor quality signal/ content for the multi-user recorded content). It is an aim of embodiments of this specification to align efficiently multi-user recorded for content processing purposes.

Figure 3 shows a schematic block diagram of a system 10 according to embodiments of the invention. Reference numerals are retained from Figures 1 and 2 for like elements.

In Figure 3, multiple end user recording devices 11 are connected to a server 14 by a first transmission channel or network 15. The user devices 11 are used for detecting an audio scene for recording. The user devices 11 may record audio and store it locally for uploading later. Alternatively, they may transmit the audio in real time, in which case they may or may not also store a local copy. The user devices 11 are referred to as recording devices 11 because they record audio, although they may not permanently store the audio locally.

The server 14 is connected to listening user devices 17 via a second transmission channel 18. The first and second channels 15 and 18 may be different channels or networks or they may be the same channel or may be different channels within a single network. The listening user devices 17 may also be termed consuming devices on the basis that audio content is consumed at those devices 17.

Each of the recording devices 11 is a communications device equipped with a microphone. Each device 11 may for instance be a mobile phone, smartphone, laptop computer, tablet computer, PDA, personal music player, video camera, stills camera or dedicated audio recording device, for instance a dictaphone or the like. The recording device 11 includes a number of components including a processor 20 and a memory 21. The processor 20 and the memory 21 are connected to the outside world by an interface 22. At least one microphone 23 is connected to the processor 20. The microphone 23 may be directional. If there are multiple microphones 23, they may have different orientations of sensitivity.

The memory 21 may be a non- volatile memory such as read only memory (ROM) a hard disk drive (HDD) or a solid state drive (SSD). The memory 21 stores, amongst other things, an operating system 24 and at least one software application 25. The memory 21 is used for the temporary storage of data as well as permanent storage. Alternatively, there may be separate memories for temporary and non-temporary storage. The operating system 24 may contain code which, when executed by the processor 20 in conjunction with the memory 25, controls operation of each of the hardware components of the device 11. The one or more software applications 25 and the operating system 24 together cause the processor 20 to operate in such a way as to achieve required functions. In this case, the functions include processing audio data, and may include recording it. As is explained below, the functions may also include processing audio data to derive basis vectors therefrom.

The audio server 14 includes a processor 30, a memory 31 and an interface 32. Within the memory 31 are stored an operating system 34 and one or more software applications 35.

The memory 31 may be a non- volatile memory such as read only memory (ROM) a hard disk drive (HDD) or a solid state drive (SSD). The memory 31 stores, amongst other things, an operating system 34 and at least one software application 35. The memory 31 is used for the temporary storage of data as well as permanent storage. Alternatively, there may be separate memories for temporary and non-temporary storage. The operating system 34 may contain code which, when executed by the processor 30 in conjunction with the memory 35, controls operation of each of the hardware components of the server 14.

The one or more software applications 35 and the operating system 34 together cause the processor 30 to operate in such a way as to achieve required functions. In this case, the functions may include processing received audio data to derive basis vectors therefrom. The functions may also include processing basis vectors to derive alignment information therefrom. The functions may also include processing alignment information and audio to render audio therefrom.

Within the listening user devices 17, a processor 40 is connected to a memory 41 and to an interface 42. An operating system 44 is stored in the memory, along with one or more software applications 45. The memory 41 may be a non-volatile memory such as read only memory (ROM) a hard disk drive (HDD) or a solid state drive (SSD). The memory 41 stores, amongst other things, an operating system 44 and at least one software application 45. The memory 41 is used for the temporary storage of data as well as permanent storage. Alternatively, there may be separate memories for temporary and non-temporary storage. The operating system 44 may contain code which, when executed by the processor 40 in conjunction with the memory 45, controls operation of each of the hardware components of the listening user device 17.

The one or more software applications 45 and the operating system 44 together cause the processor 40 to operate in such a way as to achieve required functions. In this case, the functions may include processing audio data to derive basis vectors therefrom. The functions may also include processing basis vectors to derive alignment information therefrom. The functions may also include processing alignment information and audio to render audio therefrom.

Each of the user devices 11, the audio server 14 and the listening user devices 17 operate according to the operating system and software applications that are stored in the respective memories thereof. Wherein the following one of these devices is said to achieve a certain operation or provide a certain function, this is achieved by the software and/ or the operating system stored in the memories unless otherwise stated.

Audio recorded by a recording device 11 is a time-varying series of data. The audio may be represented in raw form, as samples. Alternatively, it may be represented in a non- compressed format or compressed format, for instance as provided by a codec. The choice of codec for a particular implementation of the system may depend on a number of factors. Suitable codecs may include codecs that operate according to audio interchange file format, pulse-density modulation, pulse-amplitude modulation, direct stream transfer, or free lossless audio coding or any of a number of other coding principles. Coded audio represents a time-varying series of data in some form.

High level operation of embodiments of the invention will now be descried with reference to Figure 4, which is a flowchart illustrating steps that occur in some part of the system 10 of Figure 3.

First, in step 4.1 of Figure 4, the basis vectors are calculated for each series of content data. Next, in step 4.2, the content from various users, i.e. different series of data, is aligned using the calculated basis vectors. Finally, in step 4.3 of, the multi-user content is rendered for the end user consumption. The rendering part may include various processing such as audio mixing, view switching, or joint processing of multi-user content. The exact details for the rendering part are outside the scope of this specification but what is common to all of the available rendering processing methods is that the multi-user content is assumed to be in synchronization. The steps of Figure 4 may be implemented in various ways in the end-to-end context.

The steps shown in the flowchart of Figure 4 are performed in different parts of the system 10 in different embodiments.

In a first embodiment, step 4.1 (creating basis vectors) is performed in the recording devices 11, and steps 4.2 (aligning content) and 4.3 (rendering content) are performed in the server 14. In this first embodiment, basis vectors are transmitted from the recording devices 11 to the server 14 through the first channel 15 and the rendered content is transmitted from the server 14 to the consuming devices 17 through the second channel 18.

In a second embodiment, basic vector calculation step 4.1 is performed in the recording devices 11, aligning step 4.2 is performed in the audio server 14 and the rendering step 4.3 is performed in the consuming devices 17. In this embodiment, basis vectors are transmitted from the recording devices 11 to the audio server 14 through the channel 15 and alignment results are transmitted from the server 14 to the consuming devices 17.

In a third embodiment, basis vectors are created in step 4.1 of the recording devices 11 and the aligning step 4.2 and the rendering step 4.3 are performed in the consuming devices 17. In this embodiment, basis vectors are transmitted from the recording devices 11 to the consuming devices 17, and this transmission may or may not be via the audio server 14.

In a fourth embodiment, each of the basis vector calculation step 4.1, the alignment step 4.2 and the rendering step 4.3 are performed by the audio server 14. In this case, raw or compressed audio data is received from the recording devices 11 and rendered content is transmitted from the server 14 to the consuming device 17.

In a fifth embodiment, the basis vector calculating step 4.1 and the alignment step 4.2 are performed at the audio server 14 and the rendering step 4.3 is performed at the end user device 17. In this embodiment, raw or compressed audio data is transmitted from the recording devices 11 to the audio server 14 and alignment results are transmitted from the audio server 14 to the consuming device 17.

In a sixth embodiment, basis vector calculating step 4.1 is performed at the audio server 14 and the alignment step 4.2 and the rendering step 4.3 are performed at the consuming device 17. In this embodiment, basis vectors are transmitted from the audio server to the receiving device 17 for processing.

In a seventh embodiment, the basis vector calculation step 4.1, the alignment step 4.2 and the rendering step 4.3 are all performed at the consuming device 17. Here, raw or coded audio data is transmitted from the recording devices 11 to the consuming device 17, optionally via the server 14, for processing thereat. In the sixth and seventh embodiments, in which the alignment is performed at the consuming device 17, alignment results may be transmitted back to the server 14 for storage and for possible later use, and/or for use by other consuming devices 17. Of course, even where basis vectors are calculated by the recording devices 11 , raw or coded audio content needs to be transmitted from the recording devices 1 1 to the server 14 and/or the consuming devices 17. In embodiments in which rendering step 4.3 is performed at the audio server 14, raw or coded audio directly from one of the recording devices 11 is not usually provided to any of the consuming devices 17.

The basis vector calculation step 4.1 and the alignment step 4.2 are described in more detail below. Figure 5 shows a flow chart for determining the basis vectors for each content data. First, in step 5.1, feature data is calculated from the content data. The feature data is then converted in step 5.2 to basis vectors that describe the content data in the feature domain . Specific examples for performing the steps 5.1 and 5.2 will now be explained in detail. The feature data is calculated is step 5.1 using a transform operator as follows.

Each recording source audio signal is first transformed to a frequency domain

representation. The transform (TF) operator is applied to each signal segment according to:

X_m [bin,l] = TF(x_{m bin l T} )

where m is the recording source index, bin is the frequency bin index, /is time frame index, Tis the hop size between successive segments, and TF() is the time-to-frequency operator. In the present embodiments, Discrete Fourier Transform (DFT) is used as the transform operator as follows: TF(x_{m nJ} ) =∑(win(n) - x_m(n + l - T) - e^~^ⁿ )

„=o (2)

2 · 71 · bin

w_bin =

where N , ^ is the size of the TFQ operator transform, win(n) is a -point analysis window, such as sinusoidal, Hanning, Hamming, Welch, Bartlett, Kaiser or Kaiser- Bessel Derived (KBD) window. To obtain continuity and smooth Fourier coefficients over time, the hop size is set to T=N/2, that is, the previous and current signal segments are 50% overlapping. Naturally, the frequency domain representation may also be obtained using DCT, MDCT/MDST, QMF, complex valued QMF or any other transform that provides frequency domain representation. Equation (1) is calculated on a frame by frame basis where the size of a frame is of short duration, for example, 20ms (typically less than 50ms).

To summarise, each series of audio data is converted to another domain, here the frequency domain. A transform operator, for instance a discrete Fourier transform, is then applied. The transform operator is applied to each signal segment.

This transform based feature data extraction is just one example of the various possible implementation choices. Feature data may also be calculated, for example, based on harmonic ratio of the audio signal, low energy ratio, audio beating, or various MPEG-7 defined audio descriptors such as AudioSpectrumSpreadType. The basis vectors may be calculated from multiple feature data instances. This can increase robustness in the content alignment operation.

The feature data is then converted to basis vectors according to the following pseudo-code:

Pseudo-code 1 :

1 _wr i sResltRes + 0.5

nwmdowElements = ^L ' ^J

2 _R , \ LI n Window Ele ments + 0.5

nElements = ^L ' ^J

3

4 For t = 0 to nElements - 1 5

6 tStart = t * nWindowElements

7 tEnd = tStart + twNext * nWindowElements

8 If tEnd > L

9 tEnd = L

10

11 bV_bl t) = fD_{bi m} (tStart,tEnd)

12 Endfor where sRes describes the sampling resolution of the basis vector, tRes is the time resolution of the frames according to tRes = N / (2 * Fs), L is the number of time frames present for the signal, and twNext describes the time period each element in the basis vector is covering at each time instant.

To summarise, the number of window elements is set to a function of the sampling resolution of the basis vectors and the time resolution of frames. The number of elements is then set to a function of the number of time frames present for the series and the number of window elements. For each element, a start time is set to a function of time and the number of window elements and end time is set to a function of start time, the time period each element in the basis vector is covering at each time instant and the number of window elements. This is performed for each time frame. Then, the basis vector value is determined by applying a function to the data between the start time and end time and assigning the result of applying the function to basis vector index t. fD

The value of ^bx,m in line 11 is calculated according to

2-bx+l

fD_{bx m}{tStart,tEnd) = X_m(binIdx(bk),k), tStart≤ k < tEnd

bk=2-bx where binldxQ returns the frequency bin indices to be included in the calculation and Bk is the number of bin indices to be used. To summarise, the frequency domain in the sampling period is calculated to be the sum of all frequency bins that make up X at any instant k between start and end times.

0≤bx <—

Equation (6) is repeated for 2 _ j_{n me} present embodiments, the binldx for the

TFQ operator is set to: f{k)- 2 - N

binldx(k) - , 0≤k < Bk

Fs

\ 50 - (fc + l), A: = 0,2,4

/(*) = 150 · (£ + !) + 50, t = l,3,5,..., - 2

( ) where Fs is the sampling rate of the source signals and fQ describes the frequencies to be covered, both in Hz. As can be seen the frequencies covered by the basis vectors have a width of 50Hz and the first frequency is set to 50Hz, that is, f=50, 100, 150, 200, 250,...50*Bk.

To summarise, a frequency bin index, which depends on an integer k, is calculated as a function of the reciprocal of the sampling rate, and frequencies covered for a period N.

The basis vectors are calculated for multiple different resolutions. In the present embodiments, the basis vectors are calculated for three different resolutions with the following parameters: bV_{bx m} = 0.25seconds, sRes = bvRes₀ , twNext = 2.5s bV_f Jt) = bV_{bx m} {t},bvRes_l = 0.064-seconds, sRes = bvRes_x , twNext = 1.Os

bV_{bx m} (t);bvRes₂ = 012^>seconds ,sRes = bvRes₂ , twNext = 1.5s

In summary, the three different resolutions relate to three different time periods. The largest time period is a factor greater than the second largest, which is the same factor greater than the smallest. In this example, the largest time period is twice the second largest time period, which is twice the smallest time period, at least approximately. Figure 6 shows a flowchart for determining the alignment between the multi-user recorded content data. First, for each content data pair, correlation of the pair is calculated (step 6.1 step 6.1). Next, correlation metrics are determined for the pair to assess whether the content pair is correlated and the degree of the correlation in step 6.2. If the metrics indicate that the pair is correlated but the degree of correlation is not strong enough (step 6.), the basis vectors are changed to another resolution and the calculations are recalculated with the new basis vectors (step 6.4). The steps are repeated until correlation is found or until different resolution basis vectors have been processed. Finally, in step 6.5 of Figure 6, the relative time differences between the multi-user content are determined from the correlation metrics. Next, the elements of Figure 6 are explained in more detail.

The correlation for some arbitrary content data pair (x,y) is calculated according to xCorr_x ^_sign (k) = sign^■ xC_k , 0≤k < xLen xLen = length(x), yLen = length(y) in(xLen, yLen)- k sumO

xC

yLen ^jsuml - energy min )— I energy = y[lldx - kf

(9) where lengthQ returns the size of the specified vector and minQ returns the minimum of the specified values. In summary, the correlation of the data pair (x,y) for index k is a function of the sign variable and the cross-correlation xC. The cross-correlation is a function of normalized length of data pair (x,y) at index k and the ratio of the cross-correlation value. The nominator of the cross-correlation value is calculated to be the sum of the product of x and y at indices defined by k and length of the data pair. The denominator of the cross- correlation value is the root of the product of the sum of the delayed data vector y squared and the sum of the data vector x squared. The correlation in Equation (9) is calculated twice in order to determine whether it is the content x that needs to be delayed with respect to content y or vice versa in order to achieve synchronization. The output vector from step 6.1 is therefore: corrXY_{x y} = [xCorr_x_-_{x y}_-_{y l} xCorr

The alignment between the content pair is determined by the maximum correlation in the correlation output as follows: xyVal = msLx(corrXY_{x y} )

where maxQ returns the index that holds the maximum absolute value in the specified vector. Equation (11) is calculated for each element within the basis vectors according to xyVal_bx = sampRes_bx ^■ max(corrXF_¾ ^ )

where describes the resolution of the basis vector in time domain for the bx⁽¹

0≤bx <— element within the basis vector matrix. Equation (12) is repeated for 2 _

Next, correlation metrics are determined for the content pair according to: lagVal = xyVal , stdVal = std{xyVal ) (13) Where: kngth(z)-l

z =

length—(z)■ ∑z[k]

k=0

(14)

length : 1 z=o

The lagVal describes the average alignment needed for the content pair based on the multiple instances of the basis vector and stdVal describes the deviation of the alignment in the basis vector domain. Typically, low deviation indicates that alignment value has converged towards certain value that most likely corresponds to the time difference between the content pair.

To summarise, the mean of length 2 is a function of the reciprocal of length 2 and the sum of all 2 in the domain k. Standard deviation of length 2 is the root of the product of the reciprocal of length 2 and the sum of 2 minus the average of 2 all squared.

Then, in step 6.3 of Figure 6, the correlation metrics are assessed to see whether the content pair is correlated or whether new set of basis vectors should be taken into use to test whether a weak correlation can be converted into strong correlation.

bV

The calculations for Figure 6 start with first resolution within f .m £_{or eac}] _COntent pair. sampRes_bx = bvRes bV_fl

Let ^ , and the content data pair (x, y) be fl^ ,o _ancj bV

fldx.bx.i ^ respectively. The steps 6.1-6.4 can be summari2ed into following pseudo-code as shown in pseudo-code 2.

Pseudo-code 2:

1 CorrAlign(ml , m2)

2 {

3 tVal = 0

4 outVal = not_correlated

5

6 Set basis functions 0)

7 Calculate correlation and correlation metrics - Equations (9)- (13) 8 If stdVal < 0.2s

9 outVal = lagVal

10 Else

11 {

12 Set basis functions 1)

13 Calculate correlation and correlation metrics - Equations (9)- (13)

14 lagVal = lagVal + tVal

15 If stdVal < 0.2s

16 outVal = stdVal

17 Else

18 {

19 Set basis function 2)

20 Calculate correlation and correlation metrics - Equations (9)- (13)

21 lagVal = lagVal + tVal

22 If stdVal < 0.2s

23 outVal = lagVal

24 }

25 }

26 Return outVal

27 } where lines 6, 12, and 19 are determined according to Line 6 : Set basis function 0) :

y ⁼ b Vfldx,bx

Line 12 : Set basis function 1): fldx = fldx + 1 ^x = ^{b V}fidx ,mi (4 startl≤t < end\

y = ^{b v} _fldx,_bx,_m 2 (Λ start!≤ t < end 2

lagVal -\5s lagVal + l5s

+ 0.5 lagVal >= 0 + 0.5

startl = < lagVal >= 0 b^vRes_fldx endl■ bvRes_fldx

0, otherwise length(bV fldx,bx, otherwise lagVal -15s lagVal + 15s

+ 0.5 , lagVal < 0 + 0.5

startl = lagVal < 0 b^vRes_fldx endl = < bvRes_fldr

0, otherwise length(bV fldx, bx, ml otherwise

startl bvRes _ndx , lagVal >= 0

tVal

startl bvRes fldx - otherwise

Line 9 : Set basis function 2): fldx = fldx + 1

^: ^start ≤t < endl

b^V _fl±c ,m2 ( startl≤ t < endl

lagVal -10s lagVal + 10s

+ 0.5 , lagVal >= 0 — + 0.5

startl = < lagVal >= 0 b^vRes_fldx endl = <

0, otherwise length(bV_fldx otherwise lagVal -10s lagVal + IPs

+ 0.5 lagVal < 0 + 0.5

startl = lagVal < 0 b^vRes_fldx endl = < bvRes_fldi

0, otherwise length(bV fldx ,bx ,m2 ) otherwise startl bvRes fldx ' lagVal >= 0

tVal =

start! bvRes fldx ' otherwise three cases above the following is also used: sampRes_bx = bvRes fldx

To summarise, start time 1 is zeroed if the average alignment is less than zero and is set to a value which is a function of the average alignment and basis vector resolution if the average alignment is greater than or equal to zero. The basis vector resolution defines the time period each element in the basis vector is covering. Conversely, start time 2 is zeroed if the average alignment is greater than or equal to zero and is set to a value which is a function of the average alignment and basis vector (res) if the average alignment is less than zero. End time 1 is formulated in the same way as start time 1 , except instead of being zeroed it is set to be equal to the length when the average alignment is less than zero. End time 2 is formulated in the same way as start time 2, except instead of being zeroed it is set to be equal to the length when the average alignment is greater than or equal to zero.

As can be seen, weak correlation triggers to switch the resolution and at the same time the calculation window is positioned around the alignment value of the weak correlation. In the present embodiments, the calculation window is limited to ±15 seconds around the first resolution alignment value and +10 seconds around the second resolution alignment value. In this way, either the weak correlation is turned into strong correlation indicating that content data pair is correlated or the weak correlation is confirmed by the multi- resolution calculations and hence the content data pair is not correlated. In some variations of the embodiments, the correlation is calculated for multiple resolutions in parallel manner, that is, calculations are performed, for example, both for lines 6 and 12 even though strong correlation would already be available for the first resolution. In these embodiment variations, all calculations for the content pair are required to indicate a strong correlation before the content pair is assigned a correlated value. Furthermore, in some variations of the embodiments, the switching logic between resolutions may be improved by employing also the histogram of the correlation results. In such variations of the embodiments the pseudo-code 3 is used. Pseudo-code 3:

1 Corr Align (ml, m2)

2 {

3 Execute lines 3-7 from pseudo-code 2

4

5 If stdVal < 0.2s

6 outVal = lagVal

7 Else if his Ratio > 0.9

8 outVal = histVal

9 Else

10 {

11 Execute lines 19-21 from pseudo-code 2

12 If stdVal < 0.2s

13 outVal = stdVal

14 Else if hisfRatio > 0.8

15 outVal = histVal

16 Else

17 {

18 Execute lines 19-21 from pseudo-code 2

19 If stdVal < 0.2s

20 outVal = lagVal

21 Else if hisfRatio > 0.7

22 outVal = histVal

23 }

24 }

25 Return outVal where lines 7-8, 14-15, and 21-22 introduce new switching logic. The new logic uses histogram metrics to assess the correlation in the content data pair. The histogram metrics are histRatio, and histVal. The histVal describes the alignment value in the histogram domain and histRatio describes the ratio of the histVal items with respect to the total number of items available. The alignment value in the histogram domain may be determined, for example, defining 3 time resolutions that have a width of 0.5s, Is, and 1.5s, and determining the histogram distribution (of ^xy^^a^ ) for each of the resolutions. The histVal is the histogram item that gets most hits, and the histRatio is the corresponding ratio value. The final value for the pseudo-code 3 is the one that maximizes the ratio over all 3 time resolution widths.

The final step is then to determine the relative time differences between the multi-user content data in step 6.5 of Figure 6. For this purpose, alignment matrix is calculated for each content pair according to corrMatrix_{m n} = CorrAlign(m,n), 0≤m < M, 0≤n < M

V ~7 where M is the number of content data to be aligned and the corrMatrix holds the alignment value for the m* and η^Λ content data pair that is determined according to pseudo-code 2. Next, the alignment values of different data pairs are adjusted so that common reference index is used for all pairs. This adjustment process is as follows

W _j = invalid _ value, 0≤i < M, 0 < j < M

IV_{ = corrMatrix_{i j} - corrMatrix_{i refldx} , 0 < i < M , 0 < j < M

In summary, the matrix IV is first initialized to default values and then the difference of the matrix entries at indices (i,j) and (i, refldx) is calculated where valid.

Equation (16) changes the reference to matrix index (i, refldx) for each matrix element. Equation (16) is not determined when corrMatrix _t . != not _ correlated

corrMatrix _{i refIdx} != not _ correlated

Next, the mean alignment value is calculated for each column in the alignment matrix according to:

O t ^ Λί ' IV _j -= invalid _value

r^efldx'1 _j=0 I 0, otherwise

f 1, IV- _i != invalid _ value

count _ mOut_{refIdx i} = <

■₌₀ [0, otherwise mOut = I™⁰ _{" *}' l^COUnt - ^m0Utrefl ,,i ' ^COUnt - ^m0utre_fldx,i > 0

r^efldx'¹ invalid _value, otherwise mOut_{refldx i} = mOut_{refldx i} - mOut_refldxfi , lV_{refldx i} ! = invalid _ value

(18)

In summary, a mean value is calculated for each index (refldx, i) where only valid matrix entries are taken into account. The final output for index (refldx, i) is always the difference with respect to the first column in the corresponding row for the matrix mOut.

Equation (18) is repeated for 0 < i < _ Furthermore, Equations (16)-(18) are repeated _for 0 < refldx < M

Next, for each content data, the final alignment is calculated from the mean values according to: mOut_i != invalid _ value

alignOut_i =

otherwise

^ fl, mOut_i != invalid _value

count _ alignOut_i = _,<

~ [0, otherwise

[alignOuti count _ alignOuti , count _ alignOuti > 0

alignOuti = <

[ invalid _value, otherwise In summary, the final alignment is the mean value of the previously calculated mean values for different reference matrix entry.

Equation (19) is repeated for 0 < i < j_{t ma}y -,_e advantageous to position the alignment values such that they are with respect to the content that appears first in the timeline. For this purpose, the minimum alignment value is determined and all the values are adjusted with respect to this. In this way, the content that appears first in the timeline is assigned alignment value zero and the rest of the content are delayed with respect to this one. The re-positioning is calculated according to: minAlign = alignOut_i != invalid _value alignOut_i = alignOut_i - minAlign, 0 < i < M , alignOut_i != invalid _value _nC, where minQ returns the minimum value of the specified vector.

In summary, the alignment value for each vector element is re-positioned as a difference of the corresponding matrix element and the minimum value of the alignment vector. Repositioning is determined only for valid vector entries.

The multi-user content is now in synchronization according to alignOut values and content can be jointly processed for various rendering and analysis purposes. [Are rendering and analysis sufficiently important that we should discuss them in some detail?]

An advantage achieved by the above-described embodiments is that they do not require any special timecodes, clappers or any other special preparations for the content alignment. The above-described embodiments, as well as other embodiments operating on the same principles, can operate at low computational complexity, thereby enabling tens of content items to the aligned simultaneously.

In variations of these embodiments, only a certain portion of the basis vectors may be considered in the alignment at a time. This may be particularly advantageous for content which have different durations, for example if content A has duration of lmin and content B has duration of lOmin. In these variations, the processing flow is as per pseudo-code 4 below

The A and B content are both split into smaller duration segments as defined by refDuration, dstDuration, refAdvance, and dstAdvance. These small duration segments are aligned until alignment between the segments was found or all segments were processed. The switching to use smaller duration segments takes place if alignment is not found using the basic alignment setup with original durations.

Although the above embodiments relate to series of time-varying data that represents audio, the scope of the invention is not limited to this. For instance, the invention is applicable also to processing video and other such time-varying series of data including static images, where spatial resolutions (width & height) can be considered to be time- varying.

Claims

1. A method comprising:

aligning each of the at least three series of time-varying feature data.

2. A method as claimed in claim 1, wherein receiving basis vectors relating to each of at least three series of time-varying feature data comprises receiving basis vectors relating to each of said at least three series of time-varying feature data at at least first and second different resolutions, and wherein performing multiple correlations comprises performing a first correlation for a first pair of said at least three series at the first resolution and also performing a second correlation for the first pair of said at least three series at the second resolution.

3. A method as claimed in claim 2, wherein performing a correlation for the first pair of said at least three series at at least the two different resolutions comprises performing the first correlation for the first pair of said at least three series at a first resolution, determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion, and performing a second correlation for the first pair of said at least three series at a second resolution, higher than the first resolution, only if the metric of the degree of correlation does not meet a predetermined criterion.

4. A method as claimed in claim 3, wherein determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion comprises calculating a histogram metric from the first correlation and comparing the histogram metric to a threshold.

5. A method as claimed in claim 3 or claim 4, wherein performing the second correlation for the first pair of said at least three series at the second resolution comprises performing correlation within a calculation window that frames an alignment value of the result of the first correlation.

6. A method as claimed in claim 2, wherein performing multiple correlations comprises performing the first correlation for the first pair of said at least three series at the first resolution and also performing the second correlation for the first pair of said at least three series at the second resolution without conducting any determination as to whether a metric of degree of correlation meets a

predetermined criterion between performing the first and second correlations.

7. A method as claimed in claim 6, comprising assigning a correlated value to correlation of the first pair only if degrees of correlation of both the first and second correlations each meet a respective predetermined criterion.

8. A method as claimed in any of claims 2 to 7, comprising receiving basis vectors relating to each of the at least three series of time-varying feature data also at a third resolution, different to the first and second resolutions, and wherein performing multiple correlations for the first pair comprises performing a third correlation at the third resolution.

9. A method as claimed in any preceding claim, comprising selecting a series of time-varying data as a reference series and calculating an alignment value for each of the other series of time-varying data that is indicative of a delay between the respective series and the reference series.

10. Apparatus comprising:

means for receiving basis vectors relating to each of at least three series of time-varying feature data;

means for performing multiple correlations, each correlation being for a pair of said at least three series, the correlations linking together each of the at least three series; and means for aligning each of the at least three series of time-varying feature data.

11. Apparatus as claimed in claim 10, wherein the means for receiving basis vectors relating to each of at least three series of time-varying feature data comprises means for receiving basis vectors relating to each of said at least three series of time-varying feature data at at least first and second different resolutions, and wherein the means for performing multiple correlations comprises means for performing a first correlation for a first pair of said at least three series at the first resolution and also means for performing a second correlation for the first pair of said at least three series at the second resolution.

12. Apparatus as claimed in claim 11, wherein the means for performing a correlation for the first pair of said at least three series at at least the two different resolutions comprises means for performing the first correlation for the first pair of said at least three series at a first resolution, means for determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion, and means for performing a second correlation for the first pair of said at least three series at a second resolution, higher than the first resolution, only if the metric of the degree of correlation does not meet a predetermined criterion.

13. Apparatus as claimed in claim 12, wherein the means for determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion comprises means for calculating a histogram metric from the first correlation and means for comparing the histogram metric to a threshold.

14. Apparatus as claimed in claim 12 or claim 13, wherein the means for performing the second correlation for the first pair of said at least three series at the second resolution comprises means for performing correlation within a calculation window that frames an alignment value of the result of the first correlation.

15. Apparatus as claimed in claim 11, wherein the means for performing multiple correlations comprises means for performing the first correlation for the first pair of said at least three series at the first resolution and also means for performing the second correlation for the first pair of said at least three series at the second resolution without conducting any determination as to whether a metric of degree of correlation meets a predetermined criterion between performing the first and second correlations.

16. Apparatus as claimed in claim 15, comprising means for assigning a correlated value to correlation of the first pair only if degrees of correlation of both the first and second correlations each meet a respective predetermined criterion.

17. Apparatus as claimed in any of claims 10 to 16, comprising means for receiving basis vectors relating to each of the at least three series of time-varying feature data also at a third resolution, different to the first and second resolutions, and wherein the means for performing multiple correlations for the first pair comprises means for performing a third correlation at the third resolution.

18. Apparatus as claimed in any of claims 10 to 17, comprising means for selecting a series of time-varying data as a reference series and means for calculating an alignment value for each of the other series of time-varying data that is indicative of a delay between the respective series and the reference series.

19. Apparatus as claimed in any of claims 10 to 18, comprising means for calculating the basis vectors relating the at least three series of time-varying feature data.

20. Apparatus as claimed in any of claims 10 to 19, wherein the apparatus is at least one server.

21. Apparatus as claimed in any of claims 10 to 19, wherein the apparatus is at least one server and a data-to-audio transducing device.

22. Apparatus as claimed in any of claims 10 to 19, wherein the apparatus is at least one server, plural audio-to-data transducing devices and a data-to-audio transducing device.

23. Apparatus as claimed in any of claims 10 to 19, wherein the apparatus is a system comprising at least one server and plural audio-to-data transducing devices.

24. A computer program comprising instructions that when executed by a computer apparatus control it to perform the method of any of claims 1 to 9.

25. A non-transitory computer-readable storage medium having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to perform a method comprising:

aligning each of the at least three series of time-varying feature data.

26. A non-transitory computer-readable storage medium as claimed in claim 25 having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to perform receiving basis vectors relating to each of at least three series of time-varying feature data by receiving basis vectors relating to each of said at least three series of time-varying feature data at at least first and second different resolutions, and to perform performing multiple correlations by performing a first correlation for a first pair of said at least three series at the first resolution and also performing a second correlation for the first pair of said at least three series at the second resolution.

27. A non-transitory computer-readable storage medium as claimed in claim 26 having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to perform performing a correlation for the first pair of said at least three series at at least the two different resolutions by performing the first correlation for the first pair of said at least three series at a first resolution, determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion, and performing a second correlation for the first pair of said at least three series at a second resolution, higher than the first resolution, only if the metric of the degree of correlation does not meet a predetermined criterion.

28. A non-transitory computer-readable storage medium as claimed in claim 27 having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to perform determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion by calculating a histogram metric from the first correlation and comparing the histogram metric to a threshold.

29. A non-transitory computer-readable storage medium as claimed in claim 27 having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to perform performing the second correlation for the first pair of said at least three series at the second resolution by performing correlation within a calculation window that frames an alignment value of the result of the first correlation.

30. A non-transitory computer-readable storage medium as claimed in claim 26 having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to perform performing multiple correlations by performing the first correlation for the first pair of said at least three series at the first resolution and also performing the second correlation for the first pair of said at least three series at the second resolution without conducting any determination as to whether a metric of degree of correlation meets a

predetermined criterion between performing the first and second correlations.

31. A non-transitory computer-readable storage medium as claimed in claim 30 having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to perform a method additionally comprising assigning a correlated value to correlation of the first pair only if degrees of correlation of both the first and second correlations each meet a respective predetermined criterion.

32. A non-transitory computer-readable storage medium as claimed in claim 26 having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to perform a method additionally comprising receiving basis vectors relating to each of the at least three series of time-varying feature data also at a third resolution, different to the first and second resolutions, and to perform performing multiple correlations for the first pair by performing a third correlation at the third resolution.

33. A non-transitory computer-readable storage medium as claimed in claim 25 having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to perform a method additionally comprising selecting a series of time-varying data as a reference series and calculating an alignment value for each of the other series of time-varying data that is indicative of a delay between the respective series and the reference series.

34. Apparatus, the apparatus having at least one processor and at least one memory having computer-readable code stored thereon which when executed controls the at least one processor:

to align each of the at least three series of time-varying feature data.

35. Apparatus as claimed in claim 34 wherein the computer-readable code when executed controls the at least one processor to receive basis vectors relating to each of at least three series of time-varying feature data by receiving basis vectors relating to each of said at least three series of time-varying feature data at at least first and second different resolutions, and to perform performing multiple correlations by performing a first correlation for a first pair of said at least three series at the first resolution and also performing a second correlation for the first pair of said at least three series at the second resolution.

36. Apparatus as claimed in claim 35 wherein the computer-readable code when executed controls the at least one processor to perform a correlation for the first pair of said at least three series at at least the two different resolutions by performing the first correlation for the first pair of said at least three series at a first resolution, determining whether a metric of the degree of correlation of the first correlation meets a predetermined criterion, and by performing a second correlation for the first pair of said at least three series at a second resolution, higher than the first resolution, only if the metric of the degree of correlation does not meet a predetermined criterion.

37. Apparatus as claimed in claim 36 wherein the computer-readable code when executed controls the at least one processor to determine whether a metric of the degree of correlation of the first correlation meets a predetermined criterion by calculating a histogram metric from the first correlation and comparing the histogram metric to a threshold.

38. Apparatus as claimed in claim 36 wherein the computer-readable code when executed controls the at least one processor to perform performing the second correlation for the first pair of said at least three series at the second resolution by performing correlation within a calculation window that frames an alignment value of the result of the first correlation.

39. Apparatus as claimed in claim 35 wherein the computer-readable code when executed controls the at least one processor to perform performing multiple correlations by performing the first correlation for the first pair of said at least three series at the first resolution and also to perform the second correlation for the first pair of said at least three series at the second resolution without conducting any determination as to whether a metric of degree of correlation meets a

predetermined criterion between performing the first and second correlations.

40. Apparatus as claimed in claim 39 wherein the computer-readable code when executed controls the at least one processor to assign a correlated value to correlation of the first pair only if degrees of correlation of both the first and second correlations each meet a respective predetermined criterion.

41. Apparatus as claimed in claim 35 wherein the computer-readable code when executed controls the at least one processor to receive basis vectors relating to each of the at least three series of time-varying feature data also at a third resolution, different to the first and second resolutions, and to perform performing multiple correlations for the first pair by performing a third correlation at the third resolution.

42. Apparatus as claimed in claim 34 wherein the computer-readable code when executed controls the at least one processor to select a series of time-varying data as a reference series and calculate an alignment value for each of the other series of time-varying data that is indicative of a delay between the respective series and the reference series.