GB2491874A

GB2491874A - Synchronising audio and video content in teleconference system

Info

Publication number: GB2491874A
Application number: GB1110122.7A
Authority: GB
Inventors: David John Gwilt; Alexander Edward Nancekievill
Original assignee: BLINKPIPE Ltd
Current assignee: BLINKPIPE Ltd
Priority date: 2011-06-16
Filing date: 2011-06-16
Publication date: 2012-12-19
Anticipated expiration: 2031-06-16
Also published as: GB2491874B; GB201110122D0

Abstract

Synchronising a teleconference comprising audio carried on a telephone network 110 and a data stream (such as video) carried on a computer network (such as the internet) 103 by determining a characteristic 334 of the audio input at a first node of a teleconference system and inserting 338 it into the data stream input at the first node. The audio is transmitted over the telephone network and the data stream is transmitted over the computer network. The audio and data streams are received at a second node in the teleconference system and the audio characteristic extracted 328 from the data stream. The receiving node determines a characteristic 324 of the received audio using the same method performed on the original signal and compares 318 this with the extracted audio characteristic to determine a time offset between the first and second node. The received audio and digital data streams are synchronised by adjusting a relative time delay between the audio and digital data stream at either or both of the first and second node.

Description

Video Conferencing Synchronisation Systems

FIELD OF THE INVENTION

This invention relates to systems, methods and computer program code for teleconferencing systems, in particular for video teleconferencing systems.

Embodiments of the techniques we describe are particularly useful for achieving lip-sync in a video conferencing system.

BACKGROUND TO THE INVENTION

Any system that involves the transmission and reception of both video and audio is potentially vulnerable to the lip-sync' problem. This is the common name for the situation where users perceive a difference between the time they hear a sound, and the time they see an image associated with that sound. A study by Blakowski and Stienmetz showed that humans are very sensitive to this time difference, and particularly so if audio arrives ahead of video. As a result the Advanced Television Systems Committee recommends in report IS-191 that broadcast television systems should not permit a difference of more than 45ms if audio leads video, and 75ms if video leads audio.

Differences in time occur because the audio and video take different paths from sender to receiver. If delays in each stage of each of these paths are fixed, then correction is relatively simple and only involves applying a further fixed delay to the faster path to bring them into synchronisation. An example of this form of correction is provided in many modern home cinema amplifiers which can apply a user controlled fixed delay to correct poor lip sync between video displayed on a television and audio played back through the amplifier.

However in video conferencing systems the problem is more complex because many of the delays are not fixed, but vary over time. A good explanation of the prior art solution to this problem is given in Voice and Video conferencing Fundamentals' published by CiscoTM press. In summary, prior art video conferencing systems attach a timestamp to each frame of audio samples, and another to each video frame in the transmitter.

These timestamps are used in the receiver to calculate the time difference between the two channels and then to correct it on a frame by frame basis.

The prior art approach works well, but it can only be used if it is possible to add timestamps to both audio and video channels. However in our new video conferencing system (see also our co-pending patent application filed on the same day as the application), video is added to a pre-existing ordinary telephone call. This new approach provides a number of benefits over previous approaches. The audio part of the call retains the simplicity, familiarity and reliability of an ordinary phone call. The video is an enhancement, but it does not get in the way of placing or conducting a phone call. Users do not have to replace existing conference phones or use a different microphone for video conferences versus audio conferences. For conference calls, it becomes very easy to mix some participants with audio-only and others who have audio and video. Moreover, if the IP network used by the video channel is heavily congested and video quality is affected, communication can continue uninterrupted as audio-only.

Unfortunately using the conventional telephone network for the audio channel of a video conference makes lip-sync much harder to achieve. It is no longer practical to use the prior art technique of adding timestamps to both channels because the POTS (Plain Old Telephone System) telephone network is extremely bandwidth limited and will only reliably carry signals within the range of human hearing. Thus adding time stamps to the audio channel would substantially interfere with the audio portion of the teleconference, which is undesirable.

We will therefore describe techniques suitable for use with embodiments of our new video conferencing system which enable synchronisation of audio and a digital data stream, for example a video data stream, in embodiments without interfering with or modifying the telephone audio in any way.

SUMMARY OF THE INVENTION

According to a first aspect of the invention there is therefore provided a method of synchronising a teleconference comprising audio carried on a telephone network and a digital data stream carried on a computer network, the method comprising: inputting first audio for said teleconference at a first node of a teleconference system; processing said first audio to determine first audio characterising data characterising said first audio; providing said first audio to a telephone network providing an audio teleconference; inputting at said first node a first digital data stream for said teleconference; inserting said first audio characterising data into said first digital data stream; forwarding said first digital data stream over said computer network; receiving said first audio at a second node of said teleconference system coupled to said telephone network, wherein said second node is a participant in said audio teleconference; receiving said first digital data stream at said second node of said teleconference system via said computer network; extracting said first audio characterising data from said digital data stream at said second node; processing said received first audio at said second node in conjunction with said extracted first audio characterising data to determine a first-second node time offset between said first audio received at said second node and said first digital data stream received at said second node; and synchronising said received first audio and said received first digital data stream by adjusting one or both of i) a time delay of one of said received first audio and said received first digital data stream with respect to the other at said second node responsive to said determined first-second node time offset; and ii) a time delay of one said first audio and said first digital data stream with respect to the other at said first node responsive to said determined first-second node time offset.

The teleconference may have just two nodes, that is it may be a point-to-point call, or may have more than two connected nodes. In embodiments the first digital data stream may comprise a video data stream from a video camera, captured at the first node. Depending upon the number of other nodes connected to the teleconference system, synchronisation may either be achieved by locally delaying, for example, the received audio to align this with the received video at a receiving node or, in a system with more than two nodes, by delaying the transmitted audio from each of the nodes so that when the audio is mixed together in a telephone conference bridge, all audio channels are in time synchronisation with each other. Once this has been established, the combined audio received from the conference bridge and separate video channels received from the other nodes can be aligned by adding appropriate time delay offsets at the receiving node (noting that in embodiments a node receives the video streams separately from the other nodes).

In some preferred embodiments the time offset at the receiving node is determined by applying a corresponding audio characterising function to the received audio as that used to generate the audio characterising data at the transmitting node; this audio characterising data can then aligned to determine a time offset, for example by simple comparison, correlation or other techniques.

In embodiments the audio characterising data includes data identifying a pattern of sound level variation, and the comparing (or similar) disregards sound levels below a threshold level, or at least weights these to have a reduced significance. This facilitates alignment where multiple audio streams are present simultaneously on a single audio line. The audio provided to the telephone network may be provided via an audio characterising module or it may be provided directly to the phone network and an audio characterising module may listen in on the audio to determine the audio characterising data.

In embodiments the telephone system comprises three or more nodes, and corresponding techniques to those described above are employed to determine time offsets between each video stream received at the local node and the combined audio stream received from the audio conference bridge. Any of a range of techniques may be used to determine the time offset between each video stream and every other video stream, for example by comparing embedded timestamps against a master reference transmitted over a control channel. From these results, the time offset between the multiple audio streams mixed together in the audio received from the conference bridge can be determined. These calculated time offsets are communicated to each of the remote nodes over the computer network. Each remote node is then able to delay its outgoing audio stream so as to ensure that all audio streams arrive at the conference bridge in sync with each other. With this first aspect of synchronisation achieved, the local node can now apply delays to the combined audio feed from the conference bridge and/or each video stream received separately in order to complete the synchronisation.

Where the digital data stream comprises a video data stream, in embodiments the audio characterising data changes no faster than every half a frame duration of the video in the video data stream, for example no greater than 15Hz where the video is at frames per second, to inhibit aliasing.

The invention also provides a system for synchronising a teleconference comprising audio carried on a telephone network and a digital data stream carried on a computer network, the system comprising a plurality of nodes each having a first interface for a phone, a telephone network interface for connecting to a telephone network, and at least one computer network interface for connecting to a computer network, and wherein said nodes are configured to: input first audio for said teleconference at a first node of a teleconference system; process said first audio to determine first audio characterising data characterising said first audio; provide said first audio to a telephone network providing an audio teleconference; input at said first node a first digital data stream for said teleconference; insert said first audio characterising data into said first digital data stream; forward said first digital data stream over said computer network; receive said first audio at a second node of said teleconference system coupled to said telephone network, wherein said second node is a participant in said audio teleconference; receive said first digital data stream at said second node of said teleconference system via said computer network; extract said first audio characterising data from said digital data stream at said second node; process said received first audio at said second node in conjunction with said extracted first audio characterising data to determine a first-second node time offset between said first audio received at said second node and said first digital data stream received at said second node; and synchronise said received first audio and said received first digital data stream by adjusting one or both of i) a time delay of one of said received first audio and said received first digital data stream with respect to the other at said second node responsive to said determined first-second node time offset; and ii) a time delay of one said first audio and said first digital data stream with respect to the other at said first node responsive to said determined first-second node time offset.

The invention further provides processor control code to implement the above-described systems, nodes, and methods, for example on a general purpose computer system or on a digital signal processor (DSP). The code is provided on a physical carrier such as a disc, CD-or DVD-ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (Firmware). Code (and/or data) to implement embodiments of the invention may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as C, or assembly code. As the skilled person will appreciate such code and/or data may be distributed between a plurality of coupled components in communication with one another.

In a further aspect the invention provides a node unit for synchronising a teleconference comprising audio carried on a telephone network and a digital data stream carried on a computer network, the node unit comprising a first interface for a phone, a telephone network interface for connecting to a telephone network and at least one computer network interface for connecting to a computer network, a time offset measurement system coupled to said computer network interface and to said telephone network interface, to determine a time offset between audio on said telephone network interface and data characterising said audio in a digital data stream on said computer network interface, and wherein said node unit is configured to perform one or both of i) local control of a relative time offset between audio received from said telephone network interface and said digital data steam, and ii) transmission of data representing said time offset over said computer network to at least one other said node unit.

In embodiments the node unit includes a controllable audio delay (which may be implemented digitally) between the phone interface and telephone network interface of the node unit, having a delay control input coupled to the computer network for remote control of the delay, for example by another node unit and/or a system control server.

In embodiments this controllable delay controls a delay in outgoing or transmitted audio from the phone to the telephone network. In embodiments the node unit also includes a controllable received audio delay, also coupled between the telephone network interface and the phone interface, and having a delay control input, and, in embodiments, a controllable digital data stream (video) delay coupled between the computer network connection and a digital data output of the node unit. Both the received audio, and video controllable delays may be coupled to a skew (time offset) controller to control one or both of these delays, in particular in response to a time offset determined by comparing audio characterising data extracted from the digital data stream with corresponding audio characterising data generated from the received audio. In embodiments this skew control unit also has a time offset data output coupled to the computer network to provide time offset data to one or more of the other nodes and/or a system control server. The skilled person will appreciate that depending upon the implementation, for example, whether there are two or more than two video nodes in the system, not all of these node unit modules may be required.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the intention will now be further described, by way of example only, with reference to the accompanying figures in which: Figure la and lb show respectively, an overview and a detailed view of a video teleconferencing system according to an embodiment of the invention; Figures 2a and 2b show, respectively, first and second examples of message passing during automatic connection of computer equipment between participants in an audio teleconference, according to embodiments of the invention; Figure 3a and 3b show, respectively, a telephone interface and a node unit or endpoint' according to an embodiment of the invention; and Figures 4a and 4b illustrate a procedure which may be employed for a synchronisation process for a node unit or endpoint', according to an embodiment of the invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Broadly speaking we will describe a video conferencing system in which video is added to a pre-existing ordinary telephone call. Thus embodiments of the video conferencing system add video transmitted over an IP network to a telephone call placed using pre-existing telephone equipment and infrastructure. We will describe procedures for automatically establishing such a video conference, and also techniques for use in such a video conference for ensuring that the video and audio are matched in time, so that participant's lips move with the sound of their voices.

The approach conveys a number of benefits: The audio part of the call retains the simplicity, familiarity and reliability of an ordinary phone call. The video is an enhancement, but it does not get in the way of placing or conducting a phone call.

Users do not have to replace existing conference phones or use a different microphone for video conferences versus audio conferences. For conference calls, it becomes very easy to mix some participants with audio-only and others who have audio and video.

Moreover, if the IF network used by the video channel is heavily congested and video quality is affected, communication can continue uninterrupted as audio-only.

In a typical audio teleconference, the parties dial a number followed by a PIN.

Normally there is one PIN for the host and a separate PIN for the guests.

Teleconference service providers have local phone numbers in many geographies to reduce the cost of participation. Having dialled a normally local number, plus a PIN, the participants are able to talk to each other in a multi-way audio conference.

Often, the audio participants will want to share other electronic information with each other, which could be presentations, white-board sketches or streaming video.

Currently this is either done by email (e.g. for PowerPoint), using a service provider that gives a URL for the participants to enter into their web browser (e.g WebEx), or using a unified communications provider where such services are integrated with the audio (e.g. Skype).

Whilst many people are familiar with audio conferences, accessing associated digital content to accompany the audio is often a time-consuming process which often delays the start of multi-party meetings. This is especially noticeable in video telephony.

Where AV/IT equipment is installed in meeting rooms to facilitate rich communication, associated remote controls, touch panels and the like often serve to be a barrier to effective use of this equipment.

At a high level, therefore, we will describe techniques which enable parties to access a hosted audio conference using their existing telephone equipment and conference provider, and having accessed said audio conference will automatically connect an associated piece of IT equipment with similar pieces of IT equipment in the rooms of the other audio conference participants to facilitate the fast and easy sharing of associated digital media. Broadly the system comprises 2 parts, a phone tap device (node unit or "endpoint"), and a system control server. The phone tap or node unit has two telephone interface; one to the local telephone and one to the exchange (FBX). It also has a network interface in order to communicate with the system control server, and an interface specific to the IT equipment being connected to the hosted audio conference.

When a user wishes to join an audio conference, they take the phone off-hook and dial as they would normally do. On dialling out, the Phone Tap recognizes DTMF tones and optionally sends the dialled number to the control server. The Phone Tap acts as a pass-through, enabling normal 2-way audio communication whilst being capable of recognizing tones played on the audio conference, received from the PBX.

The control sever optionally sends a unique-in-time code back to the Phone Tap, which on receipt the Phone Tap will play as tones out to the PBX. If the Phone Tap hears any tones being played while it is playing its tones, it will back-off and retry the code until it is certain that any other audio participants can have unambiguously received the code.

On detecting a series of tones coming from the PBX, the Phone Tap will encode these and report the code back to the control or rendezvous server. Using this information, the Rendezvous Server is able to determine which Phone Taps are connected to the same audio conference without Phone Taps having to reply to each others codes over the audio channel.

Having made this determination, the rendezvous server can send a unique-in-time identifier to each Phone Tap, which the Phone Tap passes back to the associated IT equipment. The IT equipment uses this unique identifier to automatically share digital media. Examples of this identifier are a URL for joining a shared-desktop, or an IP address for transmitting or receiving streaming data.

Referring now to Figure la, this shows an overview of an embodiment of a video conferencing system 100 according to the invention. The system comprises IT (Information Technology) equipment which is automatically connected when connecting to an audio teleconference in a preferred embodiment comprising a pair of video capture/display units 107 each comprising a video camera 1 07a and a display 1 07b. The video units 107 are each connected to an IP (Internet Protocol) network 103, in particular a public IP network such as the Internet. The system 100 also includes a pair of standard audio telephones 106 each connected to a public POTS telephone network 110. The system also includes a pair of node units 105, also termed herein endpoints/phone taps. Each of units 105 is coupled between the phone 106 and the telephone network 110 and has a further network connection to a respective video unit 107. The skilled person will appreciate that in the arrangement of figure 1 a a video unit 107 may have only a single network connection rather than the pair of network connections illustrated. Broadly speaking in operation the node units 105 are able to intercept and modify, in particular delay, sound on the telephone network, and displayed video.

Referring next to Figure lb this shows further details of the system of Figure la, illustrating a third party audio conference bridge 102 connected to the telephone network 110. Typically such an audio bridge enables multiple audio conferencing parties to dial the same number with a different pin to connect to a shared audio teleconference managed by audio bridge 102. The system 100 also comprises a system control or rendezvous' server 104, in embodiments accessible via the internet, and running processor control code (software) to implement a switching' procedure as described later. The server 104 operates in conjunction with the endpoint or node units 105, which also run software as described below, for automatically establishing IT connectivity between IT equipment units 107. Typically the phone 106 and IT equipment 107 of the nodes are located in different office locations. For simplicity Figure 1 shows just two nodes but the skilled person will appreciate that there may be many such nodes in a shared teleconference.

Figures 2a and 2b, these show messages passing between the various elements of system 100 according to a method for automatic connection of the IT equipment between participants in an audio teleconference according to an embodiment of the invention. In Figures 2a and 2b the two nodes are designated A' and B'. The message passing illustrated in Figures 2a and 2b is implemented by corresponding software running on the node units 105 and server 104. Figure 2b illustrates a variant of the protocol of Figure 2a in which some messages (messages 3 and 4 and 8 and 9) are omitted.

Thus, referring to the numbered messages in Figures 2a and 2b, the protocol of Figure 2aisasfollows: 1) Telephone A dials into an audio bridge 2) Endpoint A detects the phone number that is dialled, but passes the audio through to the audio bridge 3) If Endpoint A decides that the phone number dialled may be able to share digital media, it sends a message to the Rendezvous (RV) server which includes the phone number and its own unique endpoint identifier 4) If the RV server further determines that it is likely that sharing of digital media may take place, the RV server sends a temporary unique code back to endpoint

A

5) Endpoint A announces itself in the audio bridge by playing a sequence of tones derived from the unique code received in (4). Endpoint A listens for tones from the audio bridge; if tones are heard whilst Endpoint A is announcing itself, it will back off and retry some random time later. Once tones have been successfully announced, Endpoint A enters listening' mode.

6) At some point later, Telephone B dials into the audio bridge 7) Endpoint B detects the phone number that is dialled, but passes the audio through to the audio bridge 8) If Endpoint B decides that the phone number dialled may be able to share digital media, it sends a message to the Rendezvous (RV) server which includes the phone number and its own unique endpoint identifier 9) If the RV server further determines that it is likely that sharing of digital media may take place, the RV server sends a temporary unique code back to endpoint

B

10) Endpoint B announces itself in the audio bridge by playing a sequence of tones derived from the unique code received in (9). Endpoint B listens for tones from the audio bridge; if tones are heard whilst Endpoint B is announcing itself, it will back off and retry some random time later. Once tones have been successfully announced, Endpoint B enters listening' mode.

11) The audio bridge transmits the tones played by Endpoint B back to Endpoint A over the POTS network 12) Endpoint A detects the tones, transforms them into a code and transmits that code to the RV server along with its own unique endpoint identifier 13) The RV server has been able to determine that Endpoints A and B are in the same audio conference by matching the code sent to Endpoint B with the code received from Endpoint A. The RV server generates a unique-in-time sharing-identifier which will allow IT equipment A and IT equipment B to share digital media through a 3 party sharing service. The RV server sends this sharing-identifier to Endpoints A & B 14) Endpoints A and B each pass this sharing-identifier to their attached IT equipment 15) IT Equipment A and B use their sharing identifiers to connect using an IT sharing service In the variant of Figure 2b, messages 3, 4, 8 and 9 are omitted. Then the tones that the Endpoints use to announce themselves may be some transformation of the endpoint's own unique identifier. The transformation is chosen to allow Network Address Translation (NAT) at stage (13) and may, for example, be lossless.

Figure 3a shows a telephone interface portion of a node unit 105, illustrating a connection 300 to the telephone network 110, in this example the connection comprising a two wire interface. A connection 302 to analogue phone 106 likewise comprises a two wire interface. A subscriber line interface circuit 306 converts the differential signal into an audio transmit (from the phone) line 308 and an audio receive (to the phone) line 310,. a corresponding PSTN (Public Switched Telephone Network) interface 304 re-converts the signals back to the two-wire interface 300. A switch 314 selectively connects the audio receive line 310 to either the phone interface 304 or to a suitable termination to mute the phone 106. The audio receive signal 310b is also connected to a variable delay circuit 316 which in turn is connected to a speaker 1 07c.

With the switch in the position shown, sounds (audio) received from a remote telephone are not relayed to the local handset but instead pass through the variable delay circuit 316 before being played on a speaker 10Th, which may be located next to the video display 107b so that the sound comes from the same direction as the image of the person speaking. (Alternatively the audio output of delay circuit 316 may be provided back to phone 106). With the switch in the alternative position a normal telephone call can take place where audio received from a remote telephone is relayed directly to the local handset. In this mode the output of delay circuit 316 is muted to prevent sound from being heard from the speaker 1 07c as well as the local telephone.

A second switch 319 selectively connects the audio transmit line 308b to either the phone interface 306 or to the output of a variable delay circuit 322. With this switch in the position shown audio transmitted by the local handset passes through the variable delay circuit 322 before being sent to the remote telephone. Via appropriate settings of the two switches, the interface 105 is able to selectively delay both sounds picked up by the local telephone's microphone and also sounds received from a remote telephone over the POTS network. In addition it is able to divert sounds received from the remote telephone for playback over a speaker instead of the telephone. Details of how the delays to be applied by circuits 316 and 322 are determined from the audio characteristics generated by circuits 334 and 324 are described in figure 3b.

Figure 3b shows details of an embodiment of a node unit or endpoint 105 in which like elements to those previously described are indicated by like reference numerals.

Thus referring to figure 3b an instruction to adjust the audio transmission delay is received via a logical connection 320 to lP network 103 and used to control a variable audio transmission delay module 322 coupled between the audio output of phone 106 and the POTS 110. In this way audio between 3 or more nodes in a conference call may be synchronised.

Audio received from POTS 110 and provided on line 310 is provided to an audio characterising module 324 which determines characteristic data for the audio signal from the telephone network and provides a characterising data output 326. An unload module 328 logically coupled to IP network 103, receives the video signal for the node from the network and extracts the audio characterising data from this video signal providing this data on line 330. Skew control circuit compares the data on lines 326 and 330, for example by performing a cross-correlation, to determine a time offset between the audio timing encoded on the video signal and the received audio timing, and this time offset data is provided on a set of outputs 332. One of outputs 332 controls a variable delay applied by module 316 to the received audio intended for output by phone 106. another of outputs 332 provides a delay control input to a received video delay module 334, which receives a video stream from video decoder 336 coupled to unload module 328, and provides a (delayed) video output to display 1 07b. A third output 332 is logically coupled to IP network 103 to provide data for adjusting an audio transmission delay to one or more other nodes or endpoints 105 in the video conference.

The local audio from phone 106 for transmission over the POTS 110 is similarly characterised by module 334, and this audio characterising data is buffered by module 336 and combined with video data in network payload data construction module 338.

This module also receives video from camera 107a, captured by module 340 and encoded by module 342, the digitally encoded video data and characteristic audio data being combined and provided to lP network 103.

The skilled person will appreciate that the above described modules may be implemented in hardware, software, or a combination of the two.

In operation the near end video conference unit samples sound recorded by the near end telephone and digitizes it. The digital samples are then processed to produce a simplified characteristic of the original audio signal. In its simplest form this characteristic divides the audio signal into equal length periods, and for each period record whether the average sound amplitude was greater than a threshold level of background noise. This characteristic gives a pattern of sound/no-sound values representing the transmitted audio signal. We will use this simple characteristic in the examples which follow because it makes the process of correlating the characteristics easy to visualise. However variations in gain across the telephone network, signal processing, distortion and noise contribute to reducing the reliability of using a threshold to characterise the audio signal. Preferably, rather than quantising the sound (for example, average sound amplitude) in a given period to a single bit using a threshold, it is quantised using multiple bits, for example 15 bits giving a range from 0- 32767. Greater reliability may be obtained by using more complex characteristics of the sound (for example taking a Fourier transform of the audio signal to determine the amount of energy in different frequency bands), at the cost of employing more compute power to calculate the characteristics, and at the cost of using more bandwidth in the computer network to transmit these. In theory, the entire digitised audio stream could be transmitted without modification or with a lossless compression technique.

Whichever characteristic is used, the near end video conference unit adds the characteristic to the video frame data it captured at the same time as the audio was recorded. It then transmits this characteristic with the video frame over the video transmission channel. This may be an IP network, but other transmission channels are possible. At the far end, the far end video conference unit receives the video frame with associated audio characteristic. It independently samples sound received by the far end telephone from the telephone network and then performs a substantially identical characterisation of the received sound, for comparison with the characteristic embedded in the video to determine a time offset between the two. The two characteristics may be correlated to determine the time (phase) offset.

Example 1:

Video frame rate is 30 frames per second, and each frame is embedded with an audio characteristic representing speech or no speech during the 33ms of time the video frame represents.

Over the first second of transmission, the near end produces the following characteristic from locally recorded audio: oo1o111o1001011o111o1o10001o11 Over the same second, the far end identifies the following characteristic in the received audio signal: 1o11o10001o111o1001o11o111o1o1 On comparing the two characteristics, the receiver is able to determine a unique solution for the offset between them: oololfloloolol'oll'olo'ooolofl 1o11o10001o111o1001011o111o1o1 In this example audio lags video by 7 frames which is 233ms. A 233ms delay can now be added to the video pipeline to bring the two into synchronisation.

There are two complications this approach has to handle which are introduced through the use of audio conferencing numbers.

Non-video participants If an audio conference between two video endpoints also includes a non-video participant then the characteristic derived from audio heard at the receiving endpoints will not match the characteristic transmitted with the video. The effect of audio mixing in the additional caller will be to average the audio signals for which we have characteristics with one which we do not have a characteristic for. For a simple case using a threshold characteristic, some periods of not-sound will be converted to periods of sound. The technique can still cope with this by looking for matching periods of sound, and ignoring periods of not-sound. The sound pattern encoded with the video becomes for example lxlllxlxlxlllxlxllx where x indicates "don't care". This increases the chances of a false match but this can be compensated for by using longer sequences of patterns for comparison.

Multi-way calling If there are more than two video endpoints in a conference then lip sync becomes more complex, because although each endpoint receives the video streams separately, it only gets a combined audio feed from the telephone network. Unless the audio delays from all the units to the conference bridge in the telephone network are equal it is difficult to for a receiving endpoint can correct the different audio delays mixed up in the combined audio feed. To address this endpoints apply a delay to the audio they transmit as well as the audio they receive. Each endpoint learns from the other endpoints how much delay it should apply to its outgoing audio. Using the sound characteristic technique an endpoint in a three way call will receive a sound characteristic from each of the other two endpoints in their video streams, as well as calculating a sound pattern from the audio it receives from the conference bridge.

Example 2:

Conference bridge sound characteristic llololltllollolollolol000loloo From A's video stream lxllxxxtlxlxlxx From B's video stream xlxll lxtxlxxlxl Here are the matches 11o1o11111o11o1o11o1o10001o100 lxllxxxl lxlxlxx xlxll lxlxlxxlxl In this example we have successfully identified a unique solution for both delays using just half a frame of sound characteristics.

Referring now to Figure 4a, this illustrates synchronization in a video conference with three endpoints A, B and C 105a,b,c sharing an audio conference bridge X 102, such as "MeetingZone". In Figure 4a: Vab is the end-to-end delay transmitting video from A to B. Aax is the end-to-end delay transmitting audio from A to the conference bridge X. Axa is the end-to-end delay transmitting audio from A to the conference bridge X. Each endpoint is capable of measuring the difference (skew) between an incoming audio stream and its corresponding video stream using the technique described above.

For endpoint A: Skewb = AbX+ Axa -Vab Skews = Acx+ Axa -Vac The skew will be negative if the video arrives after the audio and positive if the video arrives first.

Each endpoint is capable of inserting additional delay into its incoming audio feed, outgoing audio feed and incoming video feed(s). So endpoint A is capable of increasing: Axa Aax Vba Vca In one approach each endpoint increases the incoming video delays until they are equally skewed relative to their respective audio streams ie. For a given endpoint, the algorithm is: increase the incoming V?? delays until all the Skew? values are equal.

Then if the Skew? values are greater than zero, increase all the V?? delays until Skew? is zero. If the Skew? values are less than zero, increase the incoming audio delay AX? until they become zero. This approach does not require the endpoints to communicate with each other for synchronization as each endpoint can calculate and apply all necessary delays without referring to any other endpoint. However this simple method can increase the worst-case delay between callers. This is undesirable because as delays become longer it becomes harder for callers to maintain a natural interactive conversation. Furthermore this approach will not synchronise any video/audio pair to any other video/audio pair in the conference. It only ensures that considered individually, each stream has its audio in sync with its video.

A better solution uses the procedure of Figure 4b. In order to use this approach the endpoints need to determine the skew between the video streams. This can be achieved using a prior art technique described in Voice and Video conferencing Fundamentals'. In summary this is done by sharing a master reference clock across all the endpoints and encoding timestamps in the video streams which are relative to the master clock. With this additional information each endpoint can now calculate the skew between audio streams as follows: Skewb -Skew= AbX+ Axa -Vab -(Ac+ Axa -Vac) = AbX -Acx + Vac -Vab Acx -AbX = Vac -Vab + Skew0 -Skewb Now referring to the steps in Figure 4b: 1) Each endpoint measures the skew between the incoming audio streams contained within the mixed audio feed it receives from the conference bridge (Step 400). For example in the three way call shown in Figure 4a, endpoint A measures: SkewbC = Ab - 2) Each endpoint then determines how much the other endpoints would have to increase their outgoing audio delays in order to make this skew value zero and transmits appropriate instructions to the other endpoints over the IF network (Step 402).

3) Each endpoint will receive instructions from every other endpoint (Step 404).

Continuing this three-way example, endpoint B will receive instructions to increase its outgoing audio delay AbX from endpoints A and C. Endpoint B applies the largest delay requested by any other endpoint.

4) Audio to audio skews measured in (1) will now be zero. Each endpoint now measures the skew between each audio stream and its corresponding video stream (Step 406).

5) If the audio is ahead of video, the endpoint increases its incoming audio delay to compensate; if video is ahead of audio, the endpoint increases its incoming video delay to compensate (Step 406).

Preferred embodiments of the above described techniques are used to synchronise audio with digital data streams carrying a corresponding video. However the skilled person will appreciate that the techniques may also be employed to synchronise digital data streams carrying other forms of digital data.

No doubt many other effective alternatives will occur to the skilled person. lt will be understood that the invention is not limited to the described embodiments and encompasses modifications apparent to those skilled in the art lying within the spirit and scope of the claims appended hereto.

Claims

CLAIMS: 1. A method of synchronising a teleconference comprising audio carried on a telephone network and a digital data stream carried on a computer network, the method comprising: inputting first audio for said teleconference at a first node of a teleconference system; processing said first audio to determine first audio characterising data characterising said first audio; providing said first audio to a telephone network providing an audio teleconference; inputting at said first node a first digital data stream for said teleconference; inserting said first audio characterising data into said first digital data stream; forwarding said first digital data stream over said computer network; receiving said first audio at a second node of said teleconference system coupled to said telephone network, wherein said second node is a participant in said audio teleconference; receiving said first digital data stream at said second node of said teleconference system via said computer network; extracting said first audio characterising data from said digital data stream at said second node; processing said received first audio at said second node in conjunction with said extracted first audio characterising data to determine a first-second node time offset between said first audio received at said second node and said first digital data stream received at said second node; and synchronising said received first audio and said received first digital data stream by adjusting one or both of i) a time delay of one of said received first audio and said received first digital data stream with respect to the other at said second node responsive to said determined first-second node time offset; and ii) a time delay of one said first audio and said first digital data stream with respect to the other at said first node responsive to said determined first-second node time offset.
2. A method as claimed in claim 1 wherein said processing of said received first audio comprises determining received first audio characterising data characterising said received first audio, and correlating said received first audio characterising data and said first audio characterising data to determine said first-second node time offset.
3. A method as claimed in claim 2 wherein said first audio and said received first audio characterising data each comprise data identifying a pattern of sound level variation, wherein said correlating comprises comparing respective said patterns of second level variation, and wherein one or both of the respective said patterns is weighted such that sound levels below a threshold level have a reduced significance in said comparing.
4. A method as claimed in claim 1, 2 or 3 wherein said teleconference system further comprises a third node, the method further comprising: receiving at said second node third audio and a third digital data stream from said third node via said telephone network and said computer network respectively, wherein said third digital data stream includes third audio characterising data characterising said third audio; processing said received third audio at said second node in conjunction with said extracted third audio characterising data to determine a second-third node time offset between said third audio received at said second node and said third digital data stream received at said second node; and synchronising said received first audio, said received first digital data stream, said received third audio, and said received third digital data stream, using said first-second node time offset and said second-third node time offset.
5. A method as claimed in claim 4 wherein said synchronising comprises one or both of: i) adjusting a time delay of one of said first audio and said first digital data stream with respect to the other at said first node responsive to said data dependent on said first-second node time offset communicated to said first node, and ii) adjusting a time delay of one of said third audio and said third digital data stream with respect to the other at said third node responsive to said data dependent on said second-third node time offset communicated to said third node.
6. A method as claimed in claim 4 or 5 wherein said synchronising comprises communicating data dependent on said first-second node time offset and said second-third node time offset to said first and third nodes respectively.
7. A method as claimed in claim 6 wherein said synchronising comprises determining, at a said node, from time offset dependent data communicated to said second node from other said nodes, a maximum time offset to apply to a said audio or digital data stream received at the node from a said other node; and applying said maximum time offset to synchronise audio from said other nodes at said second node; and synchronising said digital data streams from said other nodes to said synchronised audio.
8. A method as claimed in claim 4 or 5 wherein said synchronising comprises adjusting, at said second node, a delay in one or both of said first and third digital data streams.
9. A method as recited in any preceding claim wherein a said digital data stream comprises a video data stream, and wherein said audio characterising data changes no faster than every half-a-frame duration of video in said video data stream.
10. A physical carrier carrying processor control code for: inputting first audio for said teleconference at a first node of a teleconference system; processing said first audio to determine first audio characterising data characterising said first audio: providing said first audio to a telephone network providing an audio teleconference; inputting at said first node a first digital data stream for said teleconference; inserting said first audio characterising data into said first digital data stream; forwarding said first digital data stream over said computer network; receiving said first audio at a second node of said teleconference system coupled to said telephone network, wherein said second node is a participant in said audio teleconference; receiving said first digital data stream at said second node of said teleconference system via said computer network.extracting said first audio characterising data from said digital data stream at said second node; processing said received first audio at said second node in conjunction with said extracted first audio characterising data to determine a first-second node time offset between said first audio received at said second node and said first digital data stream received at said second node; and synchronising said received first audio and said received first digital data stream by adjusting one or both of i) a time delay of one of said received first audio and said received first digital data stream with respect to the other at said second node responsive to said determined first-second node time offset; and ii) a time delay of one said first audio and said first digital data stream with respect to the other at said first node responsive to said determined first-second node time offset.
11. A system for synchronising a teleconference comprising audio carried on a telephone network and a digital data stream carried on a computer network, the system comprising a plurality of nodes each having a first interface for a phone, a telephone network interface for connecting to a telephone network, and at least one computer network interface for connecting to a computer network, and wherein said nodes are configured to: input first audio for said teleconference at a first node of a teleconference system; process said first audio to determine first audio characterising data characterising said first audio; provide said first audio to a telephone network providing an audio teleconference; input at said first node a first digital data stream for said teleconference; insert said first audio characterising data into said first digital data stream; forward said first digital data stream over said computer network; receive said first audio at a second node of said teleconference system coupled to said telephone network, wherein said second node is a participant in said audio teleconference; receive said first digital data stream at said second node of said teleconference system via said computer network; extract said first audio characterising data from said digital data stream at said second node; process said received first audio at said second node in conjunction with said extracted first audio characterising data to determine a first-second node time offset between said first audio received at said second node and said first digital data stream received at said second node; and synchronise said received first audio and said received first digital data stream by adjusting one or both of i) a time delay of one of said received first audio and said received first digital data stream with respect to the other at said second node responsive to said determined first-second node time offset; and ii) a time delay of one said first audio and said first digital data stream with respect to the other at said first node responsive to said determined first-second node time offset.
12. A node unit for synchronising a teleconference comprising audio carried on a telephone network and a digital data stream carried on a computer network, the node unit comprising a first interface for a phone, a telephone network interface for connecting to a telephone network and at least one computer network interface for connecting to a computer network, a time offset measurement system coupled to said computer network interface and to said telephone network interface, to determine a time offset between audio on said telephone network interface and data characterising said audio in a digital data stream on said computer network interface, and wherein said node unit is configured to perform one or both of i) local control of a relative time offset between audio received from said telephone network interface and said digital data steam, and ii) transmission of data representing said time offset over said computer network to at least one other said node unit.
13. A node unit as claimed in claim 11 further comprising a time delay control system coupled to said computer network interface to control an audio time delay between said first interface and said telephone network interface responsive to time offset control data received over said computer network interface.