GB2488772A - Creating annotated recordings from two devices using audio synchronisation - Google Patents

Creating annotated recordings from two devices using audio synchronisation Download PDF

Info

Publication number
GB2488772A
GB2488772A GB201103789A GB201103789A GB2488772A GB 2488772 A GB2488772 A GB 2488772A GB 201103789 A GB201103789 A GB 201103789A GB 201103789 A GB201103789 A GB 201103789A GB 2488772 A GB2488772 A GB 2488772A
Authority
GB
United Kingdom
Prior art keywords
recording
audio
annotations
time
device
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
GB201103789A
Other versions
GB201103789D0 (en
Inventor
Roger Cecil Ferry Tucker
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SONOCENT Ltd
Original Assignee
SONOCENT Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SONOCENT Ltd filed Critical SONOCENT Ltd
Priority to GB201103789A priority Critical patent/GB2488772A/en
Publication of GB201103789D0 publication Critical patent/GB201103789D0/en
Publication of GB2488772A publication Critical patent/GB2488772A/en
Application status is Withdrawn legal-status Critical

Links

Classifications

    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/102Programmed access in sequence to addressed parts of tracks of operating record carriers
    • G11B27/105Programmed access in sequence to addressed parts of tracks of operating record carriers of operating discs

Abstract

A method of providing an annotated recording comprising receiving from a first audio recording device 103 a first audio recording of an event or presentation, receiving from a note taking device 104 a second audio recording of the same event or presentation with time-sensitive user annotations, wherein the second recording is used to provide a synchronization reference such that an annotated audio stream 106 can be generated from the first recording and from the annotations associated with the second recording. The annotations and first recording can be synchronized without the need for time-stamping or clock synchronization between devices. The second recording may be of poor quality containing local noise and the first recording of good quality. Correlation using pattern matching of the first and second audio recordings can be used to calculate a time offset between the two recordings. The first audio recording may mostly replace the second audio recording.

Description

Creating Annotated Recordings from Two Devices using Audio Synchronisation

Description

Baickgrond t© the llnventon When attending a lecture, seminar, meeting or interview (an event), a person (the user) may want to record it and also make time-sensitive annotations synchronised with the recording using a computing device (the note-taking device). Annotations might be text, writing, scribbles, sketches, highlights, marks, section breaks, images, slides -any time-sensitive media that enhances the recording. There are various hardware and software products available that allow this (e.g.Livescribe, Audio Notetaker).

the problem is that mobile computing devices do not usually have good sound capture, especially at a distance from the speaker, so if the note-taking device is also used for recording, the quality of the recording is poor. Also, any local sounds -tapping on the device or talking, or (if the device is a phone) answering a call or text message -add further local noise to the recording.

A way round the problem of poor audio quality is to use a separate recording device (the recording device) whose microphone can be placed near the speaker(s), or a loudspeaker if a public address system is in use. Then afterwards the good recording from the recording device can be combined with the time-sensitive annotations from the note-taking device, along with other available material (such as the presentation slides), for a complete set of audio-visual notes.

Because of the time-sensitive nature of the annotations, it is important that the annotations and recording be correctly synchronised. To do this the relative start time of each data stream needs to be known to the process doing the combining. The current way of ensuring this is to time-stamp each data stream (eg US patent application US 2008/0276159 Al). However, for this to work, the clocks of the recording device and the note-taking device must either be exactly synchronised, or have a known relative offset.

Time-stamping is unsatisfactory for these reasons: 1. The two devices may not both be accessible to the user, so the clock offset cannot be known.

For instance, an attendee may make annotations on their personal mobile device but obtain the audio recording from the host organisatlon.

2. Even if both capture devices are accessible to the user, clocks drift out quite quickly, so they would need to be frequently synchronised, and in practice this is likely to get overlooked.

Also, it is not uncommon for users to bypass re-setting the time and date after batteries have run out, giving totally spurious time and dates to recordings.

It would therefore be desirable to have a system which doesn't require any kind of time-stamping.

Staternetrat M lirn'erthouii The present invention proposes a method of providing an annotated recording, the method comprising receiving from an audio recording device a first recording of an event; receiving from a computing device a second recording of the event/presentation, wherein the second recording is associated with time-sensitive user annotations; using the second recording to provide a synchronisation reference such that an annotated audio stream can be generated from the first recording and the annotations associated with the second recording.

This Invention provides a method for synchronising the annotations and recording without the need for time-stamping or clock synchronisation. It makes use of the audio-recording capability of the note-taking device. Although this audio is most likely poor quality and contains local noise, and will have different start and end times to the good-quality recording from the recording device, it will usually be sufficiently similar to the good-quality recording to allow the time offset of the two audio streams to be calculated using carefully applied pattern-matching techniques.

The method allows a good-quality annotated recording to be produced with no special setup and with standard, ubiquitous devices like laptops, netbooks, mobile phones and digital recorders.

Because no time-stamp or clock synchronisation is required the devices can be owned by different people, and don't need time and date to be entered correctly on the device, or maintained accurately.

The method may mostly replace the second recording with the first recording.

The method may include both the first and second recordings in the annotated audio stream through the use of two-channels of audio.

The method may allow the user to select which parts of each recording to include in the final annotated audio stream.

The method may use an automatic process to select which of the two streams to include in the final annotated stream at any given point in the event.

The method may allow the second recording not to be contiguous but consist of two or more recordings with gaps in between.

The method may allow the second recording to be parameterised, soit cannot be played as audio but can still be used as a synchronisation reference.

The method may utilise a noise model in the synchronisation process to allow for local noise in the second recording.

The method may use time-stamps to detect that two recordings probably relate to the same event, without the user having to explicitly select both the recordings.

The method may receive a first recording which also has time-sensitive annotations, which it merges with the time-sensitive annotations associated with the second recording.

The method may process time-sensitive annotations which comprise continuous media.

The method may process time-sensitive annotations which comprise discrete media.

The method may receive a second recording with associated time-sensitive user annotations which has been generated by a computing device having audio recording capabilities.

The method may receive a second recording with associated time-sensitive user annotations which has been generated by combining the outputs of two devices using the present invention.

A computer program may execute the method.

A web service may execute the method.

Brief Description of Drawings

FIgure 1 is a schematic showing the scenario for creating an annotated recording using two devices, where the present invention relates to the Combine process 105.

Figure 2 gives examples on a time-axis of the two streams 201,206 input to the Combine process and the stream 207 output from the process. It also shows how the presentation slides 208 can be added to further enhance the final annotated recording 209.

Figure 3 shows two other ways that the two Input streams, now 301 and 302, might be combined -with either a mix of the audio 303 or both audio streams 304 ending up in the output.

FIgure 4 shows how the recording, now 401, might include video, which then becomes part of the output along with either the first 403 or the second 404 audio stream.

FIgures shows that if the recording device 103 is the presenter's computer, the presenter's slides can be captured along with their audio 501 and merged with the user's annotations 502 for the final output 503.

Figure 6 shows how video, of the screen or the presenter or both, can also be captured 601 and included in the final output 603.

Figure 7 shows how the presenter can capture their presentation on their computer 102 and combine it with a separate audio recording 101 to produce a high-quality capture 103.

Figure 8 shows that the note-taking device need not capture all the audio 802, as long as there is enough for synchronisation purposes the output 803 will be unaffected.

Figure 9 gives an overview of how the time offset between the two audio streams can be calculated.

Figure 10 gives details of how a valid correlation lag can be found.

Detailed Description

Referring to Figure 1, when attending an event, a user 102 makes time-sensitive annotations using a note-taking device 104, and in addition the event is recorded using a recording device 103. The user may be in charge of the recording device, which they place near the presenter 101, or the organisers or another attendee may be in charge of it. An advantage of the current invention is that it doesn't matter how the event is recorded.

The annotations from the note-taking device are merged with the recording from the recording device using a synchronise-and-replace process 105, which is the method of the current invention.

The two are combined to form the final annotated recording 106.

Referring to Figure 2, the recording from the recording device 201 is shown on a time-axis as a series of utterances (shaded) and pauses (solid). In the same way some example annotations from the note-taking device are also shown on a time-axis 206; highlights of important spoken phrases 203, textual notes associated with a particular point in the event 204, and markers indicating every time the presentation slide changes 205.

According to the present invention, the note-taking device also makes an audio recording 202 which is included with the annotations. When the recording 201 is combined with the annotations 206 in the synchronisation process 105, the audio 202 is used to provide a synchronisation reference for the recording 201, which is then combined with the annotations to provide a final annotated recording 207 containing the annotations 203,204, 205 and the recording 203..

In the example annotations 206, because the user has marked the slide changes, the presentation slides 208 once available can be inserted into the annotated recording at the correct place to provide a final annotated recording complete with synchronised slides 209.

The synchronisation of the recording 201 with the audio recorded by the note-taking device 202 uses carefully-applied pattern matching techniques.

Pattent-Maitthhg Tedllutques The pattern matching should take account of the possible differences between the two audio streams 201 and 202. These differences come from at least these factors: 1. Different input devices, one or both of which may have only basic-quality audio input whose intended use is only voice communication with a close-talking speaker. Different frequency responses, different amounts and types of circuit noise, different amounts of non-linear distortion are all likely.

2. Different acoustic path to the microphone. The audio 202 from the note-taking device 104 Is likely to have reverberation and frequency distortion.

3. Local noise in the audio 202 from talking, typing, tapping and mechanical noise from handling the device.

Preferably the pattern matching technique should include a noise model to explicitly take account of difference number 3.

Pattern matching can either use basic correlation, or employ more sophisticated techniques such as stochastic modelling (e.g. Hidden Markov Models) or Neural Networks to better incorporate a noIse model. Those skilled in the art will know how to apply such techniques to this problem.

S

lEffidemides Figure 3 shows that instead of recording the whole event on the note-taking device, just a part of the audio can be recorded, to save using up disk space and battery unnecessarily. Preferably at least 2 minutes of audio is recorded at least 3 times during the event, preferably around the time the user makes an annotation, as shown in 803.

Instead of storing the raw audio on the note-taking device, a parameterisation of the audio can be stored to further save on disk space.

Neither efficiency measure need affect the final annotated recording 803 in any way.

Vañatii©ns out bask tuusage Two important variations are shown In Figures 4 and 5, which show how the recording device may also have annotations, which then become merged with the annotations from the note-taking device.

The figures also show how annotations from either device may include continuous media streams such as video as well as discrete media such as text, images and event times.

In Figure 4, the recording device captures both audio and video. The audio part of the recording 401 is synchronised with the note-taking device's audio, then both audio and video are included in the final annotated recording 403. Alternatively, the roles of the two devices can be reversed so that the audio 401 is discarded after synchronisation and only the video is in included in the final annotated recording 404.

In some circumstances, the annotated recording 402 could itself be the result 106 of applying the present invention to the output of two devices. For instance, if a good-quality recorder was used for audio and a video recorder with audio capability was used for video, the audio would be combined with the user annotations first, as in Figure 2, producing an annotated recording 207 which then becomes the input annotated recording 402 of Figure 4, producing the combined audio-video annotated recording of 404.

Figure 5 shows another scenario where the recording device is the presenter's computer, and the recording 501 has the presenter's slides time-synchronised to it, and these are merged with the user annotations 502 to produce a final annotated recording complete with synchronised slides 503. As in the previous paragraph, the annotated recording 502 could itself be the result of combining another audio recording and the user's annotations using the present invention.

Figure 6 combines the variations of Figures 4 and 5 to show how in addition to the presenter's slides, or perhaps as an alternative to them, the presenters computer's screen may be captured as a video along with the audio recording 601 and the resulting video included in the final annotated recording 603. In addition to the screen capture and slide capture, a camera attached to the PC may add an additional video stream to the recording. Because all these media streams are recorded on the same computer, they can all be synchronised to the user's annotations via the audio synchronisation, as they all use the same clock.

The invention can be applied in a different configuration to improve the quality of the presenter's own capture of their presentation. In this case the presenter is also the user, and the annotations are the slides and slide transition times captured, probably automatically, on the presenter's own computer. This is demonstrated in Figure 7. The presenter has used their computer as the note-taking device to capture the slides and slide timings 702 (they could also add screen capture and video as in Figure 6), and an independent good-quality audio recorder (perhaps with a microphone clipped to their lapel) as the recording device. The good-quality recording 701 then replaces the audio from their computer to create a final high-quality capture of their presentation.

From these examples it can be seen that the invention can be applied in many different ways. In all cases an extra audio stream is recorded whose primary purpose is synchronising annotations captured on that device with a recording made on a different device.

Maidng use of the extra audfio stream Although the primary purpose of the extra audio stream is to provide synchronisation, it does not have to be discarded in its entirety.

Referring to Figure 8, which once more shows the recording 801 from the recording device and the annotations 803 from the note-taking device, parts of the audio 802 may be included in the final annotated recording where beneficial, as shown In 804. For example, if there are questions from the audience, the audio 802 from the note-taking device may capture the question more clearly than the recording 801. The places the audio 802 is used might be user-selectable or automatic (based for example on low energy in the recording 801). Another use for the audio 802 is if the recording device 103 is wireless, the recording 801 may contain drop-outs, in which case the audio 802 can used instead.

In some circumstances it may be beneficial to keep the note-taking device's audio 802 in its entirety through the use of two-channel audio 805.

There may be more than one recording device 103, each producing a recording 801. Each recording 801 in turn can be pattern-matched with the note-taking device's audio 802, and once synch ronised, the recordings combined either by selecting one at any given moment (manually or automatically) or simply by producing a multi-track recording.

Auitomatk detectfion of two recordifings Although the method is designed to work without time-stamped media, in many situations the media will be time-stamped (e.g. through "Date Modified" property of the files) and the clocks synchronised to within at least a few minutes of each other. In this case, the user can request the system to scan all uploaded media and find the recording which most likely covers the same event This saves the user having to explicitly remember which files go together.

xamplle of liuweiiflion The embodiment described here is a software program which runs on a computer or as a web service. The note-taking device is a mobile smart phone with a touch screen such as the Apple iPhone, a Blackberry or one of the many other models which run the Android operating system. The smart phone runs an application which records and allows the user to simultaneously take time-synchronised notes, and stores these in a file. The recording device is a digital recorder such as the Olympus DS-40 which can record many hours of audio, stored in a file using a compressed audio format such as Windows Media Audio (wma). However, the recording device could equally well be a mobile phone as well.

The user first of all requests the program to open the file from the note-taking device. If the audio has been stored in a playable format, the user can then listen to the audio to see if the quality is sufficient, and if it is not, they can then select a "replace audio" command. Alternatively if the audio has been stored in a parameterised format and can't be played, the user would instead select an "add audio" command. In either case, the program requests an audio file, and the user specifies the audio file from the recording device.

The program then analyses the two audio files and calculates the time offset (negative or positive) between the recording device recording and the note-taking device recording. This is described below and in Figures 9 and 10. Once this offset is known, the audio in the note-taking device file is removed and the recording device recording is added in its place at the calculated offset, and new combined audio and notes presented to the user. The user can then keep the new audio and save the file, or undo the change if for any reason they are unhappy with replacement.

This embodiment can easily be extended to any of the previously described variations and efficiencies by someone skilled in the art.

IExamjpfte ©f Syirflrnisatkn PT©cess In this embodiment we use correlation to calculate the offset.

The audio is parameterised before performing the correlation process, into energy values computed over Hamming-windowed frames l6ms in length. In this embodiment these energy values are scalar (one per frame), but increased accuracy could be obtained using a vector of energy values each of which covers a different frequency band.

Before calculating the energy values, a high-pass filter with cut-off around 300Hz is applied, so that tow-frequency noise or hum in either recording system does not affect the correlation.

We implement the local-noise model via a pre-processing step which uses an energy detector to identify local-noise frames. The threshold is set so that 15% of frames in a 2 minute window are identified as noise. This simple approach is effective in most cases, taking out the worst of the local noise if there is a lot, and not removing too much of the desired signal if there is none. Clearly the detector can be improved by someone skilled in the art by incorporating expected time and frequency characteristics of the local noise.

The time offset between the two streams is computed using the straightforward algorithms described In Figures 9 and 10. I' 8

GB201103789A 2011-03-05 2011-03-05 Creating annotated recordings from two devices using audio synchronisation Withdrawn GB2488772A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
GB201103789A GB2488772A (en) 2011-03-05 2011-03-05 Creating annotated recordings from two devices using audio synchronisation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
GB201103789A GB2488772A (en) 2011-03-05 2011-03-05 Creating annotated recordings from two devices using audio synchronisation

Publications (2)

Publication Number Publication Date
GB201103789D0 GB201103789D0 (en) 2011-04-20
GB2488772A true GB2488772A (en) 2012-09-12

Family

ID=43923267

Family Applications (1)

Application Number Title Priority Date Filing Date
GB201103789A Withdrawn GB2488772A (en) 2011-03-05 2011-03-05 Creating annotated recordings from two devices using audio synchronisation

Country Status (1)

Country Link
GB (1) GB2488772A (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5729741A (en) * 1995-04-10 1998-03-17 Golden Enterprises, Inc. System for storage and retrieval of diverse types of information obtained from different media sources which includes video, audio, and text transcriptions
EP1729173A2 (en) * 2005-05-27 2006-12-06 Telegraf ApS System for generating synchronized add-on information
US20070260457A1 (en) * 1993-03-24 2007-11-08 Engate Incorporated Audio And Video Transcription System For Manipulating Real-Time Testimony
WO2007141204A1 (en) * 2006-06-02 2007-12-13 Anoto Ab System and method for recalling media
JP2008145977A (en) * 2006-12-13 2008-06-26 Yamaha Corp Content reproducing device
US7617445B1 (en) * 2001-03-16 2009-11-10 Ftr Pty. Ltd. Log note system for digitally recorded audio

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070260457A1 (en) * 1993-03-24 2007-11-08 Engate Incorporated Audio And Video Transcription System For Manipulating Real-Time Testimony
US5729741A (en) * 1995-04-10 1998-03-17 Golden Enterprises, Inc. System for storage and retrieval of diverse types of information obtained from different media sources which includes video, audio, and text transcriptions
US7617445B1 (en) * 2001-03-16 2009-11-10 Ftr Pty. Ltd. Log note system for digitally recorded audio
EP1729173A2 (en) * 2005-05-27 2006-12-06 Telegraf ApS System for generating synchronized add-on information
WO2007141204A1 (en) * 2006-06-02 2007-12-13 Anoto Ab System and method for recalling media
JP2008145977A (en) * 2006-12-13 2008-06-26 Yamaha Corp Content reproducing device

Also Published As

Publication number Publication date
GB201103789D0 (en) 2011-04-20

Similar Documents

Publication Publication Date Title
US9344606B2 (en) System and method for compiling and playing a multi-channel video
US7428000B2 (en) System and method for distributed meetings
US6188831B1 (en) Data storage/playback device and method
KR100752568B1 (en) Event-driven annotation techniques
US7870488B2 (en) Media editing system
US20070043608A1 (en) Recorded customer interactions and training system, method and computer program product
US20100183280A1 (en) Creating a new video production by intercutting between multiple video clips
US20060075347A1 (en) Computerized notetaking system and method
Hindus et al. Ubiquitous audio: capturing spontaneous collaboration
US20070223874A1 (en) Video-Audio Synchronization
EP2628047B1 (en) Alternative audio for smartphones in a movie theater.
US20070067707A1 (en) Synchronous digital annotations of media data stream
Garofolo et al. The NIST Meeting Room Pilot Corpus.
US8548618B1 (en) Systems and methods for creating narration audio
US8385588B2 (en) Recording audio metadata for stored images
US20050182627A1 (en) Audio signal processing apparatus and audio signal processing method
KR101977072B1 (en) Method for displaying text associated with audio file and electronic device
EP2628143A2 (en) Methods and systems for capturing, processing, managing and/or evaluating multimedia content of observed persons performing a task
US8818175B2 (en) Generation of composited video programming
US8406608B2 (en) Generation of composited video programming
US8818803B2 (en) Character-based automated text summarization
US9438993B2 (en) Methods and devices to generate multiple-channel audio recordings
US20080276159A1 (en) Creating Annotated Recordings and Transcripts of Presentations Using a Mobile Device
US8463875B2 (en) Synchronized playback of media players
US7848493B2 (en) System and method for capturing media

Legal Events

Date Code Title Description
WAP Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1)