GB2352845A

GB2352845A - System for recording and retrieval of multimedia conferences

Info

Publication number: GB2352845A
Application number: GB9917684A
Authority: GB
Inventors: Andrew Howard Randall; Philip Jonathan May
Original assignee: Data Connection Ltd
Current assignee: Metaswitch Networks Ltd
Priority date: 1999-07-29
Filing date: 1999-07-29
Publication date: 2001-02-07
Also published as: GB9917684D0

Abstract

A system for recording 150 and replaying 180 the contents of multimedia conferences consisting of a plurality of media types 110 including but not limited to audio, video images, graphical output of shared computer applications, text messages exchanged between conference participants, files transferred between conference participants, graphical annotations and drawings, control and conductorship information, and other data shared between conference participants. The different media types are synchronized with each other when replaying to recreate the original conference contents. Although the recording function may be implemented as part of a conference server, the recording system is not dependent on the presence of a server system, but may be implemented as part of a peer or client system. Textual or audio labels may be saved along with recorded data streams and key frames for those streams, so that selected parts of a recorded conference may be replayed.

Description

2352845

TITLE OF THE INVENTION:

SYSTEM FOR RECORDING AND RETRIEVAL OF MULTIMEDIA CONFERENCES BACKGROUND OF THE INVENTION

Field Of The Invention

The present invention relates to the field of multimedia conferencing systems, wherein two or more users communicate in real time via a plurality of media types. These media types include but are not limited to audio, video images, graphical output of shared computer applications and whiteboards, text messages, file transfers, annotations and drawings, and control and conductorship information.

Description of Prior Art

Existing multimedia conferencing systems support "live", real-time conferencing between two or more users of the system. Such systems have proven to be extremely useful in many areas of industry and commerce. One typical use is for distributed meetings, where, for example, a team of engineers working on a single project but across multiple geographical sites can quickly and efficiently discuss and update a technical design document, with the updates being communicated in real time to all participants of the meeting.

Such conferencing systems allow users to participate in distributed meetings in a very similar manner to a face-to-face meeting. Users can see one another using video images; talk using audio, either using regular analog telephone systems or, increasingly, audio encoded and transferred over a computer data network; 2 make changes to a document; or make temporary annotations to it. However, a multimedia conference has lower costs in terms of both time and money for a distributed team than organizing a face-to-face meeting.

Furthermore, multimedia conferencing systems can be of great use in the field of distance leaming or training. A teacher or trainer can deliver a presentation on a given topic using the full power of multimedia technology: for example, using video images, audio, slides or a whiteboard. Sharing a computer application and demonstrating it is a particularly effective way to train someone to use the application. Again, the use of multimedia conferencing systems for distance learning or training has much lower costs than organizing a session where all participants must attend the same location.

It would often be desirable to be able to make a permanent recording of a multimedia conference and to be able to play the recording back at a later time. For example, for distributed meetings, it may be desirable to have a permanent record of a conference, analogous to the more traditional written minutes of a meeting. This recording has multiple uses: any decisions or actions arising from the meeting are permanently documented and available for review at any time; any person unable to attend the original meeting can later play back the recording and understand exactly what took place. Indeed, in the course of a long meeting it can be useful to replay an earlier part of the meeting to review what has been said or decided. Further comments can then be added to the recording at this point.

The ability to replay recorded material could also be extremely useful for distance learning or training. A presenter could pre-record his material, allowing for a more polished performance. This would then allow for two options: first, the presenter could replay the material to an audience and bring to it additional material where required - a "live" audio comment could be added to a presentation to explain any part of it which were not clear to the current audience, for example; second, the presentation could be played multiple times, much like a training video, without the need for the presence of the original presenter. This functionality would be particularly useful in software application training.

Currently, multimedia conferencing systems do not support the recording and replay of the full plurality of conference media streams which may exist in a conference. Specifically, those systems which do allow for recording, are closely bound to one or more specific data types, generally audio or video images (see the following US patents: 5,668,863, Bieselin et aL, Sept., 1997; 5,710,59 1, Bruno et al., Jan., 1998; 5,764, 901, Skarbo et al., June, 1998). Moreover, current multimedia conferencing systems do not allow for the mixing of pre-recorded and live media streams and do not support a wide range of indexing and retrieval functions.

3 BRIEF SUMMARY OF THE INVENTION

The present invention is a multimedia conferencing system which provides the user with the ability to record and replay conferences, in addition to participating in live conferences. When replaying a conference, the user may request replay from a significant point in a conference, possibly from a point marked by a textual or audio label or possibly from a randomly selected point.

The media streams which the invention supports include but are not limited to audio, video images, graphical output of share applications and whiteboards, annotations, text messages and file transfers. The invention does not rely on a server-client architecture. A given embodiment may be implemented as a peerto-peer system or as part of a conference server system.

The invention is intended to be flexible but easy to use. In the preferred embodiment, it presents a simple user interface, partitioned into several areas. One area contains controls for regulating the recording and replay of the media streams which constitute a conference. Other areas display the data from a given media stream. Audio data is heard through a loudspeaker. The overall effect is a unified multimedia conference session.

The present invention has the following advantages over systems in the prior art:

The present invention allows the user to simultaneously view live and recorded media streams. For example, this allows live conference participants to review earlier parts of the conference.

The present invention supports the mixing of live media streams with the media streams regenerated during replay of a recorded conference. For example, this allows a presenter to add a "voice over" to a recorded presentation.

The present invention supports the recording of a multiplicity of media types. This allows a recorded conference to fully represent the ori ginal conference experienced by live participants.

The present invention supports recording with the media streams of detailed, automatically generated indexing information for ease of subsequent retrieval and playback.

4 BRIEF DESCRIPTION OF THE DRAWINGS

FIG. I is a simplified block diagram of a multimedia conferencing system which is an embodiment of this invention.

FIG. 2 is a simplified example user interface for a multimedia conferencing system which is an embodiment of this invention.

FIG. 3 is a simplified block diagram of message flows during the recording of a multimedia conference by a conferencing system which is an embodiment of this invention.

FIG. 4 is a simplified block diagram of message flows during the replay of a multimedia conference by a conferencing system which is an embodiment of this invention.

Reference Numerals in the Drawings System Components:

110: Multimedia Conference Source 120: Multimedia Stream Mixer 130: Real-Time Conference Player 140: Ul Conference Controller 150: Conference Recording Controller 160: Conference Archiver 170: Conference Archive 180: Conference Playback Controller 190: Recorded Conference Player 210: Ul Application Sharing Area 220: Ul Annotations Area 230: UI Text Char Area 240: Ul Recording/Replay Controls 250: Ul Replay Position Slider 260: Ul Replay Label Selector Message Flows:

310: Ul Conference Controller sends Record message to Conference Recording Controller 320: Conference Recording Controller sends Record message to Conference Arcbiver 330: Conference Arebiver registers with Multimedia Stream Mixer to receive live multimedia conference stream data 340: Multimedia Stream Mixer sends data to Conference Archiver 350: Conference Arcbiver writes data to Conference Arebive on a permanent storage medium 410: UI Conference Controller sends Replay message to Conference Playback Controller 420: Conference Playback Controller retrieves conference data from Conference Archive 430: Conference Playback Controller sends data to Multimedia Stream Mixer 440: Multimedia Stream Mixer sends data to Recorded Conference Player DETAILED DESCRIPTION OF THE INVENTION

This description covers a method for the recording and retrieval of multimedia conferences. In the description, there are set forth, for purposes of explanation, many specific details, to provide the reader with a thorough understanding of the invention. It will be obvious to one skilled in the art that the invention may be practiced without these details. Furthermore, certain devices and orders of processing are given in the diagrams and the description, to make understanding the invention easier. It will, again, be apparent to one skilled in the art that the specific devices and sequences are merely illustrative and may be varied whilst remaining within the spirit and scope of the present invention.

Description of the Preferred Embodiment

What follows is a description of the preferred embodiment of the present invention. Other possible embodiments will be discussed later.

6 Recording a Multimedia Conference Referring to FIG. 1, the preferred embodiment of this invention receives media data streams from one or more external Multimedia Conference Sources 110. The Multimedia Conference Sources I 10 may take one or more forms: other multimedia conferencing systems (including but not limited to embodiments of the present invention) or any other device capable of generating a media stream, for example an analog audio conferencing system.

These media streams are processed by the Multimedia Stream Mixer 120 and passed to one or more instances of the Real-Time Conference Player 130 for "display" on the User Interface (FIG. 2), where "display" may or may not involve visual output (audio will pass to the system's loudspeaker). Referring to FIG. 2, an example User Interface, application sharing data is displayed in a dedicated area 2 10; annotations are displayed in another area 220 and also over the area 2 10 which they annotate; text messages are displayed in a third area 230.

Recording of a conference session is initiated by the User Interface (FIG. 2). The user presses the record button on the UI Recording/Replay Controls (240 in FIG. 2). Referring to FIG. 1, the Ul Conference Controller 140 then sends a message to the Conference Recording Controller 150 to initiate recording (message 3 10 in FIG. 3).

The Conference Recording Controller 150 informs the Conference Archiver 160 that it should begin recording the conference media streams (message 320 in FIG. 3). The Conference Archiver sends a request to the Multimedia Stream Mixer 120 to receive the same data from the Multimedia Stream Mixer 120 as the Real-Time Conference Player 130 (message 330 in FIG. 3). The Conference Archiver 160 then prepares the media streams for storage as files and sends this data to the Conference Archive 170 for storage in memory, being any of a permanent or temporary physical memory block, magnetic disk, tape, writable optical disk or other permanent or temporary storage medium (message 350 in FIG. 3).

While the Conference Archiver is preparing media streams for storage in the Conference Archive, it periodically generates "key ftames" for the archive. Such a key frame is a special set of data items, such that the information included in the key frame for a given media stream is sufficient that when replay begins from that key frame, the reconstituted media stream will provide a complete and correct view of the data.

As an example, for the graphical output of shared applications, such a key frame contains a full color palette, a complete representation of the bitmap images and drawing orders currently applicable, the position and shape of the mouse pointer and so forth.

For some media streams a key frame may indicate that no action is necessary and that further data can be displayed immediately: text messages fall into this category.

7 The preferred embodiment of the invention stores combined key frames which contain key frame information for all active media streams.

The recording can be stopped or paused at any time. The user presses the stop or pause button on the UI Recording/Replay Controls (240 in FIG. 2). Referring again to FIG. 1, the UI Conference Controller 140 sends a message (3 10 in FIG. 3) to the Conference Recording Controller 150 to stop or pause recording.

The Conference Recording Controller 150 informs the Conference Archiver 160 that it should stop recording the conference media streams. The Conference Archiver 160 sends a request to the Multimedia Stream Mixer 120 to stop receiving the media streams.

In the case of the preferred embodiment, once the recording is complete, indexing information is added to the conference archive files in the Conference Archive 170. This information facilitates later retrieval of the archive for replay and includes, in the preferred embodiment of the invention: the conference name or identifying number; the archive file name; the archive file size; the date and starting time of the conference; which media or data stream types were recorded; the duration of the conference; the number of conference participants; the names and contact information of conference participants; the names and contact information of conference conductors and/or presenters.

Replaying a Recorded Conference Once the media streams which constitute a multimedia conference have been recorded, the conference can be replayed on demand by any user of the conferencing system.

When replaying a conference the following scenarios are possible: to replay a recording of (an earlier part of) a real-time conference while this conference is still in progress; to replay a recording of an archived conference during another real-time conference; to replay an archived conference for a single user to view; to schedule a session where a plurality of users view the broadcast replay of an archive conference but no real-time conference session otherwise exists.

Before initiating replay, in all but the first scenario, the user must specify which archived conference is required. The preferred embodiment of the invention offers the user a list of the available archived 8 conferences, displaying the following information: the archive file name, the conference name or identifying number and the conference start time and date.

Once the user has selected the conference to be replayed or wishes to replay an earlier part of the current real-time conference, replay is initiated either by pressing the play button on the Ul Recording/Replay Controls (240 in FIG. 2) or automatically upon selection of a prerecorded conference, depending on user preference. Referring now to FIG. 1, the Ul Conference Controller 130 sends a request to the Conference Playback Controller 180 to begin replay of the archived conference media streams (message 410 in FIG. 4). The Conference Playback Controller 180 retrieves the data from the Conference Archive 170 (message 420 in FIG. 4) and feeds it into the Multimedia Stream Mixer 120 (message 430 in FIG. 4). The Multimedia Stream Mixer 120 sends the media streams to the Recorder Conference Player 190 on the User Interface (message 440 in FIG. 4).

The replay of the conference can be paused or stopped at any time. This is achieved by the user selecting the pause or stop buttons on the Ul Recording/Replay Controls 240. The Ul Conference Controller 140 sends a message to the Conference Playback Controller 180 (message 410 in FIG. 4) which then ceases to retrieve data from the Conference Archive 170. Replay can be reinitiated later upon demand.

The preferred embodiment of the invention allows the user random access playback using the Replay Position Slider 250. The user may select a random point in the conference using the Replay Position Slider 250 and replay will continue from the newly selected position. The User Interface (FIG 2.) sends a message to the Conference Playback Controller 180 to inform it that replay should now continue from the specified position.

The Conference Playback Controller 180 ceases to retrieve data from the Conference Archive 170 and requests from the Archive the position of the last key frame before the specified position, for each media stream. If there is no key frame before the specified position, replay continues from the beginning of the conference.

Since the preferred embodiment of the invention stores the key frame data for different media streams as a single combined key frame, the Conference Playback Controller 180 then begins to retrieve data for all streams from the Conference Arebive 170 once more, starting from the key frame. It feeds this data to the Multimedia Stream Mixer 120 as described above.

9 The Use of Textual and Audio Labels In the preferred embodiment of the invention, significant events during the course of a multimedia conference are marked in the Conference Archive 170 with a textual or audio label. These labels may be generated by the user in addition to labels generated automatically by the conferencing system. In both cases the information is entered into the Conference Archive 170 along with a key frame. Where the labels are generated by the conferencing system they are based on a change in the title of a shared application, with the information saved being the new application title; a textual message being sent by a conference participant, with the information saved being some or all of the message content; a change in conductorship status or control of the conference or some aspect of the conference, with the information saved being the name of the conference participant taking or losing control; a change in the identity of the conference participant currently identified as the primary speaker within the audio portion of the conference, with the information saved being the name of the conference participant taking over or relinquishing the role of primary speaker; one or more files being transferred, with the information saved being the name of the files or the name of the conference participant who sent the file; the addition, modification or deletion of a graphical annotation, with the information saved being a textual description of the graphical annotation; and the addition, modification or deletion of a textual annotation, with the infon-nation saved being the textual annotation.

During replay the user can choose to continue replay from a particular label. In the preferred embodiment, there is an area on the User Interface, the Replay Label Selector 260, where textual labels are listed; the user instructs the system to continue replay from a label by clicking on it.

Referring to FIG. 1, when the user instructs the system to continue replay from a label, the UI Conference Controller 140 sends a message to the Conference Playback Controller 180 informing it that replay should continue from the label. The Conference Playback Controller 180 stops retrieving data from the Conference Archive 170 and requests the position in the archive of the label. It then begins to retrieve data from the Conference Archive 170 and passes it to the Multimedia Stream Mixer 120. Replay continues as before (see above).

Mixing of Live and Recorded Conference Media Streams The preferred embodiment of the present invention allows the user to simultaneously view real-time and recorded multimedia conference data streams.

All real-time media inputs for a given media stream are mixed at the Multimedia Stream Mixer 120 and passed to the Real-Time Conference Player 130 in the User Interface. Recorded media inputs are passed by the Multimedia Stream Mixer 120 to the Recorded Conference Player 190 in the User Interface. While recorded and real-time media streams are presented to the user separately on the User Interface, both can be viewed simultaneously.

Description of Alternative Embodiments

What follows is a description of some alternative embodiments of the present invention. One skilled in the art will easily be able to envisage other alternative embodiments of the invention.

As set out in the Summary of the Invention, a given embodiment of the invention may perform the methods described here in different ways: they may be performed by the server of a conference serving system (with or without client control), by the client of such a system or by a peer-topeer system. In the first case, data transfer between the User Interface components (in FIG. 1, the Real-Time Conference Player 130, the Recorded Conference Player 190 and the Ul Conference Controller 140) and other components of the system will occur across a network; in the later two cases the data transfer will be internal to the client or peer system.

In the case that the methods are performed by a conference server without client control, the methods of starting and stopping recording and replay are similar to the methods outlined in the description of the preferred embodiment above. However, these methods are triggered not by user instructions via the User Interface but by the system itself.

Key Frames, Indexing and Labels Some alternative embodiments of the invention may choose not to store a single, combined key frame for all media streams, but to store a separate key frame for each media stream.

If an embodiment of the invention stores the key frame data for different media streams separately rather than in a single key frame, then, when replay from a key frame occurs, the Conference Playback Controller 180 will begin retrieving data from the Conference Archive 170, starting at the earliest of the key frames whose position the Conference Archive 170 has just supplied. The Conference Playback Controller 180 then passes data for any streams for which it has so far seen a key frame to the Multimedia Stream Mixer 120 as described in the description of the preferred embodiment above, but discards data for all other streams. As the Conference Playback Controller 180 progresses through the archived data, it will reach a key frame for each media stream, until all streams are being replayed.

In the case of the preferred embodiment, indexing information is added once the recording is complete. In the case of other embodiments of the invention, this indexing information may be added at an earlier time.

A given embodiment of the invention may use a different mechanism for generating labels to that used in the preferred embodiment; or even choose not to use labeling at all. A given embodiment may also offer the user a different User Interface mechanism for selecting replay from a labeled position.

Real-Time and Recorded Conference Players In some embodiments of the invention, the Real-Time Conference Player 130 and the Recorder Conference Player 190 may be identical and "display" data for real-time and recorded streams in the same location. It is also possible that they exist as separate components but cooperate to "display" data of one or more types in the same location. For example, one embodiment might mix live and recorded audio data, so that a voiceover can be added to a pre-recorded conference, even if that conference contains an audio stream.

In the case that the Real-Time Conference Player 130 and the Recorded Conference Player 190 are identical in a given embodiment of the invention, the Multimedia Stream Mixer 120 would mix recorded media stream with the real-time media streams. Both types of stream would be presented to the user as a unified stream.

In the case that the Real-Time Conference Player 130 and Recorded Conference Player 190 are separate components in a given embodiment of the invention, they might still collaborate to mix recorded and real-time data for one or more media types. In this case, this mixing may in fact, in one embodiment, be performed by the Multimedia Stream Mixer 120 and the data passed to only one of the two Conference Players 130 and 190 (for a given stream); or, in a different embodiment, the Multimedia Stream Mixer may keep the recorded and real-time streams separate and the Conference Players 130 and 190 may mix it themselves.

12 Conclusion

In summary, the present invention is a multimedia conferencing system which, in addition to the facilities offered by existing systems, offers the following additional capabilities: recording of a live multimedia conference into a permanent archive, replaying of an archived conference, either for one user or as a broadcast for multiple users, simultaneous replaying of an arcbived conference and participation in a live conference, marking of salient points in the conference archive with audio or textual labels, either manually or automatically, for use in selective replaying of the conference.

It should be noted that the section titled "Detailed Description" is simply a description of the preferred and some additional embodiments of the invention; the full scope of the invention is determined by the claims. A systems engineer skilled in the art of developing multimedia conferencing systems would, based on the claims and the detailed description, be able to implement a wide variety of multimedia conference recording systems significantly advanced compared with the current state of the art.

13 CLAMS

Claims

We claim:

1. A method for recording a conference consisting of one or more media or data streams, said method comprising the steps of a. capturing said media or data streams or a subset of said media or data streams, and b. storing the captured media or data streams to a memory, said memory being able to store a plurality of data files.

2. The method as set forth in claim 1, with a further step of storing the recorded conference in a permanent archive upon completion of the conference.

3. The method as set forth in claim 2, with a further step of adding information about the recorded conference to an index to facilitate later retrieval.

4. The method as set forth in claim 2, further comprising the step of replaying the recorded conference, said method comprising the steps of a. accepting a user's request to retrieve recorded conference data, b. reading recorded conference data files from a memory, c. regenerating the saved media or data streams from recorded conference data files, and d. replaying the regenerated media or data streams to the requesting user.

5. The method as set forth in claim 2, further comprising the step of broadcasting the recorded conference, said step comprising the steps of a. scheduling a broadcast session which a plurality of users can attend, b. reading recorded conference data files from a memory, c. regenerating the saved media or data streams from recorded conference data files, and d. broadcasting the regenerated media or data streams to all attendees of the broadcast session.

6. The method asset forth in claim 4, where one or more of the data streams comprises an application sharing session consisting of one or more of the following data items:

14 a. graphical window updates consisting of bitmap images with corresponding position information, b. graphical window updates consisting of drawing orders including lines, rectangles, text and ellipses with associated properties such as color and thickness, c. window title information, d. shape of the cursor or pointer, e. position of the cursor or pointer, and f. details of which conference participant has control.

7. The method as set forth in claim 6, with the additional feature that at a plurality of intermediate points within the sequence of data items a special set of data items known as a "key frame" is stored such that the data stream including and following the start of the key frame provides a complete and correct view of the recorded application sharing session starting at the key frame without reference to any data items preceding the key frame.

S. The method as set forth in claim 7, further comprising the step of playing back the recorded application sharing session where a. a user is presented with an interface permitting random access to any point in the application sharing session, and b. the application sharing session is replayed by starting at the latest key frame preceding the userselected point, or the start of the session if there is no such key frame.

9. The method as set forth in claim 4, further comprising the step that during recording one or more conference participants or other users of the system can associate a textual or audio label with a particular point in time within the conference, said label being saved along with the recorded media or data streams and a key frame for these data streams.

10. The method as set forth in claim 9, further comprising the step that when replaying the conference a user of the system can, by selecting a textual or audio label, start replaying the recorded conference from the point at which the selected label was applied.

11. The method as set forth in claim 4, further comprising the step that during recording textual or audio labels are saved along with the recorded media or data streams, said labels being automatically generated by significant events within the conference or at the request of the user and containing associated information about those significant events.

12. The method as set forth in claim 11, further comprising the step that when replaying the conference a user of the system can, by selecting a textual or audio label, start replaying the recorded conference from the point at which the selected label was applied.

13. The method as set forth in claim 4, where one or more of the data streams comprises a graphical annotation ("wbiteboard") session consisting of one or more of the following data items:

a. textual annotations, with associated position, typeface, size, style and color, b. geometrical objects with associated line width, position, style and color, and c. bitmap objects with associated color palette and position.

14. The method as set forth in claim 13 with the further step that the recorded whiteboard session is associated with a recorded application sharing session such that when replayed the recorded graphical annotations appear overlaid on the recorded shared applications.

15. The method as set forth in claim 13, with the additional feature that at a plurality of intermediate points within the sequence of data items a special set of data items known as a "key frame" is stored such that the data stream including and following the start of a key frame provides a complete and correct view of the recorded whiteboard session starting at the key frame without reference to any data items preceding the key frame.

16. The method as set forth in claim 15, further comprising the step of playing back the recorded whiteboard session where a. a user is presented with an interface permitting random access to any point in the whiteboard session, and b. the whiteboard session is replayed by starting at the latest key frame preceding the user-selected point, or the start of the session if there is no such key frame.

17. The method as set forth in claim 1, where one or more of the data streams comprises a series of textual messages exchanged between conference participants, each message being saved with some or all of the following information:

a. contents of the text message, b. name of the conference participant who sent the message, c. name of the conference participant or participants to whom the message was sent, and 16 d. date and time that the message was sent.

18. The method as set forth in claim 5, further comprising the step of broadcasting one or more recorded data or media streams during a conference such that the recorded data or media streams are mixed with Or displayed together with one or more data or media streams being generated by one or more conference participants.

19. The method as set forth in claim 1, where the recording function is performed at a conference server under the control of a client of the conference server system.

20. The method as set forth in claim 1, where the recording function is performed at a conference server automatically, that is, independently of any control input from a client of the system.

21. The method as set forth in claim 1, where the recording function is performed at a peer system or a client of a conference server system at the request of a user of the peer or client system, said method enabling each conference participant to record a local conference archive independent of any other conference participant's recording mechanism.