CA2271745A1 - Method and apparatus for storing and retrieving labeled interval data for multimedia recordings - Google Patents
Method and apparatus for storing and retrieving labeled interval data for multimedia recordings Download PDFInfo
- Publication number
- CA2271745A1 CA2271745A1 CA002271745A CA2271745A CA2271745A1 CA 2271745 A1 CA2271745 A1 CA 2271745A1 CA 002271745 A CA002271745 A CA 002271745A CA 2271745 A CA2271745 A CA 2271745A CA 2271745 A1 CA2271745 A1 CA 2271745A1
- Authority
- CA
- Canada
- Prior art keywords
- interval
- intervals
- interval data
- labeled
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M3/00—Automatic or semi-automatic exchanges
- H04M3/42—Systems providing special services or facilities to subscribers
- H04M3/56—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities
- H04M3/567—Multimedia conference systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M3/00—Automatic or semi-automatic exchanges
- H04M3/42—Systems providing special services or facilities to subscribers
- H04M3/42221—Conversation recording systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M7/00—Arrangements for interconnection between switching centres
- H04M7/12—Arrangements for interconnection between switching centres for working between exchanges having different types of switching equipment, e.g. power-driven and step by step or decimal and non-decimal
Abstract
A teleconference system (200) is disclosed for digitally recording and playing a conference telephone call that includes a plurality of intervals. The teleconference system includes a skim server (55) that detects a first set of the plurality of intervals and a conference bridge (100) that detects a second set of the plurality of intervals during the conference call. An interval database server (65) generates labeled interval data for all detected intervals and stores the labeled interval data in a database. The labeled interval data includes an interval data element that defines each interval. After the conference call is recorded, the labeled interval data can be searched and retrieved based on assorted criteria. Portions of the recorded conference call associated with the retrieved labeled interval data can also be retrieved and played back. This facilitates easy retrieval and playback of desired portions or a recorded conference call. Further, during playback of the conference call (85), a user interface is generated. The user interface displays the stored labeled interval data. A user can easily select or skip to desired portions of the conference call by selecting portions of the user interface.
Description
METHOD AND APPARATUS FOR STORING AND RETRIEVING
LABELED INTERVAL DATA FOR MULTIMEDIA RECORDINGS
s ~o FIELD OF THE INVENTION
The present invention is directed to storage and retrieval of multimedia data. More particularly, the invention is directed to storage and retrieval of labeled interval data in a database.
Unlike records of written communications, records of speech communication are rarely recorded, let alone stored, even though storage of digital speech may be readily achieved. It is presently feasible to store gigabytes and even terabytes of digitally recorded speech or other types of 2o multimedia information (e.g., video). Other than for archival purposes, there is no practical reason for storing such data without having a mechanism by which a user can identify and retrieve only those portions of the stored data, which may be of interest.
The difficulty inherent in searching and retrieving digital speech 25 records stored in a database stems from the traditional approaches to querying a database to locate particular records. Most database queries are logical queries based upon the presence or absence of specified characteristics in the records being searched. Boolean logic and fuzzy logic have been used to increase the utility of database queries, but these techniques merely extend the fundamental basis of most typical database queries, whether one or more terms, indices, or other identifying characteristics are present (or absent) in the records being searched.
Digital speech records, without being converted into text by speech-to-text conversion or transcription or otherwise parsed cannot be located and/or identified using traditional database query techniques as it is not practical to determine whether a word (or phrase) appears in a selected portion of recorded speech. Therefore, review of non-transcribed digital speech records is frequently limited to listening to the digitally recorded to speech until the item or items of interest are heard. Unfortunately, this frequently requires listening to a considerable degree of extraneous or irrelevant speech which can be extremely time-consuming without providing any significant elucidation. Moreover, digital speech records frequently contain lengthy pauses and, if the digital speech record is between more than two speakers, it is frequently difficult, if not impossible, to identify the speakers, further exacerbating the problem of identifying a specific segment in recorded digital speech.
Even when a digital speech record is divided into separate digital recordings, and each recording is individually accessible and identified, the digitally recorded data is of limited use. For example, if ten conference calls were recorded in a digital storage medium, a user might be able to locate a particular conference call on a particular date, if the user were fortunate enough to know that the information he or she sought was in that specific conference call. Even, so, the user would still have to listen to the entire recording of the conference call. For a user seeking to identify a specific comment made by a specific participant to the conference call, it is extremely inefficient for the user to have to listen to the entire conference call. Moreover, if the user does not know the specific date and time of the conference call in which the person spoke, the user might have to listen to several conference call recordings before finding the desired information.
Clearly, as soon as a greater than minimal number of recordings were stored, it becomes impractical for a user to locate desired information merely by listening to the conference call recordings.
Based on the foregoing, there is a need for a method and apparatus for readily identifying, locating, and retrieving stored digital speech and other digital multimedia records.
SUMMARY OF THE INVENTION
One embodiment of the present invention is a teleconference system for digitally recording and playing a conference telephone call that includes a plurality of intervals. The teleconference system includes a skim server that detects a first set of the plurality of intervals and a conference bridge that detects a second set of the plurality of intervals during the conference call. An interval database server generates labeled interval data for a11 detected intervals and stores the labeled interval data in a database. The labeled interval data includes an interval data element that defines each 2o interval. After the conference call is recorded, the labeled interval data can be searched and retrieved based on assorted criteria. Portions of the recorded conference call associated with the retrieved labeled interval data can also be retrieved and played back. This facilitates easy retrieval and playback of desired portions of a recorded conference call.
Further, during playback of the conference call, a user interface is generated. The user interface displays the stored labeled interval data. A
._ 4 user can easily select or skip to desired portions of the conference call by selecting portions of the user interface.
BRIEF DESCRIPTION OF THE DRAWINGS
Fig. 1 illustrates a teleconference system in accordance with one embodiment of the present invention.
Fig. 2 illustrates the format of an interval data element that forms the labeled interval data associated with a recorded conference.
Fig. 3 illustrates a conference playback document in accordance to with one embodiment of the present invention.
Fig. 4 illustrates in detail how overlapping intervals are displayed.
DETAILED DESCRIPTION
In one embodiment of the present invention, intervals within recorded digital speech or other multimedia data are specifically identified and labeled. The labeled interval data provides a mechanism by which a user can specifically identify an interval within digitally recorded multimedia, and having identified that interval, retrieve it and other 2o intervals sharing desired characteristics.
Fig. 1 illustrates a teleconference system in accordance with one embodiment of the present invention. Teleconference system 200 records and stores a teleconference call and associated labeled interval data.
Teleconference system 200 further allows a recorded teleconference to be played back using the stored labeled interval data.
The main components of teleconference system 200 are a conference recorder 110, a skim server S5, an interval database ("IDB") server 65, and a Java user interface 85.
In teleconference system 200, a plurality of telephones 31, 32, and 5 33 are interconnected through the public switched telephone network "PSTN") 40. One or more individuals may participate in a teleconference through each telephone 31-33. The participants may be identified by the telephone they are calling from or, alternatively, by voice recognition or other forms of identification during the teleconference.
t o A teleconference may be initiated by a conference host accessing a WebRoom interface on a WebRooms server 50. A WebRoom interface provides a mechanism by which participants may be actively added to and/or deleted from a teleconference. In one embodiment, the WebRoom interface for a11 teleconference participants is implemented as Common i 5 Gateway Interface ("CGI") program 60 on an HyperText Transport Protocol Web Server ("Httpd") 70 that provides interactive control of the teleconference through Hyper-Text Markup Language ("HTML") documents. The HTML documents are accessible as conference pages 80 through a Web browser 90 such as Netscape~ Navigator or Internet 20 Explorer~.
At record time, the conference host uses WebRooms server 50 to dial a conference scribe. The conference scribe acts as an additional participant to the teleconference. At the same time, conference recorder 110 tells IDB Server 65 to create a new collection point, referred to as a 25 "depot" for storing a11 data related to this particular recording, and it tells skim server 55 to begin recording an audio file using, for example, a Dialogic board 57 from Dialogic Corp., or its equivalent. A depot in teleconference system 200 can be a structured query language ("SQL") database 35 coupled to an Open DataBase Connectivity ("ODBC") interface 36. While the conference is running, conference bridges 100 detects call control events (e.g., which participant is talking, new participants being added, etc. ) and sends these events through WebRooms server SO and conference recorder 110 into the new depot (i.e., SQL
database 35). Meanwhile, skim server 55 detects pauses in speech and adds these events as well to the depot. The events detected by both conference br idges 100 and skim server 65 are referred to as "intervals" .
1 o When playing back a recorded conference on teleconference system 200, the user brings up a Java user interface 85 to select a recording accessed via IDB server 65. The user interface 85 retrieves labeled interval data for the recording and uses them to display a visual time-line of events. The user enters a phone number that is passed to Skim Server t 5 55 so it can call the user's telephone for conference playback through Dialogic board 57. As the audio plays on the user's phone, Java user interface 85 continuously updates the graphical display and controls how the recording is played using skim server 55. All clients like Java user interface 85 and conference recorder 110 communicate with skim server 55 2o and IDB server 65 through a CORBA application programming interface in one embodiment of the present invention. CORBA was chosen because it allows a simple interface between programs written in different languages running on different platforms. Both servers 50 and 55 and conference recorder 110 are written in C + + and run on Sun Solaris platforms in one 25 embodiment of the present invention.
Skim server 55 performs the following functions:
1. Records audio from telephone line to file.
WO 99/l7235 PCT1US98/20446 2. Detects speech events while recording and posts them to the database.
3. Plays from file to telephone line - from any point in recording - in variable speeds - with pauses removed or not.
In one embodiment, skim server 55 is based on the same type of hardware as standard voice mail servers, and it performs many of the same functions. One difference between skim server 55 and a more traditional 1 o voice mail server is that it processes speech events and posts them to IDB
server 65, and also that it provides fine control over what parts of the audio file are played and what parts are skipped.
One function of IDB server 65 is to store and retrieve labeled interval data associated with a recorded conference. This is data that describes properties about specific intervals within the speech, such as who is talking, pauses in speech, telephone call control data, etc. This can be further extended to applications that require intervals that mark video scene changes, or relate automatic speech recognition output to a recording. The labeled interval data can be created, stored, and retrieved by a number of 2o different applications . Some are automatically derived from raw speech data, some are side effects of user activity, and others may be entered manually at record time or at playtime.
Fig. 2 illustrates the format of an interval data element 130 that forms the labeled interval data associated with a recorded conference.
Every interval during the recorded conference will be associated with an interval data element 130. In one embodiment, each interval data element 130 includes the following:
1. Recording ID or Depot 122: Refers to the recording that is associated with the interval and the collection point where the recording is stored.
2. Start time 123: Applications need both absolute time and time relative to recording start time. Relative time is more compact, and it is easy to convert to absolute as long as an absolute start time is stored with the recording.
3. Duration or end time 124.
LABELED INTERVAL DATA FOR MULTIMEDIA RECORDINGS
s ~o FIELD OF THE INVENTION
The present invention is directed to storage and retrieval of multimedia data. More particularly, the invention is directed to storage and retrieval of labeled interval data in a database.
Unlike records of written communications, records of speech communication are rarely recorded, let alone stored, even though storage of digital speech may be readily achieved. It is presently feasible to store gigabytes and even terabytes of digitally recorded speech or other types of 2o multimedia information (e.g., video). Other than for archival purposes, there is no practical reason for storing such data without having a mechanism by which a user can identify and retrieve only those portions of the stored data, which may be of interest.
The difficulty inherent in searching and retrieving digital speech 25 records stored in a database stems from the traditional approaches to querying a database to locate particular records. Most database queries are logical queries based upon the presence or absence of specified characteristics in the records being searched. Boolean logic and fuzzy logic have been used to increase the utility of database queries, but these techniques merely extend the fundamental basis of most typical database queries, whether one or more terms, indices, or other identifying characteristics are present (or absent) in the records being searched.
Digital speech records, without being converted into text by speech-to-text conversion or transcription or otherwise parsed cannot be located and/or identified using traditional database query techniques as it is not practical to determine whether a word (or phrase) appears in a selected portion of recorded speech. Therefore, review of non-transcribed digital speech records is frequently limited to listening to the digitally recorded to speech until the item or items of interest are heard. Unfortunately, this frequently requires listening to a considerable degree of extraneous or irrelevant speech which can be extremely time-consuming without providing any significant elucidation. Moreover, digital speech records frequently contain lengthy pauses and, if the digital speech record is between more than two speakers, it is frequently difficult, if not impossible, to identify the speakers, further exacerbating the problem of identifying a specific segment in recorded digital speech.
Even when a digital speech record is divided into separate digital recordings, and each recording is individually accessible and identified, the digitally recorded data is of limited use. For example, if ten conference calls were recorded in a digital storage medium, a user might be able to locate a particular conference call on a particular date, if the user were fortunate enough to know that the information he or she sought was in that specific conference call. Even, so, the user would still have to listen to the entire recording of the conference call. For a user seeking to identify a specific comment made by a specific participant to the conference call, it is extremely inefficient for the user to have to listen to the entire conference call. Moreover, if the user does not know the specific date and time of the conference call in which the person spoke, the user might have to listen to several conference call recordings before finding the desired information.
Clearly, as soon as a greater than minimal number of recordings were stored, it becomes impractical for a user to locate desired information merely by listening to the conference call recordings.
Based on the foregoing, there is a need for a method and apparatus for readily identifying, locating, and retrieving stored digital speech and other digital multimedia records.
SUMMARY OF THE INVENTION
One embodiment of the present invention is a teleconference system for digitally recording and playing a conference telephone call that includes a plurality of intervals. The teleconference system includes a skim server that detects a first set of the plurality of intervals and a conference bridge that detects a second set of the plurality of intervals during the conference call. An interval database server generates labeled interval data for a11 detected intervals and stores the labeled interval data in a database. The labeled interval data includes an interval data element that defines each 2o interval. After the conference call is recorded, the labeled interval data can be searched and retrieved based on assorted criteria. Portions of the recorded conference call associated with the retrieved labeled interval data can also be retrieved and played back. This facilitates easy retrieval and playback of desired portions of a recorded conference call.
Further, during playback of the conference call, a user interface is generated. The user interface displays the stored labeled interval data. A
._ 4 user can easily select or skip to desired portions of the conference call by selecting portions of the user interface.
BRIEF DESCRIPTION OF THE DRAWINGS
Fig. 1 illustrates a teleconference system in accordance with one embodiment of the present invention.
Fig. 2 illustrates the format of an interval data element that forms the labeled interval data associated with a recorded conference.
Fig. 3 illustrates a conference playback document in accordance to with one embodiment of the present invention.
Fig. 4 illustrates in detail how overlapping intervals are displayed.
DETAILED DESCRIPTION
In one embodiment of the present invention, intervals within recorded digital speech or other multimedia data are specifically identified and labeled. The labeled interval data provides a mechanism by which a user can specifically identify an interval within digitally recorded multimedia, and having identified that interval, retrieve it and other 2o intervals sharing desired characteristics.
Fig. 1 illustrates a teleconference system in accordance with one embodiment of the present invention. Teleconference system 200 records and stores a teleconference call and associated labeled interval data.
Teleconference system 200 further allows a recorded teleconference to be played back using the stored labeled interval data.
The main components of teleconference system 200 are a conference recorder 110, a skim server S5, an interval database ("IDB") server 65, and a Java user interface 85.
In teleconference system 200, a plurality of telephones 31, 32, and 5 33 are interconnected through the public switched telephone network "PSTN") 40. One or more individuals may participate in a teleconference through each telephone 31-33. The participants may be identified by the telephone they are calling from or, alternatively, by voice recognition or other forms of identification during the teleconference.
t o A teleconference may be initiated by a conference host accessing a WebRoom interface on a WebRooms server 50. A WebRoom interface provides a mechanism by which participants may be actively added to and/or deleted from a teleconference. In one embodiment, the WebRoom interface for a11 teleconference participants is implemented as Common i 5 Gateway Interface ("CGI") program 60 on an HyperText Transport Protocol Web Server ("Httpd") 70 that provides interactive control of the teleconference through Hyper-Text Markup Language ("HTML") documents. The HTML documents are accessible as conference pages 80 through a Web browser 90 such as Netscape~ Navigator or Internet 20 Explorer~.
At record time, the conference host uses WebRooms server 50 to dial a conference scribe. The conference scribe acts as an additional participant to the teleconference. At the same time, conference recorder 110 tells IDB Server 65 to create a new collection point, referred to as a 25 "depot" for storing a11 data related to this particular recording, and it tells skim server 55 to begin recording an audio file using, for example, a Dialogic board 57 from Dialogic Corp., or its equivalent. A depot in teleconference system 200 can be a structured query language ("SQL") database 35 coupled to an Open DataBase Connectivity ("ODBC") interface 36. While the conference is running, conference bridges 100 detects call control events (e.g., which participant is talking, new participants being added, etc. ) and sends these events through WebRooms server SO and conference recorder 110 into the new depot (i.e., SQL
database 35). Meanwhile, skim server 55 detects pauses in speech and adds these events as well to the depot. The events detected by both conference br idges 100 and skim server 65 are referred to as "intervals" .
1 o When playing back a recorded conference on teleconference system 200, the user brings up a Java user interface 85 to select a recording accessed via IDB server 65. The user interface 85 retrieves labeled interval data for the recording and uses them to display a visual time-line of events. The user enters a phone number that is passed to Skim Server t 5 55 so it can call the user's telephone for conference playback through Dialogic board 57. As the audio plays on the user's phone, Java user interface 85 continuously updates the graphical display and controls how the recording is played using skim server 55. All clients like Java user interface 85 and conference recorder 110 communicate with skim server 55 2o and IDB server 65 through a CORBA application programming interface in one embodiment of the present invention. CORBA was chosen because it allows a simple interface between programs written in different languages running on different platforms. Both servers 50 and 55 and conference recorder 110 are written in C + + and run on Sun Solaris platforms in one 25 embodiment of the present invention.
Skim server 55 performs the following functions:
1. Records audio from telephone line to file.
WO 99/l7235 PCT1US98/20446 2. Detects speech events while recording and posts them to the database.
3. Plays from file to telephone line - from any point in recording - in variable speeds - with pauses removed or not.
In one embodiment, skim server 55 is based on the same type of hardware as standard voice mail servers, and it performs many of the same functions. One difference between skim server 55 and a more traditional 1 o voice mail server is that it processes speech events and posts them to IDB
server 65, and also that it provides fine control over what parts of the audio file are played and what parts are skipped.
One function of IDB server 65 is to store and retrieve labeled interval data associated with a recorded conference. This is data that describes properties about specific intervals within the speech, such as who is talking, pauses in speech, telephone call control data, etc. This can be further extended to applications that require intervals that mark video scene changes, or relate automatic speech recognition output to a recording. The labeled interval data can be created, stored, and retrieved by a number of 2o different applications . Some are automatically derived from raw speech data, some are side effects of user activity, and others may be entered manually at record time or at playtime.
Fig. 2 illustrates the format of an interval data element 130 that forms the labeled interval data associated with a recorded conference.
Every interval during the recorded conference will be associated with an interval data element 130. In one embodiment, each interval data element 130 includes the following:
1. Recording ID or Depot 122: Refers to the recording that is associated with the interval and the collection point where the recording is stored.
2. Start time 123: Applications need both absolute time and time relative to recording start time. Relative time is more compact, and it is easy to convert to absolute as long as an absolute start time is stored with the recording.
3. Duration or end time 124.
4. Type: A code to identify the meaning of this interval. Is it Io a pause in speech, a scene change, etc.?
5. Type-specific data values 126: Depending on the type, this data could be a string of text, a number, a URL, etc.
Labeled interval data must be able to be stored, retrieved, and manipulated more than one at a time. Some applications will deal with t 5 large collections of intervals that share everything except start time and end time (e.g., a11 times when a specific person was speaking).
Applications must be able to store interval data in the database at any time: before recording has begun, during recording, and after. For example, for a teleconference it may be necessary to record caller-id and 2o ringing events before the call, record who is speaking during the call, and make annotations about the call afterwards. Some applications need to display incomplete interval data while a recording is in progress (e.g., catch up to live conference), so it should be possible to post an interval that has started but not ended yet, and post the end time later. It should also be 25 possible to adjust interval data, for example to realign them with other data.
A11 applications that post events to IDB server 65 must specify precise millisecond offsets for start and end times of each interval. All offsets are from an absolute start-time for the recording. Posting intervals from different machines in real-time requires a11 clients that are posting events have synchronized clocks, so standard network time protocol ("NTP") software is run on a11 of these machines.
Browse, search, and playback applications need to query and display subsets of interval data. Examples of queries that can be supported by the present invention include:
~ All interval data for a specific recording, sorted by time and type.
~ All intervals of a specific type with specific values, or values within a particular range.
~ Intervals within an absolute or relative time range.
~ Intervals of a specific duration.
The present invention provides for logical/set operations. For example, assume a user wants to see and/or hear only the parts of a recording when person A or person B was talking, and wants to leave a11 the pauses out. This can be expressed by making three queries: intervals 2o when A was speaking (set A), intervals when B was speaking (set B), and pause intervals (set P). The desired set can be expressed as "A union B
less P" , or if these sets are thought of as long bit masks, then they can be described as logical operations: (A B) & ( P).
Some types of intervals may not have clear start and end times.
Instead of a binary on/off state at each time increment, some data has an associated probability curve over time because the exact times of the events are not certain. Output from automatic speech recognition (e.g., phoneme _ 10 lattices) can include several overlapping hypotheses about what words are being said at any given moment. In one embodiment of the present invention, iDB server 65 provides support for "fuzzy" intervals. In another embodiment, IDB server 65 uses binary intervals along with a probability value in the type-specific numeric data field to achieve a similar effect as fuzzy intervals, but without fuzzy logical operations.
Transcriptions can be stored as interval data, perhaps one sentence per interval, or one word per interval depending on how fine a mapping is desired between words and time. The transcriptions may be produced from t o close caption text, higher quality off line transcriptions, or a lower quality automatic speech recognition system.
Teleconference system 200 provides playback of recorded conferences using conference playback documents. The system utilizes stored labeled interval data associated with the conference. Fig. 3 illustrates a conference playback document 300 in accordance with one embodiment of the present invention. Conference playback document 300 is implemented as a Java applet through Java user interface 85 of Fig. 1. It uses a visual structuring of the recording as a series of color-coded intervals (e.g., intervals 305 and 310) plotted on a horizontal time axis in an area referred to as a time-line window 315. Each participant in a call (e.g. , participants 316-320) is allocated a separate time-line for graphically depicting a11 labeled intervals that are associated with that person (e.g., dialing, connected, muted, talking, etc.).
Fig. 4 illustrates in detail how overlapping intervals are displayed.
As shown in Fig. 4, by plotting each interval type one at a time, starting with taller bars, the document displays overlapping intervals on the same line.
Referring again to Fig. 3, intervals that are not associated with an individual person are plotted separately above the participants, (e.g., hyperlinks 330, speech segments, etc.). Time-line window 315 provides a snapshot of every participants' activity, and can be used to navigate s through the recording.
In one embodiment, once users have established a phone connection to the recorded conference player, they can use a tool bar 350 below the time-line to begin playing the audio and adjust the skimming parameters.
In another embodiment, a separate phone connection is not necessary 1 o because the audio conference recording can be "streamed" in conjunction with conference playback document 300.
Toolbar 350 provides five buttons to control the player: "goto beginning 351", "jump back 352", "stop 353", "play 3S4", and "jump forward 3S5". It also contains a slider 356 for adjusting the playing speed 1s (0.7x, 1.0x, 1.3x, 1.7x, and 2.0x), a zoom menu 3S7 for selecting the zoom factor (none, 20min. , l Omin. , and Smin. ), and an on/off pause button 358 for pause removal.
As the recorded conference audio plays, a vertical red needle 360 moves across the time-line. When needle 360 moves, every participant's 2o name tag is colored to reflect that person's state at that time in the meeting.
Fig. 3 shows a one hour conference with the entire duration visible (zoom = none). In this view, the visual structures help make some details of the call immediately obvious. For example, the number and span of the light colored bars can identify the most/least dominant talkers. The initial long 2s uninterrupted talking bands show who gave the formal presentations.
Finally the point where the question and answer session began is visible roughly half way into the call, where many short talking intervals are scattered among many participants. More detailed information must be found by either listening to the audio or by searching through linked annotations, images, and other documents.
The zooming feature allows the user to narrow the duration displayed in the time-line window. A numbered scroll bar allows the user to register the zoomed-in portion with the full duration, and scroll using mouse clicks or arrow keys on the keyboard. Scrolling is independent of player location needle 360, so the user can separately glance at regions, without disrupting listening. Player needle 360 can be moved by clicking I o on the time-line, or by pressing a jump forward/backward button. When this happens, the skim server plays a short non-speech audio cue and begins to play at the new location.
Clicking the time-line near the top is used to select hyperlinks (e.g., link 330) rather than to move the needle. When a link is selected, or a "links" button 340 is pressed, a dialog displays all the links in the recording. This dialog can be used to visit a link, edit a link, or create a link both in and out of the time-line. One embodiment of the present invention supports the following types of links: annotations, audio, documents) images, and general URL. All links are implemented using 2o URLs except annotations, which store textual content as interval data.
Each type of link is displayed on the time-line with a representative icon.
Hyperlinks into and out of the time-line are stored as intervals, and contain both a beginning and ending time offset. Thus a link can refer to a particular point or region of the time-line, allowing a rich set of skimming alternatives. For example, following a link can cause play to begin at a certain point, end at a certain point, or sequence through selected regions.
This means that following a link can have multiple effects, including moving the player needle and changing the document page.
As disclosed, one embodiment of the present invention is a teleconference recorder and player. When a conference is recorded, an interval database stores labeled interval data associated with the conference. The labeled interval data allows searching and retrieving of the recorded conference, and facilitates playback of the recorded conference.
Several embodiments of the present invention are specifically 1 o illustrated and/or described herein. However, it will be appreciated that modifications and variations of the present invention are covered by the above teachings and within the purview of the appended claims without departing from the spirit and intended scope of the invention.
For example, although the embodiments disclosed are implemented over the Internet, the present invention can be implemented using a private network, or using any other known or future data communication methods.
Labeled interval data must be able to be stored, retrieved, and manipulated more than one at a time. Some applications will deal with t 5 large collections of intervals that share everything except start time and end time (e.g., a11 times when a specific person was speaking).
Applications must be able to store interval data in the database at any time: before recording has begun, during recording, and after. For example, for a teleconference it may be necessary to record caller-id and 2o ringing events before the call, record who is speaking during the call, and make annotations about the call afterwards. Some applications need to display incomplete interval data while a recording is in progress (e.g., catch up to live conference), so it should be possible to post an interval that has started but not ended yet, and post the end time later. It should also be 25 possible to adjust interval data, for example to realign them with other data.
A11 applications that post events to IDB server 65 must specify precise millisecond offsets for start and end times of each interval. All offsets are from an absolute start-time for the recording. Posting intervals from different machines in real-time requires a11 clients that are posting events have synchronized clocks, so standard network time protocol ("NTP") software is run on a11 of these machines.
Browse, search, and playback applications need to query and display subsets of interval data. Examples of queries that can be supported by the present invention include:
~ All interval data for a specific recording, sorted by time and type.
~ All intervals of a specific type with specific values, or values within a particular range.
~ Intervals within an absolute or relative time range.
~ Intervals of a specific duration.
The present invention provides for logical/set operations. For example, assume a user wants to see and/or hear only the parts of a recording when person A or person B was talking, and wants to leave a11 the pauses out. This can be expressed by making three queries: intervals 2o when A was speaking (set A), intervals when B was speaking (set B), and pause intervals (set P). The desired set can be expressed as "A union B
less P" , or if these sets are thought of as long bit masks, then they can be described as logical operations: (A B) & ( P).
Some types of intervals may not have clear start and end times.
Instead of a binary on/off state at each time increment, some data has an associated probability curve over time because the exact times of the events are not certain. Output from automatic speech recognition (e.g., phoneme _ 10 lattices) can include several overlapping hypotheses about what words are being said at any given moment. In one embodiment of the present invention, iDB server 65 provides support for "fuzzy" intervals. In another embodiment, IDB server 65 uses binary intervals along with a probability value in the type-specific numeric data field to achieve a similar effect as fuzzy intervals, but without fuzzy logical operations.
Transcriptions can be stored as interval data, perhaps one sentence per interval, or one word per interval depending on how fine a mapping is desired between words and time. The transcriptions may be produced from t o close caption text, higher quality off line transcriptions, or a lower quality automatic speech recognition system.
Teleconference system 200 provides playback of recorded conferences using conference playback documents. The system utilizes stored labeled interval data associated with the conference. Fig. 3 illustrates a conference playback document 300 in accordance with one embodiment of the present invention. Conference playback document 300 is implemented as a Java applet through Java user interface 85 of Fig. 1. It uses a visual structuring of the recording as a series of color-coded intervals (e.g., intervals 305 and 310) plotted on a horizontal time axis in an area referred to as a time-line window 315. Each participant in a call (e.g. , participants 316-320) is allocated a separate time-line for graphically depicting a11 labeled intervals that are associated with that person (e.g., dialing, connected, muted, talking, etc.).
Fig. 4 illustrates in detail how overlapping intervals are displayed.
As shown in Fig. 4, by plotting each interval type one at a time, starting with taller bars, the document displays overlapping intervals on the same line.
Referring again to Fig. 3, intervals that are not associated with an individual person are plotted separately above the participants, (e.g., hyperlinks 330, speech segments, etc.). Time-line window 315 provides a snapshot of every participants' activity, and can be used to navigate s through the recording.
In one embodiment, once users have established a phone connection to the recorded conference player, they can use a tool bar 350 below the time-line to begin playing the audio and adjust the skimming parameters.
In another embodiment, a separate phone connection is not necessary 1 o because the audio conference recording can be "streamed" in conjunction with conference playback document 300.
Toolbar 350 provides five buttons to control the player: "goto beginning 351", "jump back 352", "stop 353", "play 3S4", and "jump forward 3S5". It also contains a slider 356 for adjusting the playing speed 1s (0.7x, 1.0x, 1.3x, 1.7x, and 2.0x), a zoom menu 3S7 for selecting the zoom factor (none, 20min. , l Omin. , and Smin. ), and an on/off pause button 358 for pause removal.
As the recorded conference audio plays, a vertical red needle 360 moves across the time-line. When needle 360 moves, every participant's 2o name tag is colored to reflect that person's state at that time in the meeting.
Fig. 3 shows a one hour conference with the entire duration visible (zoom = none). In this view, the visual structures help make some details of the call immediately obvious. For example, the number and span of the light colored bars can identify the most/least dominant talkers. The initial long 2s uninterrupted talking bands show who gave the formal presentations.
Finally the point where the question and answer session began is visible roughly half way into the call, where many short talking intervals are scattered among many participants. More detailed information must be found by either listening to the audio or by searching through linked annotations, images, and other documents.
The zooming feature allows the user to narrow the duration displayed in the time-line window. A numbered scroll bar allows the user to register the zoomed-in portion with the full duration, and scroll using mouse clicks or arrow keys on the keyboard. Scrolling is independent of player location needle 360, so the user can separately glance at regions, without disrupting listening. Player needle 360 can be moved by clicking I o on the time-line, or by pressing a jump forward/backward button. When this happens, the skim server plays a short non-speech audio cue and begins to play at the new location.
Clicking the time-line near the top is used to select hyperlinks (e.g., link 330) rather than to move the needle. When a link is selected, or a "links" button 340 is pressed, a dialog displays all the links in the recording. This dialog can be used to visit a link, edit a link, or create a link both in and out of the time-line. One embodiment of the present invention supports the following types of links: annotations, audio, documents) images, and general URL. All links are implemented using 2o URLs except annotations, which store textual content as interval data.
Each type of link is displayed on the time-line with a representative icon.
Hyperlinks into and out of the time-line are stored as intervals, and contain both a beginning and ending time offset. Thus a link can refer to a particular point or region of the time-line, allowing a rich set of skimming alternatives. For example, following a link can cause play to begin at a certain point, end at a certain point, or sequence through selected regions.
This means that following a link can have multiple effects, including moving the player needle and changing the document page.
As disclosed, one embodiment of the present invention is a teleconference recorder and player. When a conference is recorded, an interval database stores labeled interval data associated with the conference. The labeled interval data allows searching and retrieving of the recorded conference, and facilitates playback of the recorded conference.
Several embodiments of the present invention are specifically 1 o illustrated and/or described herein. However, it will be appreciated that modifications and variations of the present invention are covered by the above teachings and within the purview of the appended claims without departing from the spirit and intended scope of the invention.
For example, although the embodiments disclosed are implemented over the Internet, the present invention can be implemented using a private network, or using any other known or future data communication methods.
Claims (20)
1. A system for recording and playing multimedia data that includes a plurality of intervals, said system comprising:
a skim server that detects a first set of the plurality of intervals;
an interval database server coupled to said skim server, said interval database server generating labeled interval data for the first set of the plurality of intervals detected by said skim server; and a database coupled to said interval database server and storing said labeled interval data;
wherein said labeled interval data comprises an interval data element for each of the detected plurality of intervals.
a skim server that detects a first set of the plurality of intervals;
an interval database server coupled to said skim server, said interval database server generating labeled interval data for the first set of the plurality of intervals detected by said skim server; and a database coupled to said interval database server and storing said labeled interval data;
wherein said labeled interval data comprises an interval data element for each of the detected plurality of intervals.
2. The system of claim 1, further comprising:
a conference bridge coupled to said interval database server that detects a second set of the plurality of intervals;
wherein said interval database server further generates labeled interval data for the second set of the plurality of intervals detected by said skim server.
a conference bridge coupled to said interval database server that detects a second set of the plurality of intervals;
wherein said interval database server further generates labeled interval data for the second set of the plurality of intervals detected by said skim server.
3. The system of claim 2, wherein said first set of the plurality of intervals comprise pauses in speech.
4. The system of claim 2, wherein said second set of the plurality of intervals comprise call control events.
5. The system of claim 1, wherein the multimedia data comprises a conference telephone call.
6. The system of claim 1, wherein said interval data element comprises:
a type of the detected interval;
a start time of the detected interval; and a duration of the detected interval.
a type of the detected interval;
a start time of the detected interval; and a duration of the detected interval.
7. The system of claim 6, wherein said interval data element further comprises:
a recording identification of the detected interval; and a type-specific data value of the detected interval.
a recording identification of the detected interval; and a type-specific data value of the detected interval.
8. The system of claim 1, wherein said interval database server comprises:
means for searching said stored labeled interval data.
means for searching said stored labeled interval data.
9. The system of claim 8, wherein said interval database server further comprises:
means for retrieving said stored labeled interval data and associated multimedia data.
means for retrieving said stored labeled interval data and associated multimedia data.
10. The system of claim 5, further comprising:
a user interface generated during playback of the conference call, wherein said user interface displays the stored labeled interval data.
a user interface generated during playback of the conference call, wherein said user interface displays the stored labeled interval data.
11. A method for recording and playing multimedia data that includes a plurality of intervals, said method comprising:
detecting the plurality of intervals;
generating labeled interval data for the plurality of intervals; and storing the labeled interval data in a database;
wherein said labeled interval data comprises an interval data element associated with each of the plurality of intervals.
detecting the plurality of intervals;
generating labeled interval data for the plurality of intervals; and storing the labeled interval data in a database;
wherein said labeled interval data comprises an interval data element associated with each of the plurality of intervals.
12. The method of claim 11, wherein said interval data element comprises:
a type of the associated interval;
a start time of the associated interval; and a duration of the associated interval.
a type of the associated interval;
a start time of the associated interval; and a duration of the associated interval.
13. The method of claim 12, wherein said interval data element further comprises:
a recording identification of the associated interval; and a type-specific data value of the associated interval.
a recording identification of the associated interval; and a type-specific data value of the associated interval.
14. The method of claim 11, further comprising:
storing the multimedia data in the database.
storing the multimedia data in the database.
15. The method of claim 14, further comprising:
querying said database based on one or more labeled interval data parameters; and retrieving at least one interval data element and associated multimedia data from the database.
querying said database based on one or more labeled interval data parameters; and retrieving at least one interval data element and associated multimedia data from the database.
16. The method of claim 11, wherein the multimedia data comprises a conference telephone call.
17. The method of claim 16, further comprising:
generating a user interface that displays the labeled interval data;
and playing the conference call based on selections of the user interface.
generating a user interface that displays the labeled interval data;
and playing the conference call based on selections of the user interface.
18. A method of recording and playing a teleconference telephone call, said method comprising:
detecting a plurality of intervals during the telephone call;
generating labeled interval data for each of said plurality of intervals; and storing said labeled interval data in a database.
detecting a plurality of intervals during the telephone call;
generating labeled interval data for each of said plurality of intervals; and storing said labeled interval data in a database.
19. The method of claim 18, wherein said labeled interval data comprises a plurality of interval data elements, said method further comprising:
querying said database and retrieving one or more of the stored interval data elements; and playing a portion of the teleconference telephone call that is associated with each of said retrieved interval data elements.
querying said database and retrieving one or more of the stored interval data elements; and playing a portion of the teleconference telephone call that is associated with each of said retrieved interval data elements.
20. The method of claim 18, wherein said detected intervals comprise:
an identity of a speaker;
pauses in speech; and telephone call control.
an identity of a speaker;
pauses in speech; and telephone call control.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US6061997P | 1997-10-01 | 1997-10-01 | |
| US60/060,619 | 1997-10-01 | ||
| PCT/US1998/020446 WO1999017235A1 (en) | 1997-10-01 | 1998-09-30 | Method and apparatus for storing and retrieving labeled interval data for multimedia recordings |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA2271745A1 true CA2271745A1 (en) | 1999-04-08 |
Family
ID=22030673
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002271745A Abandoned CA2271745A1 (en) | 1997-10-01 | 1998-09-30 | Method and apparatus for storing and retrieving labeled interval data for multimedia recordings |
Country Status (3)
| Country | Link |
|---|---|
| JP (1) | JP2001511991A (en) |
| CA (1) | CA2271745A1 (en) |
| WO (1) | WO1999017235A1 (en) |
Families Citing this family (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2797132B1 (en) * | 1999-07-16 | 2001-10-05 | Matra Nortel Communications | METHOD AND SYSTEM FOR SOUND RESTITUTION WITH SPATIAL EFFECT, AND TELEPHONE TERMINAL INCORPORATING SUCH A SYSTEM |
| GB2359155A (en) | 2000-02-11 | 2001-08-15 | Nokia Mobile Phones Ltd | Memory management of acoustic samples eg voice memos |
| WO2002065745A1 (en) * | 2001-02-15 | 2002-08-22 | Sivashunmugam Columbus | Context association for multimedia using mark-up intelligence |
| GB0108603D0 (en) * | 2001-04-05 | 2001-05-23 | Moores Toby | Voice recording methods and systems |
| US20040021765A1 (en) * | 2002-07-03 | 2004-02-05 | Francis Kubala | Speech recognition system for managing telemeetings |
| US7290207B2 (en) | 2002-07-03 | 2007-10-30 | Bbn Technologies Corp. | Systems and methods for providing multimedia information management |
| US20040138894A1 (en) | 2002-10-17 | 2004-07-15 | Daniel Kiecza | Speech transcription tool for efficient speech transcription |
| US7003286B2 (en) | 2002-10-23 | 2006-02-21 | International Business Machines Corporation | System and method for conference call line drop recovery |
| AU2003295834A1 (en) * | 2002-11-25 | 2004-06-18 | Telesector Resources Group, Inc. | Methods and systems for conference call buffering |
| US20040207724A1 (en) * | 2003-04-17 | 2004-10-21 | Siemens Information And Communication Networks, Inc. | System and method for real time playback of conferencing streams |
| CN1635792A (en) * | 2003-12-29 | 2005-07-06 | 皇家飞利浦电子股份有限公司 | A specific program segment construction method and apparatus |
| US7308476B2 (en) | 2004-05-11 | 2007-12-11 | International Business Machines Corporation | Method and system for participant automatic re-invite and updating during conferencing |
| EP1811759A1 (en) * | 2006-01-23 | 2007-07-25 | Hewlett-Packard Development Company, L.P. | Conference call recording system with user defined tagging |
| NO325487B1 (en) * | 2006-09-14 | 2008-05-13 | Tandberg Telecom As | Method and device for dynamic streaming / archiving configuration |
| US8838179B2 (en) | 2009-09-25 | 2014-09-16 | Blackberry Limited | Method and apparatus for managing multimedia communication recordings |
| EP2302867B1 (en) * | 2009-09-25 | 2019-06-05 | BlackBerry Limited | Method and apparatus for managing multimedia communication recordings |
| WO2013026457A1 (en) | 2011-08-19 | 2013-02-28 | Telefonaktiebolaget L M Ericsson (Publ) | Technique for video conferencing |
| WO2013055756A1 (en) * | 2011-10-10 | 2013-04-18 | Talko Inc. | Communication system |
| EP3311558B1 (en) | 2015-06-16 | 2020-08-12 | Dolby Laboratories Licensing Corporation | Post-teleconference playback using non-destructive audio transport |
| US10471348B2 (en) | 2015-07-24 | 2019-11-12 | Activision Publishing, Inc. | System and method for creating and sharing customized video game weapon configurations in multiplayer video games via one or more social networks |
| CN113259740A (en) * | 2021-05-19 | 2021-08-13 | 北京字跳网络技术有限公司 | Multimedia processing method, device, equipment and medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH02134785A (en) * | 1988-11-15 | 1990-05-23 | Sony Corp | Voice signal recording device |
| JPH052540A (en) * | 1991-06-24 | 1993-01-08 | Fujitsu Ltd | Electronic conference system having minutes forming function |
| US5550965A (en) * | 1993-12-27 | 1996-08-27 | Lucent Technologies Inc. | Method and system for operating a data processor to index primary data in real time with iconic table of contents |
| US5619555A (en) * | 1995-07-28 | 1997-04-08 | Latitude Communications | Graphical computer interface for an audio conferencing system |
| US5559875A (en) * | 1995-07-31 | 1996-09-24 | Latitude Communications | Method and apparatus for recording and retrieval of audio conferences |
-
1998
- 1998-09-30 WO PCT/US1998/020446 patent/WO1999017235A1/en not_active Application Discontinuation
- 1998-09-30 JP JP52049099A patent/JP2001511991A/en active Pending
- 1998-09-30 CA CA002271745A patent/CA2271745A1/en not_active Abandoned
Also Published As
| Publication number | Publication date |
|---|---|
| JP2001511991A (en) | 2001-08-14 |
| WO1999017235A1 (en) | 1999-04-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2271745A1 (en) | Method and apparatus for storing and retrieving labeled interval data for multimedia recordings | |
| US7848493B2 (en) | System and method for capturing media | |
| US6298129B1 (en) | Teleconference recording and playback system and associated method | |
| US7506262B2 (en) | User interface for creating viewing and temporally positioning annotations for media content | |
| US7466334B1 (en) | Method and system for recording and indexing audio and video conference calls allowing topic-based notification and navigation of recordings | |
| US6282510B1 (en) | Audio and video transcription system for manipulating real-time testimony | |
| JP4466564B2 (en) | Document creation / viewing device, document creation / viewing robot, and document creation / viewing program | |
| US8407049B2 (en) | Systems and methods for conversation enhancement | |
| US9063935B2 (en) | System and method for synchronously generating an index to a media stream | |
| US20030128820A1 (en) | System and method for gisting, browsing and searching voicemail using automatic speech recognition | |
| US20040132432A1 (en) | Voice recordal methods and systems | |
| US20070286573A1 (en) | Audio And Video Transcription System For Manipulating Real-Time Testimony | |
| US20020133513A1 (en) | Log note system for digitally recorded audio | |
| US20100100805A1 (en) | Log Note System For Digitally Recorded Audio | |
| JP2005341015A (en) | Video conference system with minute creation support function | |
| JP3437617B2 (en) | Time-series data recording / reproducing device | |
| US20020044633A1 (en) | Method and system for speech-based publishing employing a telecommunications network | |
| Roy et al. | NewsComm: a hand-held interface for interactive access to structured audio | |
| US20080167879A1 (en) | Speech delimiting processing system and method | |
| KR101783872B1 (en) | Video Search System and Method thereof | |
| KR100806225B1 (en) | The Appratus method of automatic generation of the web page for conference record and the method of searching the conference record using the event information | |
| JP4080965B2 (en) | Information presenting apparatus and information presenting method | |
| JP2004279897A (en) | Method, device, and program for voice communication record generation | |
| Clements et al. | Phonetic searching of digital audio | |
| Wellner et al. | Conference Scribe: Turning conference calls into documents |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| FZDE | Dead |