JP2004199696A - Method for displaying information stored in multiple multimedia document - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide techniques for enabling a user to "horizontally" browse or read a multimedia document. <P>SOLUTION: Techniques for providing a graphical user interface (GUI) that displays representation of stored information that may include information of one or more types is provided. The displayed representation may include representation of information of one or more types. The GUI enables the user to navigate the stored information and to analyze the contents of the stored information. The stored information may include information captured along the same timeline and information captured along different timelines. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

  The present invention relates to a user interface for displaying information, and more particularly, to a user interface for searching and displaying multimedia information stored in one or more multimedia documents.

  With rapid advances in computer technology, an increasing amount of information is stored in the form of electronic (or digital) documents. Such an electronic document is, for example, a multimedia document storing multimedia information. The term “multimedia information” is used to refer to information composed of a plurality of different types of information integrated. The different types of information included in the multimedia information include, for example, text information, graphic information, animation information, sound (audio) information, video information, slide information, whiteboard information, and other types of information. It is a combination. The multimedia information is also used to refer to information composed of one or more objects including different types of information. For example, multimedia objects included in the multimedia information include text information, graphic information, animation information, sound (audio) information, video information, slide information, whiteboard information, and other types of information. Multimedia documents are composed of video, audio, closed caption text, keyframes, presentation slides, whiteboard capture information, and other multimedia type objects. Think of it as a composite object. Examples of multimedia documents are documents that store, for example, interactive web pages, television broadcasts, videos, presentations, and the like.

  Multiple tools and applications available in the past allow a user to play, store, index, edit, or manipulate multimedia information stored in a multimedia document. Examples of such tools and / or applications include, for example, manufacturer-specific or customized multimedia players (e.g., RealNetworks RealPlayer (R), Microsoft's Microsoft Windows (R) Media Player, Apple's QuickTime (registered trademark), Shockwave multimedia player, etc.), video player, television, personal digital assistant (PDA), etc. For example, Village, Inc. (San Mateo, California) (www.virage.com) supplies various tools such as a tool for viewing and manipulating video content and a tool for creating a video database. Virage also supplies face detection tools and on-screen text recognition tools from video information.

Given the enormous number of electronic documents, readers of electronic documents are increasingly required to understand an enormous amount of information in a short time. To meet the demands placed on them, readers must read electronic documents "horizontally" instead of "vertically", that is, they read one document from beginning to end Instead of analyzing at, it has been found necessary to scan, browse and glance through portions of one or more electronic documents of interest. While there are tools that allow a user to “read horizontally” or look through an electronic document containing text / image information (eg, a reading tool described in US Pat. Multimedia documents that may contain audio information, video information, and other types of information cannot be used to “read” or look through multimedia documents. The multimedia tools described in the above patent documents do not allow the user to read or glance through multimedia documents “horizontally”.
US Non-Provisional Patent Application No. 08 / 995,616

  In view of the above, there is a need for a technique that allows a user to “read” or read a multimedia document “horizontally”. A technique that allows a user to view, analyze, and search multimedia information stored in a multimedia document is desirable.

  Embodiments of the present invention provide a graphical user interface (GUI) that displays a representation of stored information that may include one or more types of information. The displayed representation may include a representation of the one or more types of information. The GUI allows the user to navigate through this stored information and browse through it and analyze its content. This stored information may include information captured along the same timeline or may include information captured along different timelines.

  According to one embodiment of the present invention, a first representation of the first stored information is displayed. The first stored information includes a first type of information and a second type of information. The first representation includes a representation of the first type of information included in the first stored information and a representation of the second type of information included in the first stored information. One or more parts of the first representation are highlighted. One or more portions of the emphasized first representation correspond to portions of the first representation that include the first criterion.

  According to another embodiment of the present invention, in addition to displaying the first representation of the first stored information, a second representation of the second stored information is displayed. The second stored information includes a first type of information and a second type of information. The second representation includes a representation of the first type of information included in the second stored information and a representation of the second type of information included in the second stored information. One or more parts of the second representation are emphasized. One or more portions of the enhanced second representation correspond to portions of the second representation that include the first criterion.

  According to another embodiment of the present invention, a technique for displaying multimedia information is provided. A first thumbnail containing a representation of the first type of information contained in the first recorded information is displayed. A second thumbnail is displayed that includes a representation of the second type of information contained in the first recorded information. A third thumbnail containing a representation of the first type of information contained in the second recorded information is displayed. A fourth thumbnail is displayed that includes a representation of the second type of information contained in the second recorded information. According to one embodiment, one or more portions that include at least one word (or topic of interest) in a group of words of the first and third thumbnails (or of the second and fourth thumbnails). Is emphasized.

  According to yet another embodiment of the present invention, a technique for displaying information contained in a first recorded information including audio information and video information and a second recorded information including audio information and video information Is provided. A first representation of the information contained in the first recorded information is displayed. The first representation has a first thumbnail and a second thumbnail. The first thumbnail includes text information obtained from audio information included in the first recorded information, and the second thumbnail is extracted from video information included in the first recorded information. Includes the above key frames. A second representation of the information contained in the second recorded information is displayed. The second representation has a third thumbnail and a fourth thumbnail. The third thumbnail includes text information obtained from the audio information included in the second recorded information, and the fourth thumbnail is extracted from the video information included in the second recorded information. Includes the above key frames. According to one embodiment, one or more portions of the first representation and the second representation that include user criteria are highlighted. Here, the emphasized part of the first expression covers a part of the first thumbnail and the second thumbnail, and the emphasized part of the second expression is a part of the third thumbnail and the fourth thumbnail. Cover.

  According to another embodiment of the present invention, a technique for displaying information is provided. A representation of the stored information is displayed. Information representing one or more output portions of the stored information is received. In the stored information representation, one or more portions corresponding to one or more output portions of the stored information are emphasized.

  According to one embodiment of the present invention, a technique for displaying information is provided. A representation of the stored information is displayed. Information representing one or more portions of the stored information that have not yet been output is received. Of the representation of the stored information, one or more portions corresponding to one or more portions of the stored information that have not yet been output are highlighted.

  These and other features, embodiments, and advantages of the present invention will become more apparent with reference to the following specification, claims, and accompanying drawings.

  Embodiments of the present invention provide a technique for searching and displaying multimedia information. In accordance with one embodiment of the present invention, a graphical user interface (GUI) is provided that displays multimedia information that can be stored in a multimedia document. According to the method of the present invention, this GUI allows a user to search for multimedia information stored in a multimedia document. This GUI provides both a display focusing on a part of the content of the multimedia document and a display showing the entire context. Thus, this GUI allows the user to read or “glance” through the multimedia document “horizontally”.

  As described above, the word “multimedia information” refers to information including a plurality of different types of information. Different types of information included in this multimedia information include, for example, text information, graphic information, animation information, sound (audio) information, video information, slide information, whiteboard image information, and other types of information, Or a combination. For example, a television broadcast video recording may include video information and audio information. At a particular moment, the video recording may also include material related to the video information, often closed caption (CC) text information that directly represents the speech contained in the audio portion of the video recording. Multimedia information is also used to refer to information that includes one or more objects that contain different types of information. For example, multimedia objects included in multimedia information include text information, graphic information, animation information, sound (audio) information, video information, slide information, whiteboard image information, and other types of information. Including.

  As used herein, the term “multimedia document” shall refer to any electronic storage unit (eg, file, directory, etc.) that stores multimedia information. A variety of different formats can be used to store multimedia information. These formats include various MPEG formats (for example, MPEG-1, MPEG-2, MPEG-4, MPEG-7, etc.), MP3 format, SMIL format, HTML + TIME format, WMF (Windows (registered trademark) media format). ), RM (Real Media) format, Quicktime format, Shockwave format, various streaming media formats, formats developed by the engineering community, manufacturer-specific custom formats, and the like. Examples of multimedia documents are, for example, recorded videos, MPEG files, news broadcast recordings, presentation recordings, conference recordings, class recordings, television broadcast programs, and the like.

  FIG. 1 is a simplified block diagram of a distributed network 100 that may incorporate an embodiment of the present invention. As shown in FIG. 1, the distributed network 100 includes one or more client systems 102, a server system 104, and a multimedia information source (MIS) 106 connected to a communication network 108 through a plurality of communication links 110. , Including a plurality of computer systems. The distributed network 100 shown in FIG. 1 is merely illustrative of one embodiment that incorporates the present invention and does not limit the scope of the claimed invention. Those skilled in the art will recognize other variations, modifications, and alternatives. For example, the present invention may be implemented in a stand-alone system. In a stand-alone environment, the functions performed by the various computer systems shown in FIG. 1 are performed by a single computer system.

  Communication network 108 provides a mechanism that allows the various computer systems shown in FIG. 1 to communicate and exchange information. The communication network 108 may itself consist of many interconnected computer systems and communication links. In one embodiment, the communication network 108 is the Internet, but in other embodiments the communication network 108 is a local area network (LAN), a wide area network (WAN), a wireless network, an intranet, a private network. Any suitable communication network such as a network, public network, switching network, etc. may be used.

  The communication link 110 used to connect the various systems shown in FIG. 1 may vary, such as a wired link, an optical link, a satellite or other wireless communication link, a wave link, or any other mechanism for information communication. It can be of any kind. Various communication protocols can be used to facilitate information transmission over the communication link. These communication protocols include, for example, TCP / IP, HTTP protocol, extensible markup language (XML), wireless application protocol (WAP), protocols under development by industry standards bodies, vendor specific protocols, customized Protocol, etc.

  Depending on whether the computer system connected to the communication network 108 plays a role in requesting information and / or providing information and / or services, a “client” and a “server” Can be classified. Computer systems used to request information or services by a user are classified as “client” computers (or “clients”). A computer system that stores information and provides it in response to a user request received from a client computer or a computer system that performs processing to provide a service requested by a user is a “server” computer ( Alternatively, it is called a “server”). However, it is clear that a particular computer system can function as both a client and a server.

  Thus, according to one embodiment of the present invention, server system 104 is configured to perform processing to facilitate generation of a GUI that displays multimedia information in accordance with the teachings of the present invention. The GUI generated by server system 104 may be output to a user (eg, a reader of a multimedia document) through an output device connected to server system 104 or client system 102. The GUI generated by the server 104 allows the user to search and browse multimedia information that can be stored in the multimedia document. Because this GUI provides both a display that focuses on a portion of the content of the multimedia document and a display that shows the overall context, this GUI allows the user to browse through the multimedia document or “horizontally”. It becomes possible to read.

  The processing performed by the server system 104 to generate the GUI and provide various features according to the teachings of the present invention may be performed by a software module executing on the server system 104, and the server It may be implemented by a hardware module connected to the system 104, or a combination thereof. As an alternative embodiment of the present invention, this process may also be distributed among the various computer systems shown in FIG.

  The multimedia information displayed on this GUI may be stored in a multimedia document accessible by the server system 104. For example, the multimedia document may be stored in the storage subsystem of server system 104. The multimedia document may also be stored by other systems such as the MIS 106 accessible by the server 104. Alternatively, the multimedia document may be stored in a memory location accessible to the server system 104.

  As an alternative embodiment, instead of accessing a multimedia document, server system 104 receives a stream of multimedia documents (eg, streaming media signals, cable signals, etc.) from a multimedia information source such as MIS 106. Also good. According to one embodiment of the present invention, the server system 104 stores the multimedia information signal in a multimedia document and generates a GUI for displaying the multimedia information. Examples of the MIS 106 are a television broadcast receiver, a cable receiver, a digital video recorder (eg, a TIVO box), and the like. For example, the multimedia information source 106 may be implemented as a television configured to receive multimedia broadcast signals and transmit the signals to the server system 104. As an alternative embodiment, server 104 may be configured to intercept multimedia information signals received by MIS 106. The server system 104 may receive this multimedia information directly from the MIS 106 or alternatively may receive the information through a communication network such as the communication network 108.

  As described above, the MIS 106 shown in FIG. 1 represents a source of multimedia information. According to one embodiment of the invention, MIS 106 may store multimedia documents that are accessed by server system 104. For example, the MIS 106 is a storage device or server that stores multimedia documents accessible by the server system 104. As an alternative embodiment, MIS 106 may provide a multimedia information stream to server system 104. For example, the MIS 106 may be a television receiver / antenna that provides live television feed information to the server system 104. The MIS 106 is, for example, a device such as a video recorder / player, DVD player, CD player, etc. that provides recorded video and / or audio streams to the server system 104. As an alternative embodiment, MIS 106 may be a presentation or meeting recording device that may provide a stream of captured presentation or meeting information to server system 104. The MIS 106 is also configured to capture or receive multimedia information from an external source (eg, via a wireless link) and provide the captured multimedia information to the server system 104 for further processing. It may be a receiver (for example, a satellite receiving antenna or a cable receiver).

  A user can use the client system 102 to view the GUI generated by the server system 104. The user can also use the client system 102 to select a particular multimedia document and generate a GUI that displays the multimedia information stored by that particular multimedia document. You can also request it. The user can also interact with the GUI generated by the server system 104 using an input device connected to the client system 102. As an alternative embodiment, client system 102 may also perform processing to facilitate GUI generation in accordance with the teachings of the present invention. The client system 102 is a personal computer, portable computer, workstation, computer terminal, network computer, mainframe, kiosk, personal digital assistant (PDA), communication device such as a mobile phone, or Various types of data processing systems may be used.

  According to one embodiment of the present invention, a single computer system may function as both the server system 104 and the client system 102. Various other configurations of server system 104, client system 102, and MIS 106 are contemplated.

  FIG. 2 is a simplified block diagram of a computer system 200 according to one embodiment of the invention. The computer system 200 can be used as any of the computer systems shown in FIG. As shown in FIG. 2, the computer system 200 has at least one processor 202. The processor 202 communicates with a plurality of peripheral devices through the bus subsystem 204. These peripheral devices include, for example, a storage subsystem 206 including a memory subsystem 208 and a file storage subsystem 210, a user interface input device 212, a user interface output device 214, and a network interface subsystem. 216, and so on. These input / output devices allow the user to interact with the computer system 200. A user is a human user, device, process, another computer, or the like. The network interface subsystem 216 provides an interface to other computer systems and communication networks.

  The bus subsystem 204 provides a mechanism that allows the various components and subsystems of the computer system 200 to communicate with each other as intended. These various subsystems and components of the computer system 200 need not be physically located at the same location, but may be distributed at various locations in the network 100. Although the bus subsystem 204 is schematically illustrated as a single bus, multiple buses may be used as an alternative embodiment of the bus subsystem.

  The user interface input device 212 includes a keyboard, a pointing device, a mouse, a trackball, a touch pad, a graphic tablet, a scanner, a barcode scanner, a touch screen built in the display, a voice input device such as a voice recognition system, a microphone , And other types of input devices. In general, when the word “input device” is used, it is intended to include all possible types of devices and methods for inputting information using computer system 200.

  User interface output device 214 may be a display subsystem, printer, fax machine, or non-visual display such as an audio output device. The display subsystem is a flat panel device such as a cathode ray tube (CRT) or a liquid crystal display (LCD), a projection device, or the like. The display subsystem also provides a non-visual display, such as through an audio output device. In general, when the word “output device” is used, it is intended to include all possible types of devices and methods for outputting information from computer system 200. According to one embodiment of the present invention, a GUI generated in accordance with the teachings of the present invention may be presented to the user through output device 214.

  The storage subsystem 206 may be configured to store basic programs and data structures that provide the functionality of the computer system and the present invention. For example, according to one embodiment of the present invention, software modules that implement the functionality of the present invention may be stored in the storage subsystem 206 of the server system 104. These software modules can be executed by one or more processors 202 of server system 104. In a distributed environment, these software modules may be stored on multiple computer systems and executed by the processors of the multiple computer systems. The storage subsystem 206 also provides a storage room for storing various databases that can be used by the present invention. Storage subsystem 206 may include a memory subsystem 208 and a file storage subsystem 210.

  The memory subsystem 208 includes a plurality of memories such as a main random access memory (RAM) 218 that stores instructions and data during program execution, and a read only memory (ROM) 220 that stores fixed instructions. obtain. The file storage subsystem 210 provides permanent (non-volatile) storage for program and data files, such as a hard disk drive, a floppy disk drive with associated removable media, a compact Disk read only memory (CD-ROM) drives, optical drives, removable media cartridges, and other similar storage media. One or more of these drives may be located at remote locations on other connected computers.

  The computer system 200 is a personal computer, portable computer, workstation, computer terminal, network computer, mainframe, kiosk, personal digital assistant (PDA), communication device such as a mobile phone, or Various types of data processing systems may be used. Server computers typically have more storage and processing power than client systems. Because computers and networks are constantly changing in nature, the description of computer system 200 shown in FIG. 2 is intended only as a specific example for the purpose of describing a preferred embodiment of the computer system. ing. Many other computer system configurations are possible having more or fewer components than the computer system shown in FIG.

  FIG. 3 illustrates a simplified user interface 300 for viewing multimedia information generated in accordance with one embodiment of the present invention. It will be appreciated that the GUI 300 shown in FIG. 3 is merely exemplary of one embodiment incorporating the present invention and does not limit the scope of the invention as claimed. Those skilled in the art will recognize other variations, modifications, and alternatives.

  The GUI 300 displays multimedia information stored in the multimedia document. The multimedia information stored by the multimedia document and displayed by the GUI 300 may include a plurality of different types of information. As shown in FIG. 3, the GUI 300 displays multimedia information corresponding to television broadcasting including video information, audio information, and possibly closed caption (CC) text information. Television broadcasts may be stored as television broadcast recordings in memory locations accessible to server system 104. However, it is clear that the present invention is not limited to displaying television recordings. Multimedia information consisting of other types of information can also be displayed in accordance with the teachings of the present invention.

  Television broadcasts can be stored using a variety of different techniques. According to one approach, television broadcasts are recorded and stored using a satellite receiver connected to the PC-TV video card of server system 104. An application running on server system 104 then processes this recorded television broadcast to facilitate the generation of GUI 300. For example, video information contained in a television broadcast can be obtained using an MPEG capture application that creates another metafile (eg, in XML format) that includes temporal information for the broadcast and, if prepared, a closed option. Can be incorporated. Next, the GUI 300 shown in FIG. 3 is generated using the information stored in the metafile.

  As shown in FIG. 3, the GUI 300 includes a plurality of display areas 302 such as a first display area 302, a second display area 304, a third display area 306, a fourth display area 308, and a fifth display area 310. It has a display area. As an alternative embodiment, it is clear that the invention can have more or less display area than the example shown in FIG. Further, in an alternative embodiment of the present invention, one or more display areas may be combined into one display area, or the characteristic display area may be divided into a plurality of display areas. Accordingly, the display areas shown in FIG. 3 and described below are not intended to limit the scope of the present invention as set forth in the claims.

  According to an embodiment of the present invention, the first display area 302 displays one or more commands that can be selected by a user watching the GUI 300. Various user interface functions such as menu bar, drop down menu, cascade menu, button, selection bar, button, etc. can be used to display this user selectable command. According to one embodiment of the present invention, the commands provided in the first display area 302 include commands that allow the user to select a multimedia document having multimedia information displayed on the GUI. The command may also include one or more commands that allow the user to set and / or customize the manner in which multimedia information stored in the multimedia document selected by the user is displayed on the GUI 300. Various other commands may also be provided in the first display area 302.

  According to one embodiment of the invention, the second display area 304 displays a reduced representation of the multimedia information stored by the multimedia document. The user can select the magnification used to display information in the second display area 304. According to certain embodiments of the invention, a representation of the entire multimedia document (ie, multimedia information from the start time to the end time associated with the multimedia document) is displayed in the second display area 304. . In this embodiment, one end of the second display area 304 represents the start time of the multimedia document, and the other end of the second display area 304 represents the end time of the multimedia document.

  As shown in FIG. 3, according to an embodiment of the present invention, the second display area 304 has one or more thumbnail images 312. Each thumbnail image displays a representation of a particular type of information contained in the multimedia information stored by the multimedia document. For example, two thumbnail images 312-1 and 312-2 are displayed in the second display area 304 of the GUI 300 shown in FIG. The thumbnail image 312-1 displays text information corresponding to information included in the multimedia information stored by the multimedia document displayed by the GUI 300. The text displayed on the thumbnail image 312-1 represents a displayable representation of the CC text included in the multimedia information displayed by the GUI 300, for example. Alternatively, the text displayed in the thumbnail image 312-1 may represent a displayable depiction of the audio information transcript included in the multimedia information stored by the multimedia document with the content displayed by the GUI 300. Good. Various audio / text recording techniques can be used to generate a recording for audio information. The text displayed on the thumbnail image may be an expression of other types of information included in the multimedia information. For example, the text information may be an expression of a comment made when the annotation or the like added to the multimedia information is recorded or displayed.

  The thumbnail 312-2 displays a representation of video information included in the multimedia information displayed by the GUI 300. In the embodiment shown in FIG. 3, the video information is displayed using video keyframes extracted from the video information contained in the multimedia information stored by the multimedia document. This video keyframe can be extracted from the video information of the multimedia document at various times using a specified sampling rate. A special layout style that may be user configurable is used to display the extracted keyframes in thumbnail 312-2 to improve the readability of the frames.

  Depending on the type of information included in the displayed multimedia information, one or more thumbnail images may be displayed in the second display area 304. Each thumbnail image 312 displayed in the second display area 304 displays a representation of a particular type of information included in the multimedia information stored by the multimedia document. According to one embodiment of the present invention, the number of thumbnails displayed in the second display area 304 and the information type displayed by each thumbnail can be set by the user.

  According to an embodiment of the present invention, the various thumbnail images displayed in the second display area 304 are synchronized in time or aligned with each other in time series. This means that various types of information included in the multimedia information and generated almost simultaneously are displayed one after another. For example, the thumbnail images 312-1 and 312-2 are text information displayed on the thumbnail image 31201 (CC text information, audio information recording, or text of some other type of information included in the multimedia information). Video keyframes displayed in thumbnail 312-2 that occurred in the multimedia information at a particular point in time are displayed close together (eg, along the same horizontal axis). . Therefore, information having a specific time stamp is displayed adjacent to information having substantially the same time stamp. As a result, the user can determine various information types generated almost simultaneously in the multimedia information displayed by the GUI 300 by simply scanning the second display area 304 in the horizontal direction.

  According to the teachings of the present invention, a display lens or window 314 (hereinafter referred to as “thumbnail display area / lens 314”) is displayed in the second display area 304. The thumbnail display area lens 314 covers or emphasizes a part of the second display area 304. According to the teaching of the present invention, the multimedia information corresponding to the area covered by the thumbnail display area / lens 314 in the second display area 304 is displayed in the third display area 306.

  In the embodiment shown in FIG. 3, the thumbnail display area lens 314 is located above the second display area 304 and highlights the top (starting portion) of the multimedia document. The user can change the position of the thumbnail display area lens 314 by sliding or moving the lens 314 along the second display area 304. For example, in FIG. 3, the thumbnail display area lens 314 can be moved vertically along the second display area 304.

  When the position of the thumbnail display area / lens 314 is changed from the first position of the second display area 304 to the second position along the second display area 304, the multimedia displayed in the third display area 306 is displayed. The information is automatically updated so that the multimedia information displayed in the third display area 306 continues to correspond to the area highlighted by the thumbnail display area lens 314 in the second display area 304. The Thus, the user can navigate and scroll through the contents of the multimedia document displayed by the GUI 300 using the thumbnail display area lens 314. As such, the thumbnail display area lens 314 provides context and indicates the position of the multimedia information displayed in the third display area 306 within the entire multimedia document.

FIG. 4 is a simplified enlarged view of a thumbnail display area lens 314 according to one embodiment of the present invention. As shown in FIG. 4, the thumbnail display area lens 314 is divided by a first side 318 and a second side 320. The thumbnail display area / lens 314 emphasizes an area between the side 318 and the side 320 in the second display area 304. Based on the position of the thumbnail display area / lens 314 on the second display area 304, the side 318 corresponds to a specific time t ₁ in the multimedia document, and the side 320 corresponds to a specific time t ₂ ( t ₂ > t ₁ ). For example, when the thumbnail display area lens 314 is positioned at the beginning of the second display area 304 (as shown in FIG. 3), t ₁ corresponds to the start time of the displayed multimedia document, and the thumbnail display area when lens 314 is located at the end of the second display area 304, t ₂ corresponds to the end time of the multimedia document. Therefore, the thumbnail display area / lens 314 emphasizes the portion of the second display area 304 between time t ₁ and time t ₂ . According to one embodiment of the present invention, multimedia information corresponding to the time interval t ₂ -t ₁ (highlighted or covered by the thumbnail display area lens 314) is displayed in the third display area 306. The Accordingly, when the position of the thumbnail display area / lens 314 is changed along the second display area 304 according to the user input, the information displayed in the third display area 306 is transferred to the third display area 306. The displayed multimedia information is updated so as to continue to correspond to the area highlighted by the thumbnail display area / lens 314 in the second display area 304.

  As shown in FIGS. 3 and 4, the thumbnail display area / lens 314 includes a sub lens 316 that further emphasizes a part of the second display area 304 that is emphasized by the thumbnail display area / lens 314. According to an embodiment of the present invention, a portion of the second display area 304 that is emphasized or covered by the sub lens 316 is a portion of the third display area 306 that is emphasized or covered by the lens 322. Correspond. The sub lens 316 can be moved along the second display area 304 between the side 318 and the side 320 of the thumbnail display area lens 314. When the sub lens 316 is moved from the first position within the range of the thumbnail display area / lens 314 to the second position, the position of the lens 322 in the third display area 306 also corresponds to the position change of the sub lens 316. Automatically changed. Further, when the position of the lens 322 is changed from the first position to the second position on the third display area 306, the position of the sub lens 316 is automatically changed in accordance with the position change of the lens 322. Be changed. Further details regarding the lens 322 are described below.

As described above, the multimedia information corresponding to the portion highlighted by the thumbnail display area / lens 314 in the second display area 304 is displayed in the third display area 306. Therefore, the representation of the multimedia information that occurred between time t ₁ and time t ₂ (corresponding to the time interval highlighted by the thumbnail display area lens 314 in the multimedia document) is displayed in the third display area 306. Is displayed. Thus, the third display area 306 displays an expanded representation of the multimedia information stored by the multimedia document corresponding to the portion of the multimedia document highlighted by the thumbnail display area lens 314.

  As shown in FIG. 3, the third display area 306 has one or more panels 324. Each panel displays a representation of a particular type of information contained in the multimedia information that occurred during the time interval highlighted by the thumbnail display area lens 314. For example, in the GUI 300 shown in FIG. 3, two panels 324-1 and 324-2 are displayed in the third display area 306. According to one embodiment of the present invention, each panel 324 of the third display area 306 corresponds to the thumbnail image 312 displayed in the second display area 304, and among the thumbnail images, the thumbnail display area lens 314. The information corresponding to the part covered by is displayed.

  Similar to the thumbnail image 312, the panels 324 are aligned or synchronized in time with each other. Accordingly, various types of information included in the multimedia information and generated almost simultaneously are displayed one after another in the third display area 306. For example, the panels 324-1 and 324-2 shown in FIG. 3 may include text information (CC text information, audio information recording, or some other information included in the multimedia information displayed on the panel 324-1. Video keyframes displayed on panel 324-2 that occurred in the multimedia information at approximately the same time point close to each other (eg, along the same horizontal axis). Displayed). Therefore, information having a specific time stamp is displayed adjacent to other types of information having substantially the same time stamp. As a result, the user can determine various information types generated in the multimedia information almost simultaneously by simply scanning the third display area 306 in the horizontal direction.

  The panel 324-1 shown in the GUI 300 corresponds to the thumbnail image 312-1 and displays text information corresponding to the area highlighted or covered by the thumbnail display area lens 314 in the thumbnail image 312-1. . The text information displayed by the panel 324-1 may correspond to text extracted from the CC information included in the multimedia information, or may be included in the audio information recording or the multimedia information included in the multimedia information. It may represent textual representations of several other types of information. According to one embodiment of the present invention, the present invention utilizes story auto-segmentation and other features often provided as closed caption (CC) text from news broadcasts. Most news stations that provide CC text as part of the broadcast use a special syntax for CC text (eg, the “>>” delimiter indicates a storyline or theme change, and the “>>” delimiter Indicates a speaker change, etc.). Assuming this type of information is present in the CC text information included in the multimedia information, the present invention incorporates these features into the text displayed on panel 324-1. For example, a “>>” delimiter may be displayed to indicate a storyline or theme change, and a “>>” delimiter may be displayed to indicate a speaker change, clearly identifying different stories. To distinguish, additional space may be displayed between portions of text associated with different story lines. This improves the readability of the text information displayed on the panel 324-1.

The panel 324-2 displayed on the GUI 300 corresponds to the thumbnail image 312-2, and represents a representation of the video information corresponding to the area highlighted or covered by the thumbnail display area lens 314 in the thumbnail image 312-2. indicate. Therefore, the panel 324-2 is included in the multimedia information stored by the multimedia document, displaying a representation of video information occurring between times t ₁ and time t ₂ about Thumbnail Display Area lens 314. In the embodiment shown in FIG. 3, video keyframes extracted from video information included in the multimedia information are displayed on panel 324-2. In order to improve the readability of the frame, a special user-configurable layout style is used to display the extracted keyframes.

A variety of different techniques can be used to display the video keyframes on panel 324-2. According to an embodiment of the present invention, the time interval between times t ₁ and time t ₂ is divided into sub-sections of a predetermined time period. Each subsection is characterized by a start time and an end time for that subsection. According to an embodiment of the present invention, the start time of the first sub-interval corresponds to the time t _1, the end time of the last sub-section corresponding to the time t _2. The server 104 then extracts, for each subsection, one or more video keyframes from the video information stored by the multimedia document that occurred between the start time and end time associated with that subsection. For example, according to one embodiment of the present invention, for each sub-interval, the server 104 extracts video key frames at 1 second intervals between the start time and end time associated with that sub-interval.

  Server 104 then selects, for each subsection, one or more keyframes displayed on panel 324-2 from the video keyframes extracted for that subsection. The user can set the number of key frames selected to be displayed on the panel 324-2 for each sub-section. For each sub-interval, various different methods can be used to select a video keyframe to be displayed from the extracted video keyframes. For example, if the number of video keyframes extracted for one sub-section is 24 and six video keyframes are displayed for each sub-section (as shown in FIG. 3), the server 104 The first two video keyframes, the middle two video keyframes and the last two video keyframes are selected from the video keyframes extracted for the interval.

  In another embodiment, the video keyframes displayed for a sub-interval are selected based on the position order of those keyframes in the keyframe group extracted for the subinterval. For example, when the number of video key frames extracted for a certain sub-section is 24 and six video key frames are displayed for each sub-section, the first, fifth, ninth, thirteenth, The 17th and 21st key frames may be selected. In this embodiment, a fixed number of key frames are skipped.

  In yet another embodiment, the video keyframes displayed for a subinterval are selected based on the time values associated with those keyframes in the keyframe group extracted for the subinterval. For example, assume that a group of video key frames extracted for one sub-section has 24 key frames extracted at a sampling rate of 1 second, and six video key frames are displayed for each sub-section. The first key frame is selected, and then the key frame that occurs 4 seconds after the previously selected key frame is selected.

  As an alternative embodiment of the present invention, the server 104 may select a key frame from a group of key frames based on the difference in key frame content. For each sub-interval, the server 104 can use a special image processing technique to determine the difference in the content of the key frame extracted for that sub-interval. When six video key frames are displayed for each sub-interval, the server 104 selects six key frames from the extracted key frame group based on the result of this image processing method. For example, the six most different key frames may be selected as displayed on panel 324-2. It will be apparent that various other techniques known to those skilled in the art can also be used to perform video keyframe selection.

  The selected key frame is then displayed on panel 324-2. A variety of different formats can be used to display the selected keyframe on panel 324-2. For example, as shown in FIG. 3, for each sub-section, the selected key frame is placed from left to right and from top to bottom.

In an alternative embodiment of the invention, the entire multimedia document is divided into sub-intervals of a predetermined period. Each subsection is characterized by a start time and an end time for that subsection. According to an embodiment of the present invention, the start time of the first subsection corresponds to the start time of the multimedia document, and the end time of the last subsection corresponds to the end time of the multimedia document. As described above, the server 104 extracts, for each subsection, one or more video keyframes from the video information stored by the multimedia document based on the start time and end time associated with that subsection. Server 104 then selects one or more keyframes to display for each subsection. Next, depending on the position of the thumbnail display area / lens 314, a key frame generated between time t ₁ and time t ₂ related to the thumbnail display area / lens 314 is displayed on the panel 324-2. Is displayed.

  Obviously, as alternative embodiments of the present invention, various other approaches may be used to display video information on panel 324-2. According to one embodiment of the present invention, the user can set the technique used to display video information in the third display area 306.

  In the GUI 300 shown in FIG. 3, each sub-section is 8 seconds long, and video key frames corresponding to a plurality of sub-sections are displayed on the panel 324-2. Six video key frames from each sub-section are displayed. For each sub-section, the displayed key frames are placed from left to right and from top to bottom.

  As an alternative embodiment of the present invention, it will be appreciated that the number of panels displayed in the third display area 306 may be greater or less than the number of thumbnail images displayed in the second display area 304. According to one embodiment of the present invention, the number of panels displayed in the third display area 306 can be set by the user.

  According to the teachings of the present invention, a display lens or window 322 (hereinafter referred to as a “panel display area lens 322”) that covers or highlights a portion of the overview area 306 is displayed. According to the teachings of the present invention, multimedia information corresponding to the area highlighted by the panel display area lens 322 in the third display area 306 is displayed in the fourth display area 308. The user can change the position of the panel display area lens 322 by sliding or moving the lens 322 along the third display area 306. When the position of the panel display area / lens 322 is changed from the first position of the third display area 306 to the second position, the multimedia information displayed in the fourth display area 308 is changed to the fourth display area 308. The multimedia information displayed on the screen is automatically updated so as to continue to correspond to the area highlighted by the panel display area / lens 322 in the third display area 306. Therefore, the user can change the multimedia information displayed in the fourth display area 308 by using the panel display area / lens 322.

  As described above, when the position of the panel display area / lens 322 is changed, the area highlighted by the sub-lens 316 in the second display area 304 is the panel display area / lens in the third display area 306. The position of the sub lens 316 is also changed to continue to correspond to the area highlighted by 322. Similarly, as described above, when the position of the sub lens 316 is changed, the area highlighted by the panel display area / lens 322 in the third display area 306 corresponds to the position after the change of the sub lens 316. In order to continue, the position of the panel display area / lens 322 on the third display area 306 is also changed.

FIG. 5A is a simplified enlarged view of a panel display area lens 322 according to one embodiment of the present invention. As shown in FIG. 5A, the panel display area lens 322 is divided by a first side 326 and a second side 328. The panel display area / lens 322 emphasizes an area between the side 326 and the side 328 in the third display area 306. Based on the position of the panel display area lens 322 on the third display area 306, the side 326 corresponds to a specific time t ₃ in the multimedia document, and the side 328 corresponds to a specific time t ₄ ( t ₁ ≦ t ₃ <t ₄ ≦ t ₂ ). For example, when the panel display area lens 322 is positioned at the beginning of the third display area 306, t ₃ is equal to t _1, when the panel display area lens 322 is located at the end of the third display area 306, t ₂ is equal to _{t 4.} Therefore, the panel display area / lens 322 emphasizes the portion of the third display area 306 between time t ₃ and time t ₄ . According to one embodiment of the present invention, multimedia information corresponding to time intervals t ₃ -t ₄ (highlighted or covered by the panel display area lens 322) is displayed in the fourth display area 308. The When the position of the panel display area / lens 322 is changed along the third display area 306 in response to a user input, the information displayed in the fourth display area 308 is displayed in the fourth display area 308. The multimedia information is updated so as to continue to correspond to the area highlighted by the panel display area / lens 322 in the third display area 306. Thus, the panel display area lens 322 provides context and indicates the position of the multimedia information displayed in the fourth display area 308 within the entire multimedia document.

  According to one embodiment of the present invention, a particular line of text (or one or more words in the last line of text) highlighted by the panel display area lens 322 is displayed on a portion of the lens 322. For example, as shown in FIGS. 5A and 3, the last line 330 of text highlighted by the panel display area lens 322 on the panel 324-1: “Environment is a national” is displayed in bold on the panel display area lens 322. Is done.

  According to one embodiment of the present invention, the panel display area lens 322 can be equipped with special features to facilitate browsing and navigation of multimedia documents. As shown in FIG. 5A, according to one embodiment of the present invention, a “play / pause button” 332 and a “lock / release button” 334 are provided on the panel display area lens 322. The play / pause button allows the user to control the playback of video information from the panel display area lens 322. The lock / release button 334 allows the user to switch the video playback position from the area 340-1 in the fourth display area 308 to a reduced window above the panel display area / lens 322.

FIG. 5B illustrates a panel display area lens according to one embodiment of the present invention where the lock / release button 334 is activated or “locked” (ie, video playback is locked onto the panel display area lens 322). 322 is a simplified example of 322. As shown in FIG. 5B, in the lock mode, video information is played in a window 336 on the lens 322. In the embodiment shown in FIG. 5 b, the size of the portion of the panel display area lens 322 that overlaps the panel 342-2 is enlarged beyond times t ₃ -t ₄ to accommodate the window 336. According to one embodiment of the present invention, the video content displayed in window 336 corresponds to the content displayed in area 340-1 of fourth display area 308.

According to an embodiment of the present invention, the window 336 has a transparent border so that a portion of the third display area 306 below it can be seen. This helps to maintain the focus position while the user is viewing the third display area 306. The user can start and stop the video displayed in window 336 using play / pause button 332. The user can change the position of the panel display area lens 322 while playing the video in the window 336. When the position of the panel display area / lens 322 changes, the video playback in the window 336 also changes according to the new position of the panel display area / lens 322. Video playback in window 336 corresponds to new time values t ₃ and t ₄ for panel display area lens 322.

FIG. 5C is a simplified example of a panel display area lens 322 according to one embodiment of the present invention in which a representative video keyframe is displayed on the panel display area lens 322. In this embodiment, the server 104 analyzes the video keyframes of the panel 324-2 highlighted or covered by the panel display area lens 322, and the keyframes highlighted by the panel display area lens 322 are the best. A particular key frame 338 to represent is determined. This particular key frame is then displayed on the portion of the panel display area lens 322 that covers the panel 324-2. In the embodiment shown in FIG. 5C, the size of the portion covering the panel 324-2 of the panel display area lens 322 is enlarged beyond the time t ₃ and t ₄ in order to accommodate a representation of key frames 338 Yes.

As described above, the multimedia information corresponding to the portion covered by the panel display area / lens 322 in the third display area 306 (that is, the multimedia information generated in the time interval t _{3 to} t ₄ ) is the fourth. Are displayed in the display area 308. As illustrated in FIG. 3, the fourth display area 308 includes, for example, one or more sub display areas 340 (for example, 340-1, 340-2, and 340-3). According to an embodiment of the present invention, one or more of the sub-regions 340 includes a specific type of information included in multimedia information corresponding to a portion of the third display area 306 that is highlighted by the panel display area / lens 322. Is displayed.

For example, as shown in FIG. 3, video information corresponding to (or starting from) the video information highlighted by the panel display area lens 322 in the third display area 306 is displayed in the sub display area 340-1. Is displayed. According to one embodiment of the present invention, video information starting at time t ₃ (time corresponding to the upper end of the panel display area lens 322) is played in the sub display area 340-1. As an alternative embodiment, video information to be reproduced in the area 340-1 may begin at time t _4, or may start from the other time the user can set the between t ₃ and t ₄ . Video playback in the sub display area 340-1 can be controlled using the control bar 342. The control bar 342 provides a plurality of controls such as a control for playing back the video being played in the sub display area 340-1, a control for pausing, a control for stopping, a control for rewinding, and a control for fast-forwarding. In the area 340-1, the current time and the remaining reproduction time 344 are also displayed. Information 346 identifying the name of the video, date 348 when the video was recorded, and video type 350 are also displayed.

  In an alternative embodiment of the present invention, instead of playing video information, video keyframes from the video keyframes highlighted by panel display area lens 322 in panel 324-2 are displayed in sub-display area 340-1. Is displayed. According to one embodiment of the present invention, the key frame displayed in area 340-1 represents the key frame most representative of the key frame highlighted by panel display area lens 322.

  According to one embodiment of the present invention, text information (eg CC text, audio information recording, other types of audio information included in the multimedia information) highlighted in the third display area 306 by the panel display area lens 322. A text representation of information, etc.) is displayed in the sub display area 340-2. According to an embodiment of the present invention, the sub display area 340-2 displays text information displayed on the panel 324-1 and emphasized by the panel display area lens 322. As described above, various types of information can be displayed in the sub display area 340-3.

  Additional information regarding the multimedia information stored by the multimedia document can be displayed in the fifth display area 310 of the GUI 300. For example, as shown in FIG. 3, words that appear in text information included in the multimedia information displayed by the GUI 300 are displayed in an area 352 of the fifth display area 310. The frequency of each word in the multimedia document is also displayed next to each word. For example, the word “question” appears seven times in the multimedia information CC text. Various other types of information regarding multimedia information can also be displayed in the fifth display area 310.

  In accordance with an embodiment of the present invention, the GUI 300 may include text information extracted from multimedia information by a user (eg, CC text, audio information recording, text representation of other types of information included in the multimedia information, etc. ) Is provided so that one or more words appearing in (1) can be searched. For example, the user can enter one or more search words in the input field 354 and selecting the “Search” button 356 causes the server 104 to be extracted from the multimedia information stored by the multimedia document. The text information is analyzed to identify all occurrences of one or more search words entered in field 354. The presence of one or more words in the multimedia document is then highlighted when displayed in the second display area 304, the third display area 306, and the fourth display area 308. For example, according to one embodiment of the present invention, the presence of all search words is highlighted in thumbnail image 312-1, panel 324-1, and sub-display area 340-2. As an alternative embodiment of the present invention, the presence of one or more search words is determined by the second display area 304, the panel displayed in the third display area 306, and the sub display displayed in the fourth display area 308. The display area may also be emphasized.

  The user can also specify one or more words that are highlighted in the multimedia information displayed on the GUI 300. For example, the user can select one or more emphasized words from area 352. Then, the presence of all key frames selected by the user in the area 352 is emphasized in the second display area 304, the third display area 306, and the fourth display area 308. For example, as shown in FIG. 6, if the user selects the word “National” in area 352, accordingly, according to one embodiment of the present invention, the second display area 304, the third In the display area 306 and the fourth display area 308, the presence of all the words “National” is highlighted.

  According to one embodiment of the present invention, a text line 360 containing one or more words selected by the user (or a search word entered in field 354) is displayed in sub display area 340-3 of fourth display area 308. Is done. For each text line, the time 362 when the line occurred in the multimedia document (or the time stamp associated with the text line) is also displayed. The time stamp associated with the text line generally corresponds to the time stamp associated with the first word of the line.

  For each text line, one or more words surrounding the selected word or search word are displayed. According to one embodiment of the present invention, the number of words surrounding the selected word displayed in area 340-3 is selectable by the user. For example, in the GUI 300 shown in FIG. 6, the user can specify the number of surrounding words displayed in the area 340-3 using the control 364. The number designated by the user indicates the number of words that appear before the selected word and the number of words that appear after the displayed selected word. In the embodiment shown in FIG. 6, the control 364 is a slider bar that is adjustable between a minimum value “3” and a maximum value “10”. The user can specify the number of surrounding words to be displayed by adjusting the slider bar 364. For example, when the slider bar is set to “3”, three words appearing before the selected word and three words appearing after the selected word are displayed in area 340-3. These minimum and maximum values can be set by the user.

  Furthermore, the GUI 300 shown in FIG. 6 has an area 358 sandwiched between thumbnail images 312-1 and 312-2 showing the appearance locations of the search word or other words specified by the user. For example, the area 358 has a marker indicating the location of the word “National” in the thumbnail image 312-1. The user can then use the thumbnail display area lens 314 or the panel display area lens 322 to scroll to the desired location within the multimedia document. FIG. 7 shows a simplified enlarged view of the second display area 304 showing the area 358 according to one embodiment of the present invention. As shown in FIG. 7, area 358 (or channel 358) has a marker 360 that indicates where the thumbnail image 312-1 has the presence of the word “National”. As an alternative embodiment of the present invention, the marker in channel 358 may also specify the location of a user specified word or phrase in other thumbnail images displayed in second display area 304. As an alternative embodiment, the location of the search word or other words specified by the user may be displayed on the thumbnail image 312 (as shown in FIG. 20A).

  As shown in FIG. 6, the position of the thumbnail display area lens 314 has been changed with respect to FIG. When the position of the thumbnail display area / lens 314 is changed, the multimedia information displayed in the third display area 306 corresponds to the portion highlighted in the thumbnail display area / lens 314 in the second display area 304. To change. The multimedia information displayed in the fourth display area 308 also changes according to the new position of the panel display area / lens 322.

  According to one embodiment of the present invention, multimedia information related to user-specified topics of interest displayed on the GUI 300 is highlighted or annotated. Annotations or highlights visually indicate information relevant to or of interest to the user. Thus, the GUI 300 provides a convenient tool that allows a user to easily find portions of a multimedia document that are relevant to the user.

  According to one embodiment of the present invention, information specifying topics of interest to or related to the user is stored in the user profile. One or more words or phrases can be associated with each topic of interest. The presence of one or more words or phrases associated with a particular user-specified topic of interest indicates that there is information about that particular topic. For example, the user can specify two topics of interest: “George W. Bush” and “Energy Crisis”. The words or phrases associated with the topic “George Bush” include, for example, words and phrases such as “Present Bush”, “the President”, “Mr. Bush”. Words or phrases associated with the topic “Energy Crisis” are, for example, “industrial pollution”, “natural pollution”, “clean up the sources”, “amount of pollution”. Quantity) "," air pollution "," electricity "," power-generating plant ", and the like. A probability value indicative of the likelihood of the topic of interest considered from the presence of the word or phrase may be associated with each word or phrase. Various tools can be provided, such as a tool that allows a user to set a topic of interest, a tool that specifies keywords and phrases associated with the topic, and a tool that specifies probability values associated with the keywords or phrases.

  Obviously, various other techniques known to those skilled in the art can also be used to model topics of interest to the user. These techniques are, for example, using Bayesian networks, relevance graphs, or the like. Techniques for determining parts related to user-specified topics, techniques for defining topics of interest, and techniques for associating keywords and / or key phrases with probability values are hereby incorporated herein by reference in their entirety. US patent application Ser. No. 08 / 995,616 (filing date: December 22, 1997).

  In accordance with one embodiment of the present invention, to identify a location associated with a user-specified topic of interest in a multimedia document, server 104 retrieves the multimedia document and includes the interest within the multimedia document. Identify locations related to words or phrases associated with the topic. As described above, the presence of a word or phrase associated with a particular user-specified topic of interest in the multimedia document indicates that there is a particular topic associated with that user. Words and phrases that appear in multimedia documents and are associated with user-specified topics of interest are annotated or highlighted when displayed by the GUI 300.

  FIG. 8 is a simplified illustration according to an embodiment of the present invention that is highlighted (or annotated) when multimedia information related to one or more topics of interest to the user is displayed on the GUI 300. An example of the GUI 800 is shown. The GUI 800 shown in FIG. 8 is merely illustrative of one embodiment of the present invention and is not intended to limit the scope of the present invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  In the embodiment shown in FIG. 8, the user has specified four topics 802 of interest. A label 803 identifies each topic. Topics specified in the GUI 800 include “Energy Crisis”, “Assistive Technology”, “George W. Bush” and “Nepal”. . According to the teachings of the present invention, keywords and key phrases related to a specified topic are highlighted in the second display area 304, the third display area 306, and the fourth display area 308. A variety of different techniques can be used to highlight or annotate keywords and / or key phrases related to the topic of interest. According to one embodiment of the invention, different colors and styles (eg, bold, underline, different font sizes, etc.) can be used to highlight words and phrases related to user-specified topics. For example, a specific color may be assigned to each topic, and content related to the specific topic may be emphasized using a specific color assigned to the specific topic. For example, as shown in FIG. 8, the first color is used to highlight words and phrases related to the topic of interest: “Energy Crisis”, and words and phrases related to the topic of interest: “Assistive Tech”. The second color is used to highlight and the third color is used to highlight words and phrases related to the topic of interest: “George W. Bush” and the topic of interest: “Nepal” A fourth color is used to highlight related words and phrases.

  According to one embodiment of the present invention, the server 104 may extract text information extracted from the multimedia information (eg, CC text information, audio information recording, or some other type of information included in the multimedia information). To find the location of a word or phrase related to the user topic. When the server 104 finds a word or phrase associated with the topic of interest in the text information, the word or phrase is annotated or highlighted when displayed on the GUI 800. As mentioned above, a number of different techniques can be used to annotate or emphasize this word or phrase. For example, the word or phrase may be highlighted, bolded, underlined, bordered using a sidebar or balloon, and the font changed. May be.

  Keyframes (representing video information in multimedia documents) displayed by the GUI and associated with user-specified topics of interest may also be highlighted. According to one embodiment of the present invention, the server system 104 uses OCR technology to extract text from key frames extracted from video information included in multimedia information. The text output by OCR technology is then compared to words or phrases associated with one or more user-specified topics of interest. If there is a match, the key frame containing the matched word or phrase (ie, the key frame from which the matched word or frame was extracted by the OCR technique) is displayed in the GUI 800 in the second display area 304 of the GUI 800. , When displayed in the third display area 306 or the fourth display area 308, annotated or highlighted. A number of different techniques can be used to annotate or highlight this key frame. For example, a special border may be drawn around a key frame associated with a particular topic of interest. The border color may correspond to the color associated with that particular topic of interest. In addition, the matching text in the key frame may be emphasized, underlined, or displayed in reverse video. As noted above, the annotated or highlighted keyframe displayed in the second display area 304 (eg, the keyframe displayed as the thumbnail image 312-2 in FIG. 3) is in the channel area 358. Can be identified by the marker displayed on the screen. As an alternative embodiment, this key frame may be annotated or highlighted in the thumbnail image 312-2.

  According to one embodiment of the present invention, as shown in FIG. 8, a relevance indicator 804 is also displayed for each user topic. For a particular topic, the relevance indicator for that topic indicates the degree of relevance (or relevance score) of that multimedia document for that particular topic. For example, as shown in FIG. 8, the number of bars displayed on the relevance indicator associated with a particular topic indicates the degree of relevance of that multimedia document to that particular topic. Accordingly, the multimedia document displayed on the GUI 800 is most relevant to the user topic “Energy Crisis” (shown by four bars) and most relevant to the user topic “Nepal”. Does not have (indicated by one bar). Various other techniques (eg, relevance scores, bar graphs, different colors, etc.) that indicate the degree of relevance to each topical multimedia document can also be used.

  According to one embodiment of the present invention, the relevance score for a particular topic is calculated based on the frequency with which words and phrases associated with that particular topic appear in the multimedia information. It is also possible to use a probability value for a word or phrase associated with the particular topic to calculate a relevance score for that particular topic. It is also useful to determine the relevance score for a user-specified topic of interest based on the frequency with which words and phrases associated with a topic appear in the multimedia document and the probability value associated with that word or phrase. Various techniques known to those skilled in the art can also be used. Various other techniques known to those skilled in the art can also be used to calculate the degree of relevance of a multimedia document to its topic of interest.

  As described above, the relevance indicator is used to display the degree of relevance or relevance score for the user. Based on the information displayed on the relevance indicator, the user can easily determine the relevance of the multimedia information stored by the multimedia document to the topic that the user can specify.

  FIG. 9 illustrates a simplified user interface 900 that defines topics of interest for one embodiment of the present invention. The user interface 900 can be invoked by selecting an appropriate command from the first display area 302. The GUI 900 shown in FIG. 9 is merely an example of an embodiment of the present invention and does not limit the scope of the present invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  The user can specify the topic of interest in field 902. A label identifying the topic of interest can be specified in field 910. The label specified in field 910 is displayed on a GUI generated in accordance with the teachings of the present invention to identify the topic of interest. A list of words and / or phrases associated with the topic specified in field 902 is displayed in area 908. The user can add new keywords to the list, modify one or more keywords in the list, and delete one or more keywords from the keyword list associated with the topic of interest. . The user can specify a new keyword or phrase associated with the topic of interest in field 904. Selecting the “Add” button 906 adds the keyword or phrase specified in the field 904 to the keyword list associated with the previous topic. The user can specify the color used to annotate or highlight information related to the topic of interest by selecting a color in area 912. For example, in the embodiment shown in FIG. 9, the location associated with “Assistive Technology” in the multimedia document is annotated or highlighted in blue.

  According to the teachings of the present invention, the GUI generated by the server 104 can display a variety of different types of information included in the multimedia information. FIG. 10 illustrates a simplified user interface 1000 that displays multimedia information stored by conference recording according to one embodiment of the present invention. It will be appreciated that the GUI 1000 shown in FIG. 10 is merely illustrative of one embodiment that incorporates the present invention and does not limit the scope of the invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  The multimedia information stored by the conference recording has video information and audio information, and possibly CC text information, slide information, and other types of information. The slide information may include information related to slides presented during the meeting (eg, PowerPoint presentation slides). For example, the slide information includes a slide image presented at the meeting. As shown in FIG. 10, the second display area 304 includes three thumbnail images 312-1, 312-2, and 312-3. Text information extracted from the conference recording multimedia information (eg, CC text information, recording audio information included in the conference recording, or text representation of other types of information included in the conference recording) is a thumbnail image 312-1. Is displayed. Video key frames extracted from the video information included in the conference recording multimedia information are displayed in the thumbnail image 312-2. The slide extracted from the slide information included in the multimedia information is displayed on the thumbnail image 312-3. The thumbnail images are arranged in time series with each other. The information displayed in the thumbnail image 312-4 provides additional context for video and text information in that the user can view presentation slides presented at various times during the conference recording.

  The third display area 306 has three panels 324-1, 324-2, and 324-3. The panel 324-1 displays text information corresponding to a portion highlighted or covered by the thumbnail display area lens 314 in the thumbnail image 312-1. Panel 324-2 displays video keyframes corresponding to the portion of thumbnail image 312-2 that is highlighted or covered by thumbnail display area lens 314. The panel 324-3 displays one or more slides corresponding to a portion of the thumbnail image 312-3 highlighted or covered by the thumbnail display area lens 314. These panels are arranged in time series with each other.

The fourth display area 308 has three sub display areas 340-1, 340-2, and 340-3. The sub display area 340-1 displays video information corresponding to a portion of the panel 324-2 covered by the panel display area / lens 322. As described above, the sub-display area 340-1 may display a key frame corresponding to the highlighted portion of the panel 324-2. Alternatively, video based on the portion of the panel display area lens 322 may be played in area 340-1. According to one embodiment of the present invention, time t ₃ associated with lens 322 is used as the time to start playing the video in area 340-1 of fourth display area 308. For example, a panoramic shot of a conference room recorded using a 360 degree camera is also displayed in the area 340-1 of the fourth display area 308. The text information emphasized by the panel display area / lens 322 in the panel 324-1 is displayed in the area 340-2 of the fourth display area 308. One or more slides highlighted by the panel display area / lens 322 in the panel 324-3 are displayed in the area 340-3 of the fourth display area 308. According to one embodiment of the present invention, the user can also select a slide from panel 324-3 by clicking on a particular slide. The selected slide is then displayed in area 340-3 of fourth display area 308.

  According to an embodiment of the present invention, the user can specify the type of information displayed on the GUI among the information included in the multimedia document. For example, the user selects or deselects the “slide” button 1004 to display slide-related information (ie, thumbnail image 312-3, panel 324-3, and fourth) displayed on the GUI. Information displayed in the area 340-3 of the display area 308) can be turned on / off. When the user deselects the slide information, the GUI 1000 does not display the thumbnail 312-3 and the panel 324-3. When the user selects the button 1004, the GUI 1000 displays a thumbnail 312-3 and a panel 324-3. As described above, the button 1004 functions as a switch for switching between display / non-display of slide information. Similarly, the user can control other types of information displayed by the GUI generated in accordance with the teachings of the present invention. For example, a function for turning on / off video information, text information, and other types of information displayed by the GUI 1000 can be provided.

  FIG. 11 illustrates a simplified user interface 1100 that displays multimedia information stored by a multimedia document in accordance with one embodiment of the present invention. It will be appreciated that the GUI 1100 illustrated in FIG. 11 is merely illustrative of one embodiment that incorporates the present invention and does not limit the scope of the invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  The multimedia document in which the included content is displayed on the GUI 1100 includes video information, audio information, CC text information, slide information, and whiteboard information. Whiteboard information may include text and graphic images written on the whiteboard. As shown in FIG. 11, the second display area 304 includes four thumbnail images 312-1, 312-2, 312-3, and 312-4. Text information extracted from the multimedia document (e.g., CC text information, or recording of audio information included in the conference recording, or other type of text representation included in the multimedia information) is the thumbnail image 312-1. Is displayed. Video key frames extracted from the video information contained in the multimedia document are displayed in the thumbnail image 312-2. The slide extracted from the slide information included in the multimedia information is displayed in the thumbnail image 312-3. The whiteboard image extracted from the whiteboard information included in the multimedia document is displayed in the thumbnail image 312-4. These thumbnail images are arranged in time series with each other.

  The third display area 306 has four panels 324-1, 324-2, 324-3, and 324-4. The panel 324-1 displays text information corresponding to a portion highlighted or covered by the thumbnail display area lens 314 in the thumbnail image 312-1. Panel 324-2 displays video keyframes corresponding to the portion of thumbnail image 312-2 that is highlighted or covered by thumbnail display area lens 314. The panel 324-3 displays one or more slides corresponding to a portion of the thumbnail image 312-3 highlighted or covered by the thumbnail display area lens 314. The panel 324-4 displays one or more whiteboard images corresponding to a portion of the thumbnail image 312-4 highlighted or covered by the thumbnail display area lens 314. These panels are arranged in time series with each other.

The fourth display area 308 has three sub display areas 340-1, 340-2, and 340-3. The area 340-1 displays video information corresponding to a portion of the panel 324-2 covered with the panel display area / lens 322. As described above, the sub display area 340-1 may display a key frame corresponding to the highlighted portion of the panel 324-2, and may play a corresponding video. According to one embodiment of the present invention, time t ₃ (described above) associated with lens 322 is used as the start time for playback of the video in area 340-1 of fourth display area 308. A panorama shot 1102 at the location where the multimedia document was recorded (eg, recorded using a 360 degree camera) is also displayed in area 340-1 of fourth display area 308. The text information emphasized by the panel display area / lens 322 in the panel 324-1 is displayed in the area 340-2 of the fourth display area 308. The slide emphasized by the panel display area / lens 322 in the panel 324-3 or the whiteboard image emphasized by the panel display area / lens 322 in the panel 324-4 is displayed in the area 340-3 of the fourth display area 308. It may be displayed. In the embodiment shown in FIG. 11, the whiteboard image corresponding to the portion of the panel 324-4 covered by the panel display area / lens 322 is displayed in the area 340-3. According to one embodiment of the present invention, a user can select a slide or whiteboard image by clicking on a specific slide on panel 324-3 or a specific whiteboard image on panel 324-4. . The selected slide or whiteboard image is then displayed in area 340-3 of fourth display area 308.

  As described above, according to an embodiment of the present invention, the user can specify the type of information displayed on the GUI among the information from the multimedia document. For example, the user can turn on / off certain types of information displayed by the GUI. The user uses the “WB” button 1104 to display the whiteboard related information (that is, the thumbnail image 312-4, the panel 324-4, and the area 340-3 of the fourth display area 308) displayed by the GUI 1000. Can be turned on / off.

  FIG. 12 illustrates a simplified user interface 1200 that displays the contents of a multimedia document according to an embodiment of the present invention. It is clear that the GUI 1200 shown in FIG. 12 is merely illustrative of one embodiment incorporating the present invention and does not limit the scope of the invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  As shown in FIG. 12, preview areas 1202 and 1204 are provided above and below the third display area 306. In this embodiment, the panel display area lens 322 can be moved along the third display area 306 between the end 1206 of the preview area 1202 and the end 1208 of the preview area 1204. Preview areas 1202 and 1204 allow the user to preview the content displayed in the third display area 306 when scrolling through the multimedia document using the panel display area lens 322. For example, when the user is scrolling down a multimedia document using the panel display area lens 322, the user can see the next content in the preview 1204 and leave the third display area 306. The content can be viewed in preview 1202. When the user is scrolling up the multimedia document using the panel display area lens 322, the user can see the next content in the preview 1202 and the content leaving the third display area 306. It can be seen in the preview 1204. According to an embodiment of the present invention, the user can change and customize the size (or length) of each preview area. For example, in the GUI 1200 shown in FIG. 12, a handle 1210 used by the user to change the size of the preview area 1204 is provided. According to one embodiment of the invention, a preview area may also be provided in the second display area 304.

  FIG. 13 illustrates a simplified user interface 1300 for displaying the contents of a multimedia document according to one embodiment of the present invention. It will be appreciated that the GUI 1300 shown in FIG. 13 is merely illustrative of an embodiment incorporating the present invention and does not limit the scope of the invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  As shown in FIG. 13, the text information is compressed and displayed on the panel 324-1 of the third display area 306, that is, the margin between lines is deleted. Thereby, the readability of text information improves. The text line displayed on panel 324-1 is then used to determine the video frame displayed on panel 324-2. According to one embodiment of the present invention, a time stamp is associated with each line of text displayed on panel 324-1. The time stamp associated with one text line represents the time when the text appeared in the multimedia document displayed by the GUI 1300. In one embodiment, the time stamp associated with a text line matches the time stamp associated with the first word of that text line. The text lines displayed on panel 324-1 are then grouped so that each group has a predetermined number of lines.

  Then, for each row group, video keyframes are extracted from the video information stored by the multimedia document in response to the time stamp associated with the group row. According to one embodiment of the invention, the server 104 determines a start time and an end time associated with each of the row groups. The start time for a group matches the time associated with the first (oldest) row of the group, and the end time for a group is the time associated with the last (newest) row of the group Matches. To determine the keyframe displayed on panel 324-2 corresponding to a particular text line group, server 104 appears between the start time and end time associated with that particular group of video information. One or more video keyframes are extracted from the obtained portion. One or more key frames are then selected from the extracted video key frames to be displayed on panel 324-2 for that particular group. The one or more selected key frames are then displayed on panel 324-1 adjacent to the row group displayed on panel 324-1 from which the key frame was extracted.

  For example, in FIG. 13, the lines displayed on panel 324-1 are divided into groups such that each group has 4 text lines. For each group, the timestamp associated with the first row of the group matches the start time for the group, and the timestamp associated with the fourth row of the group is the end time for the row group. Match. In the embodiment shown in FIG. 13, three video key frames are displayed on panel 324-2 for each group of four text lines displayed on panel 324-1. According to one embodiment of the present invention, these three video keyframes corresponding to a particular row group are keyframes extracted from video information between the start time and end time of that particular group. Corresponds to the first, middle, and last keyframes. As described above, various other techniques can be used to select the video keyframes displayed on panel 324-2. For each row group displayed on panel 324-1, the key frames corresponding to that row group are displayed such that the key frames are aligned with the row group in time. In the embodiment shown in FIG. 13, the height of the key frame for a row group is approximately equal to the vertical height of that row group.

  The number of text lines included in one group can be set by the user. Similarly, the number of video keyframes extracted for a particular row group can be set by the user. In addition, the user of the present invention can also set the video keyframes displayed on panel 324-2 for each row group.

  The user can also set how the extracted key frame is displayed on the panel 324-2. Various techniques can be used to indicate the relationship between a particular row group and the video keyframes displayed for that particular row group. For example, according to one embodiment of the present invention, the particular row group displayed on panel 324-1 and the corresponding video keyframe displayed on panel 324-2 are the same color to indicate the relationship. May be color-coded or displayed. Various other techniques known to those skilled in the art can also be used to indicate this relationship.

(GUI generation method according to an embodiment of the present invention)
Hereinafter, a GUI (for example, GUI 300 illustrated in FIG. 3) generation method according to an embodiment of the present invention will be described. For convenience, it is assumed that the multimedia information displayed on the GUI includes video information, audio information, and CC text information. When the task for generating the GUI 300 is disassembled, (a) a task for displaying a thumbnail 312-1 for displaying text information extracted from the multimedia information in the second display area 304, and (b) a video included in the multimedia information. A task for displaying a thumbnail 312-2 for displaying video keyframes extracted from the information, (c) highlighting a part of the second display area 304, and highlighting the highlighted part of the second display area 304 The thumbnail display area / lens 314 for displaying the corresponding information in the third display area 306 and a part of the third display area 306 are emphasized, and the third display area 306 corresponds to the highlighted part. A task for displaying a panel display area / lens 322 for displaying information in the fourth display area 308, and (d) a fifth display. It can be divided into the rear 310 task for displaying information, the.

  FIG. 14 is a simplified high-level flowchart 1400 illustrating a method for displaying thumbnail 312-1 in second display area 304 according to one embodiment of the present invention. The method illustrated in FIG. 14 may be executed by the server 104, may be executed by the client 102, or may be executed jointly by the server 104 and the client 102. For example, the method may be performed by a software module executing on the server 104 or client 102, may be performed by a hardware module connected to the server 104 or client 102, or a combination thereof But you can. In the embodiment described below, the method is performed by the server 104. The method illustrated in FIG. 14 is merely illustrative of an embodiment incorporating the present invention and is not intended to limit the scope of the invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  As shown in FIG. 14, the method begins when the server 104 accesses multimedia information displayed on the GUI (step 1402). As already mentioned, the multimedia information may be stored in a multimedia document accessible by the server 104. As part of step 1402, server 104 may receive information identifying a multimedia document (eg, the filename of the multimedia document) and the location of the multimedia document (eg, a directory path). The user of the present invention may provide multimedia document identification information. In that case, the server 104 can access the multimedia document based on the provided information. Alternatively, the server 104 may receive multimedia information displayed on the GUI in the form of streaming media signals, cable signals, etc. from multimedia information sources. In that case, the server system 104 stores the multimedia information signal as a multimedia document, and then uses the stored document to generate a GUI according to the teachings of the present invention.

  Next, the server 104 extracts text information from the multimedia information accessed in Step 1402 (Step 1404). If the multimedia information accessed in step 1402 includes CC text information, this text information corresponds to the CC text information extracted from the multimedia information. If the multimedia information accessed in step 1402 does not include CC text information, in step 1404, the audio information included in the multimedia information accessed in step 1402 is recorded, and a text recording of the audio information is generated. This text recording represents the text information extracted in step 1404. Further, the text information extracted in step 1404 may be a text representation of other types of information included in the multimedia information.

  The text information determined in step 1404 includes a set of rows where each row has one or more words. Each word is associated with a time stamp indicating the time when the word appeared in the multimedia information. Time stamp information for each word is included in the CC text information. Alternatively, if the text represents a recording of audio information, the time stamp information for each word may be determined during the audio recording process. Alternatively, if the text information represents a text representation of another type of information included in the multimedia information, a time stamp associated with the other type of information may be determined.

  As part of step 1404, each line is assigned a start time and an end time based on the words contained in that line. The start time of a line matches the time stamp associated with the first word appearing on that line, and the end time of a line matches the time stamp associated with the last word appearing on that line.

  The text information determined in step 1404, such as timing information, is then stored in a memory location accessible by the server 104 (step 1406). In one embodiment, a data structure (or memory structure) that includes a linked list of row objects is used to store text information. Each line object has a linked list of words contained in that line. Also, text information associated with words and lines is stored in this data structure. The information stored in this data structure is then used to generate the GUI 300.

Next, the server 104 determines the length or height of a panel (hereinafter referred to as “text canvas”) for drawing the text information in units of pixels (step 1408). In order to determine the length of the text canvas, the duration (“duration”) of the multimedia information (or of the multimedia document storing the multimedia document) is determined in seconds. Also defined is the pixel per second (“pps”) value. This “pps” determines the distance between lines of text drawn on the text canvas. Thus, the value of pps depends on how close the user wants the text lines to be when they are displayed and the size of the font used to display the text. According to one embodiment of the invention, using a 6 point font, a pps value = 5 is specified. The total height in pixels of the text canvas (“textCanvasHeight”) is determined as follows.
textCanvasHeight = duration * pps
For example, if the multimedia information has a duration of 1 hour (ie, 3,600 seconds) and a pps value of 5, the height of the text canvas (textCanvasHeight) is 18,000 pixels (= 3,600). * 5)

A multiplier is then calculated that converts pixel positions in the text canvas into seconds and converts seconds into pixel positions in the text canvas (step 1410). A multiplier “pix_m” is calculated that converts a given time value (in seconds) to a particular pixel location on the text canvas. The multiplier pix_m can be used to determine the pixel location of the text canvas corresponding to a particular time value. The value of pix_m is determined as follows.
pix_m = textCanvasHeight / duration
For example, when duration = 3,600 seconds and text CanvasHeight = 18,000 pixels, pix_m = 18,000 / 3,600 = 5.

A multiplier sec_m is calculated that converts a particular pixel location on the text canvas into a corresponding time value. The multiplier sec_m can be used to determine a time value corresponding to the pixel position of the text canvas. The value of sec_m is determined as follows.
sec_m = duration / textCanvasHeight
For example, when duration = 3,600 seconds and text CanvasHeight = 18,000 pixels, sec_m = 3,600 / 18,000 = 0.2.

  The multiplier calculated in step 1410 can then be used to convert pixels to seconds and seconds to pixels. For example, the pixel position on the text canvas of an event that occurred at time t = 1,256 seconds in the multimedia information is 1,256 * pix_m = 1,256 * 5 = 6,280 pixels from the top of the text canvas. By the way. The number of seconds corresponding to pixel location p = 231 in the text canvas is 231 * sec_m = 231 * 0.2 = 46.2 seconds.

  Based on the height of the text canvas determined in step 1408 and the multiplier generated in step 1410, the position coordinates (horizontal coordinate X and vertical coordinate Y) are calculated for the word of the text information extracted in step 1404. (Step 1412). As described above, information related to words and lines, and time stamps associated with them, can be stored in a data structure accessible to server 104. The position coordinate values calculated for each word may also be stored in this data structure.

The Y (vertical) coordinate (W _y ) for a word is calculated by multiplying the timestamp in seconds (W _t ) associated with that word by the multiplier pix_m determined in step 1410. Therefore,
W _y (pixel unit) = W _t * pix_m
It becomes. For example, if a particular word has W _t = 539 seconds (ie, the word appeared 539 seconds after entering the multimedia information), W _y = 539 * 5 = 2,695 pixels from the top of the text canvas It is.

The X (horizontal) coordinate (W _x ) for a word is calculated based on the in-line position of that word and the width of the previous word in that line. For example, if a particular row (L) has 4 words, ie L: W ₁ W ₂ W ₃ W ₄ ,
W _{x of} W ₁ = 0
W ₂ of the _W x = _{(W 1} of the _W x) + (width of _{W 1)} + (interval of the word)
W ₃ W _x = (W ₂ W _x ) + (W ₂ width) + (word spacing)
W ₄ of _W x = _{(W 3} of _W x) + (width of _{W 3)} + (interval of the word)
The word of text information is then drawn at the position on the text canvas determined by the X and Y coordinates calculated for the word in step 1412 (step 1414).

  Next, the server 104 determines the height of the thumbnail 312-1 for displaying text information in the second display area 304 of the GUI 300 (step 1416). The height of the thumbnail 312-1 depends on the height of the GUI window used to display multimedia information and the height of the second display area 304 in the GUI window. The value of ThumbnailHeight is set so that the thumbnail 312-1 fits the GUI of the second display area 304.

  The thumbnail 312-1 is then generated by reducing the text canvas so that the height of the thumbnail 312-1 is equal to ThumbnailHeight and the thumbnail fits perfectly within the size constraints of the second display area 304. (Step 1418). Next, a thumbnail 312-1 representing a reduced version of the text canvas is displayed in the second display area 304 of the GUI 300 (step 1420).

A multiplier is then calculated that converts the pixel position in thumbnail 312-1 to seconds and converts the seconds to pixel positions in thumbnail 312-1 (step 1422). A multiplier “tpix_m” is calculated that converts a given time value (in seconds) to a particular pixel location in the thumbnail 312-1. The multiplier tpix_m can be used to determine the pixel position in the thumbnail corresponding to a particular time value. The value of tpix_m is determined as follows.
tpix_m = ThumbnailHeight / duration
For example, when duration = 3,600 seconds and ThumbnailHeight = 900, tpix_m = 900/3, 600 = 0.25.

A multiplier “tsec_m” is calculated that converts a particular pixel location in the thumbnail 312-1 to a corresponding time value. The multiplier tsec_m can be used to determine the time value for a particular pixel location in the thumbnail 312-1. The value of tsec_m is determined as follows.
tsec_m = duration / ThumbnailHeight
For example, when duration = 3,600 seconds and ThumbnailHeight = 900, tsec_m = 3,600 / 900 = 4.

  Multipliers tpix_m and tsec_m can then be used to convert pixels to seconds and seconds to pixels in thumbnail 312-1. For example, the pixel position in the thumbnail 312-1 of the word that occurred at time t = 1,256 seconds in the multimedia information is 1,256 * tpix_m = 1,256 * 0.25 = 314 pixels from the top of the thumbnail 312-1. It is the place. The number of seconds represented by pixel position p = 231 in thumbnail 312-1 is 231 * tsec_m = 231 * 4 = 924 seconds.

  FIG. 15 is a simplified high-level diagram illustrating a method of displaying a thumbnail 312-2 showing video keyframes extracted from video information in the second display area 304 of the GUI 300 according to one embodiment of the present invention. Flowchart 1500. The method shown in FIG. 15 may be executed by the server 104, may be executed by the client 102, or may be executed jointly by the server 104 and the client 102. For example, the method may be performed by a software module executing on the server 104 or client 102, may be performed by a hardware module connected to the server 104 or client 102, or a combination thereof But you can. In the embodiment described below, the method is performed by the server 104. The method illustrated in FIG. 15 is merely illustrative of an embodiment incorporating the present invention and is not intended to limit the scope of the invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  For convenience, it is assumed that the thumbnail 312-1 for displaying text information has already been displayed according to the flowchart shown in FIG. As shown in FIG. 15, the server 104 extracts a key frame group from the video information included in the multimedia information (step 1502). These video keyframes can be extracted from the video information by sampling the video information at a specific sampling rate. According to one embodiment of the invention, key frames are extracted from the video information at a sampling rate of 1 frame per second. Accordingly, if the multimedia information has a duration of 1 hour (3,600 seconds), in step 1502, 3,600 video key frames are extracted from the video information. Each key frame extracted in step 1502 is associated with a time stamp indicating the time at which the key frame appears in the multimedia information.

  The video keyframes extracted in step 1502 and the time stamp information associated with them are stored in a data structure (or memory structure) accessible to the server 104 (step 1504). The information stored in this data structure is then used to generate thumbnail 312-2.

  The video keyframes extracted at step 1504 are then grouped (step 1506). A time period (“groupTime”) that can be set by the user is used to group the key frames. According to one embodiment of the present invention, groupTime is set to 8 seconds. In this embodiment, each group has video keyframes extracted within an 8 second time period window. For example, if the duration of the multimedia information is 1 hour (3,600 seconds) and 3,600 video key frames are extracted from the video information using a sampling rate of 1 frame per second, the groupTime is 8 If set to seconds, 3,600 key frames are divided into 450 groups, each group having 8 video key frames.

  For each frame group, a start time and an end time are calculated (step 1508). For a particular frame group, the start time of that particular group is the timestamp associated with the first video keyframe of that group (ie, the keyframe with the oldest timestamp in the group) The end time of that particular group is the time stamp associated with the last video key frame of that group (ie, the key frame with the latest time stamp in the group).

  For each keyframe group, server 104 determines a pixel segment on the keyframe canvas that renders one or more keyframes of the keyframe group (step 1510). Similar to the text canvas, the keyframe canvas is a panel on which keyframes extracted from video information are drawn. The height of the keyframe canvas (“keyframeCanvasHeight”) is the same as the height of the text canvas described above (“textCanvasHeight”) (ie, keyframeCanvasHeight = textCanvasHeight). As a result, the multipliers pix_m and sec_m (described above) can be used to convert time values to pixel positions on the keyframe canvas and specific pixel positions on the keyframe canvas to time values.

Pixel segments that draw keyframes from a particular group on the keyframe canvas are calculated based on the start and end times associated with that particular group. The starting vertical (Y) pixel coordinates (“segmentStart”) and ending vertical (Y) coordinates (“segmentEnd”) of a pixel segment for a particular keyframe group in the keyframe canvas are calculated as follows: The
segmentStart = (start time of group) * pix_m
segmentEnd = (group end time) * pix_m
Thus, the height in pixels of each segment of the text canvas (“segmentHeight”) is
segmentHeight = segmentEnd-segmentstart
It becomes.

  Next, the number of key frames from each frame group to be drawn in each pixel segment on the text canvas is determined (step 1512). The number of keyframes drawn on a keyframe canvas for a particular group depends on the height of the segment corresponding to that particular group (“segmentHeight”). When the value of segmentHeight is small, only a small number of key frames may be drawn on the segment so that the user can easily understand the key frames drawn when displayed on the GUI. The value of segmentHeight depends on the value of pps. When pps is small, segmentHeight is also small. Therefore, as the number of key frames drawn per segment increases, a larger value of pps is selected.

  According to one embodiment of the present invention, if segmentHeight is equal to 40 pixels and each keyframe group contains 8 keyframes, 6 of those 8 keyframes are drawn on each segment on the text canvas. Is done. The number of key frames drawn in a segment is generally the same for all key frame groups. For example, in the embodiment shown in FIG. 3, six key frames are drawn for each segment on the text canvas.

  Once the number of keyframes to be drawn on each segment of the text canvas is determined, for each keyframe group, the server 104 can generate one or more drawn on the keyframe canvas from the keyframe group keyframes. A key frame is specified (step 1514). A variety of different techniques can be used to select video keyframes that are displayed in a segment for a particular frame group. According to one approach, if each video keyframe group has 8 keyframes and 6 video keyframes are displayed for each segment on the keyframe canvas, the server 104 Select the first two video keyframes, the middle two video keyframes, and the last two video keyframes from the keyframe group as drawn on the keyframe canvas. As described above, various other techniques can also be used to select one or more key frames to be displayed from a key frame group. For example, the key frame may be selected based on the sequential position of the key frame in the key frame group, may be selected based on the time value associated with the key frame, or based on other criteria. It may be selected.

  According to another approach, the server 104 may determine the similarity or dissimilarity between key frames in each key frame group using special image processing techniques. If six video key frames are displayed from each group, the server 104 selects six key frames from each key frame group based on the results of the image processing technique. According to one embodiment of the present invention, the six least similar key frames of each group are selected for rendering on the key frame canvas. Obviously, various other techniques known to those skilled in the art can also be used to perform the selection of video keyframes.

  The key frames identified from the key frame group in step 1514 are drawn into their corresponding segments on the key frame canvas (step 1516). A variety of different formats can be used to draw the selected keyframe on a particular segment. For example, as shown in FIG. 3, for each segment, the selected key frames are arranged from left to right and from top to bottom, with 3 frames as one line. Various other techniques known to those skilled in the art for drawing key frames on a key frame canvas can also be used. The size of each individual key frame drawn on the key frame canvas depends on the height of the segment on which the key frame is drawn (segmentHeight) and the number of key frames drawn on the segment. As described above, the height of one segment depends on the value of pps. Therefore, the size of each individual key frame drawn on the key frame canvas also depends on the value of pps.

  The server 104 then determines the height (or length) of the thumbnail 312-2 that displays the video keyframe on the GUI 300 (step 1518). According to the teachings of the present invention, the height of the thumbnail 312-2 is set to be the same as the height of the thumbnail 312-1 that displays the text information (i.e., the height of the thumbnail 312-2 is set to ThumbHeight). )

  The thumbnail 312 is then reduced by reducing the keyframe canvas so that the height of the thumbnail 312-2 is equal to ThumbnailHeight and the thumbnail 312-2 fits perfectly within the size constraints of the second display area 304. -2 is generated (step 1520). A thumbnail 312-2 representing a reduced version of the keyframe canvas is then displayed in the second display area 304 of the GUI 300 (step 1522). The thumbnail 312-2 is displayed in the GUI 300 next to the thumbnail image 312-1 and arranged in time series or synchronized with the thumbnail 312-1 (as shown in FIG. 3). Therefore, the upper end of the thumbnail 312-2 is aligned with the upper end of the thumbnail 312-1.

  A multiplier is calculated for the thumbnail 312-2 that converts the pixel location in the thumbnail 312-2 to seconds and converts the seconds to pixel locations in the thumbnail 312-2 (step 1524). Since the thumbnail 312-2 has the same length as the thumbnail 312-1 and is aligned with the thumbnail 312-1, the multipliers “tpix_m” and “tsec_m” calculated for the thumbnail 312-1 are the thumbnail 312-2. Can also be used. These multipliers can then be used to convert pixels to seconds and seconds to pixels in thumbnail 312-2.

  According to the method displayed in FIG. 15, the size of each individual video keyframe displayed in thumbnail 312-2 is the length of thumbnail 312-2 and the length of video information, in addition to other criteria. Also depends on. Assuming that the length of the thumbnail 312-2 is fixed, the height of each individual video keyframe displayed on the thumbnail 312-2 is inversely proportional to the length of the video information. Therefore, as the length of the video information increases, the size of each key frame displayed in the thumbnail 312-2 decreases. As a result, the longer the multimedia document, the smaller the size of each keyframe can be so that the video keyframe displayed in thumbnail 312-2 is no longer visible to the user. To avoid this, various techniques for displaying video keyframes on thumbnail 312-2 can be used to make thumbnail 312-2 more readable and recognizable to the user.

  FIG. 16 is a simplified high-level flowchart 1600 illustrating another method for displaying thumbnail 312-2 according to one embodiment of the present invention. The method shown in FIG. 16 reduces the number of video keyframes drawn on the keyframe canvas and displayed in the thumbnail 312-2, thereby making the information displayed in the thumbnail 312-2 easier to understand and use. Maintain ease. The method shown in FIG. 16 may be executed by the server 104, may be executed by the client 102, or may be executed jointly by the server 104 and the client 102. For example, the method may be performed by a software module executing on the server 104 or client 102, may be performed by a hardware module connected to the server 104 or client 102, or a combination thereof But you can. In the embodiment described below, the method is performed by the server 104. The method shown in FIG. 16 is merely illustrative of an embodiment incorporating the present invention and is not intended to limit the scope of the invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  As shown in FIG. 16, steps 1602, 1604, 1606, and 1608 are the same as the above-described steps 1502, 1504, 1506, and 1508 shown in FIG. After step 1608, one or more groups whose video keyframes are drawn on the keyframe canvas are selected from the groups determined in step 1606 (step 1609). A variety of different techniques can be used to select groups in step 1609. According to one approach, the group determined in step 1606 is selected based on a “SkipCount” value that can be set by the user. For example, if SkipCount is set to 4, every 4 groups are selected in step 1609 (ie, 4 groups are skipped). The value of SkipCount can be adjusted based on the length of the multimedia information. According to one embodiment of the present invention, the value of SkipCount is directly proportional to the length of the multimedia information. That is, SkipCount is set to a higher value as the multimedia document is longer.

  For each group selected in step 1609, the server 104 identifies one or more key frames to be drawn on the key frame canvas from that group (step 1610). As described above, various methods can be used to select a key frame to be drawn on the key frame canvas.

  The keyframe canvas is then divided into a plurality of equally sized row portions. The number of row parts is equal to the number of groups selected in step 1609 (step 1612). According to one embodiment of the present invention, the height of each row portion is approximately equal to the height of the keyframe canvas divided by the number of groups selected in step 1609 (“keyframeCanvasHeight”).

  Next, for each group selected in step 1609, the row portion of the keyframe canvas that renders one or more video keyframes from that group is identified (step 1614). According to one embodiment of the invention, row portions are associated with groups in chronological order. For example, the first row is associated with the group with the oldest start time, the second row is associated with the group with the second oldest start time, and so on.

  Then, for each group selected in step 1609, one or more keyframes from that group (identified in step 1610) are placed in the row portion on the keyframe canvas determined for that group in step 1614. Drawing is performed (step 1616). The size of the key frame selected for each group is reduced to fit the row portion of the key frame canvas. According to one embodiment of the present invention, the height of each row portion is greater than the height of the selected key frame, and the height of the selected key frame is increased to fit the row portion. This increases the size of the selected keyframes, making them easier to see when drawn on the keyframe canvas. In this way, key frames from the group selected in step 1609 are drawn on the key frame canvas.

  The keyframe canvas is then reduced according to steps 1618, 1620, and 1622 to form a thumbnail 312-2 that is displayed in the second display area 304. Since the height of the key frame drawn on the key frame canvas is increased according to an embodiment of the present invention, as described above, the key frame is also more easily recognized when displayed on the thumbnail 312-2. Become. The multiplier is then calculated according to step 1624. Steps 1618, 1620, 1622, and 1624 are similar to steps 1518, 1520, 1522, and 1524 described above shown in FIG. As described above, selecting a portion of a group reduces the number of keyframes that are drawn on the keyframe canvas and displayed in thumbnail 312-2. This increases the height of each individual video keyframe displayed in thumbnail 312-2, making it easier to recognize when they are displayed.

  FIG. 17 shows a thumbnail display area lens 314 according to an embodiment of the present invention, information displayed by the thumbnail display area lens 314 is displayed in the third display area 306, and a panel display area lens 322 is displayed. In a simplified high-level flowchart 1700 showing how to display the information highlighted by the panel display area lens 322 in the fourth display area 308 and display the information in the fifth display area 310. is there. The method illustrated in FIG. 17 may be executed by the server 104, may be executed by the client 102, or may be executed jointly by the server 104 and the client 102. For example, the method may be performed by a software module executing on the server 104 or client 102, may be performed by a hardware module connected to the server 104 or client 102, or a combination thereof But you can. In the embodiment described below, the method is performed by the server 104. The method illustrated in FIG. 17 is merely illustrative of an embodiment incorporating the present invention and is not intended to limit the scope of the invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  As shown in FIG. 17, the server 104 first determines the height (“PanelHeight”) of each panel displayed in the third display area 306 of the GUI 300 (step 1702). The value of PanelHeight depends on the height (or length) of the third display area 306. Since the panels are aligned with each other, the height of each panel is set to PanelHeight. According to one embodiment of the present invention, PanelHeight is set to the same value as ThumbnailHeight. However, as an alternative embodiment of the present invention, the value of PanelHeight may be different from the value of ThumbnailHeight.

Next, a section of the text canvas (generated in the flowchart shown in FIG. 14) equal to PanelHeight is identified (step 1704). The section of the text canvas identified in this step 1704 is determined by the vertical pixel coordinate (P _start ) marking the start pixel position of the section and the vertical pixel coordinate (P _end ) marking the end pixel position of the section. Characterized.

Next, a time value corresponding to the border (marked by pixel locations P _start and P _end ) of the section of the text canvas identified at step 1704 is determined (step 1706). The multiplier sec_m is used to calculate this corresponding time value. The time t ₁ (seconds) corresponding to the pixel position P _start is calculated as follows.
t ₁ = P _start * sec_m
The time t ₂ (seconds) corresponding to the pixel position P _end is calculated as follows.
t ₂ = P _end * sec_m
Next, a section corresponding to the selected section of the text canvas among the keyframe canvas is identified (step 1708). Since the height of the keyframe canvas is the same as the height of the keyframe canvas, the selected section of the keyframe canvas also has a pixel position P _start in the keyframe canvas corresponding to time t ₁ and time t _2. And _Pend .

  The part specified in step 1704 of the text canvas is displayed on the panel 324-1 of the third display area 306 (step 1710). The portion of the key frame canvas specified in step 1708 is displayed on the panel 324-2 of the third display area 306 (step 1712).

A panel display area / lens 322 covering a part of the third display area 306 is displayed (step 1714). Panel Display Area lenses 322, time _{t 3} and _{t 4 (t 1 ≦ t 3} <t 4 ≦ t 2) one third panel is displayed on the display area 306 of between 324 - and 324-2 The part is displayed so as to emphasize or cover the part. The upper end of the panel display area lens 322 corresponds to the time t _3, the lower end of the panel display area lens 322 corresponding to the time t _4. The height (in pixels) of the panel display area lens 322 is equal to (vertical pixel position corresponding to t ₄ in the text canvas) − (vertical pixel position corresponding to t ₃ in the text canvas). . The width of the panel display area lens 322 is approximately equal to the width of the third display area 306 (as shown in FIG. 3).

Next, a part of the thumbnail 312-1 corresponding to a part of the text canvas displayed on the panel 324-1 and a part of the key frame canvas displayed on the panel 324-2 of the thumbnail 312-2. The corresponding part is determined (step 1716). A portion of the thumbnail 312-1 corresponding to a part of the text canvas displayed on the panel 324-1 includes a vertical pixel coordinate (TN _start ) that marks the start pixel position of a part of the thumbnail, and the thumbnail of the thumbnail. Characterized by a vertical pixel location (TN _end ) marking some end pixel locations. The multiplier tpix_m is used to determine the pixel positions TN _start and TN _end .
TN _start = t ₁ * tpix_m
TN _end = t ₂ * tpix_m
Since the thumbnails 312-1 and 312-2 are the same length and are arranged in time series with each other, the portion of the thumbnail 312-2 corresponding to a part of the key frame canvas displayed on the panel 324-2 Is also located between the pixel position TN _start and TN _end on the thumbnail 312-2.

Next, in the thumbnails 312-1 and 312-2, a part corresponding to a part of the text canvas displayed on the panel 324-1 and a part of the key frame canvas displayed on the panel 324-2. A thumbnail display area / lens 314 covering the corresponding portion is displayed (step 1718). A thumbnail display area lens 314 that covers a portion of the thumbnails 312-1 and 312-2 between the thumbnail pixel positions TN _start and TN _end is displayed. The height of the thumbnail display area lens 314 in pixels is equal to (TN _end −TN _start ). The width of the thumbnail display area lens 314 is approximately equal to the width of the second display area 306 (as shown in FIG. 3).

Next, a portion of the second display area 304 corresponding to a part of the third display area 306 highlighted by the panel display area / lens 322 is determined (step 1720). In step 1720, server 104 determines a portion of the part of the thumbnail 31201 corresponding to the time period between _{t 3} and _{t 4} and thumbnail 312-2. The portion corresponding to the time window between _{t 3} and _{t 4} of the thumbnail 312-1, corresponds to the time _{t 3,} the vertical pixel coordinate of marking a portion of the starting vertical pixel position of the thumbnail _{(TNSub start)} , Corresponding to time t ₄ and characterized by a vertical pixel position (TNSub _end ) marking the end pixel position of a portion of that thumbnail. The multiplier tpix_m is used to determine the pixel location TNSub _start and TNSub _end .
TNSub _start = t ₃ * tpix_m
TNSub _end = t ₄ * tpix_m
Thumbnail 312-1 and 312-2, the same length, because it is arranged in time series to each other, also the part corresponding to the time period between _{t 3} and _{t 4} of the thumbnail 312-2 thumbnail 312-2 It is located between the upper pixel positions TNsub _start and TNSub _end .

Then, corresponding to the time window between _{t 3} and _{t 4} of the thumbnail 312-1 and 312-2 (i.e., corresponding to the highlighted portion by a panel display area lens 322 of the third display area 306 ) A sub-lens 316 covering the portion is displayed (step 1722). The sub lens 316 is displayed so as to cover a portion between the pixel positions TNSub _start and TNSub _end of the thumbnails 312-1 and 312-2. Height in pixels of the sub-lens 316 _is equal to (TNSub end _{-TNSub start).} The width of the sub lens 316 is substantially equal to the width of the second display area 304 (as shown in FIG. 3).

The multimedia information corresponding to the portion highlighted by the panel display area / lens 322 in the third display area 306 is displayed in the fourth display area 308 (step 1724). For example, video information starting at time t ₃ is reproduced in the area 340-1 of the fourth display area 308 of the GUI 300. As an alternative embodiment, the start time of the video playback may be set to any time between t ₃ and t ₄ including t ₃ and t _4. A text gun corresponding to the time window between t ₃ and t ₄ is displayed in area 340-2 of the fourth display area 308.

  Next, the multimedia information is analyzed, and the result of the analysis is displayed in the fifth display area 310 (step 1726). For example, text information extracted from the multimedia information is analyzed to identify the frequencies of words and individual words that are transferred to the text information. These words and their frequencies may be printed in the fifth display area 310 (for example, information printed in the area 352 of the fifth display area 310 as shown in FIG. 3). As described above, information extracted from the multimedia information can be stored in a data structure accessible to the server 104. For example, text information and video keyframe information extracted from multimedia information may be stored in one or more data structures in a data structure accessible by the server 104. The server 104 can analyze the multimedia information using the information stored in these data structures.

(Multimedia information navigation)
As described above, the user of the present invention can navigate and scroll through the multimedia information stored by the multimedia document and displayed on the GUI 300 using the thumbnail display area lens 314 and the panel display area lens 322. it can. For example, the user can change the position of the thumbnail display area lens 314 by moving the thumbnail display area lens 314 along the length of the second display area 304. When the position of the thumbnail display area / lens 314 is changed from the first position of the second display area 304 to the second position along the second display area 304, the multimedia displayed in the third display area 306 is displayed. The information continues to correspond to the area where the multimedia information displayed in the third display area 306 is highlighted by the thumbnail display area lens 314 in the second position of the second display area 304. Updated automatically. Similarly, the user can change the position of the panel display area lens 322 by moving the panel display area lens 322 along the length of the third display area 306. When the position of the panel display area / lens 322 is changed, the position of the sub-lens 316 and possibly the position of the thumbnail display area / lens 314 are also updated so as to continue to correspond to the new position of the panel display area / lens 322. The information displayed in the fourth display area 308 is also updated to correspond to the new position of the panel display area / lens 322.

  FIG. 18 is a simplified diagram illustrating a method for automatically updating information displayed in the third display area 306 in response to a change in the position of the thumbnail display area lens 314 according to one embodiment of the present invention. High level flowchart 1800. The method shown in FIG. 18 may be executed by the server 104, may be executed by the client 102, or may be executed jointly by the server 104 and the client 102. For example, the method may be performed by a software module executing on the server 104 or client 102, may be performed by a hardware module connected to the server 104 or client 102, or a combination thereof But you can. In the embodiment described below, the method is performed by the server 104. The method shown in FIG. 18 is merely illustrative of an embodiment incorporating the present invention and does not limit the scope of the invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

As shown in FIG. 18, when the server 104 detects a change in the position of the thumbnail display area / lens 314 from the first position on the second display area 304 to the second position, It starts (step 1802). Next, the server 104 determines a portion of the second display area 304 that is highlighted by the thumbnail display area lens 314 at the second position (step 1804). As part of step 1804, the server 104 determines in the thumbnail 312-1 the pixel location (TN _start and TN _end ) corresponding to the end of the thumbnail display area lens 314 at the second location. TN _start marks the starting vertical pixel position in the thumbnail 312-1 and TN _end marks the ending vertical pixel position in the thumbnail 312-1. Since the thumbnails 312-1 and 312-2 have the same length and are arranged in time series, the portion of the thumbnail 312-2 corresponding to the second position of the thumbnail display area lens 314 is also the pixel position TN. Located between _start and TN _end .

The server 104 then determines a time value corresponding to the second position of the thumbnail display area lens 314 (step 1806). A time value t ₁ corresponding to the pixel position TN _start is determined, and a time value t ₂ corresponding to the pixel position TN _end is determined. The multiplier tsec_m is used to determine these time values.
t ₁ = TN _start * tsec_m
t ₂ = TN _end * tsec_m
Server 104 then determines the pixel location corresponding to the time value determined in step 1806 in the text canvas and keyframe canvas (step 1808). The pixel position P _start in the text canvas is determined based on time t ₁ , and the pixel position TN _end in the keyframe canvas is determined based on time t ₂ . A multiplier pix_m is used to determine these pixel positions.
P _start = t ₁ * tpix_m
P _end = t ₂ * tpix_m
Since the text canvas and the keyframe canvas are the same length, the time values t ₁ and t ₂ correspond to the pixel positions P _start and P _end in the key frame.

A section of the text canvas between pixel positions P _start and P _end is displayed on panel 324-1 (step 1810). The section of the text canvas displayed on panel 324-1 corresponds to the portion of thumbnail 312-1 that is highlighted by thumbnail display area lens 314 at the second position.

Section between the pixel position _{P start} and _{P end The} one keyframe canvas is displayed on the panel 324-2 (step 1812). The section of the key frame canvas displayed on the panel 324-2 corresponds to the portion of the thumbnail 312-2 that is highlighted by the thumbnail display area lens 314 at the second position.

When the thumbnail display area lens 314 is moved from the first position to the second position, the sub lens 316 also moves along the thumbnail display area lens 314. At that time, the server 104 determines a portion emphasized by the sub lens 316 in the second position in the second display area 304 (step 1814). As part of step 1814, the server 104 determines a pixel location (TNSub _start and TNSub _end ) corresponding to the end of the sub-lens 316 at the second location in the thumbnail 312-1. TNSub _start marks the starting vertical pixel position in thumbnail 312-1 and TNSub _end marks the ending vertical pixel position in thumbnail 312-1. Since the thumbnails 312-1 and 312-2 have the same length and are arranged in time series with each other, the portion of the thumbnail 312-1 corresponding to the second position of the sub lens 316 also has pixel positions TNSub _start and TNSub _end. Located between.

Next, the server 104 determines a time value corresponding to the second position of the sub-lens 316 (step 1816). A time value t ₃ corresponding to the pixel position TNSub _start is determined, and a time value t ₄ corresponding to the pixel position TNSub _end is determined. The multiplier tsec_m is used to determine these time values.
t ₃ = TNSub _start * tsec_m
t ₄ = TNSub _end * tsec_m
Server 104 then determines pixel locations corresponding to the time values determined in step 1816 in the text canvas and keyframe canvas, respectively (step 1818). The pixel position Psub _start in the text canvas is calculated based on time t ₃ , and the pixel position Psub _end in the keyframe canvas is calculated based on time t ₄ . The multiplier pix_m is used to determine these positions as follows.
Psub _start = t ₃ * tpix_m
Psub _end = t ₄ * tpix_m
Since the text canvas and the keyframe canvas are the same length, the time values t ₁ and t ₂ correspond to the pixel positions Psub _start and Psub _end in the key frame canvas.

A panel display area / lens 322 covering a portion of the third display area 306 between the pixel positions Psub _start and Psub _end is drawn on the third display area 306 (step 1820). The multimedia information displayed in the fourth display area 308 is then updated to correspond to the new position of the panel display area lens 322 (step 1822).

  FIG. 19 shows the information displayed in the fourth display area 308 and the positions of the thumbnail display area / lens 314 and the sub-lens 316 according to the change of the position of the panel display area / lens 322 according to the embodiment of the present invention. 6 is a simplified high-level flowchart 1900 illustrating a method for automatically updating. The method shown in FIG. 19 may be executed by the server 104, may be executed by the client 102, or may be executed jointly by the server 104 and the client 102. For example, the method may be performed by a software module executing on the server 104 or client 102, may be performed by a hardware module connected to the server 104 or client 102, or a combination thereof But you can. In the embodiment described below, the method is performed by the server 104. The method illustrated in FIG. 19 is merely illustrative of an embodiment incorporating the present invention and is not intended to limit the scope of the invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

As shown in FIG. 19, when the server 104 detects a change in the position of the panel display area / lens 322 on the third display area 306 from the first position to the second position, It is started (step 1902). Next, the server 104 determines a time value corresponding to the panel display area / lens 322 at the second position (step 1904). In step 1904, the server 104 determines the pixel positions of the upper and lower ends of the panel display area lens 322 at the second position. These pixel positions are then converted into time values using a multiplier sec_m. A time value t ₃ corresponding to the upper end of the panel display area / lens 322 in the second position is determined, and a time value t ₄ corresponding to the lower end of the panel display area / lens 322 is determined.
t ₃ = (panel display area / pixel position of the upper end of the lens 322) * sec_m
t ₄ = (panel display area / pixel position of the lower end of the lens 322) * sec_m
Next, the server 104 determines a pixel position corresponding to the time value determined in Step 1904 in the second display area 304 (Step 1906). (Either 312-1 or 312-2; since they because it is aligned with the same length) the second thumbnail of the display area 304 pixel locations _{TNSub start} of is determined on the basis of the time _{t 3,} thumbnails pixel position _{TNSub end} is determined on the basis of the time _{t 4.} Using the multiplier tpix_m, these pixel positions are determined as follows.
TNSub _start = t ₃ * tpix_m
TNSub _end = t ₄ * tpix_m
The sub-lens 316 is then updated to highlight the portion of the thumbnail 312 of the second display area 304 between the pixel locations determined in step 1906 (step 1908). As part of step 1908, if the pixel positions TNSub _start and TNSub _end are outside the border of the lens 314 when the panel display area lens 322 is in the first position, the thumbnail display area The position of the lens 314 may also be updated. For example, when the user scrolls the third display area 306 beyond the PanelHeight using the panel display area / lens 322, the position of the thumbnail display area / lens 314 is updated accordingly. When the second position of the panel display area / lens 322 is located within the panel height, only the sub lens 316 is moved so as to correspond to the second position of the panel display area / lens 322, and the thumbnail display area / lens 314 is moved. Not moved.

  As described above, the information displayed in the third display area 306 can be scrolled using the panel display area / lens 322. For example, the user can move the panel display area / lens 322 to the lower end of the third display area 306 so that the contents of the third display area 306 are automatically scrolled upward. Similarly, the user can move the panel display area / lens 322 to the upper end of the third display area 306 so that the contents of the third display area 306 are automatically scrolled downward. The positions of the thumbnail display area / lens 314 and the sub-lens 316 are updated according to the occurrence of scrolling.

  Next, multimedia information corresponding to the second position of the panel display area / lens 322 is displayed in the fourth display area 308 (step 1910). For example, video information corresponding to the second position of the panel display area / lens 322 is displayed in the area 340-1 of the fourth display area 308, and text information corresponding to the second position of the panel display area / lens 322 is displayed. Is displayed in the area 340-2 of the third display area 306.

According to one embodiment of the present invention, in step 1910, server 104 selects a time “t” having a value equal to t ₃ or t ₄ or a value between t ₃ and t ₄ . Time “t” may be referred to as “location time”. This location time may be settable by the user. According to an embodiment of the present invention, location-time is set to t _4. Next, this location time is used as the start time of video information reproduction in the area 340-1 of the fourth display area 308.

  According to one embodiment of the present invention, the GUI 300 can operate in two modes: a “full update” mode and a “partial update” mode. The user of the GUI can select the operation mode of the GUI.

  When the GUI 300 is operating in the “full update” mode, the positions of the thumbnail display area / lens 314 and the panel display area / lens 322 reflect the position of video playback in the area 340-1 of the fourth display area 308. So that it is automatically updated. Accordingly, in the “full update” mode, the thumbnail display area lens 314 and the panel display area lens 322 hold or reflect the position of the video played in the fourth display area 308. This video can be played back or played back using the controls shown in area 342 of the fourth display area 308, with the thumbnail display area / lens 314 and panel display area / lens 322 positions accordingly. Change. Also, the multimedia information displayed on the panel 324 of the third display area 306 is automatically updated so as to reflect the current position of the video corresponding to the position of the thumbnail display area / lens 314 (upward). Shifted to).

  When the GUI 300 is operating in the “partial update” mode, the positions of the thumbnail display area / lens 314 and the panel display area / lens 322 reflect the position of video playback in the area 340-1 of the fourth display area 308. Will not be updated. In this mode, the positions of the thumbnail display area / lens 314 and the panel display area / lens 322 remain stationary when the video is played in the area 340-1 of the fourth display area 308. Since the position of the thumbnail display area / lens 314 does not change, the multimedia information displayed in the third display area 306 is not updated. In this mode, a “location pointer” for reflecting the current video playback position in the area 340-1 of the fourth display area 308 is displayed in the second display area 304 and the third display area 306. Also good. The location pointer location is continuously updated to reflect the video location.

(range)
According to an embodiment of the present invention, the present invention provides a technique for selecting or designating a portion of multimedia information displayed on a GUI. Each part is called a “range”. The range may be manually specified by the user of the present invention, or alternatively may be automatically selected by the present invention based on the range criteria provided by the user of the present invention.

The range refers to a portion of the multimedia information between the start time (R _S ) and the end time (R _E ). Thus, each range is characterized by R _S and R _E that define the temporal boundaries of that range. A range includes or identifies the portion of multimedia information that occurred between times R _S and R _E associated with that range.

  FIG. 20A illustrates a simplified user interface 2000 that displays a range according to one embodiment of the present invention. It will be appreciated that the GUI 2000 shown in FIG. 20A is merely illustrative of one embodiment that incorporates the present invention and does not limit the scope of the claimed invention. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  As shown in FIG. 20A, the GUI 2000 has various functions (buttons, tabs, etc.) that allow the user to manually specify one or more ranges or set the GUI 2000 to automatically generate ranges. )I will provide a. In the embodiment shown in FIG. 20A, the user can manually specify the range by selecting the “New” button 2002. After selecting the button 2002, the user can specify a range by selecting a portion displayed in the second display area 2004 of the thumbnail. By selecting different parts of the thumbnail, one or more ranges can be specified. For example, six ranges 2006-1, 2006-2, 2006-3, 2006-4, 2006-5, and 2006-6 are displayed in FIG. 20A. The user can manually specify one or more of these ranges by selecting or marking a portion of thumbnail 2008-2.

  As an alternative embodiment, instead of selecting a portion of the thumbnail, the user can specify a range by clicking on a location in the thumbnail. The range is then automatically generated by adding a predetermined buffer time before and after the currently clicked position. Thus, the range can be specified with a single click. Using this technique, a plurality of ranges can be specified.

In FIG. 20A, each designated range is indicated by a bar displayed on thumbnail 2008-2. Also, an identifier or label that uniquely identifies the range may be associated with each range. In FIG. 20A, each range is identified by a number associated with the range displayed at the top left of the range. These numbers serve as labels for the ranges. Thus, information stored for a range may include the start time (R _S ) of the range, the end time (R _E ) of the range, and a label or identifier that identifies the range. Information identifying a multimedia document storing information corresponding to the range may also be stored for the range.

Each range designated by selecting a part of the thumbnail 2008-2 has an upper end (R _top ) and a lower end (R _bottom ) as boundaries. The R _S and R _E times for a range may be determined from the pixel locations of R _top and R _bottom as follows:
R _S = R _top * tsec_m
R _E = R _bottom * tsec_m
Obviously, other techniques can be used to specify the range. For example, as an alternative embodiment of the present invention, a user may specify a range by providing a start time (R _S ) and end time (R _E ) of the range.

In the GUI 2000 shown in FIG. 20A, information related to the range displayed on the GUI is displayed in the area 2010. Information displayed for each range in area 2010 includes a label or identifier 2012 that identifies the range, a start time (R _S ) for the range, an end time (R _E ) for the range, a time span 2018 for the range, And a video key frame group 2019 extracted from a portion of the multimedia information associated with the range. The time span for a range is calculated by determining the difference between the end time R _E and the start time R _S associated with that range (ie, time span for a range = R _E −R _S is there). In the embodiment shown in FIG. 20A, the first, last, and middle keyframes extracted from the multimedia information corresponding to each range are displayed. Various other techniques can also be used to select the keyframes displayed for a range. The information shown in FIG. 20A is not intended to limit the scope of the present invention. As alternative embodiments of the present invention, various other types of information about the range can also be displayed.

According to the teachings of the present invention, various operations can be performed on the range displayed on the GUI 2000. The user can edit the range by changing the R _S and R _E times associated with the range. Editing a range can change the time span of that range (ie, the value of (R _E −R _S )). In the GUI 2000 shown in FIG. 20A, the user can correct or edit the displayed range by selecting an “Edit” button 2020. After selecting the “Edit” button 2020, the user can edit that range by dragging the top and / or bottom of the bar representing the particular range. When the upper end position is changed, the start time (R _S ) of the range is corrected, and when the lower end position is changed, the end time (R _E ) of the range is corrected.

The user can also edit the range by selecting the range in area 2010 and then selecting the “Edit” button 2020. In this scenario, by selecting the “Edit” button 2020, a dialog box (for example, the dialog box 2050 shown in FIG. 20B) is displayed to the user. Then, the user can be changed by inputting each value of R _S and R _E value associated with the selected range in the field 2052 and 2054. The time span of the selected range is displayed in area 2056 of the dialog box.

In addition, the user can move the position of the displayed range by changing the position of the displayed range along the thumbnail 2008-2. Moving a range changes R _S and R _E associated with that range, but maintains the time span of that range. In the GUI 2000, the user can move the range by first selecting the “Move” button 2022, and then selecting and moving the range. As described above, the time span for a range can be edited by selecting the “Edit” button and then dragging the end of the bar representing that range.

  The user can delete or delete a previously specified range. In the GUI 2000 shown in FIG. 20A, the user can delete the displayed range by selecting the “Remove” button 2024 and then selecting the range to be deleted. When the “Clear” button 2026 is selected, all ranges designated for the multimedia information displayed on the GUI 2000 are deleted.

As described above, each range refers to a portion of multimedia information that occurs between times R _S and R _E associated with the range. Multimedia information corresponding to a certain range is output to the user by selecting a “Play” button 2028. After selecting the “Play” button 2028, the user selects a specific range that has multimedia information to be displayed on the GUI 2000 and to be output to the user. Then, the portion corresponding to the selected range in the multimedia information is output to the user. Various different techniques known to those skilled in the art can be used to output multimedia information to the user. According to one embodiment of the present invention, video information corresponding to multimedia information associated with the selected range is played to the user in area 2030. Text information corresponding to the selected range may be displayed in area 2032. The position of the thumbnail display area / lens 314 and panel display area / lens 322 and the information displayed in the third display area 306 correspond to the selected range having information output to the user in area 2030. Automatically updated.

  In addition, the user can reproduce information corresponding to the selected range by selecting a range in the area 2010 and then selecting a “Pally” button 2020. Then, multimedia information corresponding to the selected range is displayed in area 2030.

  The user also instructs the GUI 2000 to sequentially output information associated with all ranges specified for the multimedia information displayed by the GUI 2000 by selecting a “Preview” button 2034. You can also When the “Preview” button 2034 is selected, multimedia information corresponding to the displayed range is sequentially output to the user. For example, when six ranges are displayed as shown in FIG. 20A, multimedia information corresponding to the range identified by the label “1” is output first, and then identified by the label “2”. The multimedia information corresponding to the range continues until this multimedia information corresponding to all six ranges is output to the user. The order in which the ranges are output to the user may be settable by the user.

  Also, multimedia information associated with a range can be stored in memory. For example, in the embodiment shown in FIG. 20A, the user can select a “Save” button 2036 and then select one or more ranges to save. Then, the multimedia information corresponding to the range selected to be saved by the user is saved in a memory (for example, a hard disk, a storage unit, a floppy disk, etc.).

  Various other operations on the range can also be performed. For example, according to an embodiment of the present invention, multimedia information corresponding to one or more ranges can be printed on a paper medium. Details describing a technique for printing multimedia information on paper media have been filed on November 19, 2001, the entire contents of which are hereby incorporated by reference, US patent application Ser. No. 10 / 001,895 (lawyer). (Reference number: 15358-006500 US).

  Also, multimedia information associated with a certain range can be transmitted to a user-specified recipient. For example, a user can select a particular range and request that multimedia information corresponding to that range be communicated to a user-specified recipient. The multimedia information corresponding to the range is then transmitted to the recipient. Various communication techniques known to those skilled in the art (for example, communication techniques such as facsimile, e-mail, wireless communication, etc.) can be used to transmit the range information to the recipient.

  In addition, multimedia information corresponding to a certain range may be provided as input to another application program such as a search program, a browser, a graphic application, a MIDI application, or the like. The user can select a specific range and then specify the application to which the information is to be provided. In response to the user's selection, multimedia information corresponding to the range is then provided as input to the application.

  As described above, the range may be manually specified by the user or automatically selected by the present invention. This automatic selection of the range may be performed by a software module executing on the server 104, may be performed by a hardware module connected to the server 104, or a combination thereof. FIG. 21 is a simplified high-level flowchart 2100 illustrating a method for automatically creating a range in accordance with one embodiment of the present invention. The method illustrated in FIG. 21 may be executed by the server 104, may be executed by the client 102, or may be executed jointly by the server 104 and the client 102. For example, the method may be performed by a software module executing on the server 104 or client 102, may be performed by a hardware module connected to the server 104 or client 102, or a combination thereof But you can. In the embodiment described below, the method is performed by the server 104. The method illustrated in FIG. 21 is merely illustrative of an embodiment incorporating the present invention and does not limit the scope of the invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

As shown in FIG. 21, the method begins when the server 104 receives a criteria for creating a range (step 2102). The user of the present invention can specify criteria through the GUI 2000. For example, in GUI 2000 shown in FIG. 20A, area 2040 displays various options that can be selected to specify the criteria for automatic range creation. In the GUI 2000 shown in FIG. 20A, the user can select “topic” or “word” as the range reference. When the user selects “topic”, information related to the topic that the user is interested in (displayed in area 2042) is identified as a range creation criterion. When the user selects “word”, one or more words selected by the user in the area 2044 of the GUI 2000 are used as a reference for automatically creating a range. As an alternative embodiment, the criteria for automatically creating a range may be stored in a memory location accessible to the server 104. For example, the reference information may be stored in a file accessible by the server 104. Various other types of criteria can also be specified in accordance with the teachings of the present invention. The multimedia information stored in the multimedia document was then analyzed and received in step 2102 within the multimedia information. A position satisfying the criterion (referred to as “hit”) is specified (step 2104). For example, if the user specifies that one or more words selected by the user in area 2044 are used as a range creation criterion, the position of the selected word is identified in the multimedia information. Similarly, when the user specifies a topic of interest as the range creation criterion, the server 104 analyzes the multimedia information and identifies a location related to the topic of interest specified by the user in the multimedia information. As described above, the server 104 can analyze the multimedia information and determine the location of the word or phrase associated with the topic of interest specified by the user. Information related to the topic of interest can be stored in a user profile file accessible by the server 104. Various other techniques known to those skilled in the art can also be used to identify locations in the multimedia information that meet the range criteria received in step 2102.

  Next, one or more ranges are created based on the hit position identified in step 2104 (step 2106). Various different approaches can also be used to form the range based on the hit position. According to one approach, one or more ranges are created based on the time associated with the hit. Hits can be grouped into ranges based on their proximity to each other. One or more ranges created based on the hit position can be combined to form a larger range.

  Next, the range created in step 2106 is displayed to the user using GUI 2000 (step 2108). Various K techniques can be used to display the range to the user. In FIG. 20A, each range is indicated by a bar displayed on thumbnail 2008-2.

  FIG. 22 is a simplified high-level flowchart 2200 illustrating a method for automatically creating a range based on hit locations in multimedia information, in accordance with one embodiment of the present invention. The process shown in FIG. 22 is executed in step 2106 shown in FIG. The method illustrated in FIG. 22 may be executed by the server 104, may be executed by the client 102, or may be executed jointly by the server 104 and the client 102. For example, the method may be performed by a software module executing on the server 104 or client 102, may be performed by a hardware module connected to the server 104 or client 102, or a combination thereof But you can. In the embodiment described below, the method is performed by the server 104. The method illustrated in FIG. 22 is merely illustrative of an embodiment incorporating the present invention and is not intended to limit the scope of the invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

As shown in FIG. 22, the method begins by determining the time associated with the first hit in the multimedia information (step 2202). The first hit in this multimedia information corresponds to the hit with the oldest associated time (ie, hits that occurred before other hits in the multimedia information). The R _S for the new range was then set to the time of occurrence of the first hit, and the R _E for the new range was set to some time value after the time of occurrence of the first hit. A new range is created (step 2204). According to one embodiment of the invention, _RE is set 5 seconds after the time of occurrence of the first hit.

  The server 104 then determines whether there are any further hits in the multimedia information (step 2206). If there is no further hit in the multimedia information, the process ends. In that case, the range created for the multimedia information is displayed to the user according to step 2108 shown in FIG. If it is determined in step 2206 that there are more hits in the multimedia information, the time associated with the next hit is determined (step 2208).

Next, the server 104 determines whether or not the time difference between the end time of the range including the previous hit and the time determined in Step 2208 exceeds the threshold (Step 2210). Accordingly, in step 2210, the server 104
(Time determined in step 2208) − (R _{E of the} range including the previous hit)> GapBetweenHits
Determine whether or not. Here, GapBetweenHits represents the threshold time value. This threshold can be set by the user. According to one embodiment of the present invention, GapBetweenHits is 60 seconds.

If it is determined in step 2210 that the time difference between the end time of the range including the previous hit and the time determined in step 2208 exceeds the threshold, the _RS for the new range is set to the time determined in step 2208. is set, R _E about the new range to be set to a time value some later time determined in step 2208, the new range is created that contains the next hit (step 2212). According to one embodiment of the invention, _RE is set 5 seconds after the hit occurrence time. Thereafter, the processing proceeds to step 2206.

If the time difference between the end time and the time determined in step 2208 of the range including the previous hit in step 2210 it is determined not to exceed the threshold value, an end time R _E range including the previous hit in step 2208 By changing to the determined time, the range is extended (step 2214). Thereafter, the processing proceeds to step 2206.

  According to the method shown in FIG. 22, one range is created for hits in multimedia information that occur within a threshold (“GapBetweenHits”) from the previous range. At the end of the method shown in FIG. 22, one or more ranges are automatically created based on the range criteria.

  According to one embodiment of the present invention, after forming one or more ranges based on the time associated with the hit (eg, according to flowchart 2200 shown in FIG. 22), one or more created based on the hit position A range is combined with other ranges to form a larger range. According to one embodiment of the present invention, if a small range is identified and the time difference between the small range and the adjacent range is within a user-configurable time period threshold, the small range is combined with the adjacent range. Is done. If there are two adjacent time ranges within the time period threshold, the adjacent range closest to the small range is combined with the small range. The adjacent range need not be a small range. Combining smaller ranges to form a larger range is based on the assumption that a larger range is more useful to the user than multiple smaller ranges.

  FIG. 23 is a simplified high-level diagram showing how to synthesize one or more ranges based on their size and proximity to adjacent ranges according to one embodiment of the invention. It is a flowchart 2300. The process shown in FIG. 23 is executed after the process according to the flowchart 2200 shown in FIG. 22 in step 2106 shown in FIG. The method shown in FIG. 23 may be executed by the server 104, may be executed by the client 102, or may be executed jointly by the server 104 and the client 102. For example, the method may be performed by a software module executing on the server 104 or client 102, may be performed by a hardware module connected to the server 104 or client 102, or a combination thereof But you can. In the embodiment described below, the method is performed by the server 104. The method shown in FIG. 23 is merely illustrative of an embodiment incorporating the present invention and does not limit the scope of the invention as claimed. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

In order to explain the processing executed in FIG. 23, it is assumed that N ranges (N ≧ 1) have been created for the multimedia information displayed by the GUI. These ranges are created according to the processing shown in the flowchart 2200 of FIG. 22, for example. Each range R _i (1 ≦ i ≦ N) in the “N” ranges has a start time R _S and an end time R _E associated with it. Regarding the range R _i , the adjacent ranges of the range are defined as a range R _(i−1) and a range R _{(i + 1)} . Range _{R E} of _{R (i-1)} is generated before the _{R S} of range _{R i, R E} of the range _{R i} is generated before the _{R S} of range _{R (i + 1).} The range R _(i-1) is referred to as a range that occurs before the range R _i . The range R _{(i + 1)} is referred to as a range generated after the range R _i .

As shown in FIG. 23, the method begins by initializing variable “i” to 1 (step 2303). Next, the range R _i is selected (step 2304). When first going through the flowchart 2300, the first of the “N” ranges is selected. In subsequent processing, the subsequent range is selected.

Next, the server 104 determines whether or not the range R _i selected in step 2304 is appropriate as a small range. According to one embodiment of the present invention, if a threshold “SmallRangeSize” is defined and the time span of the range is less than or equal to the threshold SmallRangeSize, the range is considered a small range. Accordingly, to determine whether the range R _i is suitable as a small range, the time span of the range R _i selected in step 2304 is compared with the time threshold “SmallRangeSize” (step 2306). The value of SmallRangeSize can be set by the user. According to one embodiment of the present invention, SmallRangeSize is set to 8 seconds.

If it is determined in step 2304 that the range R _i selected in step 2304 is not suitable for a small range (ie, the time span (R _E −R _S ) of the range R _i is greater than the threshold SmallRangeSize), Is not a candidate for a range to be combined with another range. The value of variable “i” is then incremented by 1 to facilitate the selection of the next range among the “N” ranges. Therefore, according to the teachings of the present invention shown in FIG. 23, only a range suitable for a small range is eligible to be combined with other adjacent ranges.

After step 2308, server 104 determines whether all “N” ranges have been processed. This is done by determining whether the value of “i” is greater than the value of “N” (step 2310). If the value of “i” is greater than “N”, it indicates that all ranges for that multimedia information have been processed and the process of flowchart 2300 is complete. If it is determined in step 2310 that “i” is equal to or less than “N”, it indicates that there are at least one range that has not been processed according to the flowchart 2300 among the “N” ranges. In that case, the process proceeds to step 2304 and the next range R _i is selected.

In step 2306, appropriate for the selected range _{R i} is smaller ranges at step 2304 (i.e., the time span of the range _{R i (R} E -R _S) is less than the threshold SmallRangeSize) If it is determined that the present invention range R _i may be the synthesizing of range range adjacent to R _i (i.e., the range that occurs immediately before or after the selected range R _i in step 2304) to perform a process of identifying. To identify such range, the server 104 initializes the variables, to facilitate selection of the range adjacent to the range R _i selected in step 2304 (step 2312). The variable “j” is set to the value (i + 1), and the variable “k” is set to the value (i−1). Range variable "j" is adjacent to the range R _i, used to refer to a range that occurs after the range R _i, the variable "k", which is adjacent to the range R _i, occurs before the range R _i Used to refer to FIG. 24 shows a simplified diagram illustrating the relationship between the ranges R _i , R _j , and R _k . As shown in FIG. 24, range R _i occurs after range R _k (ie, R _S of R _i occurs before R _E of R _k ) and occurs before range R _j (ie, , _{R E} of _{R i} occurs before structured _{R S} of _{R j).}

Then, the server 104, the range and range R _i which occurs before the range R _i a neighboring range of the range R _i chosen created "N" range in step 2304 for the multimedia information judges whether included a range occurring after range R _i a neighbor range. This is done by determining the values of the variables “j” and “k”. If the value of “j” is greater than “N”, it is the range R _i selected in step 2304 is the last of the “N” ranges created for that multimedia information, and the range R _This indicates that there is no range that occurs after _i . If the value of “k” is 0, it means that the range R _i selected in step 2304 is the first of the “N” ranges created for that multimedia information, and before the range R _i Indicates that there is no range generated.

Accordingly, the server 104 determines whether the range R _i has an adjacent range that occurred before R _i and an adjacent range that occurred after R _i . This is done by determining whether the value of “j” is less than “N” and the value of “k” is not 0 (step 2314). If the condition is met in step 2314, it is adjacent to the range R _i selected in step 2304 in the "N" range, a range that occurs before the range R _i, adjacent to the range R _i and shows that the range occurring after range R _i are present. In this case, the process proceeds to Step 2316. If the condition in step 2314 is not met, it is that the range R _i selected in step 2304 is the first of the “N” ranges, and no range has occurred before the range R _i ; And / or indicates that the range R _i selected in step 2304 is the last of the “N” ranges, and that no range occurs after the range R _i . In this case, the process proceeds to Step 2330.

If it is determined that the condition in step 2314 is true, the server 104 determines the time difference between the ranges R _i and R _k and between the ranges R _i and R _j (step 2316).
The time difference between ranges R _i and R _k (denoted by G _ik ) is calculated by determining the time between R _{S in} range R _i and R _E in R _k (see FIG. 24). That is,
G _ik = (the _{R i R S) - (R} k of _R E)
It is. The time difference between ranges R _i and R _j (denoted as G _ij ) is calculated by determining the time between R _{E in} range R _i and R _S in R _j (see FIG. 24). That is,
G _ij = (R _{S of} R _j ) − (R _{E of} R _i )
It is.

The teachings of the present invention teach that the small range is combined with the adjacent range only if the difference between the small range and the adjacent range is less than or equal to the difference threshold. This difference threshold can be set by the user. Thus, the server 104 then determines the size of the time difference for determining whether the range R _i can be combined with one of its adjacent ranges.

Server 104 then determines which time difference is greater by comparing the values of time difference G _ik and time difference G _ij (step 2318). If it is determined in step 2318 that G _ik is greater than G _ij , it indicates that the range R _i selected in step 2304 is closer to range R _j than to range R _k , and processing is Proceed to step 2322. On the other hand, if it is determined in step 2318 that G _ik is not greater than G _ij , it means that the time difference between range R _i and range R _k selected in step 2304 is the time difference between range Ri and range R _j. The following is indicated, and the process proceeds to step 2320.

If it is determined in step 2318 that G _ik is not larger than G _ij , the server 104 determines whether or not the time difference (G _ik ) between the range R _i and the range R _k is equal to or smaller than the difference threshold “GapThreshold”. Judgment is made (step 2320). The value of GapThreshold can be set by the user. According to one embodiment of the present invention, GapThreshold is set to 90 seconds. Obviously, various other values can be used for GapThreshold.

If it is determined in step 2320 that the time difference (G _ik ) between the range R _i and the range R _k is less than or equal to the difference threshold GapThreshold (ie, G _ik ≦ GapThreshold), the ranges R _i and R _k are combined, A range is formed (step 2324). Synthesis process range _{R i} and _{R k} changes the end time of the range _{R k} to the end time of the range _{R i} (i.e., sets the _{R E} of _{R k} in _{R E} of _{R i),} the range _{R i} Including erasing. Thereafter, the process proceeds to step 2308, and the value of the variable “i” is incremented by one.

If it is determined in step 2320 that the time difference G _ik is greater than GapThreshold (ie, G _ik > GapThreshold), it indicates that both ranges R _j and R _k are outside the difference threshold, and as a result , Range R _i cannot be combined with either of ranges R _j and R _k . For this scenario, processing proceeds to step 2308 where the value of variable “i” is incremented by one.

Returning to step 2318, if it is determined that G _ik is greater than G _ij , the server 104 determines whether or not the time difference (G _ij ) between the range R _i and the range R _j is less than or equal to the difference threshold “GapThreshold”. Judgment is made (step 2322). As described above, the value of GapThreshold can be set by the user. According to one embodiment of the present invention, GapThreshold is set to 90 seconds. Obviously, various other values can be used for GapThreshold.

If it is determined in step 2322 that the time difference (G _ij ) between the ranges R _i and R _j is less than or equal to the difference threshold GapThreshold (ie, G _ij ≦ GapThreshold), the ranges R _i and R _j are combined and 1 Two ranges are formed (step 2326). The synthesis process of the ranges R _i and R _j changes the start time of the range R _{j to} the start time of the range R _i (ie, sets R _S of R _j to R _S of R _i ), and sets the range R _i to Including erasing. Thereafter, the process proceeds to step 2308, and the value of the variable “i” is incremented by one.

If it is determined in step 2322 that the difference G _ij is greater than GapThreshold (ie, G _ij > GapThreshold), it indicates that both ranges R _j and R _k are outside the difference threshold, and as a result , Range R _i cannot be combined with either range R _{j or} R _k . For this scenario, processing proceeds to step 2308 where the value of variable “i” is incremented by one.

If the server 104 determines that the condition in step 2314 is not satisfied, the server 104 determines whether or not the value of “k” is 0 (step 2330). If the value of “k” is 0, it means that the range R _i selected in step 2304 is the first of the “N” ranges created for that multimedia information, and the range R _i of This indicates that no previously generated range exists. In this scenario, the server 104 determines whether or not the value of the variable “j” is greater than “N” (step 2332). If the value of “j” is also greater than “N”, it is not only that the range R _i selected in step 2304 is the first of the “N” ranges created for that multimedia information. It is also the last range, indicating that there is no range that occurs after range R _i . If “k” is determined to be 0 in step 2330 and “j”> N in step 2332, it means that the range group for that multimedia information includes only one range (ie, N = 1). Next, the process shown in the flowchart 2300 ends because there is no range that can be combined.

If “k” is determined to be equal to 0 in step 2330 and “j” is determined not to be greater than “N” in step 2332, then it is determined that the range R _i selected in step 2304 is for that multimedia information. It represents the first range in the created "N" range, which range group is adjacent to the range R _i, including at least one range R _j occurring after the range R _i, indicating that. In this case, a time difference G _ij between the range R _i and the range R _j is determined (step 2334). As described above, G _ij is calculated by determining the time between R _{E in} range R _i and R _S in R _j . That is,
G _ij = (R _{S of} R _j ) − (R _{E of} R _i )
It is. The process then proceeds to step 2322 described above.

If it is determined in step 2330 that “k” is not equal to 0, it represents the last range in the “N” ranges for which the range R _i selected in step 2304 was created for that multimedia information; they range group is adjacent to the range R _i, including at least one range R _k that occurs before the range R _i, which information indicates that. In this case, a time difference G _ik between the range R _i and the range R _k is determined (step 2336). As described above, G _ik is calculated by determining the time between R _{S in} range R _i and R _E in R _k . That is,
G _ik = (the _{R i R S) - (R} k of _R E)
It is. The process then proceeds to step 2320 described above.

FIG. 25A shows a simplified diagram illustrating one range created by combining ranges R _i and R _k shown in FIG. 24, according to one embodiment of the present invention. FIG. 25B shows a simplified diagram illustrating one range created by combining the ranges R _i and R _j shown in FIG. 24, according to one embodiment of the present invention.

  As described above, the process shown in FIG. 23 can be executed after one or more ranges are created based on the time associated with the hit, according to the flowchart 2200 shown in FIG. According to one embodiment of the present invention, after the ranges are synthesized according to the flowchart 2300 shown in FIG. 23, the ranges are displayed to the user in the GUI 2000 according to step 2108 of FIG.

  According to an alternative embodiment of the present invention, after combining ranges according to the flowchart shown in FIG. 23, buffer time is added to the start time and end time of each range. The user can set the amount of time added to the start time of each range (BufferStart) and the amount of time added to the end time of each range (BufferEnd). The buffer time is added to the range so that the range does not start immediately from the first hit of the range and does not end immediately at the last hit of the range. Buffer time provides a better context for the range because it provides lead-in and trailing-off for information contained in the range.

Buffer, by varying the R _S time range as follows, are provided at the start of the range.
Range of _R S = _(R S of the previous range adding the buffer) -BufferStart
A buffer is provided at the end of the range by changing the range's _RE time as follows.
Range of _R E = _(R E of the previous range adding the buffer) -BufferEnd
FIG. 26 shows an expanded version of GUI 2000 showing a range automatically created according to one embodiment of the present invention. A plurality of hits 2602 that meet the criteria provided by the user are marked in the thumbnail 2008-1 displaying text information. According to one embodiment of the invention, these hits represent words and / or phrases associated with the topic of interest specified by the user. As shown in FIG. 26, two ranges 2006-2 and 2006-3 are automatically created based on the positions of these hits. The range 2006-2 is created by merging a plurality of small ranges in accordance with the teaching of the present invention (for example, the flowchart 2300 shown in FIG. 23).

(Display multimedia information from multiple multimedia documents)
The above embodiments of the present invention display a representation of recorded (or captured) information along a common timeline. These recorded information may include various types of information such as audio information, video information, closed caption (CC) text information, slide information, whiteboard information, and the like. These various types of information may be captured by one or more capture devices.

  As described above, a multimedia document can comprise a holding for storing recorded or captured information. The multimedia document may be a file that stores recorded information including a plurality of types of information. A multimedia document may be a file that includes references to one or more other files that store recorded information. These referenced files may store one or more types of information. The multimedia document may be a location where one or more types of recorded information is stored. For example, the multimedia document may be a directory that stores files containing information captured or recorded in a common timeline. According to one embodiment of the invention, each file in the directory stores a particular type of information. That is, each file stores a specific information stream. Therefore, various types of information may be stored in one file for recorded information including multiple types of information (for example, first type, second type, etc.). It may be stored in separate files.

  Since various types of information are captured along a common timeline, the representation of the information can be displayed so that the display is temporally aligned with each other when displayed by the GUI. For example, the interface 300 shown in FIG. 3 displays multimedia information stored by a television broadcast recording multimedia document. Various types of information stored in a broadcast record are, for example, video information, audio information, and possibly closed caption (CC) text information. Video information, audio information, and CC text information are all captured along the same (or common) timeline, possibly by different capture devices. For example, audio information is captured using an audio information capturing device (for example, a microphone), and video information is captured using a video information capturing device (for example, a video camera). Audio information and video information may be captured by a single information capturing device.

  As described above, the interface 300 displays text information that is a representation of audio information or CC text information included in a broadcast record (or a text representation of other types of information included in multimedia information). The interface 200 displays video key frames extracted from video information included in the broadcast record. The displayed video keyframe is a representation of the video information stored in the multimedia document. Since audio information and video information are captured along the same timeline, a representation of the information can be displayed so that they are time aligned or synchronized with each other. For example, as described above, the thumbnail images 312-1 and 312-2 may include text information (eg, audio information or CC text information recording in the thumbnail image 312-1 or other types of information included in the multimedia information). And a video key frame generated at a particular point in time displayed on thumbnail 31202 are displayed in close proximity to each other along the same horizontal axis. As a result, the user can determine various types of information in the television broadcast recording generated substantially simultaneously by simply scanning the thumbnail image on the horizontal axis. Similarly, panels 324-1 and 324-2 are time aligned or synchronized with each other so that representations of various types of information that occur substantially simultaneously in television broadcast recording are displayed in close proximity to each other. .

  Embodiments of the present invention can also display recorded multimedia information stored in multiple multimedia documents. Multimedia information in a plurality of multimedia documents may be captured along different timelines. For example, embodiments of the present invention provide a representation of multimedia information from a television news broadcast (eg, a morning news broadcast) captured or recorded during a first timeline, and a first timeline. Can display a multimedia information representation from another television news broadcast (eg, evening news broadcast) captured during a different second timeline. Accordingly, embodiments of the present invention can display multimedia information stored in one or more multimedia documents that can store multimedia information captured along different timelines. Each multimedia document may include various types of information such as audio information, video information, CC text information, whiteboard information, slide information, and so on.

  Also, a plurality of multimedia documents having displayed information may include documents that store information captured along the same timeline. For example, the plurality of multimedia documents may include a first television program recording from a first channel captured during the first timeline and the same timeline as the first television program recording (ie, the first timeline). A second television program record from a second channel captured in the line). Accordingly, embodiments of the present invention can display representations of information from multiple multimedia documents that store information captured along the same or different timelines.

  FIG. 27 illustrates a simplified startup user interface 2700 that can display information that can be stored in one or more multimedia documents in accordance with one embodiment of the present invention. Interface 2700 is merely illustrative of an embodiment of the invention and does not limit the scope of the invention. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  As shown in FIG. 27, the interface 2700 has a toolbar 2702 that includes a plurality of buttons that can be selected by the user. These buttons include, for example, a button 2704 for loading to display a multimedia document, a button 2706 for deleting one or more previously loaded multimedia documents, and from one or more loaded multimedia documents. A button 2708 for printing multimedia information on a paper medium, a button 2710 for setting user preferences, a user executing an action, and a method for displaying information from one or more multimedia documents , Other buttons that allow customization or control, etc. Additional features of interface 2700 are described in more detail below.

  To load one or more multimedia documents for display, the user selects a load button 2704. FIG. 28 illustrates a simplified window 2800 that is displayed when the user selects the load button 2704 in one embodiment of the invention. Window 2800 facilitates selection of one or more multimedia documents that are loaded and displayed in accordance with the teachings of the present invention. As shown in FIG. 28, information identifying one or more loadable multimedia documents is displayed in box 2802 of window 2800. Each multimedia document can be identified by an identifier (eg, a location identifier such as a file name, directory name, etc.). In the embodiment shown in FIG. 28, each multimedia document is identified by a 5-digit code identifier. The user can select one or more multimedia documents to be loaded by highlighting the identifier corresponding to the multimedia document in box 2802 and then selecting the “Add” button 2804. The highlighted identifier for that multimedia document is then moved from box 2802 and displayed in box 2806, which displays the multimedia document selected for loading. By highlighting the identifier for the multimedia document in box 2806 and then selecting the “Remove” button 2808, the previously selected multimedia document can be deselected.

  Information related to the multimedia document corresponding to the highlighted identifier (in box 2802 or 2806) is displayed in information area 2810. In the embodiment shown in FIG. 28, the displayed information includes information 2812 indicating the duration of the multimedia document, information 2814 indicating the date when the information of the multimedia document was captured or recorded, and the recorded time. Information 2816, information 2818 identifying the television channel on which the information is recorded, and information 2820 indicating the type of recording. Other descriptive information available about the multimedia document (eg, television program name) may be displayed in the description area 2821.

  The user can select the “Load” button 2822 to load and display the content of the multimedia document identified by the identifier displayed in box 2806. As shown in FIG. 28, three multimedia documents have been selected, and these three multimedia documents are loaded when the “Load” button 2822 is selected. These selected multimedia documents may store multimedia information captured along the same timeline or may store multimedia information captured along different timelines. Each selected multimedia document may have one or more types of information (eg, audio information, video information, CC text information, whiteboard information, slide information, etc.). The information type stored by one multimedia document may be different from the information type stored by another selected multimedia document. The user can cancel the load operation by selecting a “Cancel” button 2824.

  Other techniques can be used to select and load multimedia documents. For example, according to one approach, the user can scan a particular identifier (eg, a barcode). A multimedia document (or a portion of the information stored by the multimedia document) corresponding to the scanned barcode is selected and loaded.

  FIG. 29A illustrates the user interface 2900 after one or more multimedia documents have been loaded and displayed in one embodiment of the invention. Interface 2900 is merely illustrative of an embodiment of the present invention and does not limit the scope of the present invention. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  As shown in FIG. 29A, the contents of three multimedia documents are loaded and displayed. For each multimedia document, a representation of the various types of information stored by that multimedia document is displayed in a thumbbar corresponding to that multimedia document. A video window is also displayed for each multimedia document. The thumb bar for each multimedia document includes one or more thumbnail images that display representations of the various types of information contained in the multimedia document.

  For example, in FIG. 29A, thumb bar 2902 displays a representation of the information stored by the first multimedia document, thumb bar 2906 displays a representation of the information stored by the second multimedia document, and thumb bar 2910 Displays a representation of the information stored by the third multimedia document. Video window 2904 is displayed for the first multimedia document, video window 2908 is displayed for the second multimedia document, and video window 2912 is displayed for the third multimedia document. The In the embodiment shown in FIG. 29A, the first, second, and third multimedia documents are television program recordings, each including audio information, video information, and possibly CC text information. However, this is not intended to limit the scope of the invention. A multimedia document displayed in accordance with the teachings of the present invention can include various types of information.

  Each thumb bar displayed in FIG. 29A includes one or more thumbnail images. Each thumbnail image displays a representation of one information type stored in the multimedia document. The three multimedia documents loaded into interface 2900 have audio information, video information, and possibly CC text information, and thumbbars 2902, 2906, and 2910, respectively, are audio from the corresponding multimedia document. Thumbnail images that display information or text information that is a representation of CC text information (or a text representation of other types of information contained in a multimedia document), and a video keyframe that represents video information in the corresponding multimedia document Including thumbnail images. For example, the thumbnail 2902 includes a thumbnail image 2914 that displays text information representing audio information (or CC text information) from the first multimedia document, and a video image extracted from the video information of the first multimedia document. And a thumbnail image 2916 for displaying key frames. A thumbnail 2906 is a thumbnail image 2918 that displays text information (or a text representation of other types of information included in the multimedia information) representing audio information (or CC text information) from the second multimedia document. A thumbnail image 2920 displaying video keyframes extracted from the video information of the second multimedia document. A thumb bar 2910 is a thumbnail image 2922 that displays text information (or a text representation of other types of information included in the multimedia information) representing audio information (or CC text information) from the third multimedia document. A thumbnail image 2924 displaying video keyframes extracted from the video information of the third multimedia document. The method for generating and displaying thumbnail images has already been described. Each thumb bar is similar to the second display area shown in FIG.

  Thumbnail images in the thumb bar are aligned so that representations of information that occur simultaneously in time in the multimedia document are displayed in close proximity to each other along the same horizontal axis. Each thumb bar represents information captured according to a common timeline. However, the timeline corresponding to one thumbbar may be different from the timeline corresponding to another thumbbar.

  For each thumb bar, a lens (“thumbnail display area lens”) that covers or highlights a portion of that thumb bar is displayed. As shown in FIG. 29A, the thumbnail display area lens 2926 covers a partial area of the thumb bar 2902, the thumbnail display area lens 2928 covers a partial area of the thumb bar 2906, and the thumbnail display area lens 2930 includes a thumbnail bar. Covers a partial area of 2910. The thumbnail display area lens is initially located at the top of the thumb bar (ie, the beginning of the multimedia document), as shown in FIG. 29A. As already described with respect to FIG. 3, each thumbnail display lens can be moved along a corresponding thumb bar and can be used to navigate and scroll through the contents of the multimedia document displayed in the thumb bar. The method for displaying the thumbnail display area / lens and the method for navigating and scrolling the contents of each multimedia document using the thumbnail display area / lens have already been described. Each thumbnail display area / lens may or may not have a sub lens such as the sub lens 316 shown in FIG.

  Also, descriptive information related to each multimedia document can be displayed on a thumb bar corresponding to the multimedia document. This information includes information indicating the duration of the multimedia document, information indicating the date on which the information of the multimedia document was captured or recorded, information indicating the recording time, the television channel or TV on which the information was recorded Information such as information for identifying a program and information indicating the type of recording may be included. As shown in FIG. 29A, descriptive information 2932 about each multimedia document is displayed along one side of the corresponding thumb bar.

  For each multimedia document, video information may be played in a video window corresponding to the multimedia document. Also, audio information accompanying the video information may be output through the audio output device. For example, video information from a first multimedia document is played in video window 2904, video information from a second multimedia document is played in video window 2908, and video information from a third multimedia document. Is played in the video window 2912. Each video window is provided with a control bar that controls the playback of video information in the video window. For example, the playback of video information in video window 2904 is controlled by the control provided by control bar 2934, and the playback of video information in video window 2908 is controlled by the control provided by control bar 2936, Playback of video information in the video window 2912 is controlled by the controls provided by the control bar 2938.

The content of video information displayed in the video window for a multimedia document depends on the position of the thumbnail display area lens on the thumb bar corresponding to the multimedia document. For example, the content of the video information displayed in the video window 2904 depends on the position of the thumbnail display area lens on the thumb bar 2902. As described above, each thumbnail display area lens is characterized by an upper end corresponding to time t ₁ and a lower end corresponding to time t ₂ . Playback of video information in the video window is started at time t ₁ or t ₂ or any time between times t ₁ and t ₂ . When the position of the thumbnail display area lens on the thumb bar is changed, the video playback in the corresponding video window is also performed at the time t ₁ or t ₂ or the time corresponding to the current position on the thumbnail bar of the thumbnail display area lens. to start playback from any time between t ₁ and t _2, it may be changed. Note that each thumbnail display area lens that covers a portion of a different thumb bar can be repositioned along the thumb bar independently of each other.

  Each video window can also display information related to the multimedia document for which video information content is displayed in the video window. This information is, for example, information for identifying a television program for the recording, information for identifying a time in a multimedia document corresponding to the currently reproduced content, and the like.

  According to one embodiment of the present invention, a list of words (ie, common words) 2940 found in all loaded multimedia documents is also displayed in an area of the interface 2900. This word list 2940 includes words found in one or more types of information included by the loaded multimedia document. For example, the word list displayed in FIG. 29A includes words found in the first multimedia document, the second multimedia document, and the third multimedia document. According to one embodiment of the invention, the textual representation of the information contained by the loaded multimedia document is searched to find common words. This text information can represent CC text information, a recording of audio information, or a text representation of other types of information stored in a multimedia document. According to another embodiment of the invention, the word list includes words determined from the video information included by the multimedia document. For example, video keyframes extracted from video information are searched to find common words. Keyframes are searched for those words. In addition, the number of times the word appears in the multimedia document is indicated.

FIG. 29B shows the interface 2900 when the position of the thumbnail display area lens is changed from their initial position according to one embodiment of the present invention. As shown, thumbnail display area lenses 2926, 2928, and 2930 have been changed from their positions shown in FIG. 29A. Since the position of the thumbnail display area lens affects the video information played in the corresponding video window, the content in the video windows 2904, 2908, and 2912 has also changed. When the user moves the thumbnail display area lens on the thumb bar, a window such as window 2942 is displayed on the lens. One video key frames selected from a plurality of video key frames extracted from the video information of the multimedia document between times t ₁ and t ₂ of the thumbnail display area lens, as shown in FIG. 29B, the window 2942. Window 2942 disappears when the user releases the thumbnail display area lens.

  In accordance with one embodiment of the present invention, a user can specify criteria, and the content of the multimedia document loaded and displayed on the user interface identifies locations within the multimedia document that meet the criteria specified by the user. Searched to find in. Sections or locations within the multimedia document that meet the user-specified criteria are highlighted and displayed on the interface 2900. According to an embodiment of the present invention, the user-specified criteria is, for example, a user-specified word or phrase, a search query including one or more words, a topic of interest, and the like.

  In the interface 2900 shown in FIG. 29C, the user enters a word or phrase into the input area 2944 and selects a “Find” button 2946 to enter the content of the multimedia document for the word or phrase specified by the user. You can request to search for. Also, the word or phrase to be searched may be selected from the common word list 2940. In FIG. 29C, the user has specified the word “Stewart”.

  The content of the multimedia document is then searched to identify the location and presence of the user specified word or phrase. According to one embodiment of the present invention, a textual representation of the information stored by the multimedia document is searched to find the location of the word or phrase specified by the user. Video key frames may also be searched for that word or phrase. All occurrences (“hits”) 2950 of user specified words or phrases in various thumb bars (ie, thumbnail images in the thumb bar) are highlighted, as shown in FIG. 29C. A variety of different techniques can be used to highlight hits in a multimedia document. For example, individual hits may be highlighted. Also, a range may be determined based on the hit (described above) and this range may be emphasized. In addition, other methods such as a mark (see FIG. 29D below) can be used to mark the approximate position of the hit. According to one embodiment of the present invention, as shown in FIG. 29C, a colored rectangle is drawn around the line in the thumbnail image to highlight the line containing the search word or phrase. Also, the video keyframe displayed on the thumb bar containing the search word or phrase is highlighted by drawing a colored box around the video keyframe. Various other types of techniques can also be used. For example, if the multimedia document includes slide information, the slide displayed on the thumb bar containing the search word or phrase is highlighted. Also displayed is the total number 2952 of words or phrases present in various multimedia documents. For example, in FIG. 29C, the word “Stewart” appears 31 times in three multimedia documents.

  The user can also form a search query that includes multiple words (eg, multiple words and / or phrases). As shown in FIG. 29D, words or phrases included in the search query are displayed in area 2954. The user adds a word or phrase to the search query by typing a word or phrase in the input area 2944 (or selecting a word from the common word list 2940) and selecting the “Add” button 2956 Can do. The word or phrase is then added to the search query and displayed in area 2954. In FIG. 29D, the word “Stewart” has been added to the search query. The user can delete or delete the word or phrase from the search query by selecting the word or phrase in area 2954 and selecting the “Del” button 2960. The user can use the “Reset” button 2962 to reset or clear the search query.

  The user can also specify a Boolean connector that connects the words to each other in the search query. For example, in the embodiment shown in FIG. 29D, search query words or phrases may be ANDed or ORed based on the selection of radio button 2958. If the words are ORed, all positions in the various multimedia documents of the search query word or phrase are searched and highlighted. When words are ANDed, only the portion of the multimedia document that contains all the words or phrases in the search query within close proximity is highlighted as relevant. This proximity unit can be set by the user. According to one embodiment of the present invention, the unit of proximity matches the number of words. For example, the location of search query words or phrases in a multimedia document is highlighted if they occur within a predetermined number of words from each other. Proximity can also be based on time. In this embodiment, the location of search query words or phrases in a multimedia document is highlighted if they occur within a predetermined length of time.

  In FIG. 29D, the hit position is indicated by a mark 2964 displayed on the thumb bar. Each mark 2964 identifies a line in the text information printed on the thumbnail containing the search query word.

  In the embodiment shown in FIG. 29D, a range is formed based on the hit position. The method of forming a range based on the hit position has already been described (see FIGS. 20A, 20B, 21, 22, 23, 24, 25A, 25B, and 26, and related descriptions). The position of the range is displayed using a colored rectangle 2966. Each rectangle identifies a range. Thus, a rectangular box representing a range satisfies a user-specified criteria (eg, word, phrase, topic of interest, etc.) used to search the multimedia document among the multimedia documents, or Identify relevant parts.

In the embodiment of the present invention where only the content of one multimedia document is displayed (eg, FIG. 3), the range is the start time (R _S ) and end that define the boundaries of the range, as described above. It is identified by the time (R _E ). In the embodiment of the present invention in which information from a plurality of multimedia documents is displayed, the range includes a start time (R _S ), an end time (R _E ), and a multimedia document in which the range exists. It is defined by the identifier to identify. Further, as described above, identifiers (eg, text codes, numbers, etc.) can be used to identify each range. The range identifier for a range may be displayed in a rectangular box corresponding to that range, or may be displayed at other locations on the user interface.

  Also, each thumb bar in FIG. 29D is related to a search criterion (for example, a word or phrase specified by the user, a search query, a topic of interest, etc.) specified by the user of the multimedia document whose content is displayed on the thumb bar. A relevance indicator 2968 indicating the degree of sex is also included. A method for determining the relevance score or degree of relevance has already been described. According to one embodiment of the present invention, the degree of relevance for a certain thumb bar is based on the hit frequency in the multimedia document whose content is displayed on that thumb bar. In the relevance indicator shown in FIG. 20D, the degree of relevance of a multimedia document to a user-specified criterion is indicated by the number of bars displayed in the relevance indicator. Thus, the first and third multimedia documents whose contents are displayed in the thumb bars 2902 and 2910 (four bars shown in the respective relevance indicators) are the second multimedia documents displayed in the thumb bar 2906. It is more relevant to the current criteria specified by the user (ie, a search query such as the word “Stewart”) than to the document (only one bar is shown in the relevance indicator). Various other techniques (eg, relevance score, bar graph, different colors, etc.) can also be used to indicate the degree of relevance of the multimedia document.

  As described above, various operations can be performed on the range. The operations performed on the range include, for example, printing a representation of the range content on a paper document, saving the range content, and transmitting the range content. Ranges can also be annotated, emphasized, and grouped into range groups. In addition, the ranges in the range group can be ranked or rearranged according to some standard that can be set by the user. For example, the ranges can be ranked based on their relevance to the search criteria specified by the user of each range. According to one embodiment of the present invention, a range with a higher number of hits is ranked higher, and a range with a lower number of hits is ranked lower. Other techniques can also be used to rank and / or reorder ranges.

The user can also select one or more ranges displayed by the user interface and perform operations on the selected ranges. According to an embodiment of the present invention, a user can select a range by clicking a rectangle representing the range using an input device such as a mouse. In FIG. 29E, range 2970 has been selected by the user. The rectangle representing range 2970 is highlighted to indicate that it has been selected (eg, in a different color than the rectangles representing other ranges). In the embodiment shown in FIG. 29E, a web page 2971 is generated for the range selected by the user and displayed in the window 2969 of the user interface 2900. Web page 2971 is text information representing audio information or CC text information (or a text representation of other types of information contained in multimedia information) for the selected range (ie, time R of the selected range). generated audio information between the _S and R _E, CC text information or text information) 2972 representing other information, one or more video key frame or extracted from the video information corresponding to the selected range An image (ie, a video keyframe extracted from video information generated between times R _S and R _E for a selected range) 2973. According to one embodiment of the present invention, each image 2973 on the web page 2971 is a hypertext link that, when selected, starts playing video information from the time associated with that image.

  As shown in FIG. 29E, a barcode 2977 may be printed on each image. This bar code represents the time associated with the image, and when this bar code is scanned using a bar code reader or scanner, video information is played back from the time represented by the bar code associated with that image. . This playback can be displayed in the video window of the multimedia document corresponding to the selected range. Information 2974 identifying the multimedia document for which the range has been selected can also be displayed on the web page 2971. Each barcode 2977 can also identify the start time and end time for the range. By scanning such a barcode using a barcode reader or a scanner, information corresponding to the range can be reproduced. The barcode 2977 can also represent a label or identifier that identifies the range. When scanning such a percode, the range identifier represented by the scanned barcode can be used to determine the start and end times of the range, and the information corresponding to the range is reproduced. be able to.

  When a user identifies user-specified criteria (eg, a word or phrase, a topic of interest, a search query, etc.) for searching a multimedia document, the presence of the user-specified criteria in web page 2971 is emphasized. For example, in FIG. 29E, because the user specified a search query that includes the word “Stewart”, the presence 2975 of the word “Stewart” on the web page 2971 is highlighted (eg, bolded). ).

  Also included in web page 2971 is a hypertext link 2976 named “Complete Set”. Selecting the “Complete Set” link 2971 generates and displays a web page based on the various ranges of content shown on the thumb bar across all multimedia documents displayed in the interface 2900.

  As an alternative embodiment, types of documents other than web pages may be generated and displayed for the range. According to one embodiment of the present invention, a printable representation of one or more selected ranges is generated and displayed. Further details relating to the generation and display of such printable representations of multimedia information are filed on Nov. 19, 2001, the entire contents of which are hereby incorporated by reference. 001,895.

  FIG. 29F shows an interface 2900 in which the search query includes multiple words (ie, “Stewart”, “Imclone”, and “Waksal”) connected by an OR Boolean operator. All occurrences 2980 of search query words are highlighted (eg, bolded) on web page 2971. All occurrences or hits of words in the thumbbar are also marked using markers 2964. A range is formed and displayed based on the hit position.

  As already mentioned, user-specified criteria for searching multimedia documents can also include topics of interest. Thus, according to one embodiment of the invention, the content of one or more multimedia documents is searched to identify portions of the multimedia document that are relevant to the topic of interest that may be specified by the user.

  FIG. 29G shows a simplified user interface 2900 in which portions of a multimedia document that are relevant to a topic of interest specified by the user are highlighted. As shown in FIG. 29G, three topics of interest 2981 are defined: “Airlines”, “mstewart”, and “baseball”. Of the multimedia document, sections related to these topics of interest and displayed on the thumb bar corresponding to the multimedia document are highlighted using a marker 2983. A range is formed and displayed based on the hit position. Thus, the range identifies portions of these multimedia documents that are determined to be relevant to the topic of interest. Portions of the web page 2971 relating to these topics of interest are also highlighted. Techniques for specifying topics of interest and techniques for determining portions of a multimedia document related to one or more topics of interest are described in US patent application Ser. No. 10 / 001,895, filed Nov. 19, 2001, and It is also described in US patent application Ser. No. 08 / 995,616, filed Dec. 22, 1997. The entire contents of these patent documents are hereby incorporated by reference. A relevance indicator 2982 indicating the relevance of various multimedia documents to the topic of interest is also displayed for each topic of interest.

  According to one embodiment of the invention, a particular style or color can be associated with each topic of interest. For example, a first color may be associated with the topic of interest “Airlines”, a second color may be associated with the topic of interest “mstewart”, and a third color may be associated with the topic of interest “baseball”. The portion of the multimedia document that is determined to be related to a particular topic of interest can be highlighted using the style or color associated with that particular topic of interest. Thereby, the user can easily determine a part related to a specific topic of interest in the multimedia document.

  FIG. 29H illustrates another method of displaying portions of a thumb bar that relate to or meet or match user-specified criteria (eg, words, phrases, search queries, topics of interest, etc.) according to one embodiment of the invention. . As shown in FIG. 29H, a rectangular box is drawn on the thumbnail image of the thumb bar to identify the portion of the multimedia document related to the topic of interest 2981. For each rectangular box that covers a portion of the thumb bar, the word or phrase or image that selected that portion of the thumb bar is displayed on the rectangular box. This not only makes it easier for the user to see which part of the multimedia document is relevant to the topic of interest (or any other user-specified criteria), but also the word or word that resulted in the selection of that part. Phrases or images can be easily determined.

  In FIG. 29I, the playback of video information for a particular multimedia document has been moved from the video window corresponding to that multimedia document to a larger video window. As shown in FIG. 29I, video playback of the third multimedia document has been moved from the video window 2912 to the larger video window 2984. The mode of displaying a larger video window 2984 is activated by selecting the “Video” tab 2986. Switching the display from video window 2912 to window 2984 can be performed by selecting a control 2938a provided by control bar 2938. Also displayed below the larger video window 2984 is a control bar 2984 that includes controls that control the playback of video information in the window 2984. Moving the playback of video information from the small video window 2912 (or 2904 or 2908) to the large video window 2984 makes it easier for the user to view the video information playback. This video playback can be returned from window 2984 to small window 2912 by selecting control 2938a from control bar 2938 or by selecting control 2985a from control bar 2985.

  Also, text information 2987 corresponding to video playback (eg, CC text information, audio information recording, or text representation of other types of information included in multimedia information) is displayed below the large video window 2984. The Text information 2987 scrolls with video playback. Each word in the text information 2987 can be searched so that when the user clicks on a certain word, the number of times the selected word appears in the content of the multimedia document and the position where the word appears are known. As with video playback in the small video window, the content of the video playback in the large video window 2984 is a thumbnail of a thumb bar that displays a representation of the content of the multimedia document from which video information was played in the large video window 2984. It is affected by the display area and lens position.

As already mentioned, the user can manually define ranges for one or more multimedia documents. We have already explained how to manually define ranges for multimedia documents. In the embodiment shown in FIG. 29I, a button 2988 is provided that, when selected, initiates an operation mode in which the user can manually define the range. According to one embodiment of the invention, selecting button 2988 invokes a window, such as window 2050 shown in FIG. 20B. In addition to the information shown in FIG. 20B, this window also includes an input field where the user can enter information identifying a multimedia document whose range is defined from a plurality of multimedia documents loaded by the user interface. The user can also specify the start time and end time of the range. As an alternative embodiment, selecting button 2988 initiates a mode in which the user can manually specify the range by clicking on a portion of the thumb bar shown on interface 2900 using an input device such as a mouse. Clicking on a portion of the thumbbar displays a rectangular box that represents the range. The user can set the start time (R _S ) and end time (R _E ) of the range by operating the upper and lower ends of the rectangular box. The rectangular box itself can also be moved along the thumb bar.

  According to one embodiment of the present invention, a rectangular box representing an automatically generated range (eg, a range generated based on hits for user specified criteria) and a manual box specified by a user. Can be simultaneously displayed by the interface 2900. Different colors or styles can be used to display the rectangular box representing the automatic range and the rectangular box representing the manual range to distinguish between the manually generated range and the automatically generated range.

  FIG. 29K shows a user interface 2900 in which the portion of the multimedia document viewed or played by the user is highlighted according to one embodiment of the present invention. As an alternative embodiment, portions of the multimedia document that the user has not seen or played may be highlighted. As shown in FIG. 29K, a rectangular box 2990 is drawn over a portion of the thumb bar that identifies the portion of the multimedia document displayed on the user interface 2900 that the user has viewed or played. This portion may be played or viewed in a small video window 2904, 2908, or 2912, or may be played or viewed in a large video window 2984, or some output device It may be played back or viewed using. In this embodiment, information for identifying a portion of stored multimedia information that has been output to the user (or information for identifying a portion of stored multimedia information that has not been output to the user) Is stored. With this feature of the present invention, the user can easily see which sections of the multimedia document have already been viewed and which portions the user has not yet viewed. Boxes that have already been viewed are displayed in a specific color to distinguish them from other boxes displayed on the interface 2900, such as boxes that represent ranges.

  FIG. 30A illustrates another simplified user interface 3000 that displays the content of one or more multimedia documents according to one embodiment of the present invention. As shown in FIG. 30A, the contents of three multimedia documents are displayed, and three thumb bars 3002, 3004, and 3006 and three small video windows 3008, 3010, and 3012 are displayed. The contents of these multimedia documents were searched for a search query that included the words “Stewart”, “Imclone”, and “Faksal”. A portion of the thumb bar that includes content related to the search query is identified by marker 3014. Ranges are formed based on these hits, and a rectangular box 3016 representing these ranges is displayed. These functions have already been explained.

  In addition, the user interface 3000 includes a plurality of web pages 3020-1, 3020-2, 3020-3, etc. generated by rectangular boxes for the various ranges displayed on the thumb bar. According to one embodiment of the present invention, one web page is generated for each range displayed on the thumb bar. The web page shown in FIG. 30A (hereinafter referred to as a “palette” view) is generated and displayed by selecting a “Palette” button 3022. Thus, the palette of web pages includes web pages generated for various ranges. The palette of web pages can be displayed as a scrollable list as shown in FIG. 30A. The user can annotate the palette, add information to the web page of the palette, or add capture information to a portion of the multimedia document. Annotations can also be added to the range itself. For example, an annotation can be added to a range by adding a comment to a web page that displays information about the range.

In the embodiment shown in FIG. 30A, each web page 3020 for a particular range is text information that represents audio information, CC text information, or text representations of other information types included for that particular range (ie, , Recorded audio information appearing between times R _S and R _E of that particular range or text information representing CC text information). The web page also includes one or more video keyframes or images extracted from the video information corresponding to that particular range. These image and text information are synchronized or aligned in time. The web page image is a hypertext link, and when selected, the video information can be played back from the time associated with the selected image.

  The barcode can be printed and associated with each image printed on the web page. The bar code represents the time associated with the image, and when a bar code is scanned using a bar code reader or scanner, the bar code is associated with the image and video information is played from the time represented by the bar code. .

  For each range, information identifying the range and information identifying the multimedia information from which the range was selected can also be displayed on the web page corresponding to that range. For example, in the embodiment shown in FIG. 30A, the start time and end time 3018 of each range are displayed on the web page. An identifier 3021 for identifying the range is also displayed. Because the multimedia document of FIG. 30A supports television video recording, each web page corresponding to a range also displays an icon 3023 associated with the TV network that broadcasts information about that range.

  In each web page, the presence of user-specified criteria is emphasized. For example, in the embodiment shown in FIG. 30A, the search query includes the words “Stewart”, “Imclone”, and “Faksal”, and the presence of these words on the web page is highlighted. A variety of different techniques for emphasizing words such as bolding, using colors, changing styles, using balloons, surroundings, etc. can also be used. As already mentioned, the search query word may also be highlighted in the expression displayed on the thumb bar.

  A lens 3024 is displayed that highlights or covers the area of the web page that corresponds to the currently selected range. For example, in the embodiment shown in FIG. 30A, since a range is selected in the thumb bar 3006, a lens 3024 is displayed that covers the portion of the web page 3020-4 that corresponds to the selected range. The portion of the web page 3020-4 covered by the lens 3024 is displayed in the large window 3026. The user can change the position of the lens 3024 along the length of the web page 3020-4. The portion of the web page displayed in window 3026 is changed so that it continues to correspond to the portion of web page 3020-4 covered by lens 3024. In this way, the user can use the lens 3024 to navigate the contents of the selected web page. Also, the user can scroll the web page displayed in the window 3026 using the scroll bar provided by the window 3026. The position of lens 3024 on web page 3020-4 is changed so that it continues to correspond to the portion of the web page displayed in window 3026.

  The user can select that range by clicking on another range in the thumb bar using an input device such as a mouse (ie, by clicking on a rectangle representing that range). In response, the position of lens 3024 is changed and lens 3024 is displayed on the web page in a palette view corresponding to the newly selected range. The portion of the web page covered by lens 3024 in the palette view is then displayed in window 3026. For example, as shown in FIG. 30B, another range 3030 is selected by the user. The user can select this range by clicking on the rectangular box corresponding to this range in the thumb bar 3002. In response, a lens 3024 that covers the portion of web page 3020-1 corresponding to range 3030 is drawn. The portion of web page 3020-1 covered or highlighted by lens 3024 is displayed in window 3016.

  According to one embodiment of the present invention, a user can also select a range by selecting a web page corresponding to a range from the palette of web pages. The user can select a web page by clicking on the web page using an input device such as a mouse. In response, lens 3024 is displayed on the selected web page. A portion of the web page in the palette view covered by the lens 3024 is displayed in the window 3026. The rectangular box representing the range corresponding to this newly selected web page is also highlighted to indicate that the range has been selected.

  As described above, embodiments of the present invention also allow information representations stored by one or more multimedia documents recorded during the same timeline to be represented by one or more multimedia documents recorded during different timelines. A representation of the stored information can also be displayed. Users can specify criteria such as words, phrases, search queries that contain multiple words, topics of interest, etc., and the portion of the multimedia document that is related to or includes this user-specified criteria represents a marker, range Emphasized using boxes, etc. Thus, the content of multiple multimedia documents can be compared using embodiments of the present invention.

  For example, recordings of three different television programs such as “Closing Bell”, “60 Minute II”, and “Business Center” are displayed as shown in FIG. Search queries that include words, topics of interest, etc.). Of these three programs, the parts that are relevant or coincide with the user-specified criteria are highlighted. The ability to search across such multiple multimedia documents is not provided by conventional tools. Further, based on the search results displayed by the interface, the user can easily determine the relevance of the television program to the user criteria. As described above, the contents of a plurality of multimedia documents can be analyzed with each other using the embodiment of the present invention. Visualization of these search results is often useful to gain an understanding of the content of these multiple multimedia documents.

  As another example, if the user is interested in “Imclone / Martha Stewart scandal”, he / she can use the words “Stewart”, “Imclone”, and “Waksal” (or other words related to this scandal). A search query can be formed that includes and is displayed by the user interface in the representation of the multimedia document, and the portion containing the search query word is highlighted using markers, colors, and the like. Ranges are also formed based on search hits and are shown on the interface using colored boxes to highlight relevant sections. By viewing the highlighted portion of the multimedia document in the interface, the user can easily determine how much information about the scandal is contained in the multimedia document and the location of the related information in the multimedia document. Can do. The user can also determine the distribution of the related information in the multimedia document. Also, multiple multimedia documents can be compared with each other for search queries. Thus, embodiments of the present invention provide a valuable tool for users who want to analyze multiple multimedia documents.

  Analysis and review of multiple multimedia documents is further facilitated by generating and displaying web pages corresponding to the range displayed on the interface (automatically generated or manually specified). The web pages generated for these ranges allow the user to extract, organize and condense related portions in multiple multimedia documents.

  The embodiment of the present invention also allows the user to view a set of multimedia documents at the same time. For example, a user can view the contents of multiple video recordings or video clips. Various controls are provided for controlling the playback of multimedia information. The portion of the multimedia document played by the user can be emphasized. Therefore, the user can easily determine the part that the user has already seen and the part that has not been seen yet in the multimedia document.

  As described above, a range can be used to perform multiple operations. These operations include, for example, printing a representation of the range content on a paper document, saving the range content, communicating the range content, annotating the range, and the like. It is also possible to group ranges (eg, into range groups) and perform operations on those groups. For example, the ranges of a certain range group can be ranked or rearranged based on some criteria that the user may be able to set. For example, ranges may be ranked based on their relevance to user-specified search criteria. According to an embodiment of the present invention, a range with a higher number of hits is ranked higher, and a range with a lower number of hits is ranked lower. Other techniques for ranking and / or reordering the ranges can also be used.

(Printing multimedia information)
As described above, multimedia information from one or more multimedia documents displayed by the user interface can be printed on a paper medium to create a multimedia paper document. Accordingly, a multimedia paper document can be generated for one or more multimedia documents. The term “paper” or “paper medium” can refer to any tangible medium capable of printing, writing, drawing, stamping, embossing, and the like.

  According to one embodiment of the invention, for each multimedia document, a printable representation of the recorded information stored by that multimedia document is generated. This recorded information can store various types of information such as audio information, video information, closed caption (CC) text information, slide information, whiteboard information, etc. According to this, the printable representation of the recorded information may include a printable representation of one or more information types. Printable representations for recorded information that can include printable representations for one or more information types that make up the recorded information can be printed on a paper medium to generate a multimedia paper document. A variety of different techniques can be used to generate printable representations for multimedia information. An example of generating a printable representation and printing the printable representation on a paper medium to create a multimedia paper document was filed on November 19, 2001, the entire contents of which are incorporated herein by reference. It is described in US patent application Ser. No. 10 / 001,895.

  The printable representation can then be printed on a paper medium. The word “print” includes printing, writing, drawing, stamping, embossing, and the like. According to one embodiment of the invention, the printable representation is communicated to a paper document output device (printer, copier, etc.) configured to print a printable version on a paper medium to generate a paper document. Is done. A variety of different techniques can be used to print the printable representation on the paper medium. According to one embodiment of the present invention, printing is performed in accordance with the teachings set forth in US patent application Ser. No. 10 / 001,895, filed Nov. 19, 2001, the entire contents of which are hereby incorporated by reference. Is executed.

  In another embodiment of the invention, instead of generating a multimedia paper document for the entire content of the multimedia document, the multimedia paper document is generated only for the range displayed in the graphical user interface. In this embodiment, a printable representation is generated for the multimedia information corresponding to the range, and then the printable representation is printed on a paper medium. Since the multimedia information corresponding to the range may include one or more types of information, the printable representation of the multimedia information corresponding to the range may include one or more types of printable representation. Various different techniques can be used to generate a printable representation for multimedia information corresponding to a range. For example, those described in US patent application Ser. No. 10 / 001,895 filed on Nov. 19, 2001 can be used.

  FIG. 31 illustrates a simplified user interface 3100 that can be used to print content corresponding to the content or range of one or more multimedia documents in accordance with one embodiment of the present invention. The interface 3100 shown in FIG. 31 is merely illustrative of one embodiment of the present invention and does not limit the scope of the present invention. Other variations, modifications, and alternatives will be apparent to those skilled in the art. The graphical user interface 3100 can be invoked by selecting a command or button provided by the interface.

  As illustrated in FIG. 31, the user can specify that only information corresponding to the range is printed by washing the check box 3101. If the check box 3101 is not selected, it means that all content of one or more loaded multimedia documents is printed. The user can indicate that information corresponding to all displayed ranges should be printed by selecting check box 3102. Alternatively, the user can specifically specify the range to be printed by inputting the range identifier in the input box 3104. For example, if a range is identified by a number assigned to the range, the user may enter a number in box 3104 that corresponds to the range to be printed. If the range identifier is a sequential number, a range list can be specified.

  Selecting a “Print” button 3106 starts printing the contents of the range or the contents of the loaded multimedia document. The user interface 3100 can be canceled by selecting a “Cancel” button 3108.

  A number of options are provided to control how information corresponding to the range or information from the loaded multimedia document is printed. For example, a format style for printing information on a paper medium can be selected. In the embodiment shown in FIG. 31, the user can select one of three different styles 3110 by selecting a check box corresponding to the style. 32A, 32B, and 32C show pages printed according to three styles selectable from the interface 31 shown in FIG. 31 according to one embodiment of the present invention. FIG. 32A shows a page printed according to “Style 1”. FIG. 32B shows a page printed according to “Style 2”. FIG. 32C shows a page printed according to “Style 3”. Various other styles can also be used as alternative embodiments of the present invention.

  The user can also select different styles 3112 for printing key frames extracted from the video information. For example, in the embodiment shown in FIG. 31, the user can either print one key frame for each barcode (or print one barcode for each printed key frame), or each It is possible to select a style in which multiple (eg, four) key frames are printed per barcode. 33A and 33B show a page printed according to two keyframe styles selectable from the interface 31 shown in FIG. FIG. 33A shows a page on which one key frame is printed for each barcode. FIG. 33B shows a page on which four key frames are printed for each barcode. Other styles can be used as other embodiments of the invention.

  A list 3116 of printers (or any other paper document output device capable of generating a printed representation of multimedia information, such as a copier or facsimile machine) is displayed. The user can print the multimedia information on a paper medium by selecting one or more printers from the list 3116. The user selects a specific copier to perform printing (or copying) by selecting the “Send to Copier” check box 3113 and identifying the copier to be used in box 3114. can do.

  According to one embodiment of the present invention, a printable representation of multimedia information corresponding to a selected range or multimedia document can be stored in memory. For example, the printable representation can be stored as a PDF file. A file name can be specified in the input box 3118.

  In accordance with one embodiment of the present invention shown in FIG. 31, when a printable expression is printed on a paper medium, the user includes a section containing words or phrases that meet or match user-specified criteria in the printable expression. Has an option to indicate whether or not to emphasize. The user can activate this option by selecting checkbox 3120. When this option is selected, the word or phrase of the multimedia information corresponding to the selected range that matches the multimedia document or the topic of interest or matches the word or phrase specified by the user or the search query word is: Emphasized when printed on paper. A variety of different techniques can be used to emphasize words or phrases on paper.

  By selecting check box 3122, a text marker can be printed that associates the barcode with the printed text information.

  As described above, the user can specify that only information corresponding to the range should be printed by selecting the check box 3101. If desired, for each range, the user can specify in the input box 3126 a buffer time period to be added to the beginning and end of that range. For example, when the buffer time period is specified as 5 seconds, for each range, information corresponding to 5 seconds before the beginning of the range and information corresponding to 5 seconds after the end of the range are information corresponding to the range. Printed with.

  Embodiments of the present invention can also print cover sheets for printed information (for information corresponding to a range or for information corresponding to multimedia document content). In addition to printing the contents of the range or multimedia document, the user can specify that the cover sheet should be examined by selecting check box 3128. This cover sheet can provide an overview or summary of the printed content of the multimedia document or range.

  A variety of different techniques can be used to print the cover sheet. Various styles 3130 can be selected for the cover sheet. 34A, 34B, and 34C show examples of cover sheets that can be printed according to one embodiment of the present invention. An example of a technique for generating and printing a cover sheet is described in US patent application Ser. No. 10 / 001,895, filed Nov. 19, 2001, the entire contents of which are hereby incorporated by reference. Examples of various cover sheets are also described in US patent application Ser. No. 10 / 001,895, filed Nov. 19, 2001.

  The cover sheet can be used for a number of different purposes. As already mentioned, the cover sheet provides an overview or summary of the printed content of the multimedia document or range. The cover sheet can also provide a summary of the information stored on the storage device. For example, for multimedia information stored on a CD, a cover sheet summarizing what the content of the CD is based on the content of the CD can be generated. For example, as shown in FIG. 34C, the cover sheet is generated and used as a cover for a jewelry box that can store a CD. In the embodiment shown in FIG. 34C, barcode information printed on a CD can be used to access and index multimedia information stored on that CD. A technique for accessing multimedia information using a barcode printed on a cover sheet is described in US patent application Ser. No. 10 / 001,895, filed Nov. 19, 2001. Various other uses of the cover sheet are also within the scope of the present invention.

  Further, the user can select to print only the cover sheet without printing the contents of the range or multimedia document by selecting the check box 3132 of FIG. This is useful, for example, when a cover sheet is generated to provide an index to information stored on the storage device.

  The cover sheets shown in FIGS. 34A, 34B, and 34C each have a limited number of key frames sampled from the multimedia information from which the cover sheet was generated (eg, sampled uniformly every N seconds). indicate. The sampling interval can be specified by the user. For example, in the embodiment shown in FIG. 31, the user can enter a sampling interval in the input box 3134.

  The user can also control the print image quality. In the embodiment shown in FIG. 31, the user can select one of three options 3136.

  35A, 35B, 35C, 35D, and 35E show a paper document printed for a range according to one embodiment of the present invention. As described above, the range may be automatically generated, or may be manually designated by the user. Information corresponding to a range can be stored in one or more multimedia documents. The pages shown in FIGS. 25A, 35B, 35C, 35D, and 35E are merely illustrative of one embodiment of the invention and do not limit the scope of the invention. Other variations, modifications, and alternatives will be apparent to those skilled in the art.

  The documents shown in FIGS. 35A, 35B, 35C, 35D, and 35E are printed for a range selected from three multimedia documents. These three multimedia documents are TV program recordings, ie, the program “Money and Markets” captured from the CNN / fn channel (channel 358) and the program “Closing Bell” captured from the CNBC channel (channel 355). ”, And the program“ street sweep ”taken from the CNN / fn channel (channel 358).

  As shown in FIGS. 35A-E, the content for that range of the three recorded programs is printed continuously. Range content from the “Money and Markets” program recording multimedia document is printed on the page shown in FIGS. 35A and 35B, and range content from the “Closing Bell” program recording multimedia document is shown in FIGS. 35C and 35D. The range content from the “Street Sweep” program recording multimedia document is printed on the page shown in FIG. 35E.

  Information 3500 identifying the multimedia document with the range selected is printed as shown in FIGS. 35A, 35C, and 35E. In the embodiment shown in FIGS. 35A-E, the information identifying each multimedia document includes the name of the television program, information identifying the channel on which the program was recorded, the length of the recording, the recording date and time, and the like. . Other types of information related to multimedia documents may also be printed.

The head of each range is indicated by a bar 3502. Thus, two ranges of content are printed from the “Money and Markets” multimedia document, four ranges of content are printed from the “Closing Bell” multimedia document, and three from the “street sweep” multimedia document. One range of content is printed. Each bar 3502 is also printed with information 3504 related to the range. In the embodiment shown in FIGS. 35A-E, information related to a range includes an identifier for the range, a start time (R _S ) and end time (R _E ) for the range, a span for the range, and the like. Including. Other types of information associated with each range may also be printed.

  Information printed for each range includes text information 3506 and one or more images 3508. The text information is a printable representation of audio information (or CC text or text representation of other types of information contained in the multimedia information) corresponding to the range. Appears in the printed text information and highlights the presence of words or phrases related to the topic of interest, or words or phrases that match a user specified word or phrase or search query. For example, for the embodiment shown in FIGS. 35A-E, the user has defined a search query that includes the words “Stewart”, “Imclone”, and “Waksal”. Thus, the presence of all of these search query words is highlighted (using underline) in the printed text section for the various ranges. A variety of different techniques for emphasizing words such as making text bold, changing the font or size, making text italic, etc. can also be used.

  An image 3508 printed for each range represents an image extracted from the video information for that range. A number of different techniques can be used to extract video keyframes from a range of video information and identify the keyframes to be printed. Examples of these techniques have already been described and are described in US patent application Ser. No. 10 / 001,895 filed on Nov. 19, 2001. A variety of different styles can be used to print the information. For example, the user can select from the styles 3110 and 3112 shown in FIG.

  A barcode 3510 is also printed for each range. In the embodiment shown in FIGS. 35A-E, a barcode 3510 is printed for each image 3508 and placed under the image. A variety of different styles can be used to print the barcode. For example, in the embodiment shown in FIG. 31, two different styles 3112 are provided for printing barcodes, the first style being one barcode per keyframe (as shown in FIGS. 35A-E). In the second style, one bar code is printed every four key frames.

  According to one embodiment of the invention shown in FIGS. 35A-E, each barcode printed below an image represents the time associated with that image. Barcode 3510 provides a mechanism for readers of the paper document to access multimedia information using the paper document. According to an embodiment of the present invention, reproduction of multimedia information from a multimedia document corresponding to a barcode is represented by the barcode by scanning the barcode using a device such as a scanner or a barcode reader. It starts from the set time. This reproduction can be performed on any output device. For example, the information may be reproduced in one window of the above-described GUI displayed on the computer screen.

  Each bar code 3510 can also identify the start time and end time for the range. By scanning such a barcode using a barcode reader or scanner, information corresponding to the range is reproduced. Each barcode 3510 can also represent a label or identifier that identifies the range. In this embodiment, when scanning such a barcode, the range identifier represented by the scanned barcode can be used to determine the start and end time of that range, and then correspond to that range. Information can be reproduced.

  Thus, the documents shown in FIGS. 35A-E provide a paper interface for accessing stored multimedia information. Further information relating to using a paper interface to access multimedia information is described in US patent application Ser. No. 10 / 001,895, filed Nov. 19, 2001. Also, as alternative embodiments of the present invention, other user-selectable identifiers such as watermarks, glyphs, text identifiers, etc. may be used in place of the parcode. The user selectable identifier may be printed so as not to reduce or affect the overall readability of the paper document.

  In addition, a barcode group 3512 is printed at the lower end of each paper page of the paper document shown in FIGS. The barcode group 3512 allows the user to start and control the reproduction of multimedia information using a paper document. According to one embodiment of the invention, each barcode corresponds to a command that controls the playback of multimedia information. In the embodiment illustrated in FIGS. 35A-E, five control barcodes 3512 are printed. The control barcode 3512-1 allows the user to play back or pause playback. For example, the user can start reproducing information from the time represented by the operated barcode 3510 by scanning the barcode 3510 and then scanning the barcode 3512-1. The user can pause playback by rescanning the bar code 3512-1. The user can execute a rewind operation by selecting the baud code 3512-3. By selecting bar code 3512-4, playback can be performed in enhanced mode. Enhanced mode provides additional viewing controls and information (e.g., a specialized timeline is displayed, controls are provided so that the information being played can be navigated using on-screen buttons on the PDA, etc. ) Providing an alternative GUI. Further details regarding enhanced mode display are described in US patent application Ser. No. 10 / 174,522, filed Jun. 17, 2002, the entire contents of which are hereby incorporated by reference. By selecting the barcode 3512-5, a specific operation mode can be entered. As alternative embodiments of the present invention, bar codes for various other operations may also be provided. Information regarding barcodes that control the reproduction of information is described in US patent application Ser. No. 10 / 001,895, filed Nov. 19, 2001.

  While specific embodiments of the invention have been described, various modifications, alternatives, alternative configurations, and equivalents are also included within the scope of the invention. The described invention is not limited to operation within a particular specific data processing environment, but can operate freely within a plurality of data processing environments. In addition, although the present invention has been described using a specific series of interactions and processes, as will be apparent to those skilled in the art, the scope of the present invention is not limited to the series of interactions and processes described. For example, the process of generating a GUI in accordance with the teachings of the present invention may be performed by the server 104, may be performed by the client 102, may be performed by another computer, or may be a variety of connected It may be executed by a computer system.

  Furthermore, although the invention has been described using a particular combination of hardware and software, it should be apparent that other combinations of hardware and software are within the scope of the invention. The present invention may be implemented with hardware alone, software alone, or a combination thereof.

  The specification and drawings are accordingly to be regarded in an illustrative rather than restrictive sense. However, it will be apparent that additions, substitutions, deletions and other repairs and modifications may be made without departing from the broader spirit and scope of the claimed invention.

1 is a simplified block diagram of a distributed network that may incorporate an embodiment of the present invention. FIG. FIG. 2 is a simplified block diagram of a computer system according to an embodiment of the present invention. FIG. 3 illustrates a simplified user interface 300 for viewing multimedia information according to an embodiment of the present invention. It is an expansion simplification figure of a thumbnail display area lens concerning one embodiment of the present invention. 1 is a simplified diagram of a panel display area lens according to an embodiment of the present invention. 1 is a simplified diagram of a panel display area lens according to an embodiment of the present invention. 1 is a simplified diagram of a panel display area lens according to an embodiment of the present invention. FIG. 6 illustrates a simplified user interface according to an embodiment of the present invention with a user selected word annotated or highlighted. It is an expansion simplification figure of the 2nd display area of GUI which concerns on one Embodiment of this invention. FIG. 6 illustrates a simplified GUI according to an embodiment of the present invention annotated or highlighted with multimedia information related to one or more topics of interest to a user. FIG. 6 illustrates a simplified user interface according to an embodiment of the invention that defines target topics. FIG. 6 is a simplified user interface displaying multimedia information stored by conference recording according to an embodiment of the present invention. FIG. 4 illustrates a simplified user interface displaying multimedia information stored by a multimedia document according to an embodiment of the present invention. FIG. 4 illustrates a simplified user interface displaying multimedia information stored by a multimedia document according to an embodiment of the present invention. 6 is a simplified high-level flowchart illustrating a method for displaying thumbnail display text information in a second display area of a GUI according to an embodiment of the present invention. 6 is a simplified high-level flowchart illustrating a method for displaying thumbnail display text information in a second display area of a GUI according to an embodiment of the present invention. 6 is a simplified high-level flowchart illustrating a method for displaying a thumbnail indicating a video keyframe extracted from video information in a second display area of a GUI according to an embodiment of the present invention. 6 is a simplified high-level flowchart illustrating another method of displaying thumbnail 312-2 according to one embodiment of the present invention. The thumbnail display area / lens 314 is displayed, the information highlighted by the thumbnail display area / lens 314 is displayed in the third display area 306, the panel display area / lens 322 is displayed, and the panel is displayed in the fourth display area 308. 6 is a simplified high-level flowchart illustrating a method according to an embodiment of the present invention for displaying information highlighted by display area lens 322 and displaying information in a fifth display area 310. 6 is a simplified high-level flowchart illustrating a method according to an embodiment of the present invention for automatically updating information displayed in a third display area 306 in response to a change in position of a thumbnail display area lens 314; . A simplified view showing how to automatically update the information displayed in the fourth display area 308 and the positions of the thumbnail display area / lens 314 and sub-lens 316 in response to the position change of the panel display area / lens 322. It is a level flowchart. FIG. 6 illustrates a simplified user interface displaying a range according to an embodiment of the present invention. FIG. 6 illustrates a simplified dialog box for editing a range according to one embodiment of the present invention. 6 is a simplified high-level flowchart illustrating a method for automatically creating a range according to an embodiment of the present invention. 6 is a simplified high-level flowchart illustrating a method according to an embodiment of the present invention for automatically creating a range based on the hit location of multimedia information. 6 is a simplified high-level flowchart illustrating a method according to an embodiment of the present invention for combining one or more ranges based on the size of the range and the proximity of the range and adjacent ranges. It is a simplification figure showing the relation between adjacent ranges concerning one embodiment of the present invention. FIG. 25 is a simplified diagram illustrating a range created by combining ranges R _i and R _k shown in FIG. 24 according to an embodiment of the present invention. FIG. 25 is a simplified diagram illustrating a range created by combining ranges R _i and R _j illustrated in FIG. 24 according to an embodiment of the present invention. FIG. 5 is an enlarged version of a GUI showing a range automatically created according to one embodiment of the present invention. FIG. 6 illustrates a simplified startup user interface displaying information that can be stored in one or more multimedia documents according to an embodiment of the present invention. FIG. 6 shows a simplified window according to an embodiment of the present invention displayed when a user selects a load button. FIG. 3 is a diagram illustrating an example of a user interface that displays stored information according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a user interface that displays stored information according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a user interface that displays stored information according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a user interface that displays stored information according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a user interface that displays stored information according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a user interface that displays stored information according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a user interface that displays stored information according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a user interface that displays stored information according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a user interface that displays stored information according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a user interface that displays stored information according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a user interface that displays stored information according to an embodiment of the present invention. FIG. 6 illustrates a simplified user interface displaying content of one or more multimedia documents according to an embodiment of the present invention. FIG. 6 illustrates a simplified user interface displaying content of one or more multimedia documents according to an embodiment of the present invention. FIG. 6 illustrates a simplified user interface that can be used to print content corresponding to the content or range of one or more multimedia documents according to one embodiment of the present invention. FIG. 32 is a diagram showing a page printed according to a style selectable from the interface shown in FIG. 31 according to an embodiment of the present invention. FIG. 32 is a diagram showing a page printed according to a style selectable from the interface shown in FIG. 31 according to an embodiment of the present invention. FIG. 32 is a diagram showing a page printed according to a style selectable from the interface shown in FIG. 31 according to an embodiment of the present invention. FIG. 32 is a diagram illustrating a page printed using a key frame style selectable from the interface 31 illustrated in FIG. 31 according to an embodiment of the present invention. FIG. 32 is a diagram illustrating a page printed using a key frame style selectable from the interface 31 illustrated in FIG. 31 according to an embodiment of the present invention. It is a figure which shows an example of the cover sheet which can be printed according to one Embodiment of this invention. It is a figure which shows an example of the cover sheet which can be printed according to one Embodiment of this invention. It is a figure which shows an example of the cover sheet which can be printed according to one Embodiment of this invention. FIG. 3 is a diagram illustrating a paper document printed for a range according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a paper document printed for a range according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a paper document printed for a range according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a paper document printed for a range according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a paper document printed for a range according to an embodiment of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Distributed network 102 Client system 104 Server system 106 Multimedia information source 108 Communication network 110 Communication link 200 Computer system 202 Processor 204 Bus subsystem 206 Storage subsystem 208 Memory subsystem 210 File storage subsystem System 212 User interface input device 214 User interface output device 216 Network interface subsystem 218 Random access memory (RAM)
220 Read-only memory (ROM)

Claims (82)

A computer-implemented method of displaying information comprising:
Displaying a first representation of the first stored information;
Emphasizing one or more portions of the first representation,
The first stored information includes a first type of information and a second type of information;
The first expression includes an expression of the first type of information included in the first stored information, and an expression of the second type of information included in the first stored information. Including
The method of claim 1, wherein one or more portions of the enhanced first representation correspond to portions of the first representation that include a first criterion. The method of claim 1, comprising:
Displaying a second representation of the second stored information;
Further emphasizing one or more portions of the second representation,
The second stored information has a first type of information and a second type of information;
The second representation includes a representation of the first type of information included in the second stored information, and a representation of the second type of information included in the second stored information. Including
One or more portions of the enhanced second representation correspond to portions of the second representation that include the first criterion. The method of claim 2, comprising:
The first criterion includes a word;
Emphasizing one or more parts of the first expression includes emphasizing one or more parts of the first expression that include the word;
Emphasizing one or more parts of the second expression includes emphasizing one or more parts of the second expression that contain the word. The method of claim 2, comprising:
The first criterion includes the topic of interest;
Emphasizing one or more portions of the first representation includes highlighting one or more portions of the first representation that are related to the topic of interest;
Emphasizing one or more portions of the second representation includes highlighting one or more portions of the second representation that are related to the topic of interest. The method of claim 2, comprising:
The first type of information included in the first stored information is audio information;
The second type of information included in the first stored information is video information;
The first type information included in the second stored information is audio information;
The method of claim 2, wherein the second type of information included in the second stored information is video information. 5. The method of claim 5, comprising:
The representation of the first type of information included in the first representation includes text information obtained by recording the audio information included in the first stored information;
The representation of the second type of information included in the first representation includes one or more video keyframes extracted from the video information included in the first stored information;
The representation of the first type of information included in the second representation includes text information obtained by recording the audio information included in the second stored information;
The representation of the second type of information included in the second representation includes one or more video keyframes extracted from the video information included in the second stored information; Feature method. The method of claim 6, comprising:
Emphasizing one or more portions of the first representation includes emphasizing one or more lines including the first criterion in the text information included in the first representation;
Emphasizing one or more portions of the second representation includes emphasizing one or more lines including the first criterion in the text information included in the second representation. Feature method. The method of claim 6, comprising:
Emphasizing one or more portions of the first representation includes emphasizing one or more video keyframes that include the first criterion in the video information included in the first representation. ,
Emphasizing one or more portions of the second representation includes emphasizing one or more video keyframes that include the first criterion in the video information included in the second representation. A method characterized by that. The method of claim 5, comprising:
The method further comprising the step of playing back the video information included in the first stored information and the video information included in the second stored information. The method of claim 2, comprising:
Receiving a signal indicative of a first portion selected from the highlighted one or more portions of the first representation or the second representation;
Displaying the representation of the content of the first selected portion. The method of claim 10, comprising:
The method of claim 1, wherein the content representation of the first selected portion is a web page generated based on the content of the first selected portion. The method of claim 2, comprising:
Further comprising displaying a representation for each portion of the highlighted one or more portions of the first representation and the second representation;
The representation of the portion is generated based on the content of the portion. The method of claim 2, comprising:
The method further comprising the step of reproducing the first type of information contained in the first stored information and the first type of information contained in the second stored information. A computer implemented method for displaying multimedia information comprising:
Displaying a first thumbnail including a representation of the first type of information contained in the first recorded information;
Displaying a second thumbnail containing a representation of the second type of information contained in the first recorded information;
Displaying a third thumbnail containing a representation of the first type of information contained in the second recorded information;
Displaying a fourth thumbnail containing a representation of the second type of information contained in the second recorded information. 15. The method of claim 14, comprising
Receiving information identifying a group of words including one or more words;
Highlighting one or more portions of the first and third thumbnails that include at least one word in the word group. The method of claim 15, comprising:
The method further comprises highlighting one or more portions of the second thumbnail and the fourth thumbnail that include at least one word in the word group. The method of claim 16, comprising:
Receiving information indicative of a first portion selected from the highlighted one or more portions of the second or fourth thumbnail;
Outputting the second type of information corresponding to the first part,
The method of claim 1, wherein the first portion corresponds to a portion of the second type of information included in the first or second recorded information. 15. The method of claim 14, comprising
Receiving information identifying a group of topics of interest including one or more topics of interest;
Highlighting one or more portions of the first thumbnail and the third thumbnail that include at least one topic of interest in the group of topics of interest. The method of claim 18, comprising:
The method further comprising highlighting one or more portions of the second thumbnail and the fourth thumbnail that include at least one topic of interest in the group of topics of interest. 15. The method of claim 14, comprising
The first type information included in the first recorded information is audio information;
The second type of information included in the first recorded information is video information;
The first type information included in the second recorded information is audio information;
The second type of information included in the second recorded information is video information, and the first thumbnail is text information obtained from the audio information included in the first recorded information. Including
The second thumbnail includes one or more video keyframes extracted from the video information included in the first recorded information;
The third thumbnail includes text information obtained from the audio information included in the second recorded information;
The method of claim 4, wherein the fourth thumbnail includes one or more video keyframes extracted from the video information included in the second recorded information. 15. The method of claim 14, comprising
The first type of information contained in the first recorded information is closed caption text information;
The second type of information included in the first recorded information is video information;
The first type of information contained in the second recorded information is closed caption text information;
The second type of information included in the second recorded information is video information, and the first thumbnail is obtained from the closed caption text information included in the first recorded information. Text information provided,
The second thumbnail includes one or more video keyframes extracted from the video information included in the first recorded information;
The third thumbnail includes text information obtained from the closed caption text information included in the second recorded information,
The method of claim 4, wherein the fourth thumbnail includes one or more video keyframes extracted from the video information included in the second recorded information. A method for displaying information contained in a first recorded information including audio information and video information and a second recorded information including audio information and video information,
Displaying a first representation of the information contained in the first recorded information;
Displaying a second representation of the information contained in the second recorded information,
The first representation has a first thumbnail and a second thumbnail;
The first thumbnail includes text information obtained from the audio information included in the first recorded information,
The second thumbnail includes one or more key frames extracted from the video information included in the first recorded information;
The second representation has a third thumbnail and a fourth thumbnail;
The third thumbnail includes text information obtained from the audio information included in the second recorded information;
The method of claim 4, wherein the fourth thumbnail includes one or more key frames extracted from the video information included in the second recorded information. 23. The method of claim 22, wherein
Receiving information identifying user criteria;
Further emphasizing one or more portions of the first representation and the second representation that include the user criteria,
The highlighted portion of the first representation covers a portion of the first thumbnail and the second thumbnail;
The emphasized portion of the second representation covers a portion of the third thumbnail and the fourth thumbnail. 24. The method of claim 23, comprising:
The step of highlighting the one or more portions of the first representation and the second representation identifies a location that includes the user criterion in the first representation and the second representation, and the identified location And determining one or more portions of the first representation and the second representation to be emphasized. 24. The method of claim 23, comprising:
The method of claim 1, wherein the user criteria is one or more words or phrases. 24. The method of claim 23, comprising:
The user criteria is one or more topics of interest. 24. The method of claim 23, comprising:
Further comprising displaying a document for each of the highlighted portions of the first representation and the second representation;
The document for the portion includes text information from the portion of the first or third thumbnail covered by the portion and one from the portion of the second or fourth thumbnail covered by the portion. A method comprising the above key frames. 24. The method of claim 23, comprising:
Receiving information indicating a first highlighted portion selected from the one or more highlighted portions of the first representation;
Further comprising displaying a document about the first highlighted portion,
The document about the first enhancement portion includes text information from a portion of the first thumbnail covered by the first enhancement portion and the second thumbnail covered by the first enhancement portion. Including one or more keyframes from a portion of. 24. The method of claim 23, comprising:
Receiving information indicating a first highlighted portion selected from the one or more highlighted portions of the first representation;
Outputting the video information corresponding to the first emphasized portion from the first recorded information. A computer-implemented method of displaying information comprising:
Displaying a representation of the stored information;
Receiving information representing one or more output portions of the stored information;
Highlighting one or more portions of the stored representation of the stored information corresponding to one or more output portions of the stored information. The method of claim 30, comprising:
The method of claim 1, wherein the stored information is video information. The method of claim 30, comprising:
The method of claim 1, wherein the stored information is audio information. A computer-implemented method of displaying information comprising:
Displaying a representation of the stored information;
Receiving information representing one or more portions of the stored information that have not yet been output;
Highlighting one or more portions of the stored representation of the stored information corresponding to one or more portions of the stored information that have not yet been output. A device for displaying information,
A processor;
A display,
The processor is
Displaying a first representation of the first stored information on the display;
Set to highlight one or more portions of the first representation displayed on the display;
The first stored information includes a first type of information and a second type of information;
The first expression includes an expression of the first type of information included in the first stored information, and an expression of the second type of information included in the first stored information. Including
The apparatus of claim 1, wherein one or more portions of the enhanced first representation correspond to portions of the first representation that include a first criterion. 35. The apparatus of claim 34, wherein
The processor further includes:
Displaying a second representation of second stored information on the display;
Is set to emphasize one or more parts of the second representation;
The second stored information comprises a first type of information and a second type of information,
The second representation includes a representation of the first type of information included in the second stored information, and a representation of the second type of information included in the second stored information. Including
The apparatus characterized in that one or more portions of the enhanced second representation correspond to portions of the second representation that include the first criterion. 36. The apparatus of claim 35, wherein
The first criterion includes a word;
The processor is configured to highlight one or more portions of the first representation that include the word;
The apparatus, wherein the processor is configured to highlight one or more portions of the second representation that contain the word. 36. The apparatus of claim 35, wherein
The first criterion includes the topic of interest;
The processor is configured to highlight one or more portions of the first representation related to the topic of interest;
The apparatus, wherein the processor is configured to highlight one or more portions of the second representation related to the topic of interest. 36. The apparatus of claim 35, wherein
The first type of information included in the first stored information is audio information;
The second type of information included in the first stored information is video information;
The first type information included in the second stored information is audio information;
The apparatus of claim 2, wherein the second type of information included in the second stored information is video information. 40. The apparatus of claim 38, wherein
The representation of the first type of information included in the first representation includes text information obtained by recording the audio information included in the first stored information;
The representation of the second type of information included in the first representation includes one or more video keyframes extracted from the video information included in the first stored information;
The representation of the first type of information included in the second representation includes text information obtained by recording the audio information included in the second stored information;
The representation of the second type of information included in the second representation includes one or more video keyframes extracted from the video information included in the second stored information; Features device. 40. The apparatus of claim 39, comprising:
The processor is configured to highlight one or more lines including the first criterion in the text information included in the first representation;
The apparatus, wherein the processor is configured to highlight one or more lines including the first criterion in the text information included in the second representation. 40. The apparatus of claim 39, comprising:
The processor is configured to highlight one or more video keyframes that include the first criterion among the video keyframes included in the first representation;
The apparatus, wherein the processor is configured to highlight one or more video keyframes that include the first criterion among the video keyframes included in the second representation. 40. The apparatus of claim 38, wherein
The apparatus, wherein the processor is configured to play back the video information included in the first stored information and the video information included in the second stored information. 36. The apparatus of claim 35, wherein
The processor is
Receiving a signal indicative of a first portion selected from the enhanced one or more portions of the first representation or the second representation;
An apparatus configured to display a representation of the content of the first selected portion on the display. 44. The apparatus of claim 43, comprising:
The apparatus, wherein the representation of the content of the first selected portion is a web page generated based on the content of the first selected portion. 36. The apparatus of claim 35, wherein
The processor is configured to display a representation on the display for each portion of the highlighted one or more portions of the first representation and the second representation;
The representation of the portion is generated based on the content of the portion. 36. The apparatus of claim 35, wherein
The processor is configured to reproduce the first type of information included in the first stored information and the first type of information included in the second stored information; Features device. A device for displaying multimedia information,
A processor;
A display,
The processor is
Displaying a first thumbnail including a representation of the first type of information contained in the first recorded information on the display;
Displaying a second thumbnail containing a representation of the second type of information contained in the first recorded information on the display;
Displaying a third thumbnail on the display including a representation of the first type of information contained in the second recorded information;
An apparatus configured to display a fourth thumbnail including a representation of a second type of information included in the second recorded information on the display. 48. The apparatus of claim 47, wherein:
The processor is
Receiving information identifying a group of words including one or more words;
The apparatus is configured to emphasize one or more portions including at least one word in the word group among the first thumbnail and the third thumbnail. 49. The apparatus of claim 48, comprising:
The apparatus, wherein the processor is configured to emphasize one or more portions of the second thumbnail and the fourth thumbnail that include at least one word in the word group. 50. The apparatus of claim 49, comprising:
The processor is
Receiving information indicating a first portion selected from the one or more highlighted portions of the second or fourth thumbnail;
Set to output the second type of information corresponding to the first part,
The apparatus of claim 1, wherein the first portion corresponds to a portion of the second type of information included in the first or second recorded information. 48. The apparatus of claim 47, wherein:
The processor is
Receiving information identifying a group of topics of interest, including one or more topics of interest;
An apparatus configured to highlight one or more portions of the first thumbnail and the third thumbnail that include at least one topic of interest in the group of topics of interest. 52. The apparatus of claim 51, comprising:
The processor is configured to highlight one or more portions of the second thumbnail and the fourth thumbnail that include at least one topic of interest in the group of topics of interest. Device to do. 48. The apparatus of claim 47, wherein:
The first type information included in the first recorded information is audio information;
The second type of information included in the first recorded information is video information;
The first type information included in the second recorded information is audio information;
The second type of information included in the second recorded information is video information, and the first thumbnail is text information obtained from the audio information included in the first recorded information. Including
The second thumbnail includes one or more video keyframes extracted from the video information included in the first recorded information;
The third thumbnail includes text information obtained from the audio information included in the second recorded information;
The apparatus, wherein the fourth thumbnail includes one or more video keyframes extracted from the video information included in the second recorded information. 48. The apparatus of claim 47, wherein:
The first type of information contained in the first recorded information is closed caption text information;
The second type of information included in the first recorded information is video information;
The first type of information contained in the second recorded information is closed caption text information;
The second type of information included in the second recorded information is video information, and the first thumbnail is obtained from the closed caption text information included in the first recorded information. Text information provided,
The second thumbnail includes one or more video keyframes extracted from the video information included in the first recorded information;
The third thumbnail includes text information obtained from the closed caption text information included in the second recorded information,
The apparatus, wherein the fourth thumbnail includes one or more video keyframes extracted from the video information included in the second recorded information. An apparatus for displaying information included in a first recorded information including audio information and video information and a second recorded information including audio information and video information,
Display,
A processor, and
The processor is
Displaying a first representation of the information contained in the first recorded information on the display;
Set to display on the display a second representation of information contained in the second recorded information;
The first representation has a first thumbnail and a second thumbnail;
The first thumbnail includes text information obtained from the audio information included in the first recorded information,
The second thumbnail includes one or more key frames extracted from the video information included in the first recorded information;
The second representation has a third thumbnail and a fourth thumbnail;
The third thumbnail includes text information obtained from the audio information included in the second recorded information;
The apparatus of claim 4, wherein the fourth thumbnail includes one or more key frames extracted from the video information included in the second recorded information. 56. The apparatus of claim 55, wherein
The processor is
Receive information identifying user criteria,
Set to highlight one or more portions of the first representation and the second representation that include the user criteria;
The highlighted portion of the first representation covers a portion of the first thumbnail and the second thumbnail;
The apparatus of claim 2, wherein the highlighted portion of the second representation covers a portion of the third thumbnail and the fourth thumbnail. 57. The apparatus of claim 56, comprising:
The processor identifies a position including the user criterion in the first representation and the second representation, and based on the identified position, the enhancement of the first representation and the second representation. An apparatus characterized in that it is set to determine one or more parts to be played. 57. The apparatus of claim 56, comprising:
The apparatus wherein the user criteria is one or more words or phrases. 57. The apparatus of claim 56, comprising:
The apparatus wherein the user criteria is one or more topics of interest. 57. The apparatus of claim 56, comprising:
The processor is configured to display a document on the display for each of the highlighted portions of the first representation and the second representation;
The document about the portion includes text information from the portion of the first or third thumbnail covered by the portion and one from the portion of the second or fourth thumbnail covered by the portion. An apparatus comprising the above key frame. 57. The apparatus of claim 56, comprising:
The processor is
Receiving information indicating a first highlighted portion selected from the one or more highlighted portions of the first representation;
Set to display a document about the first highlighted portion on the display;
The document about the first enhancement portion includes text information from a portion of the first thumbnail covered by the first enhancement portion and the second thumbnail covered by the first enhancement portion. One or more key frames from a portion of the device. 57. The apparatus of claim 56, comprising:
The processor is
Receiving information indicating a first highlighted portion selected from the one or more highlighted portions of the first representation;
An apparatus configured to output video information corresponding to the first emphasized portion from the first recorded information. A device for displaying information,
A processor;
And a memory connected to the processor,
The memory is configured to store a program for controlling the processor;
The processor is under the control of the program,
Display a representation of the stored information,
Receiving information representing one or more output portions of the stored information;
An apparatus configured to highlight one or more portions of the stored information representation corresponding to one or more output portions of the stored information. A device for displaying information,
A processor;
And a memory connected to the processor,
The memory is configured to store a program for controlling the processor;
The processor is under the control of the program,
Display a representation of the stored information,
Receiving information representing one or more portions of the stored information that have not yet been output;
An apparatus configured to highlight one or more portions of the stored information representation corresponding to one or more portions of the stored information that have not yet been output. A program stored on a computer-readable recording medium and displaying information,
A code for displaying a first representation of the first stored information;
A code that highlights one or more portions of the first representation;
The first stored information includes a first type of information and a second type of information;
The first expression includes an expression of the first type of information included in the first stored information, and an expression of the second type of information included in the first stored information. Including
One or more portions of the emphasized first expression correspond to a portion including the first reference in the first expression. A program according to claim 65, wherein
A code for displaying a second representation of the second stored information;
A code that highlights one or more portions of the second representation;
The second stored information has a first type of information and a second type of information;
The second representation includes a representation of the first type of information included in the second stored information, and a representation of the second type of information included in the second stored information. Including
One or more portions of the emphasized second expression correspond to a portion including the first reference in the second expression. A program according to claim 66,
The first criterion includes a word;
The code that emphasizes one or more parts of the first expression includes a code that emphasizes one or more parts of the first expression including the word,
The program that emphasizes one or more parts of the second expression includes a code that emphasizes one or more parts including the word in the second expression. A program according to claim 66,
The first criterion includes the topic of interest;
The code for highlighting one or more portions of the first representation includes a code for highlighting one or more portions of the first representation related to the topic of interest;
The code for emphasizing one or more parts of the second expression includes a code for emphasizing one or more parts related to the topic of interest in the second expression. A program stored on a computer readable recording medium for displaying multimedia information,
A code for displaying a first thumbnail containing a representation of the first type of information contained in the first recorded information;
A code for displaying a second thumbnail containing a representation of the second type of information contained in the first recorded information;
A code for displaying a third thumbnail containing a representation of the first type of information contained in the second recorded information;
And a code for displaying a fourth thumbnail including a representation of the second type of information included in the second recorded information. A program according to claim 69, wherein
A code for receiving information identifying a group of words including one or more words;
A program further comprising: a code for emphasizing at least one portion including at least one word in the word group among the first thumbnail and the third thumbnail. A program according to claim 70, wherein
The program further comprising a code for emphasizing at least one portion including at least one word in the word group among the second thumbnail and the fourth thumbnail. The program according to claim 71, wherein
A code for receiving information indicative of a first portion selected from the highlighted one or more portions of the second or fourth thumbnail;
A code for outputting the second type of information corresponding to the first portion;
The program according to claim 1, wherein the first part corresponds to a part of the second type of information included in the first or second recorded information. A program for displaying information contained in a first recorded information including audio information and video information and a second recorded information including audio information and video information stored on a computer-readable recording medium. ,
A code for displaying a first representation of information contained in the first recorded information;
A code for displaying a second representation of the information contained in the second recorded information,
The first representation has a first thumbnail and a second thumbnail;
The first thumbnail includes text information obtained from the audio information included in the first recorded information,
The second thumbnail includes one or more key frames extracted from the video information included in the first recorded information;
The second representation has a third thumbnail and a fourth thumbnail;
The third thumbnail includes text information obtained from the audio information included in the second recorded information;
The program according to claim 4, wherein the fourth thumbnail includes one or more key frames extracted from the video information included in the second recorded information. A program according to claim 73,
A code for receiving information identifying the user criteria;
A code that highlights one or more portions of the first representation and the second representation including the user criteria;
The highlighted portion of the first representation covers a portion of the first thumbnail and the second thumbnail;
The emphasized portion of the second expression covers a part of the third thumbnail and the fourth thumbnail. A program according to claim 74, wherein
The code that emphasizes the one or more portions of the first expression and the second expression is identified by a code that specifies a position including the user criterion in the first expression and the second expression. And a code for determining one or more emphasized portions of the first expression and the second expression based on the position. A program according to claim 74, wherein
A code for receiving information indicating a first highlighted portion selected from the one or more highlighted portions of the first representation;
And a code for outputting video information corresponding to the first emphasized portion from the first recorded information. A program stored on a computer-readable recording medium and displaying information,
A code that displays a representation of the stored information;
A code for receiving information representing one or more output portions of the stored information;
And a code for emphasizing one or more portions corresponding to one or more output portions of the stored information in the stored information representation. A program stored on a computer-readable recording medium and displaying information,
A code that displays a representation of the stored information;
A code for receiving information representing one or more portions of the stored information that have not yet been output;
And a code for emphasizing one or more portions of the stored information corresponding to one or more portions of the stored information that have not yet been output. A device for displaying information,
Means for displaying a first representation of the first stored information;
Means for highlighting one or more portions of the first representation;
Means for displaying a second representation of the second stored information;
Emphasizing one or more portions of the second representation,
The first stored information includes a first type of information and a second type of information;
The first expression includes an expression of the first type of information included in the first stored information, and an expression of the second type of information included in the first stored information. Including
One or more portions of the emphasized first representation correspond to portions of the first representation that include a first criterion;
The second stored information comprises a first type of information and a second type of information,
The second representation includes a representation of the first type of information included in the second stored information, and a representation of the second type of information included in the second stored information. Including
The apparatus characterized in that one or more portions of the enhanced second representation correspond to portions of the second representation that include the first criterion. A device for displaying multimedia information,
Means for displaying a first thumbnail containing a representation of the first type of information contained in the first recorded information;
Means for displaying a second thumbnail including a representation of a second type of information contained in the first recorded information;
Means for displaying a third thumbnail containing a representation of the first type of information contained in the second recorded information;
Means for displaying a fourth thumbnail containing a representation of the second type of information contained in the second recorded information. An apparatus for displaying information included in a first recorded information including audio information and video information and a second recorded information including audio information and video information,
Means for displaying a first representation of information contained in the first recorded information;
Means for displaying a second representation of the information contained in the second recorded information,
The first representation has a first thumbnail and a second thumbnail;
The first thumbnail includes text information obtained from the audio information included in the first recorded information,
The second thumbnail includes one or more key frames extracted from the video information included in the first recorded information;
The second representation has a third thumbnail and a fourth thumbnail;
The third thumbnail includes text information obtained from the audio information included in the second recorded information;
The apparatus of claim 4, wherein the fourth thumbnail includes one or more key frames extracted from the video information included in the second recorded information. A device for displaying information,
Means for displaying a representation of the stored information;
Means for receiving information representing one or more output portions of the stored information;
Means for highlighting one or more portions of the stored information representation corresponding to one or more output portions of the stored information.

Images