JP2005100415A - Multimedia print driver dialogue interface - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a user interface allowing a user to interact with a media contents analysis, a media expression generating process. <P>SOLUTION: The media print interface makes it possible for the user to interact with a multimedia transformation process, and also to format multimedia data in order to generate expressions of multimedia data. A user interface makes it possible for the user to interact with media content analysis and media expression generation. A media analysis software module receives commands for media contents analysis from the user by the user interface, and the media analysis software module analyzes and recognizes features of the media contents such as, for example, faces, contents of conversations, texts or the like. The media expressions can be generated in one of a paper-based format, a digital format, or any another expression format. The user interface includes various fields capable of displaying the media contents and modifying the generated media expressions by the user. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a system and method for providing a multimedia printing interface. In particular, the present invention relates to a system and method for providing a print driver dialog interface that allows a user to format multimedia data for generating a representation of the multimedia data.

  Printers in today's modern systems are not designed to generate multimedia documents. Currently, there is no effective method for easily readable representation of multimedia content, either in paper format or digital format. Several different technologies and tools can be utilized to access and manipulate multimedia information (eg, existing multimedia players). However, none of them can provide the user with an option to generate a multimedia document that the user can easily examine and gain access to the multimedia content.

  Printers in today's modern systems are also generally not designed to facilitate interaction with multimedia content or printed content. The standard printer dialog box provides the user with some common formatting options in a print job, such as the number of pages to print, the number of copies to be performed, etc. However, printer drivers in modern operating systems are not designed to facilitate interactive information gathering. The operating system does not facilitate user interaction because the print job is redirected to another printer, or the print protocol does not allow such interactive sessions.

  Because of these limitations in printer interaction, the user cannot define more detailed print performance in standard printing. Furthermore, since such printing capabilities are not currently available, it is not possible to define any printing performance for everything related to multimedia content. Therefore, the user cannot use the current print dialog box to select a segment of multimedia content that is of interest to printing. The current print dialog box also does not allow the user to preview any multimedia content. Furthermore, there is no way for the user to search with redundant multimedia segments for the particular feature of interest. For example, a user cannot currently search for a new segment for content that covers a particular topic, or the user cannot search for a specific face or event in the new segment. Furthermore, there is no way to define a print format for a selected segment of multimedia content, and there is no way to examine or modify the print format directly through the print dialog box.

  Accordingly, there is a need for systems and methods that allow a user to interact and control the generation of multimedia representations that overcome the limitations recognized in the prior art.

  The present invention overcomes the limitations and deficiencies of the prior art using a system and method that provides a user interface that allows a user to interact with a media content analysis process and a media representation generation process. The system of the present invention includes a user interface to allow a user to control media content analysis and media representation generation. The media analysis software module analyzes and recognizes the characteristics of the media content. In addition, the system can include an output device driver module that receives instructions from the user to facilitate media content analysis and media representation generation. For example, the media software analysis module recognizes features such as faces, utterances, and texts. The system can also include an enhanced output device for generating the media representation. Processing logic manages the display of the user interface that allows the user to control the generation of the multimedia representation. Processing logic also implements the generation of printable multimedia representations. The representation can be generated in a paper-based format, a digital format, or any other representation format. The user interface includes a number of fields that allow a user to view media content and modify the generated media representation.

  The method of the present invention includes interacting with a user interface for controlling media data analysis and media representation generation. The method facilitates media representation generation by analyzing media data characteristics for media representation generation, facilitating media data analysis, receiving instructions and sending instructions regarding media representation parameters. Further comprising the step of:

  Systems and methods for providing a graphic user interface or print driver dialog interface that allow a user to interact with multimedia representation generation are described. In accordance with an embodiment of the present invention, a graphical user interface is provided for displaying multimedia information that can be stored in a multimedia document. In accordance with the teachings of the present invention, the interface allows a user to operate with multimedia information stored in a multimedia document.

  For the purposes of the present invention, the terms “media”, “multimedia”, “multimedia content”, “multimedia data” or “multimedia information” refer to text information, graphic information, animation information, audio (audio). Information, video information, slide information, whiteboard image information, and other types of information, or a combination thereof. For example, video recording of a television broadcast can be composed of video information and audio information. In a particular example, the video recording is also composed of closed captioned (CC) text information, which is composed of elements related to the video information, often the audio portion of the video recording. Is an accurate representation of the utterances contained in. Multimedia information is also used to reference information composed of one or more objects, which contain different types of information. For example, a multimedia object included in the multimedia information can be composed of text information, graphic information, animation information, audio information, video information, slide information, whiteboard image information, and other types of information. .

  For the purposes of the present invention, the term “print” or “printing” refers to printing on one type of media, printing, writing, drawing, imprinting, embossing, digital format generation and other types of data. It is intended to include the generation of expressions. Also, for the purposes of the present invention, the output generated by the system is “media representation”, “multimedia document”, “multimedia representation”, “document”, “paper document” or “video paper” or “ “Audio paper”. While word documents and paper are referred to in the above terms, the output of the system in the present invention is not limited to physical media such as paper media, for example. Instead, the above term can refer to any output that is fixed to some tangible medium. In some embodiments, the output of the system of the present invention can be a representation of multimedia content printed in a physical paper document. In paper format, multimedia documents take advantage of the high resolution and portability of paper to provide a readable representation of multimedia information. In accordance with the teachings of the present invention, a multimedia document can also select, retrieve, and access multimedia information. In other embodiments, the system can output to a digital format or some other tangible medium. Further, for the output of the present invention, any storage unit (eg, file) that stores multimedia information in digital format can be referenced. A variety of different formats can be used to store multimedia information. Those formats are MPEG format (for example, MPEG1, MPEG2, MPEG4, MPEG7, etc.), MP3 format, SMIL format, HTML + TIME format, WMF (Windows (registered trademark) Media Format), RM (Real Media) format, Quicktime format, Includes formats developed by the Shockwave format, various streaming media formats, engineering communities, proprietary and conventional formats, and other formats.

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of the present invention. However, one of ordinary skill in the art appreciates that the invention may be practiced without these specific details. In other instances, structures and devices are shown in block diagram form in order to avoid obscuring the present invention.
For example, certain features of the present invention will be described primarily in the context of video content. However, the features of the present invention can be applied to any type of media content including audio content, even if only the features related to video information are described.

  Criteria in defining “one embodiment” or “embodiment” mean that a particular feature, configuration, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. ing. Where the phrase “in one embodiment” in various parts of the specification, there is not necessarily all referring to the same embodiment.

  Now referring to FIG. 1, an exemplary system 100 that provides a graphical user interface that allows a user to format multimedia data to generate a multimedia representation is shown. In this embodiment, an extended output device or printer 102 for generating a multimedia representation is shown. The printer 102 displays multimedia data, such as audio data or video data, which the user can manage and edit via a user interface or print driver dialog interface (PDDI) 122. While the term “print driver dialog interface” or “PDDI” is used to refer to a graphic user interface, the graphic user interface is not limited to a printer, but any graphic user that provides the functionality described below. It can be an interface. The multimedia information displayed on the PDDI 122 can be stored in a multimedia document that is accessible to the system 100. The multimedia information can be stored directly in the system 100, and an external storage device or server (not shown) where the multimedia information can be accessed by the system 100 via the connection 140. ) Can be stored information.

  In other embodiments, instead of a multimedia document, the system 100 can receive a stream of multimedia information (eg, streaming media signal, cable signal, etc.) from a multimedia information source. In accordance with an embodiment of the present invention, the system 100 stores the multimedia information signal in the multimedia document and then generates an interface 122 that displays the multimedia information. Sources that can provide multimedia information to the system 100 include, for example, televisions, television broadcast receivers, cable receivers, video recorders, digital video recorders, portable digital terminals (PDAs), and the like. For example, the source of multimedia information can be embodied as a television configured to receive multimedia broadcast signals and transmit the signals to the system 100. In this example, the information source can be a television receiver / antenna that provides live television supply information to system 100. The information source can also be a device, such as a video recorder / player, DVD player, CD player, etc., that supplies video and / or audio streams recorded in the system 100. In other embodiments, the source of information can be a presentation or meeting recorder that can provide a stream of presentation or meeting information captured by the system 100. Further, the source of multimedia information may capture or receive multimedia information from an external source and then provide the captured multimedia information to system 100 for further processing (eg, via a wireless link). It can be a receiver (for example, a satellite broadcasting antenna or a cable receiver). The multimedia content can originate from a dedicated or customized multimedia player, such as RealPlayer®, Microsoft Windows® Media Player, for example.

  In other embodiments, the system 100 can be configured to intercept a multimedia information signal received by a multimedia information source. System 100 can receive multimedia information directly from a multimedia information source and can receive information over a communications network.

The extended output device or printer 102 is comprised of a number of components including a conventional printer 103, media analysis software module 104, processing logic 106, and digital media output 108. The conventional printer 103 components of the printer 102 can include all or part of the capabilities of a standard or conventional printing device, such as an inkjet printer, laser printer, or other printing device. Thus, the conventional printer 103 has the capability to print paper documents, and can include fax machines, copiers, and other devices for generating physical documents. More information about the printing system
Entitled “Networked Printing System Having Embedded Functionality for Printing Time-Based Media” and provided in US Patent Application Publication No. 10 / 814,948, filed March 30, 2004 by Hart et al. This document is replaced by a part of the description of the invention.

Media analysis software module 104 also includes audio and video content recognition and processing software. The media analysis software module 104 can be located in the printer 102 or can be remotely located, for example, in a personal computer (PC) or the like. Some examples of such multimedia analysis software are video event detection, video foreground / background segmentation, face detection, face image matching, face recognition, face cataloging, video text localization, video optical character recognition (OCR), language conversion, frame classification, clip classification, image stitching, audio reformatter, speech recognition, audio event detection, audio waveform matching, audio caption alignment, video OCR and caption alignment It is not a thing. Once the user selects “print” in the system 100, the system 100 analyzes the multimedia content using one or more of these techniques and generates an analysis result that allows the user to generate a document. Can be provided to the user,
In the embodiment shown in FIG. 1, the printer 102 additionally includes processing logic 106 that controls the PDDI 122 to manage the generation of the multimedia document 120 or media representation by the printer 102. For example, the processing logic 106 manages the display of the PDDI 122 that allows the user to manage certain printer actions, such as processing multimedia content or formats in which the multimedia content is displayed in a multimedia representation. Alternatively, the functionality of the PDDI 122 can be provided by a web interface that allows the user to manage printer actions such as issue formatting.

  In the example shown in FIG. 1, PDDI 122 displays a user selection including printing of multimedia document 120 displaying video content. In this example, the user selects to have multimedia content printed in video paper format, and the video paper displays one frame per scene. The interface 122 also includes a preview field 124 that displays a preview of the multimedia representation being generated by the user. In the example of FIG. 1, PDDI 122 shows a thumbnail picture 142 of the display frame.

  In addition, PDDI 122 allows the user to set the preferred formatting associated with the generated multimedia document. In some embodiments, the user can format and layout the document, font type and size, information displayed in each line, information displayed in the header, schedule column size and position, font color, line Preferences regarding spacing, number of characters per line, bolding and capitalization technology, language in which the document is printed, paper size, paper type, etc. can be set. For example, the user can choose to have a multimedia document that includes a large, bold header that indicates the name of the multimedia content to be displayed (eg, a segment of CNN news). The arrangement of video frames displayed per page can be selected.

  As shown in the embodiment of FIG. 1, a data structure called a document format specification (DFS) 112 is generated by print driver software. DFS 112 represents the conversion of multimedia data. DFS 112 is used to store PDDI 122 and is modified by system 100. The DFS 112 determines feature extraction options provided to the user, and the feature extraction options can be applied to the multimedia data. The DFS 112 also determines the format guidelines used to generate the output document. The DFS 112 can be supplied by an external application, such as a print driver in a PC, or can be determined internally in the printer 102.

The DFS 112 may include metadata information about the multimedia file, such as information about the title of the multimedia content, producer / publisher of the multimedia content, etc. The DFS 112 also specifies the start time and end time of the multimedia segment (eg, the start time and end time of the audio recording) and the designation of the graphical representation of the multimedia data that can be displayed along the timeline (eg, time). Other information such as (a waveform indicating the amplitude of the audio signal). DFS 112 provides metadata and timestamp markers for each timestamp (eg, text tag or barcode) that can be displayed along the timeline, and layout parameters that determine the appearance of physical multimedia document 120. A designation for can be further included. For more information and examples about DFS 112, see “
“Printable Representations for Time-Based Media” and is provided in US Patent Application Publication No. 10 / 814,844, filed March 30, 2004, by Hull et al. Substitute part of the description of the invention.

  The multimedia document 120 generated by the printer 102 can be composed of various formats. For example, the multimedia document 120 may be composed of a paper document such as a video paper of the form shown in FIG. The multimedia document 120 provided by the printer 102 can also be stored on the digital media 144. As shown in FIG. 1, this printer 102 embodiment includes a digital media output device or interface 108. The digital media writing hardware can include, for example, a network interface card, a DVD (Digital Video Disc) writer, a secure digital (SD) writer, a CD (Compact Disc) writer, and the like. The multimedia content can be stored in digital media 144 such as flash media, DVD, CD, for example.

  Multimedia document 120 has many different types of layouts and can display various types of information. FIG. 1 shows an example of a video paper document that displays video frames from one or more news segments. In the example of FIG. 1, the video paper document includes thumbnail images or frames 132 extracted from the video information and can display video content that the user can preview. In this embodiment, the user can specify a preferred formatting for the video paper document via PDDI 122. The layout and format information can specify the sampling rate for extracting the media frame 132, the number of frames 132 extracted from the video information, the order and position of the frames 132 in the media, and other information. is there. For video information, the printer 102 can extract frames that capture salient features of the video (or information-providing frames) for a particular segment of multimedia information. Further, as described above, the printer 102 allows the user to search in the video segment for items of interest such as, for example, specific facial images, specific words displayed as text, Feature recognition capabilities (eg, face recognition, face detection, OCR, etc.) can be included. For example, the printer 102 may use a face recognition technique to extract a frame that displays an image of a particular person's face that the user is interested in seeing.

  In other embodiments of the invention, a user selectable identifier 134 (eg, a barcode) is associated with each frame 132. In the example of FIG. 1, user selectable identifiers 134 are displayed below each frame 132, but they can also be displayed somewhere on this page. User selectable identifier 134 serves as an interface that allows a user to access or retrieve multimedia content displayed in multimedia document 120. A user selects a user-selectable identifier by scanning the appropriate barcode in a printed paper document using any type of device that has an integrated barcode scanner, such as a cell phone or PDA. . For example, by scanning the bar code of FIG. 1, the user can cause the video clip to be displayed on a display device (eg, television, PC monitor, cell phone screen, PDA, etc.), and the user can view the content. Can see. As another example, the paper multimedia document 120 can also include a numeric identifier included in each frame 132, and the user is associated with a device that allows the system 100 to display a video clip on the display device. You can type those numbers on the touchpad or keypad. Also, if the video paper document shown in FIG. 1 is in a digital format, the system 100 allows the user to select a frame 132 that causes the video content to be displayed on the display device (ie, a mouse or other device). Can be configured (by clicking on the frame with the selection device).

  The printer 102 can retrieve multimedia information corresponding to the user selectable identifier 134. A signal communicated to the printer 102 from a selection device (ie, a device having a keypad or barcode scanner for entering a numeric identifier) is sent to the multimedia content frame 132 selected by the user, the location of the displayed multimedia content. Information relating to the multimedia paper document from which the segment is selected, preferences and / or one or more multimedia display devices (eg, television sets) selected by the user, and requested multimedia information It is possible to identify other information to facilitate the search. For example, the system 100 can access a video file stored on the PC, and the system can play this video content upon user command.

  The example of FIG. 1 further shows the text information following each frame in the multimedia document 120. The text information includes a field for displaying a speaker name field 126 or a person's name (for example, Brit Human) shown in the video frame 132. The text information further includes a subject field 128 that displays information about the subject of the video segment (eg, Intro Intel-gate). The text information also includes a time field 130, which displays the length of time of the video segment (eg, 3 minutes 52 seconds).

  The user can also choose to include a portion of the audio information for frame 132 in multimedia document 120, and the portion of the audio information is displayed as text. For example, the user may choose to have a portion of the multimedia segment copy (ie, a copy of the news program segment) displayed next to the multimedia frame 132. As another example, the user can choose to include a summary or text description of the content in each frame 132, such as a summary of a particular television segment or program, in the printed document. The user uses the technology used to convert the speech information into text information (ie, the technology for generating a text copy for the speech information) and the format and style for printing the speech copy (print the text information). The print driver dialog interface 122 can be used to identify the format and style for printing a summary list of multimedia content, etc. it can. In addition, information about retrieving multimedia information and annotating multimedia information is provided to the video paper application.

  Now referring to FIG. 2, the architecture of an embodiment of the present invention is shown. In this embodiment, system 200 includes a printer 102 coupled to a data processing system, which is a PC 230 in the embodiment of FIG. 2, but is also a portable computer, workstation, computer terminal, network. It can be a computer, mainframe, kiosk, standard remote control, PDA, game controller, mobile phone or any other data system. The printer 102 can also be optically coupled to the application server 212 in a network environment.

  In the example of FIG. 2, the printer 102 includes the following components. That is, the conventional printer 103, processor 214, multimedia storage 202, and digital media input / output unit 108. As described above, the conventional printer 103 has the standard printing capability that the conventional printer generally has.

  The processor 214 is composed of various computing architectures that process data signals and include a CISC (Complex Instruction Set Computer) architecture, a RISC (Reduce Instruction Set Computer) architecture, or an architecture that executes a combination of instruction sets. Is possible. Although only one signal processor is shown in FIG. 2, it is possible to include multiple processors. Main memory (not shown) may store instructions and / or data that can be executed by processor 214, which includes software and other components of system 200. The instructions and / or data can be comprised of code for performing any and / or all of the techniques described herein. The main memory (not shown) can be a DRAM (Dynamic Random Access Memory) device, an SRAM (Static Random Access Memory) device, or other memory device known in the art.

  As described above, the printer 102 accesses or receives multimedia information such as audio or video files from a certain source. In one embodiment, the multimedia file is stored in a data processing system, such as PC 230, which is coupled to printer 102 by signal line 248. In the embodiment of FIG. 2, the multimedia file can be stored in a multimedia file store 264 in the PC 230. Multimedia files can also be accessed from some remote source (not shown). As another example, the multimedia file can itself be stored in the printer multimedia storage 202 of the printer 102 itself, and the file is accessed from this storage.

  As described above, a user can view multimedia content on a display device (not shown) in order to select content specifically for printing using printer 102. The display device (not shown) includes a flat panel device such as a cathode ray tube (CRT) and a liquid crystal display device (LCD), a projection device, and the like. In other embodiments, the printer 102 includes an LCD display panel or other type of display panel that allows a user to display the printer's own multimedia content.

  In the embodiment of FIG. 2, the user uses a multimedia rendering application (MRA) 204 on the PC 230 that allows the user to play, store, index, edit or manipulate multimedia information. Can see. Examples of MRA 204 are dedicated or customized multimedia players (eg, RealPlayer (registered trademark) from RealNetworks, Microsoft Windows (registered trademark) Media Player from Microsoft), QuickTime (registered trademark) player from Apple, Inc. Shockwave multimedia player, etc.), video player, television, PDA and the like. In the embodiment of FIG. 2, MRA 204 is coupled to multimedia file store 264 by bus 244. The stored multimedia content can be accessed by MRA 204 and transferred to MRA 204 for viewing by the user. Further information on multimedia visualization is the following US patent application publications, each of which is incorporated by reference in its entirety. That is, these documents are entitled: “Multimedia Visualization and Integration Environment”, US Patent Application Publication No. 10 / 081,129 filed by Graham on February 21, 2001; “Multimedia Visualization and Integration”. No. 10 / 701,966 entitled “Environment” and filed by Graham on Nov. 4, 2003; entitled “Interface For Printing Multimedia Information”, 18 June 2003 And US Patent Application Publication No. 10 / 465,027 filed by Graham et al .; “Tech "Request For Displaying Information Stored In Multiple Multimedia Document", published on June 18, 2003 by Graham et al., "Television-Based Visualization 200" United States Patent Application Publication No. 10 / 174,522 filed by Graham on June 17, and entitled “Multimedia Visualization and Integration Environment”, filed by Graham on March 3, 2004 US Patent Application Publication.

  In the embodiment of FIG. 2, the system 200 also includes an output device driver module or printer driver software module 208 that can be located at the PC 230 or elsewhere. The printer driver software module 208 is set during installation to perform a specific function. The printer driver software module 208 adds a “print” function to an existing MRA 204 such as Windows (registered trademark) Media Player. An optional application plug-in 206 is required to add a “print” function. Alternatively, the user can install a separate MRA 204 designated for this purpose. When the printer 102 is invoked (ie, the user selects the print button on the MRA 204), the printer driver software module 208 receives a print request from the MRA 204 along with multimedia data and other relevant information through the signal line 246. . The printer driver software module 208 sends multimedia data to the printer 102 via the bus 248 and instructs the printer to apply a specified conversion routine (eg, face recognition). The printer driver software module 208 can further instruct the user to confirm the layout decisions and results made by the user, if necessary.

  When printer 102 receives a print request, the request and associated multimedia data are forwarded to processor 214. The processor 214 interprets the input and activates the appropriate module. The processor 214 is coupled to and controls a multimedia conversion software module (MTS) (not shown) for converting multimedia content. If processor 214 receives the print request, processor 214 can then activate an MTS (not shown) depending on whether the user has requested conversion of multimedia data. is there. The conversion to multimedia content can be applied at the printer 102, PC 230 (ie, by software installed using the print driver 208) or other location. TMS (not shown) applies a conversion function specified for a predetermined audio or video file. MTS (not shown) generates an appropriate document-based representation and interacts with the user via a print driver dialog interface to modify the parameters of the conversion and preview the results. The results and parameters of the multimedia conversion are expressed in the state of the document format specification (Document Format Specification (DES)) described above.

  As described above, the printer 102 can include a multimedia storage unit 202 for storing multimedia data such as video or audio files. The processor 214 is coupled to the multimedia storage unit 202 and can transmit multimedia data to the multimedia storage unit 202 via the bus 251. This data can be stored while the print job is being processed. The multimedia storage unit 202 may include a number of memory types including a main RAM (Random Access Memory) for storing instructions and data during program execution and a ROM (Read Only Memory) that stores the defined instructions. Is possible. The multimedia storage unit 202 may also be, for example, a hard disk drive, floppy disk drive, CD-ROM drive, DVD-ROM device, DVD-RAM device, DVD-RAM device, DVD-RW device, or the art. It is possible to include fixed (non-volatile) storage for programs and data files, such as other known similar storage devices. One or more drives or devices can be located at remote locations in other connected components.

  The processor 214 also controls the digital media input / output unit 108. The processor 214 transfers information to and receives information from the digital media input / output unit 108 via the bus 250. The generated multimedia document can be converted to some type of digital format as described above. The digital media writing hardware can include, for example, a network interface card, a DVD (Digital Video Disc) writer, a secure digital (SD) writer, a CD (Compact Disc) writer, and the like. The digital output 260 document can be stored on digital media including CD, DVD, flash media, and the like. Thus, the user can generate a digital output 260 version of the input audio or video file, and can view this digital output 260 version on a designated target device such as a PC, mobile phone or PDA, for example.

  The processor 214 also manages the generation of multimedia documents such as video or audio paper documents. The multimedia information can also be displayed in a paper document or multimedia document 120, as shown in FIG. The processor 214 communicates with the conventional printer 103 via the bus 252 and transmits print job information thereto, and the conventional printer 103 generates a paper output. The generated multimedia document 120 includes a paper representation of input audio or video file information obtained from recognition software. The multimedia document embodiment of FIG. 2 also includes a user selectable identifier, such as a barcode, and other links to multimedia data stored in the online database stored or designated by the printer 102. Can do.

  The processor 214 also controls external communication hardware via a network interface or the like. The processor 214 can transfer information to and receive information from the application server 212 via the bus 254. The printer 102 can also communicate with and obtain information from an application server 212 (eg, a “web service” system or a “grid computing” system).

  In one embodiment, the system 200 includes a communication monitoring module or user interface listener module 210 (UI Listener). In the embodiment of FIG. 2, UI Listener 210 is located at PC 230, but UI Listener 210 may alternatively be located at printer 102, application server 212, or some other remote location. The UI listener 210 is coupled to and communicates with the MRA 204 and can send and receive data to and from the bus 240. Specifically, the UI listener 210 receives a print request from the user to the MRA, and transmits a request to the user from a remote component (for example, the printer 102, the application server 212, etc.). UI listener 210 may also be coupled to and communicate with printer 102 to send and receive data to bus 242. Specifically, the UI listener 210 transmits a print request to the printer and receives a request for further information from the user from the printer 102. In addition, the UI listener 210 is coupled to and can communicate with the application server 212 in the network and can send and receive data over a network connection (not shown). The UI listener 210 receives information from the application server 212, such as a request for information from the user, and the UI listener 210 can send back a response. The UI Listener 210 and its functions are described in detail below.

  Referring now to FIG. 3, a graphical representation of interactive communication with the printer 102 in the system 200 is shown. Printer drivers typically do not facilitate interactive information collection. Once the initial printer settings are captured, further interaction with the printer 102 is generally not allowed. One way to deal with this problem is to incorporate sleeping data into the print stream itself. However, the printer 102 needs to request more information from the user in response to calculations performed from data supplied by the user. In addition, the printer 102 itself can delegate some tasks to other application servers 212, and such other application servers 212 may also require additional information from the user 302. .

  An additional mechanism as shown in FIG. 3 can be configured to allow this interaction without modifying the basic operating system printer driver architecture. One solution is UI Listener 210, which builds a program that listens to network sockets, accepts requests for information, interacts with the user to obtain such data, and returns data to the requester. is there. Such a program has a fixed set of valid dialogs or can allow flexible instruction syntax, which allows the requester to display many different requests. To do. An example of such an instruction syntax is the ability of a standard web browser to display HTML forms. Such a form is generated by a remote server and displayed by a browser, which then sends the results back to the server. In this embodiment, the UI listener 210 differs from the browser in that the user 302 does not generate an initial request to view the form. Instead, the remote machine generates this request. Therefore, the UI listener 210 is a server, not a client.

  Because this type of network transaction tends to cause many complex error conditions, a timeout system provides efficient operation. Each message sent across the network is typically a response or a unilateral message. A message requesting a response may have a timeout or a time limit for the time allowed for a response to arrive. In the present invention, the embedded metadata includes metadata about the UI listener 210 that allows a request for further information. Such metadata has at least a network address, a port number, and a timeout period. Such metadata also avoids malicious attempts to retrieve information from the user 302 because it is not known whether the request was made by the printer 102, a delegated server or a malicious agent. It is possible to include specified authentication information. When the printer 102 or the entrusted application server 212 requests further information, the printer 102 or the entrusted application server 212 can use the above information in response to a request that the UI listener 210 requests information necessary for the user. . The UI Listener 210 program can be located at the user 302's interactive device (eg, PC, mobile phone, PDA), printer 102 (ie, for user interaction on the LCD panel located on the printer) or other remote location. .

  FIG. 3 shows an example of the above interactive communication system. In the example of FIG. 3, the user 302 selects a “print” option in the system 200, which “print” option clicks a print button added to the MRA 204 or other method of selecting the “print” option. Can be included. By selecting “Print”, the user 302 sends a print request 304 to the MRA 204 or other application. For example, user 302 may request to print a video paper representation of a user selected video frame from a news segment about CNN. The MRA 204 transmits a request notification 306 message to the UI listener 210, and requests that the UI listener 210 notify the printer 102 of the print request 304. Print job 308 information, such as user-defined layout and formatting preferences for video paper, is then sent from the MRA 204 to the printer 102. The print job 308 can include embedded information such as, for example, the network address of the UI listener 210, authentication information, and information about the latest time that the client views for the request.

  In the example of FIG. 3, the printer 102 sends a request for information to the UI Listener 210 program located in the user 302 interaction device. For example, the printer 102 can request further information about a particular layout preference selected by the user for a video paper print job, or a new default layout needs to be used. It is possible to confirm. The UI listener 210 then provides this request to the user 302, and the dialog box is displayed 312 to the user 302, allowing the user 302 to respond to the request by selecting information in the dialog box. . A response 314 of the user 302 is transmitted to the printer 102 in a state of an answer to the request of the printer 102 for the information 310.

  Further, in the example of FIG. 3, the printer 102 transmits a request 316 for information to the application server 212. For example, the printer 102 can request specific data needed for print options from a database, which needs to collect further information from the user. In the example of FIG. 3, the application server 212 sends a request 318 for information to the UI Listener 210, which then forwards the request 318 to the user 302. A dialog box is displayed 320 to the user, allowing the user 302 to respond to the request 318. The UI listener 210 may then forward the user 302 response 322 to the application server 212, which may then send a response 324 to the printer 102 in response to the printer 102 request 316 for information.

  Referring now to FIG. 4, a graphical representation of MRA 204 having a “print” button 402 added to MRA 204 is shown. In this example, the MRA 204 box is a Windows® Media Player (WMP) application, but as described above, other types of MRA 204 can be used. The user can select print options by clicking a print button 402 that causes the printer to generate a multimedia document. Printing options can be added to WMP version 9 by utilizing plug-in functionality supplied by Microsoft. Plug-in functionality allows developers to somehow develop applications that supplement WMP. Several types of plug-in functions can be developed, such as “view”, “settings”, “metadata”, “windows and backgrounds”, etc. Microsoft explains what a plug-in is and how to build it. By using one of the user interface plug-in styles, buttons or panels can be added to the WMP screen. For more information on adding print options to applications, see “Printing System with Embedded Audio / Video Content Recognition and Processing”, published in US Patent Application Publication No. Hull et al. 10 / 813,950, which is incorporated herein by reference in its entirety.

  Optionally, system 200 provides a method for printing multimedia content. The user selects a print option in the MRA and an initial print driver dialog interface (PDDI) 122 is presented to the user. The initial PDDI 122 includes information about the capabilities of the printer 102 for transferring multimedia data. The initial PDDI 122 can display options that are available to the user to transfer data, or can indicate the results of performing a default transformation with a default set of parameters. The user can select between two options that the user prefers, and the user's preferences can also be set in the characteristics of the printer 102. The flow of operation for each such option is shown in FIGS. 5 and 6, described below. Further information on the different transformations that can be performed and the options available to the user for the transformations, entitled “Printer with Embedded Audio / Video Content Recognition and Processing”, on 30 March 2004, Hull In U.S. Patent Application Publication No. 10 / 813,950, filed by E. et al., Which is incorporated herein by reference in its entirety.

  Referring now to FIG. 5, a flowchart illustrating the operational flow in the system 200 when displayed to the user before any multimedia conversion is performed is shown. In this embodiment, the user enters a “print” command into the system by pressing 502 a print button (eg, FIG. 4) in the MRA. The user can use the initial PDDI 122 to define preferences associated with multimedia documents that are generated before any conversion is performed. The user selects 506 parameters for conversion to be applied to the multimedia content. For example, the user can select to have a document showing a specific number of video frames displayed in a user-defined array.

  The system 200 then waits 508 for the user to press an OK or update button on the PDDI 122. If the user selects the cancel button, the system 200 ends and the PDDI 122 disappears from view. Once the user selects the update button or the OK button, the system 200 sends 510 parameters and other user selection information to the printer 102. System 200 determines whether multimedia data has already been transferred to printer 102. As described above, this multimedia data can be located on a PC, mobile phone, PDA or other device that can include multimedia content. If the multimedia data has not yet been transferred to the printer 102, the system 200 transfers 512 the multimedia data to the printer 102 and the operational flow continues. If the multimedia data has already been transferred to the printer 102, the system 200 determines whether a multimedia conversion with user-defined parameters has already been performed. If not, the printer performs a conversion 514 on the multimedia data. If yes, the system 200 determines whether the user has pressed the update button after entering the parameters, or alternatively, has pressed the OK button. If the user has not pressed the update button, and instead presses the OK button, the printer 102 generates 516 control data that links to the document, multimedia data, and a paper document having multimedia data. In addition, the system 200 assigns identifiers (eg, barcodes) to multimedia data and provides an interface to the user depending on which multimedia content is accessed. If necessary, before the document is generated, the printer 102 may first prompt the user for further information related to the print job. Metadata about multimedia data and commands input to the PDDI 122 is represented in the DFS 112.

  If the user presses the update button instead of the OK button, the user has not yet requested that the printer 102 generate a multimedia document. Rather, when the user modifies user selection parameters in the PDDI 122, the user presses the update button and the user asks for a preview field of the PDDI 122 to be updated. If the user presses the update button, the system 200 interactively returns 518 the results for the display in the interactive PDDI 122. This allows the user to preview how the multimedia document will appear with modification of newly added parameters. The flow of operation then returns to the point where the user has a 506 opportunity to select parameters, and the system 200 can again cycle through the flow until the final document is generated in the interactive PDDI 122. Continue to modify parameters.

  Referring now to FIG. 6, a flow diagram for illustrating the operational flow in the system 200 when the system transfers multimedia data, performs default conversion, and displays the results in the PDDI 122 is shown. ing. In this embodiment, the user presses 602 a print button in the multimedia rendering application. The system 200 reads 620 parameters and default conversions from printer characteristics information stored in the printer 102, PC or other location. System 200 then determines whether multimedia data has already been transferred to printer 102. If the multimedia data has not yet been transferred to the printer 102, the system 200 transfers the multimedia data to the printer 102 612 and continues the operational flow. If the multimedia data has already been transferred to the printer, the system 200 determines whether a transfer with the defined parameters has already been performed. If not, the printer performs a conversion 614 on the multimedia data. If so, the system 200 displays 604 PDDI 122 to the user and indicates the conversion result. The user modifies 606 parameters for the transformation applied to the multimedia content. The system 200 then waits 608 for the user to press the update or OK button on the PDDI 122. If the user selects the cancel button, the system 200 ends and the PDDI 122 disappears from view. If the user presses the OK button, the printer 102 generates the document, multimedia data 616, controls the data linked to the paper document having the multimedia data, and the system 200 assigns an identifier to the multimedia data. If the user presses the update button, the system 200 sends the parameters for conversion 610 to the printer 120, and then the flow of operations cycles again.

  FIGS. 7-19 show examples of PDDI 122 displayed to the user in the method of FIG. 6 in which default conversion is first applied to multimedia data before the user views PDDI 122, as described below. Show. Further, the example of FIGS. 7-19 can also be the interactive PDDI 122 shown in the method of FIG. 5, which allows the user to modify the result of the conversion. In the method of FIG. 5, an example of the initial PDDI 122 that is initially displayed to the user is entitled “Printer with Embedded Audio / video Content Recognition and Processing”, filed by Hull et al. On Mar. 30, 2004. U.S. Patent Application Publication No. 10 / 081,129, which is incorporated herein by reference, and replaces part of the description of the invention.

Audio FIG. 7 shows graphic conversion of the PDDI 122 for printing a user selected area in an audio file. A user can enter information into fields in the PDDI 122 to specify user preferences for layout, segmentation, and the like. The PDDI 122 shown in the embodiment of FIG. 7 includes some fields found in a standard printer dialog box such as, for example, a printer field 704, a print area field 706, and a copy and adjust field 708. However, PDDI 122 also displays fields that are not found in standard printer dialog boxes, such as Advanced Options Field 710, Preview Field 712, and Content Selection Field 714.

  As shown in the standard printer dialog box, the top of PDDI 122 includes a file name field 702 that displays the name of the multimedia file to be printed (eg, “lomotion.mp3”). In the printer field 704, the user can select which printer performs the print job and other options such as print job, printing as an image or file, print order, and the like. Further, the printer field 704 displays the status of the selected printer, the type of printer, where the printer is located, and the like. The print range field 706 allows the user to select which part of the document is to be printed and so forth. The copy and adjust field 708 allows the user to specify the number of copies to be generated in the print job, the size of the print job page associated with the paper, the alignment of the print job page on the paper, and the like. Although not shown, the dialog box can also include any various combinations of video, audio, or other conventional printing parameters associated with outputting a representation of a document.

  In the embodiment of FIG. 7, the advanced options field 710 provides the user with specific options for the layout and formatting of the media media content. In this embodiment, the user can select the type of segmentation that the user wanted to apply to the multimedia content. In this embodiment of the invention, the user clicks on the arrow in the segmentation type field 716 to bring up a drop-down menu that can display a list of types that the user can select. Segmentation types include, for example, speech event detection, speaker segmentation, speaker recognition, speech source localization, utterance recognition, profile analysis, video event detection, color histogram analysis, face detection, clustering, face recognition, optical character These include but are not limited to recognition (OCR), motion analysis, distance prediction, foreground / background segmentation, scene segmentation, car recognition and license plate recognition. In this example, the user has not selected any segmentation type in the segmentation type field 716, so the segmentation type is indicated as “NONE”. Accordingly, in this example, the user manually selects the audio range in the audio waveform timeline 734 by moving the selector 736 in the content selection field 714.

  Each segmentation type can have a confidence level associated with each of the events selected in the segmentation. For example, if the user applies audio event detection that segments the audio data according to the applause event that occurs in the audio data, each applause event has a corresponding confidence level that defines the confidence that the applause event is accurately detected. . In the advanced options field 710, the user can define or adjust a threshold for a confidence value associated with a particular segmentation. The user sets the threshold by typing the threshold in the threshold field 718. For example, the user can set 75% of the threshold, and only display events that are greater than this threshold (ie, the probability that the event will be accurately detected as a clap event is greater than 75%). In other embodiments, a threshold slider (not shown) is included in PDDI 122 and the user moves the slider along a threshold bar that moves from 0% to 100% to select a particular threshold within that range. Can move.

  In one embodiment, the user can also make layout selections associated with the generated multimedia representation. The user sets the number of pages on which the audio waveform timeline 734 is displayed in the “fit on field 720. The user also sets the number of timelines displayed on each page in the timeline number selection field 722. Further, the user selects the display direction (eg, vertical or horizontal) of the timeline in the multimedia representation in the direction field 724. For example, as shown in FIG. You can choose to have one timeline displayed on one page in the horizontal direction, which displays the overall audio waveform timeline 734 in the horizontal direction on the page. The user can see four displayed vertically in two pages (ie two timelines per page) It can be selected to have a speech waveform timeline 734 is divided into minute.

  In the embodiment of FIG. 7, various buttons including an update button 726, a page setting button 728, an OK button 730, and a cancel button 732 are shown. As described in connection with FIGS. 5 and 6, when the user modifies the print job parameters in PDDI, the user can select the update button 726 and the user can see how the multimedia representation appears. Hope to see the updated image. The image of the multimedia document shown in preview field 712 is updated to display any new changes that the user has made in PDDI 122. The system can also be specified to automatically change the preview field 712 that any time change is made in the PDDI 122. In one embodiment, when the user selects the page setup button 728, a different dialog interface box is displayed to the user, providing the user with various print formatting options. This will be explained in more detail below. The embodiment of FIG. 7 also includes an OK button 730, when the user selects this button, the printer prepares to generate a multimedia document under the current user-defined preferences set in the PDDI 122. If the user selects the cancel button 732 at any point in the process, the print job end generation and PDDI 122 disappear.

  In the embodiment of FIG. 7, the content selection field 714 shows an audio information waveform in a timeline that displays audio data selected by the user for conversion and printing. In this example, the upper part of the audio waveform timeline indicates time “00:00:00” or the start time of this audio content. The lower part of the audio waveform timeline 734 indicates time “00:07:14” or the end time of the audio content. Therefore, the audio information in this example is 7 minutes 14 seconds. A user can slide selector 736 along audio waveform timeline 734 to select a particular segment of audio content to be displayed for the multimedia document for which the corresponding marker or identifier was generated. For example, the user can use a mouse or other selection device to click and slide the selector 736 over the segment 740, which is shown as the selected segment in FIG. In one embodiment, once the selector 736 is positioned on the segment of audio content that the user wishes to select, the user can click or double-click the selector 736 to select the segment. In this example, the user can select a longer segment of audio content by clicking and dragging selector 736 across the amount of movement of the audio segment that the user wishes to select. The audio waveform timeline 734 can also be a number of alternatives, such as showing a horizontal timeline, showing two or more timelines next to each other, showing different waveform displays, showing waveforms colored according to a particular scheme, etc. Can be displayed.

  In the embodiment shown in FIG. 7, the user has selected three regions of the audio waveform timeline 734 to be carded in the multimedia document. The user has selected segment 740, segment 742 and segment 744. Each of these selected segments has a separate corresponding marker or identifier 766 (eg, barcode) displayed in the print preview of the multimedia document. For example, an image of the multimedia document is shown in the preview field 712 of FIG. This document shows one page containing one horizontal timeline displayed with the beginning of the audio timeline for the left and the end for the right. In this example, the full audio waveform timeline 736 is displayed on the multimedia document page. Also, the timeline displayed in the preview field 712 is three markers or identifiers 766, one for segment 740, one for segment 742, and one for segment 744. Is about. Each marker 766 includes a barcode and a time stamp that gives the location of the segment in the audio content. Furthermore, the example of FIG. 7 shown in preview field 712 includes a preview field that includes a header that may include information about the audio content (eg, the title of the audio content, the musician who created the audio content, and the date of the audio content). 712 is shown. The multimedia document further includes a play identifier or play marker 760 that can be located somewhere in the document (ie, bottom, center).

  Users can play audio content in many ways. For example, the user can click the play selector or play arrow 750 in the audio waveform timeline 750 to start the segment playing. Further, the system can be configured such that selection of the play arrow 750 causes all audio content in the audio waveform timeline 734 to begin playing. The user can also right click on any one selected segment to find the corresponding marker in the multimedia document. The paper multimedia representation can also provide an interface for playing audio content. The user can select any marker (ie, scan a barcode) for any selected segment in the paper document. For example, a user can scan a barcode using a cell phone or PDA device that has a barcode scanner. The user can listen to selected excerpts on the mobile phone or PDA, or the user can listen to the content with a sound card on the user's PC. The user can also select a play marker 760 that functions as a pause button, so if the user selects any marker on the page to play the corresponding audio content, the user selects the play marker 760 By doing so, the audio content can be stopped. The user can resume playing the audio content by selecting play marker 760 again, or the user can select other markers on the page to play the corresponding audio content. .

  Referring now to FIG. 8, there is shown a graphical representation of the PDDI 122 for a set multimedia document page or a page with a dialog interface 800 set. When the user selects the page settings button 728 as described above, the page settings dialog interface 800 is displayed and the user can select formatting options. In the paper field 802, a user can select a paper size (eg, letter) and a paper source (eg, automatic selection) for a multimedia print job. In the direction field 804, the user can specify whether to direct the document in portrait or landscape format. In the preference field 806, the user can set the title, text font type and size (eg, Helvetica, 22 inch size), media type (eg, video), and marker position (eg, above the waveform) on the page. The user can also decide whether to print the waveform, center title, timeline barcode and its frequency, and time label. Also, each of the page setting options shown in the page setting dialog interface 800 can be incorporated into a main PDDI such as the PDDI shown in FIG. The page setup options are not limited to the options as shown in FIG. 8, and in other embodiments, a variety of different page setup options are provided to the user.

  Referring now to FIG. 9, there is shown a graphic representation of the PDDI 122 that generates a two page summary of the audio file. PDDI 122 is similar to the PDDI shown in FIG. 7, but the user has selected in timeline number selection field 722 to include three timelines to be printed for each page of the multimedia document. In addition, the user has selected in page fit field 720 to print the multimedia document over two pages. The user has selected four segments of audio content in which the marker is displayed in the content selection field. Selected segments include segment 902, segment 904, segment 906 and segment 908.

  The multimedia document shown in the preview field 712 of FIG. 9 displays three timelines on the page, indicating that two pages have been generated. A time stamp 910 at the beginning of each horizontal timeline indicates the start time of the timeline. The timeline shown on the upper page in the preview field 712 corresponds to half of the audio waveform timeline 734 shown in the content selection field 714. More specifically, the multimedia document displays the upper half of the speech waveform and is divided into three separate timelines. Markers corresponding to the selected segments 902 and 904 are displayed on the page shown in the preview field 712. Markers corresponding to the selected segments 906 and 908 are displayed on the second page and their contents are not visible in the preview field 712.

The document in preview field 712 of FIG. 9 additionally includes a timeline marker 912 near the beginning and end of each of the three timelines displayed on the page. These timeline markers provide the user with additional intermediate interface points in the printed document that allow the user to access multimedia content. The timeline marker 912 indicates the position in the audio content corresponding to the beginning or end of each printed timeline, and the user selects the marker for the audio content to start playing at that position in the audio file (ie. By scanning the barcode as described above, these positions can be accessed.
Although the timeline markers 912 in FIG. 9 are displayed below the timelines, these timeline markers 912 can also be displayed above or near the timelines. These timeline markers 912 also provide other interfaces that are displayed in the printed document and allow the user to access multimedia content at defined locations.

  FIG. 10 shows a graphical representation of the PDDI 122 with the timeline divided into two parts. PDDI 122 is similar to the PDDDI shown in FIG. 7, but the user has selected in the “fit on” field that the output is limited to one page. The timeline number selection field 722 specifies two timelines per page. Accordingly, the audio waveform timeline 734 shown in the content selection field 714 is divided into two, half of which are displayed in the multimedia document as two horizontal timelines. The user selects segments again in the audio waveform timeline 734 and the markers corresponding to those segments are displayed in the multimedia document. Specifically, the user has selected segments 1002, 1004, 1006 and 1008.

Referring now to FIG. 11, a graphical representation of the PDDI 122 is shown with the timeline divided into two vertical parts and the segmentation type and threshold level applied. In this example, the user has selected in the timeline number selection field 722 to include two timelines printed per page of the multimedia document. In addition, the user has selected in the “fit on” field 720 to print the multimedia document over two pages. The user also selected in direction field 724 to display a vertical timeline in the multimedia document. Therefore, the audio waveform timeline 734 shown in the content selection field 714 is divided in half and the upper half is shown on the page shown in the preview field 712.
The lower half is shown on the second page and its content is not shown in the preview field.

  In the example of FIG. 11, instead of manually selecting a segment of the audio waveform timeline 734 having a selector 736, the user applied a segmentation type to the audio data. The user has selected in segmentation type field 716 to perform audio detection for applause events in the audio data. The system 200 searches for all applause events in the audio data. However, the user also chose to apply a 75% threshold in the threshold selection field 718. Therefore, only audio events with a probability of being applause events greater than 75% are displayed in PDDI 122. The applause event is displayed in the segmented display field 1102. Each event segment 1104 shown in the segmented display field 1102 corresponds to an event with a probability of applause event greater than 75%.

The event segment 1104 is shown in a staggered box in FIG. However, it is also possible to provide a line or other visual indicator that extends across the segmented display field 1102. The user can click on any one of the event segments 1104 to detect the event segment 1104. A marker corresponding to the applause event segment 1104 (ie, a bar code, RFID tag, URL or other indication about where the multimedia data can be retrieved) 1120 is shown in the multimedia document shown in the preview field 712. It is. In this example, the time stamp 1122 also accompanies each marker 1120. The user can click on an arrow 750 positioned near each event segment 1104 to play audio content that may include applause. In this way, the user can check the indicated event segment 1104 before printing the document to ensure that the event segment 1104 really corresponds to the applause event. Further, the user can select a marker in the printed document corresponding to the applause event to play the applause content. In addition to the audio detection event example described in FIG. 11, there are many other segmentation types that can be applied to audio content or other types of multimedia content. Each of these segmentation types can be displayed in a menu in the segmentation type field 716 and the user can select from the menu to which the segmentation type is applied. A summary of examples of various different segmentation types that can be applied follows. Speaker segmentation is an example, in which each segment corresponding to a different speaker is indicated by a different color or by a different icon. Segments given by the same speaker are shown in the same color or by the same icon. Speaker recognition is another example, and in the speaker recognition, each speaker's name is attached with an accurately detected reliability. PDDI 122 includes a series of check boxes that allow the user to select which speakers to display. The user can also apply audio source localization, in which the direction in which the audio was detected is displayed as a circular sector. Each sector is accompanied by a confidence that the voice was correctly detected. The user interface includes a series of check boxes arranged around the circumference of the prototype circle that allow the user to select which direction is displayed. Speech recognition is another example of segmentation type, in which the timeline displays a confidence value and text for each word or sentence spoken during the optional audio content.

Video FIG. 12 shows a graphic display of PDDI 122 for generating a video paper document. Similar to the PDDI 122 for generating an audio document, a user can enter information into fields in the PDDI 122 to generate a video document. The PDDI 122 shown in the embodiment of FIG. 12 includes several fields found in a standard printer dialog box such as, for example, a printer field 704, a print range field 706, and a copy and adjust field 708. However, PDDI 122 also displays fields that are not found in the standard printer dialog box, such as, for example, Advanced Options Field 710, Preview Field 712, Content Selection Field 714, and the like.

  In the embodiment of FIG. 12, the advanced option field 710 provides the user with specific options for the layout and formatting of multimedia content. In this embodiment, the user selects a segmentation type in the segmentation type field 1202 that the user wishes to apply to the video content. Segmentation types for generating a video document include at least the segmentation types already described above in connection with generating the audio document in FIG. In this example, the user has not selected any segmentation type in the segmentation type field 1202, and therefore the segmentation type 1202 is indicated as “none”. Thus, in this example, the user moves the selector 1222 in the content selection field 714 and clicks on the portion of the video timeline display that the user wishes to select to start a segment for a given video file. Select time and end time manually.

  In the advanced options field 710, the user can define or adjust a threshold for a confidence value associated with the segmentation of the measurement, as described above. The user sets the threshold by typing it in the threshold field 1204. For example, the user can set the threshold to 75% and only display frames that are greater than this threshold (ie, the probability that the frame contains a face in the face detection analysis is greater than 75%). In other embodiments, a threshold slider is included in PDDI 122 and the user can move the slider along a threshold bar that runs from 0% to 100% to select a particular threshold within the range. Further, the buttons shown in the embodiment in FIG. 12, including an update button 726, a page setting button 728, an OK button 730, and a cancel button 732, function in a manner similar to the corresponding buttons described in connection with FIG. .

  In the embodiment of FIG. 12, the content selection field 714 shows the text and video frames in the timeline, and the timeline was extracted at regular intervals throughout a given video content. For example, the system stores video frames of CNN news segments per second and the video timeline displays at least some or all of the stored frames in the video time frame. The extracted frame is a frame that continues to be displayed along the timeline in the CNN news segment at the top of the timeline, starting at time “00:00:00” and ending at time “00; 12: 19”. Displayed with it. In this example, “00:00:00” at the top of the video timeline or the start time of the video content displayed on the timeline is shown. At the bottom of the video timeline, the time “00; 12:19” or the end time of the video content is shown. In some embodiments, video frames can be displayed in reverse order along the timeline.

  There are also three columns 1250, 1252 and 1254 displayed in the content selection field 714. One column 1250 displays text information, and the other two columns 1252 and 1254 display video frames. The video frames displayed in FIG. 12 in two columns are displayed next to each other. For example, the selected first frame is displayed in the upper left part of the timeline, and the selected second frame is displayed at the top of the first frame. The selected third frame is displayed below the first frame, and the selected fourth frame is displayed below the second frame. The video frame display is continued along the timeline in this pattern. In other embodiments, the video frames can be displayed in different patterns, or can be displayed in two or more columns along one column or timeline. A copy of the text is also displayed along the time frame in FIG. 12, from top to bottom, generally near the corresponding video frame. In other embodiments, the text is displayed in two or more columns or other sides of the video frame, or displayed in all in the timeline.

  A user can slide a selector 1222 along the video timeline to select a particular segment of video content, which is displayed in the generated multimedia document. In one embodiment, once the selector 1222 is positioned in the segment of video content that the user wishes to select, the user can click the selector 1222 to select the segment 1226. The video timeline can also be displayed in a variety of alternative ways, for example, displaying a horizontal timeline, displaying two or more adjacent timelines, displaying different video frames, etc. . As noted above, while the video timeline in the embodiment of FIG. 12 displays both the video frame and the corresponding text, the video timeline also displays only the video frame in the associated text in some embodiments. Can be displayed. In those embodiments where the timeline displays only video frames, the generated multimedia representation may still include both text and video frames, or may be limited to video frames only. it can.

  In the example shown in FIG. 12, the user has selected four areas of the video timeline carked in the multimedia document. The user has selected segment 1226, segment 1228, segment 1230 and segment 1232. Each of these selected segments is displayed as one or more video frames and corresponding text in the multimedia document. The example document of FIG. 12 shows a page including five video frames 1206 and associated text 1216. Each displayed video frame 1206 and associated text corresponds to the segment selected in the content selection field 714. For example, the first two video frames 1206 starting at the top left of the multimedia document correspond to the selected segment 1226. In the example of FIG. 12, the video frame 1206 shown in the upper left corner of the multimedia document corresponds to the segment 1228 selected in the video timeline. The video frame 1206 in the upper right corner of the document corresponds to the selected segment 1230, and the video frame in the lower right corner corresponds to the selected segment 1232.

  Further, the position of each displayed video frame in the video timeline is displayed as a time marker 1240 on each video frame. In FIG. 12, each time marker 1240 corresponds to a segment in the range of “00:00:00” to “00:12:19” full length of the video content displayed on the video timeline. For example, the video frame 1206 in the upper left corner of the multimedia document for the CNN news segment includes a time marker of “00:04:21”. In this way, the video content corresponding to this video frame 1206 starts at 4 minutes 21 seconds in the CNN news segment. Furthermore, the text 1216 corresponding to this video frame 1206 displays a copy of the video frame 1206, which starts at 4 minutes 21 seconds in the CNN news segment.

  The user can also play video content in various ways. For example, the user can click the play arrow 1224 next to each selected segment in the video timeline to cause the segment to begin playing. In the embodiment of FIG. 12, each video frame 1206 displayed for a multimedia document has a corresponding marker or identifier 1208 (eg, a barcode) positioned below the video frame 1206. These identifiers 1208 can also select any identifier 1208 to play the video content (ie, scan the barcode), as described above in connection with the audio segment. Then, the selected video segment is played.

  When the user selects the identifier 1208, the corresponding video content starts playing at the time displayed at the corresponding time marker 1240. In the embodiment of FIG. 12, the dialog associated with video frame 1206 begins at the beginning of the associated copy of text 1216. For example, if the user scans the barcode shown below the video frame in the upper left corner of the multimedia document shown in the preview field 712 of FIG. The second news show begins.

  The multimedia document shown in the embodiment of FIG. 12 further shows markers or identifiers to control video content display. In FIG. 12, a play marker 1210, a fast forward (FF) marker 1212, and a rewind marker 1214 are shown. The user can select a play marker in the printed document (i.e., by scanning the barcode using a mobile phone or other device), and the play marker functions as a pause button. If the user selects any identifier 1208 on the print page and the corresponding video content is being played on a type of display device (not shown), such as a mobile phone, the user can click play marker 1210. This play can be paused by selecting. The user can resume playing the video content by selecting the play marker 1210 in the printed document, or the user can select another identifier 1208 of the page to play the corresponding video content. it can. Furthermore, if the user selects any identifier 1208 on the printed page, the user can fast forward or rewind through the video clip by selecting the fast forward marker 1212 or the rewind marker, respectively.

  In the example of FIG. 12, the multimedia document shown in preview field 712 also includes a header, which may include information about the video content (eg, video content title and video content date). For example, the header of FIG. 12 identifies video content as “CNN news”, and the news segment is played on “September 19, 2001”.

  The preview field 712 shown in the embodiment of FIG. 12 further includes a preview content field 1220. The preview content field 1220 marks whether the user previews the paper version of the multimedia document or the video frame 1206 displayed in the multimedia document also previews the associated video content. The user can preview the selected video content associated with the video frame by a video player embedded in the PDDI 122 by selecting the “Video” radio button.

  In the embodiment of FIG. 12, the multimedia document is shown in preview field 712 according to one particular layout. However, the document can be arranged in a variety of different forms. For example, the document can include no header, the time marker 1240 can be displayed below the video frame 1206, and the identifier 1208 can be displayed above the video frame. Yes, etc.

  FIG. 12 also shows a preview window 1280 displayed for the video timeline in the content selection field 714. In some embodiments, the preview window 1280 appears when the user moves the selector 1222 along the video timeline. The preview window 1302 displays an image of the video frame where the selector 1222 is positioned. The preview window 1280 can appear directly in the selected segment, or alternatively can appear next to, around, or below the segment.

  Here, referring to FIG. 13, the graphic representation of the PDDI 122 of FIG. 12 is shown, and in that graphic representation, the user previews the video clip. The user can select a play arrow 1224 positioned near each selected segment along the video timeline in the content selection field 714 that causes the clip to be played. The system can also be configured such that when the play arrow 1224 is selected, all video content represented by the video timeline begins to play. When a video is played in the preview field 712, the corresponding segment along the timeline (eg, segment 1226) is highlighted and the play arrow 1224 next to that segment indicates that the segment is being played. Change to a form (for example, double line). Also, when the user selects a video frame and clicks or double-clicks the specific video frame in the multimedia document in the preview field 712 or clicks the frame in the video timeline, the video frame is played. The system can be specified so that you can get started. Further, in some embodiments, if the user right-clicks a segment (eg, 1224) in the video time frame, the dialog box provides an option to the user to play the video (start at the beginning of the segment). Appears for. The user can select a play option in the playbox and the video frame starts playing in the preview field 712.

  When the user selects a particular video segment for preview, the media player embedded in PDDI 122 starts playing the video segment in preview field 712 from the start of the video segment. For example, in FIG. 13, the video segment can begin playing at the news segment at 4 minutes 21 seconds, which is selected to run from “00: 04: 21-00: 6: 35.” Corresponds to the start of a clip. As described above, video content can also start playing from “00:00:00” in the video timeline rather than in a specific clip. The system can also be specified so that the media player does not begin playing the video clip until the user selects the play button 1304. In this way, when selecting a video segment to be previewed, the media player appears with a slider 1308 at the start of the segment and the user needs to actually click the play button 1304 so that the content begins to play. is there.

  The media player in the preview field 712 also includes, for example, a pause button 1310 to stop / pause display of the video clip, a rewind button to rewind in the video content, and fast forward to fast forward in the video content. Includes features of many standard multimedia players (eg, Microsoft Windows® Media Player) such as buttons 1314 and a volume adjuster 1306 for setting the volume for display. A slider 1308 is also included, which allows the user to move around in the video content. The 1308 slider bar 1316 along which the slider moves can correspond to the length of the entire video content displayed along the timeline, or only the length of the clip. The user can drag or click slider 1308 along slider bar 1316 to move around in the video content. The fast forward button 1314 and rewind button 1312 allow the user to move only in selected segments, or alternatively, allow the user to move in all video content associated with the video timeline. To do. The media player may not include any one of the control buttons shown in FIG. Other buttons can be included to control the display of the video.

  FIG. 13 also shows a preview window 1280 displayed in the video timeline of the content selection field 714, similar to the preview window shown in FIG. This allows the user to preview the video content in the content selection field 714.

  Referring now to FIG. 14, a graphical representation of the PDDI 122 where a video clip is displayed in the preview field 712 is shown. FIG. 14 shows the generation of segments by using the start marker button 1402 and the end marker button 1404 included in the media player in this embodiment. The media player can be specified such that the start of the slider bar 1316 corresponds to the start of the video content associated with the video timeline (eg, at “00:00:00”) or the start of the selected clip. When video content is played, the user can use a start marker button 1402 and an end marker button 1404 to mark the segment of interest. For example, if the user is interested in video content in a news segment shown for a particular actor, the user can play the video content of the media player. When the user reaches the segment shown for the actor, the user can click on the marker button to mark its position. When the segment ends, the user can click on the end marker button 1404. The user can continue this throughout the news segment and can mark segments of interest to be printed or for other purposes.

  FIG. 15 shows a graphical representation of the PDDI 122 shown in FIG. 12, in which a mouse or other printer device was used to right-click on the video timeline shown in the content selection field 714. . When the user right-clicks a video segment in the video timeline, a dialog box 1502 appears, providing the user with options regarding the video frame. The user can select in the dialog box 1502 to play the video corresponding to the segment, or if the video is currently being played, the video can be paused. The user can also choose to edit the segment. Editing options are explained in the study below. Dialog box 1502 appears anywhere in the selected segment in the video timeline or near the video timeline. The dialog box 1502 can further include other control options in addition to the options shown in FIG. 15 such as a rewind option or a fast forward option.

  Referring now to FIG. 16, a graphical representation of PDDI 122 of FIG. 15 with the user selecting an edit option in dialog box 150 is shown. Dialog box 1 appears and allows the user to select further options. In the edit dialog box 1602, the user can modify the start time or end time for a segment by modifying the start time field 1604 and end time field 1606. In this way, the user can select to have a segment that includes some time before or after the segment. For example, the user can have some introductory or prefix that the user is interested in, so that he sees the video content generated in the CNN news segment 45 45 seconds before the predetermined start time of the segment. It is possible to be interested in In addition, the user can be interested in viewing video content that occurs several seconds after a predetermined end time for the segment. The user can also modify the segment start and end times to shorten the segment and remove extraneous content that the user is not interested in. After modifying the start time, end time, or both, the user can select the OK button to apply the modification, or the user can select the cancel button to end the task and edit dialog box 1602. Can disappear.

  In the embodiment of FIG. 17, instead of manually selecting a segment of the video timeline using the selector 1222, the user applied a segmentation type to the video data. The user selects to perform face detection in the segmentation field, and in the face detection, the system searches for a face image in the video content. When face detection is selected, PDDI 122 shows a segment along the timeline that includes the face image. Each segment can be assigned by a confidence value and an integer representing the number of faces detected in the clip. The user also chose to apply an 80% threshold at the threshold selection yield 1204. In this way, only video frames that may contain face images with a probability greater than 80% are displayed on the PDDI 122. The face detection result is displayed in the segmented display field 1702. Each event segment 1704 shown in the segmented display field 1702 corresponds to a set of video frames or one video frame that may contain face images with a probability greater than 80%.

  Event segment 1704 is shown as a staggered box in FIG. However, it also includes a line that extends across the segmented display field 1702 or other visual indicator. The user can right click on any one of the event segments 1704 to detect the event segment 1704. A marker (eg, a barcode) corresponding to the event segment 1704 can be shown in the multimedia document displayed in the preview field 712. Also, when the user moves the selector 1222 along the video timeline, a preview window 1302 appears, providing the user with an opportunity to view the video frame in the event segment 1704 to ensure that a face image is present. In addition, the system can combine video content showing the same face image in one video frame, rather than displaying separate video frames for each example of a particular face image.

  In addition to the face detection example of FIG. 17, there are many other segmentation types that can be applied to video content or other types of multimedia content. Each of these segmentation types can be displayed in a menu in the segmentation type field 1202 and the user can select from the menu to which the segmentation type needs to be applied. A summary of examples of the various different segmentation types that are applied follows. Video event detection is a segmented type that can be applied by the user, and in that video event detection, PDDI 122 shows the result of applying a video event detection algorithm along the timeline. Examples of video events include when a person stands up during a meeting or when a person enters a room. Color histogram analysis is another segmentation type that can be applied by the user, in which PDDI 122 shows the result of applying a color histogram analysis algorithm along the timeline. For example, the PDDI 122 can show a hue diagram every 30 seconds interval that allows an experienced user to immediately locate a portion of the video including sunset. Further, clustering can be applied to cluster face images so that multiple instances of the same face are combined into a single representation of the face image.

  Face recognition is another segmentation type, in which PDDI 122 indicates the name along the timeline that results from applying face recognition to the video frame corresponding to the point along the timeline. . A series of check boxes are also provided that allow the user to select a clip by selecting a name. Optical character recognition (OCR) is a segmentation type, where OCR is performed on each frame in the video content, and each frame is subsampled (ie, once every 30 frames). The result is displayed along the timeline. A text entry dialog box is also provided to allow the user to enter words to be searched for in the OCR results. Clips included in the entered text are shown along the timeline. Further, clustering can be applied so that similar results in performing OCR for each frame are combined. Clusters containing the entered text are displayed along the timeline.

  In addition to the segmentation types described above, there are other examples that can be applied. Motion analysis is another segmentation type, in which PDDI 122 shows the result of applying a motion analysis algorithm along the timeline. The result can be shown, for example, as a waveform having a magnitude that indicates the amount of motion detected. This allows, for example, an experienced user to instantly locate a portion of an image that includes a person running across the camera's field of view. Distance prediction is another segmentation type, in which the PDDI 122 shows the result of applying a distance prediction algorithm along the timeline. For example, in surveillance camera applications using two cameras that are separated by a known distance, the distance of each point from the camera can be predicted. Based on their distance from the camera, the user can set a threshold for selecting portions of a given video file for printing. For example, the user may wish to see only objects that are 50 yards or more away from the camera. Foreground and background segmentation can also be applied, and in that foreground and background segmentation, PDDI 122 shows the result of applying the foreground / background segmentation algorithm along the timeline. At each point, the foreground object is displayed. Clustering and combining algorithms can be applied over groups of adjacent frames to reduce the number of individual objects displayed. The user can set a threshold to select a portion of a given video file at a point based on the foreground / background segmentation confidence value as well as the combining algorithm. Scene segmentation is another type that a user can apply, in which PDDI 122 shows the result of applying a shot segmentation algorithm along a time frame. Each segment can be accompanied by a confidence value that the segmentation is accurate.

  A segmentation type for recognizing automobiles or license plates can also be applied. Car recognition is useful, for example, for a user operating a camera with a lot of time generating very boring images. Such users often need to find and print only those parts that contain a particular object, such as a red cadillac. For this purpose, each frame of the video is input to the car recognition technology and the results obtained are displayed along the timeline. License plate recognition is also useful for a user operating a surveillance camera and is necessary to retrieve video about the part containing the number of a particular license plate. For this purpose, each frame of the video is input into license plate recognition technology and the results obtained (number of license plate, province, plate color, name and address of plate owner, unresolved arrest Status, plate owner's criminal history, etc.) along the timeline. Using either car or license plate recognition, the user can set a threshold to select a portion of a given video file to be printed based on a confidence value with the car or license plate recognition result. it can. A text entry dialog box is also provided to allow the user to enter the manufacturer, model, color, car year, license plate number, county, year, etc. into the license plate. These text inputs are searched for the range of recognition results. Clips containing input information are displayed along the timeline.

  Referring now to FIG. 18, a graphical display of a PDDI 122 is shown that includes video content from multiple sources displayed in one or more timelines. For example, video content can come from two different CNN news segments, or it can come from both a CNN news segment and a CSPAN news segment. The system can print video frames from one news segment on one page of the multimedia document and video frames from other news segments on other pages. FIG. 18 shows two separate content selection fields 714a and 714b, each having a separate video timeline displaying extracted video frames and corresponding text doors. Each video timeline includes event segments 1704 representing frames in which face images are detected with a probability of 80% or higher and selectors 1222a and 1222b for making selections in the video timeline. Each video timeline also includes a separate segmented display field 1102 corresponding to each video timeline showing segments 1704 resulting from the application of face detection to the video source. Thus, until the user views preview window 1302 and determines which video frame to select for display in the multimedia document, the user independently selects selectors 1222a and 1222b around each timeline. Can move. While FIG. 18 shows two video timelines, it also allows the user to compare multiple sources, thus allowing multiple timelines to be generated in PDDI 122.

  Users can apply different segmentation types to video content using PDDI. The user can, for example, choose to apply both voice detection and speaker recognition to one 20 minute long CNN news show. FIG. 19 shows the PDDI 122 of FIG. 17 to which face recognition is applied. However, FIG. 19 shows the result of applying both to the face detection video OCR. In some embodiments, the system includes a drop-down menu in the segmentation type field. The menu can list each segmentation type one by one in the loss menu. Therefore, in this embodiment, the user can click on more than one segmentation type in the menu to apply all selected segmentation types.

  In other embodiments, the menu can also include many different combinations of options, allowing a user to select an item in a menu that includes more than one segmentation type. For example, the combination of voice detection and speaker recognition can be one combination item in the menu. By selecting this option in the menu, the user causes voice detection and speaker recognition to be performed on the multimedia content. This menu item combination can be pre-configured in the printer 102 as a default list of segmentation types and segmentation combination types. In addition, the user can define his own combination type. When the user creates a user-defined segmentation type, the user can give the segmentation type a name and this option will appear in the segmentation type drop-down menu. The segmentation type in FIG. 19 is named “Combo1”, which is a user-defined combination of a single segmentation type. In addition, the threshold field 1204 is invalid because a large number of parameters can be generated that can be adjusted by a combination of two-to-brain segmentation techniques. Therefore, each technology combination can have a default set of parameter values that have proven to work well. However, the user can modify these default sets in a dialog box (not shown) that appears when option button 1906 is clicked.

  As shown in FIG. 19, the content selection field 714 includes two segmentation display fields 1102, one of each segmentation type being applied to video content. In this example, there is one segmented display field 1102 for each of the two segmentation types that make up “Combo1”. A segmented display field 1102 on the left shows the result applied to the face detection shown in FIG. However, the event segments 1704 are not staggered as shown in FIG. 17 and are instead arranged in a line. A segmented display field 1102 on the right side shows the result of applying the video OCR to the video content. The event segment 1704 shown in the segmented display field 1102 is displayed differently from the event segment shown in the segmented display field 1102 on the left side. However, in some embodiments, event segments 1704 shown in different segmented display fields 1102 can be displayed as well. In some embodiments, event segments 1704 are arranged in different formats in the two segmentation display fields 1102 or PDDI 1704 shows one event segment 1704 for all applicable segmentation types. Only the segmented display field 1102 can be included.

  There are many other segmentation type combinations that can be implemented besides the example of FIG. 19 where segmentation types are applied (eg, Combo1). Each of these segmentation type combinations can be displayed in a menu in the segmentation type field 1202 and the user can select from a menu where the segmentation type needs to be applied. An overview of examples of various different segmentation type combinations is described next, but many other combinations not described below are also possible. The user can apply motion analysis in combination with distance prediction, in which PDDI 122 applies the motion analysis algorithm and the distance prediction algorithm along one timeline or two separate timelines. Results are shown. The motion analysis timeline can include a waveform representing the amount of motion detected in magnitude. The user can set a threshold that allows the user to select the location of a predetermined video file to be printed based on the amount of motion detected and the distance of that motion from the camera. Scene segmentation and face detection is another combination that a user can apply, in which PDDI 122 shows the result of applying a shot segmentation algorithm along the timeline. For example, a color or a specific icon can indicate a segment in a timeline that includes a face image. Each segment can be accompanied by a confidence value that the scene segmentation is accurate, and can be accompanied by an integer representing the number of detected faces as well as the confidence value. Scene segmentation and OCR are other combinations that can be applied, in which PDDI 122 shows the results of the shot segmentation algorithm along the timeline. OCR is also performed on each frame of video content, and the content is subsampled. The resulting results are displayed along the same timeline or different timelines. The user can also perform a text search on the OCR results and display segments containing the search terms along the timeline.

  When applying a combination of segment types to multimedia content, the user is not limited to applying only two types in the combination. The user can apply three or more segmentation types, such combinations can be shown in the segmentation type menu by default, and those combinations can be created by the user. A combination of scene segmentation, OCR and face recognition can be applied, in which case PDDI 122 shows the result of applying a shot segmentation algorithm along the timeline. Each frame in the video has an OCR on which they are executed and subsampled, and the resulting results are displayed along the same timeline or different timelines. Names resulting from the application of face recognition to video frames are also shown in the same timeline or in different timelines. A series of check boxes are also provided to allow the user to select a clip by selecting a name. The user can set a threshold for the result, and the user can select a portion of the predetermined video file to be printed based on the confidence value with shot segmentation, OCR and face recognition results. to enable. Instead. The user can apply face detection with OCR and scene segmentation. PDDI 122 displays the results of OCR and scene segmentation as described above. Ona timeline or a different timeline can also include segments containing facial images. Each segment can be attached with an integer representing the number of faces detected in the clip, as well as a confidence value.

  Car recognition and motion analysis is a combination of other alternative segmentation types, in which each frame of video is input into car recognition technology and the resulting results are displayed along a timeline. Motion analysis techniques are also applied to the video to predict the speed of the car from one frame to the next. A text entry dialog box also allows the user to enter manufacturer, model, color, car year, license plate number, county, year, etc. into the license plate. For these items, car recognition and motion analysis results and input information are shown along the timeline, and clips including the input information are displayed along the timeline.

  FIG. 19 shows an example of more than one segmentation type (eg Combo1) applied to video content, while more than one segmentation into any other type of audio content or multimedia content It is possible to apply types. An overview of the different combinations of segmentation types that can be applied is described below, but many other combinations that can be implemented are not described below. Voice event detection and classification is an example of one combination. The PDDI 122 shows the results of applying audio event detection, such as applause, cheer, laughter, etc. along the timeline. Each detected event is accompanied by a correctly retrieved confidence. Speaker segmentation and speaker recognition are examples of other combinations. Each segment is shown along the timeline with a different color or with a different icon, and segments generated by the same speaker are shown with the same color or with the same icon. The result of speaker recognition includes text and optionally confidence for each speaker's name. Multiple speaker names can be assigned to each segment. Audio source localization and audio event detection can alternatively be applied to the user. The direction in which the sound is detected is displayed as a circular sector. Each sector is accompanied by a confidence that the voice was correctly detected. The user interface includes a series of check boxes arranged around the circumference of a prototype circle that allows the user to select which direction should be displayed. Each detected audio event is accompanied by a confidence that the audio was accurately detected, and PDDI 122 includes a series of check boxes that allow the user to select which events should be displayed. The user can alternatively apply a combination of speech recognition and profile analysis. The timeline in the PDDI 122 shows the text and a confidence value for each word or sentence spoken as an option. The result of speech recognition is adapted to a prior text-based profile representing the user's interest. The user can adjust the paving stone with respect to the confidence value, and the user can also adjust the threshold with respect to the matching word eye between the profile and the speech recognition result. Speech recognition and voice event detection are examples of other combinations that can be applied. The tie-line includes text and a confidence value for each spoken word or sentence, along with the results of the speech event detection.

  When applying a combination of segmentation types to multimedia content, the user is not limited to applying only two types in combination. The user can apply three or more segmentation types, and such combinations can be indicated in the segmentation type by default or can be created by the user. Speech recognition, voice event detection, and speaker recognition can be applied in combination. The result of speech recognition includes text and optionally a confidence value for each word or sentence. The detected audio events are shown on the same timeline or on different timelines. PDDI 122 also displays the name of each detected speaker, accompanied by a confidence that the audio event was detected correctly. The user interface includes a series of check boxes that allow the user to select which speakers are to be displayed. A combination of speech recognition, voice event detection and speaker segmentation can be applied as an alternative. Its application is similar to the above except that the speaker segmentation event is shown instead of the speaker recognition event. Each speaker segment is shown in a different color or with a different icon, and segments generated by the same speaker are shown in the same color or with the same icon. As another example, speaker recognition, voice event detection, and voice localization can be applied in combination. The timeline shows the text and, optionally, the confidence value for each word or sentence along with the detected audio event. The timeline also displays the direction in which the audio was detected as a circular sector. Each sector is accompanied by a confidence that the voice was correctly detected. The user interface includes a series of check boxes arranged around the circumference of a prototype circle that allows the user to select which direction should be displayed.

  Referring now to FIG. 20, a multimedia representation (eg, a video paper document) that displays a representation of another embodiment of a multimedia document that can be generated by the system is shown. This document 2000 shows eight video frames 1206, some of which are attached with text 1216, which can be generated by PDDI in FIG. 19, a copy of a dialog, video content It is possible to make it a summary. A partition 2004 separates each video frame 1206 in this embodiment, and a time stamp 2006 indicating from the start time to the end time of each segment of the video content is shown in each partition 2004. In addition, a header 2002 that displays information about video content is shown. In this example, the header 2002 includes the title or CNN news, the time of the news show (eg, 10:00 am), the date of the show (eg, September 19, 2001) and the duration of the show (eg, 12 minutes 19 Second).

An identifier 1208 is shown below each video frame 1206, and the user can use any one of those on-identifiers 1208 that causes the video content attached with the video frame 1206 to begin playing. Video frame 1206 can begin playing when the speaker begins to receive a copy of the associated text 1216. A video frame 1206 indicating that there is no text or the expression “no text” is displayed.
Can include video content that a person in the clip is not speaking, or can indicate an example where the user has chosen to show text.

  The multimedia document shown in the embodiment of FIG. 20 further shows control markers or identifiers to control video content display. In FIG. 20, a play marker 1210, a fast-forward marker 1212, and a rewind marker 1214 are shown. These markers provide an interface to the multimedia data as described above.

  Although the present invention has been described with reference to certain preferred embodiments, those skilled in the art will recognize that various modifications are possible. Variations and improvements to the preferred embodiment are provided by the present invention, which is defined only by the appended claims.

1 is a block diagram of a system for generating a print driver dialog interface that allows a user to format multimedia data before generating a multimedia representation. FIG. FIG. 2 is a block diagram for an exemplary architecture of one embodiment of the system of FIG. FIG. 4 is a diagram illustrating an example of interactive communication with a printer. FIG. 2 is a graphical representation of an exemplary multimedia rendering application with a print selection button embedded in the application. FIG. 5 is a flow diagram of a method for system control by a user before any multimedia conversion is performed. FIG. 4 is a flow diagram of a method for system control when the system converts multimedia, performs default conversion, and displays the results in a graphic user interface. FIG. 5 is a diagram illustrating a graphical representation of an exemplary graphic user interface illustrating generation of a representation that includes a user-selected range of an audio file. FIG. 6 illustrates a graphic representation of a graphic interface that provides exemplary options for formatting a multimedia representation. FIG. 4 is a diagram illustrating a graphic representation of an exemplary graphic user interface illustrating the generation of a two-page gist of an audio file. FIG. 4 is a graphical representation of an exemplary graphic user interface showing the division of an audio file timeline into two parts. FIG. 6 shows a graphical representation of an exemplary graphic user interface showing the division of an audio file timeline into two vertical parts. FIG. 6 is a diagram illustrating a graphic representation of an exemplary graphic user interface illustrating generation of a representation including a user-selected range of a video file. FIG. 6 illustrates a graphical representation of an exemplary graphic user interface showing a preview field for previewing multimedia content. FIG. 5 illustrates a graphical representation of an exemplary graphic user interface showing the use of segment marker buttons in a preview field. FIG. 5 illustrates a graphical representation of an exemplary graphic user interface showing a dialog box providing options to a user. FIG. 6 is a diagram illustrating a graphical representation of an exemplary graphic user interface showing a dialog box for editing a video segment. FIG. 6 is a diagram illustrating a graphical representation of an exemplary graphic user interface illustrating generation of a video representation to which face detection technology has been applied. FIG. 5 is a diagram illustrating a graphical representation of an exemplary graphic user interface illustrating the generation of video representations from multiple sources. FIG. 5 is a diagram illustrating a graphical representation of an exemplary graphic user interface illustrating generation of a video representation to which a user-defined combination of analysis techniques is applied. FIG. 3 is a diagram illustrating an exemplary multimedia representation of a graphic representation including a video frame and text.

Explanation of symbols

100 System 102 Printer 103 Conventional Printer 104 Media Analysis Software Module 106 Processing Logic 108 Digital Media Output 112 Document Format Specification ((DFS))
120 multimedia document 122 PDDI
124 preview field 126 speaker name field 128 subject field 130 time field 132 frame 134 user selectable identifier 140 connection 142 thumbnail picture 144 digital media 200 system 202 multimedia store 204 multimedia rendering application (MRA)
206 Application Plug-in 208 Printer Driver Software Module 210 Communication Monitoring Module or User Interface Listener Module (UI Listener)
212 Application server 214 Processor 230 Personal computer (PC)
240 Bus 242 Bus 244 Bus 246 Multimedia data and signal line 248 Bus 251 Bus 260 Digital output 264 Multimedia file store 302 User 304 Print request 306 Request notification 308 Print job 310 Information 312 Application server 314 Response 316 sent to printer Information Request 318 Information Request 320 Dialog Box Displayed 322 User Response to Application Server 324 Response to Printer 102 402 Print Button 702 File Name Field 704 Printer Field 706 Print Range Field 708 Copy and Adjustment Field 710 Advanced Options Field 712 Preview field 714 Content selection field 714a Content selection field 714b Content selection field 716 Segmentation type field 718 Threshold field 720 Fit on field 722 Timeline number selection field 724 Direction field 726 Update button 728 Page setting button 730 OK button 732 Cancel button 734 Audio waveform timeline 736 Selector 740 Segment 742 Segment 744 Segment 750 Audio waveform timeline 760 Play identifier or play marker 766 Marker or identifier 800 Multimedia document page or dialog interface 802 Paper field 804 Direction field 806 Preference field 902 Segment 904 Segment 906 Segment 908 Segment 910 Time stamp 912 Timeline Manufacturers 1002 segment 1004 Segment 1006 segment 1008 Segment 1120 marker 1122 timestamp 1202 segmentation type field 1204 the threshold field 1206 video frame 1208 marker or identifier 1210 play marker 1212 fast forward (FF) marker 1212
1214 Rewind marker 1216 Text 1220 Preview content field 1222 Selector 1222a Selector 1222b Selector 1224 Play arrow 1226 Segment 1228 Segment 1230 Segment 1232 Segment 1240 Time marker 1250 Column 1252 Column 1254 Column 1280 Preview window 1302 Preview window 1304 Play button 1306 Volume adjuster 1308 Slider 1310 Pause button 1312 Rewind button 1314 Fast forward button 1316 Slider bar 1402 Start marker button 1404 End marker button 1502 Dialog box 1602 Edit dialog box 1604 Start time field 1606 End time field 1702 segmented display field 1704 event segment 1906 option buttons 2000 document 2002 header 2004 partition 2006 time stamp

Claims (48)

A system that enables interaction with media data analysis and media representation generation:
A user interface for enabling a user to control media content analysis and media representation generation; and a media analysis software module for analyzing the characteristics of the media content for receiving media content analysis instructions A media analysis software module communicatively coupled to the user interface;
A system characterized by comprising.   The system of claim 1, wherein the media analysis software module further comprises content recognition software for recognizing features in media content.   The system of claim 1, further comprising processing logic for controlling display of a user interface.   The system of claim 1, further comprising processing logic for controlling the generation of a media representation.   The system of claim 1, further comprising hardware for writing a media representation in a digital format.   6. The system according to claim 5, further comprising a storage medium for storing a media representation written in a digital format.   The system of claim 1, wherein the media representation is generated in a paper format.   8. The system of claim 7, wherein the paper format includes at least one user selectable identifier that allows a user to access and control media content.   The system of claim 8, wherein the at least one user-selectable identifier further comprises at least one barcode printed for a media representation.   9. The system of claim 8, wherein the at least one user selectable identifier further comprises at least one play identifier that can be selected to play corresponding media content. Feature system.   The system of claim 1, further comprising a data structure for representing the conversion of media content.   The system of claim 1, wherein the communication monitoring module is for monitoring communication between components of the system, wherein the communication monitoring module transfers a request for information and responds to a request between stem components. A system further comprising modules.   The system of claim 1, wherein the user interface further comprises a selection menu that allows a user to select a feature analysis to be performed on the media content.   The system of claim 1, wherein the user interface further comprises a field for setting a threshold for a confidence value corresponding to a result of media content.   The system of claim 1, wherein the user interface further comprises at least one field for managing and modifying the display of media information for media representations.   The system of claim 1, wherein the user interface further comprises a preview field for previewing active media frames in home-based media content.   The system of claim 1, wherein the user interface further comprises a preview field for previewing a generated media representation.   The system of claim 1, wherein the user interface comprises at least one content selection field for selecting a segment of media content from at least one source displayed in a media representation. System.   19. The system of claim 18, wherein the content selection field further comprises a selector that a user can slide along the content selection field to select a segment to be displayed in a media representation. , A system characterized by that.   19. The system of claim 18, wherein the content selection field further comprises a graphical display of media content that allows a user to display media content and select a segment of media content. System.   21. The system of claim 20, wherein the graphical display of media content further comprises an audio waveform timeline that displays audio content.   21. The system of claim 20, wherein the graphical display of the media content further comprises a video timeline that displays video frames extracted from the video content.   21. The system of claim 20, wherein the graphic display of the media content further comprises a video timeline that displays text extracted from the video content.   19. The system according to claim 18, wherein the content selection field is a field for displaying a result of media content analysis, and the result is displayed as a segment defined along a timeline. The system further comprising:   The system of claim 1, wherein the output device driver module for facilitating the media content analysis and media representation generation is communicatively coupled to a user interface to receive user instructions. Further comprising an output device driver module.   26. The system of claim 25, wherein the extended output device for generating a media representation is communicatively coupled to the media analysis software module to receive converted media data. A system further comprising an extended output device communicatively coupled to the output device driver module to receive instructions for media representation generation. A method that enables interaction between media data analysis and media representation generation:
Interacting with the interface to control media data analysis and media representation generation;
Analyzing the characteristics of media data for media representation generation;
Facilitating media data analysis; and facilitating media representation generation by receiving and transmitting instructions relating to media representation parameters;
A method comprising:   28. The method of claim 27, further comprising the step of generating a media representation.   28. The method of claim 27, wherein analyzing the characteristics of the media data further comprises a procedure for performing speech recognition on the media data.   28. The method of claim 27, wherein analyzing the characteristics of the media data further comprises a procedure for performing optical character recognition on the media data.   28. The method of claim 27, wherein analyzing the characteristics of the media data further comprises a procedure for performing face recognition on the media data.   28. The method of claim 27, wherein analyzing the characteristics of the media data further comprises a procedure for performing speech recognition on the media data.   28. The method of claim 27, wherein analyzing the characteristics of the media data further comprises a procedure for performing speaker detection on the media data.   28. The method of claim 27, wherein analyzing the characteristics of the media data further comprises a procedure for performing face detection on the media data.   28. The method of claim 27, wherein analyzing the characteristics of the media data further comprises a procedure for performing event detection on the media data.   28. The method of claim 27, further comprising adding a printing function to a media rendering application for performing media representation.   28. The method of claim 27, further comprising storing the media content on a storage medium accessible to the extended output device.   28. The method of claim 27, wherein interacting with an interface for controlling media data analysis and media representation generation further comprises using a user interface for displaying media content formatting options to a user. A method characterized by comprising.   28. The method of claim 27, wherein interacting with an interface for controlling media data analysis and media representation generation is a procedure for selecting an analysis technique to be applied to media content, the analysis technique A method characterized by further comprising a procedure for recognizing features defined in the media content.   28. The method of claim 27, wherein interacting with an interface for controlling media data analysis and media representation generation is applied to a confidence level corresponding to a defined feature recognized in the media content. The method further comprising the step of selecting a threshold value.   28. The method of claim 27, wherein interacting with an interface for controlling media data analysis and media representation generation is generated in a preview field that displays the media representation as the media representation is generated. A method further comprising the step of previewing the media representation.   28. The method of claim 27, wherein interacting with an interface for controlling media data analysis and media representation generation includes modifying an update field after modifying content in a user interface to update a preview field. The method further comprising the step of selecting.   28. The method of claim 27, wherein interacting with an interface for controlling media data analysis and media representation generation includes selecting media content in a field of the user interface by a selector along a timeline for displaying media content. The method further comprising the step of selecting a segment.   28. The method of claim 27, further comprising the step of interacting with an interface for controlling media data analysis and media representation generation selecting a play option at a user interface for playing media content. A method characterized by comprising.   28. The method of claim 27, further comprising selecting print options in a media rendering application, wherein a user interface appears and the user selects parameters for media content conversion. A method characterized by that.   28. The method of claim 27, wherein selecting a print option in a media rendering application displays a user interface in which default media content conversion is performed and the media representation is shown in a preview field of the user interface. The method further comprising:   28. The method of claim 27, wherein generating the printable multimedia representation further comprises the step of printing the media representation in a paper-based format.   48. The method of claim 47, further comprising the step of selecting user selectable identification in a paper-based format for playing corresponding media content.