KR20020060964A - System to index/summarize audio/video content - Google Patents

Abstract

The "background information" available during the production of the content material correlates with the content material to facilitate selective access to the source of the audio / video content material. Such production information includes, for example, camera settings used during the collection of content data. Other production information includes sound controls, screen identifiers, source identifiers, instructions to communicate with the staff producing the content material, and the like. In order to facilitate indexing, the production information is processed and filtered to produce a collection of symbolic representations, each symbol corresponding to a determined event or characteristic. This preference making information is preferably combined with other information that indexes and summarizes the information to provide a set of annotations in the content material that facilitates efficient and effective selection retrieval.

Description

System for indexing / summarizing audio / video content {System to index / summarize audio / video content}

2. Description of the prior art

Broadcast News Navigation using story segmentation, 381-389 As is demonstrated by Page, improvements continue in the area of segmentation and identification of automation stories. Using BNE, news broadcasts are automatically split into individual story segments, and the first line of closed caption text associated with that segment is used as a summary of each story. Keywords from closed caption text or audio are used as a summary of each story. The BNN allows the customer to enter search terms, and the BNN classifies the story segments by the number of keywords in each story segment that match the search terms. Based on the frequency of occurrences that match the keywords, the user selects stories of interest. Similar search and search techniques are commonplace in the art. For example, conventional text search techniques could be applied to a computer based on a television guide, allowing people to search for a specific title, a specific performer, a particular type of shows, and the like.

These indexing and retrieval techniques are also being developed for recorded entertainment materials such as audio and video CDs and DVDs. Users may be able to search for a particular scene, for example, by specifying the actor's name, such as "monologue, John Smith," and the characteristics of the scene. In response, the search device will present one or more scenes including John Smith who plays the monologue.

One of the difficulties encountered by sorting and indexing data for search is the need to "annotate" the data with relevant information to facilitate efficient search. Although manual processing can be used to add indexing information to be recorded and broadcast content of content respectively, such processing is a costly effort and therefore there will be a need for previously mentioned automated indexing systems such as BNE. For example, conventionally, by searching for frames that are substantially different from the immediately preceding frame, the automated indexing system first recognizes or cuts each change in the scene. Then, if the frame contains a close-up face shot, and the context of the program is "news broadcast," the next sequence of frames is "Newscaster ( newscaster "clip, whereas if the frame contains a full-figure shape, then the image of the next frame can be identified as an" on-location "clip. . As noted above, any closed caption text is also used to identify and classify scenes. Although these techniques turn out to be somewhat efficient, they rely heavily on the content of material such as pictures and sounds in each scene to determine the appropriate setting of indexing parameters that characterize the material.

The MPEG-7 standard addresses the need for effective indexing and search capabilities, and standardizes a set of description schemes and descriptors, a description definition language, and a configuration for coding the description. Request the intended "Multimedia Content Description Interface." In particular, the MPEG-7 standard includes pre- and post-production scripts, information captured or annotated during shooting, and post production edit lists in video streams at various video production stages. Request the ability to associate descriptive information. Adding such information during the production of the video material is expected to substantially improve the quality and efficiency of the annotations compared to post-production annotations of the material in the video archive.

1. Field of Invention

The present invention relates to the field of customer electronics, and more particularly to a system that facilitates indexing and summarization of audio / video content for efficient retrieval and navigation of selections.

1 illustrates an exemplary production scene for gathering audio / video information.

2 shows an exemplary block diagram of a production recording system according to the invention.

3 illustrates an exemplary block diagram of an indexing / summary system according to the present invention.

4 shows an exemplary block diagram of a production recorder in accordance with the present invention.

Brief summary of the invention

It is an object of the present invention to improve the efficiency of content indexing and summarizing systems for audio / video content material. It is yet another object of the present invention to provide additional supplemental information to facilitate indexing and summarization of content material.

These and other objects are achieved by correlating "backgroud information" available with the content material with the content material during use. Such production information includes, for example, camera settings used during the collection of content data. Other production information includes sound controls, scene identifiers, source identifiers, instructions for communicating with a staff that produces content material, and the like. Instructions from the director to each camera operator can provide insight into the content of subsequent images from the cameras, for example. In the same way, commands automatically generated from automated camera systems can also provide insight. Other production information includes sound controls, scene identifiers, source identifiers, and the like. In order to facilitate indexing, production information is processed and filtered to produce a collection of symbolic representations of production information, each symbol corresponding to a determined event or characteristic. This preference making information is preferably combined with other sources of indexing and summary information to provide setting of annotations in content material that facilitates efficient and efficient selection retrieval. The techniques presented herein are also well suited for annotating the content data of video conferencing, in particular to facilitate identification of key segments of videoconference recording.

The invention is explained by way of example in more detail with reference to the accompanying drawings.

In all figures, like reference numerals indicate similar or corresponding features or functions.

For ease of reference and understanding, the terms "indexing" and "summarizing" are used herein to refer to specific applications of the present invention. These inventions address a method and device for providing information associated with content material, and are not limited by how the information is used. Although such provided information may be particularly well suited for use as an index to facilitate searching for a material or as a synopsis to facilitate quick review and preview of the material. Those skilled in the art will recognize that they are not limited to these specific applications.

1 is a diagram illustrating an exemplary production scene for gathering audio / video information. The scene includes directors 110 and 131 and objects 140 as well as director 110 instructing the actions of camera operators 120 and 121. Example scenes represent scenes indicated. As will be apparent to those skilled in the art, the present invention is also applicable to undirected scenes, such as recording of news or sports events. In a live sports broadcast, for example, the production director decides successively that the camera is an "on-line" camera depending on the current action, and to capture specific scenes for potential selection as an "online" camera. Command "off-line" cameras or potential use for "instant replay". In a news broadcast, the production director also performs a similar selection of source material for transmission. In video conferencing, participants on both sides can adjust camera settings that are typically coordinated with video conferencing activities.

As can be imagined, director 110 of FIG. 1 states, “Camera 1 follows when Joe 130 runs into the crowd. Camera 2 follows Jim 131. Make sure there are briefcases in view of exchanging the bags. ” This invention conveys a substantial amount of information for interpreting pictures of scenes such that "behind-the-scene" instructions form part of the production information corresponding to those scenes. For example, the above three command sentences give meaning to the scene of FIG. 1 which is difficult to reason based solely on the image contents. And in the absence of any conversations associated with the scene of FIG. 1, the use of closed-caption text will provide minimal assistance in identifying this meaning. The published scripts, outlines, and scene edit lists cited above for MPEG-7 may contain appropriately described information, but the published information does not reflect those events as they actually occurred at the production site. You may not. On the other hand, the three example sentences above convey information that Joe and Jim are in the scene, this is the scene where the briefcases are exchanged, and so on. This production information also conveys information that can be used in an indirect manner to facilitate the interpretation of other sources of information. For example, the image processing system will probably identify such a scene as a "group" scene, and while the example sentences highlight that the crowd is simply a "background", it may identify Joe or Jim in that group and identify it. You may not be able to. In the same way, the presence of the cars 140 in the scene can also be interpreted as non-essential background information, based on the lack of a reference to them in the production instructions. That is, the production information not only conveys direct information about the contents of the following pictures, but also provides signals that facilitate efficient processing by indexing or other sources of summary information.

2 shows an exemplary block diagram of a production recording system 200 according to the present invention. Production recorder 210 receives information from various sources 220, 230, 240, and produces a database 215 of production information that captures production information in an efficient format for subsequent processing. The primary source of production information is a vocal input 220. Various spoken sources 220 typically provide production information. For example, in a news broadcast, the production booth provides a source of spoken information; An on-site report can provide instructions to an on-site camera operator; Before the broadcast delivery, the news host may advise studio newscasters or field reporters.

Production recorder 210 is configured to process each of these sources of information, extract the appropriate information, and record the appropriate information for subsequent use in the retrieval process. Depending on the complexity of the analysis, this processing may be performed in real time as the images being recorded, or as a later processing task. In a preferred embodiment, production recorder 210 also directly records information from sources 220-240 to facilitate processing and the next selectable degree of gemstone. For example, real-time analysis of production information uses a 'primary' source of production information, such as voice information from a microphone associated with the director, while other sources of production information may be required later for more detailed analysis and evaluation. Is treated as.

In addition to speech commands, the production recording system 200 of the preferred embodiment includes an input 230 corresponding to parameters associated with separate cameras. For example, the camera's zoom setting can be used to provide the characteristics of the scene. Narrow angles, high zoom settings typically emphasize a focus pointed to a person or event, or a change in zoom setting that indicates a change in emphasis on the person or event. Wide angle or low zoom settings are typically associated with a "background" or "emotion setting" scene. The identification of a series of images captured at a low zoom setting can be used, for example, as an image based classification system for "skip-ahead" to higher zoom setting pictures. Alternatively, in a learning system, each new scene can be evaluated regardless of the zoom setting, and if the image processor finds little or no recognizable information with a particular zoom setting, the 'skip forward' function is disabled. It can be activated. Assuming consistency in video capture techniques, especially in the same production, the decision to 'skip forward' can be made faster and faster, thus improving the efficiency of the picture based classification system. In the same way, azimuth or rate of change of azimuth can also be used to characterize the scene. For example, in sports, end-line-rush or strike-out will probably not include a change in camera orientation, while kick-off, forward Capturing a forward-pass or home-run will involve a relatively rapid change in camera orientation.

During recording, other sources 240 of production information such as the location of the sound booms and other means of identifying the 'focus' of the scene are also used to facilitate indexing or summarization of the recorded content material. In the same way, a 'source' of content can convey information about the scene. For example, in a news broadcast, identification of a scene coming from a "file footage" can be used for either minimizing the processing of the scene or providing a link to previous processing and features of such footage. . Note that as production tasks are increasingly automated or at least managed through computer resources, the source of that production information becomes significant. For example, it can be expected that sequencing and selection of sources during the news broadcast will be controlled via a computer. Capturing this information for each production substantially increases the efficiency and efficiency of other content indexing and summarization tools.

Note that the selection of sources 220-240 of fabrication information is arbitrary and need not be adapted to traditional fabrication techniques. For example, the US patent application "HANDS-FREE HOME VIDEO PRODUCTION CAMCORDER" of serial number 09 / 532,820, agent document number US000063, filed March 21, 2000, by Mi-Suen Lee captures scenes that may be of interest. Techniques and devices for automatically adjusting the field of view of a camera field are described, and are referred to herein. The adjustment is for example based on the purpose of tracking techniques, sound position and focusing, etc., and incorporates knowledge-based system techniques comparable to the operations of a seasoned camera operator. As used in the present invention, the camera settings that are automatically generated as a result provide the camera input 230 to the playback recorder 210 described above with or without any other sources of production information. In the same way, US patent application "MULTIMODAL VIDEO TARGET ACQUISITION AND RE-" of serial number 09 / 488,028, attorney docket US000015, filed Jan. 20, 2000 by Eric Cohen-Solal and Mi-Suen Lee, co-pending DIRECTION SYSTEM AND METHOD "discloses a technique and device for adjusting the field of view of a camera field based on gestures and key words, and is referenced herein. Similarly, US patent application "METHOD AND APPARATUS FOR TRACKINGMOVING OBJECTS USING COMBINED VIDEO AND" issued by co-pending Hugo Strubbe and Mi-Suen Lee, Serial No. 09 / 548,734, filed April 13, 2000, Representative Document No. US000103. AUDIO INFORMATION IN VIDEO CONFERENCING AND OTHER APPLICATION "describes techniques and devices for adjusting the field of view of a camera field based on analysis of video and audio content being recorded. As used in the present invention, analysis of the resulting camera settings and gestures, voices and movements, or gestures, voices or movements used to cause these settings may be used in the production of production information database 215. Can be provided to the production recorder 210 for.

Camera settings during video conferencing may similarly be used to facilitate the characteristics of the video conferencing session. Improved video conferencing systems are expected to include the automation and semi-automated camera control features described above, and even relatively simple systems allow participants in video conferencing to adjust the field of view of the camera field at their or remote locations. Alternatively, the camera operator may be provided at the center position of the video conference, the position of the key speakerr, or the like. A fixed camera position with some fluctuations, perhaps upon zooming for an extended duration, may indicate a keynote address, especially when correlated with audio content from each video conferencing position. In the same way, continuous camera rotation back and forth may indicate a key discussion period. As noted above, this combination of production information with content material may provide insights that may not be readily apparent from the content material, thereby improving the quality and efficiency of providing summaries for each video conference. For example, once the camera settings corresponding to each of the plurality of speakers have been determined (or only provided explicitly), the camera settings corresponding to the images are used to previously filter the selections of characters provided in the character identification process. , Character identification in the images becomes quite simple. Similarly, speaker identification processing can be similarly improved by providing camera settings corresponding to each audio track. In the same way, a quick determination that the audio track does not match the identified participant in the field of view corresponding to the current camera can be used to change the current camera settings to retrieve the current speaker. These and other synergetic effects will be apparent to those skilled in the art when the use of the present invention is commonplace.

To facilitate synchronization of production information with content data, a time reference 201 is associated with the recorded production information 215. As will be apparent to those skilled in the art, some production information, such as camera settings 240, will coincide with the scenes of the content material in time. Other information, such as speech commands 220, typically precedes the scenes they apply. Knowledge-based and heuristic techniques are used to determine the correlation between specific instructions 220 and content data. For example, the occurrence of a 'cut' in the command 220 or scenes prior to significant adjustment of the camera settings 230 is intended to include information about the next clip. Otherwise, for example, if there is no significant change in production recorder 210 or other inputs to the content material, the instructions will probably be appropriate for the current clip. Other techniques for determining cause and effect relationships are common in the art. Other scene identification and synchronization inputs 202 may also be provided, in particular, to the indicated scenes, and an explicit identification of that scene (eg, “Rocky Ⅸ, Scene 32, Take 3”) is available.

3 shows an exemplary block diagram of an indexing / summary system 300 in accordance with the present invention. As shown, an indexer / summarizer 310 preferably accesses the characteristics of the content information, or various information 215, 320-323 that facilitates indexing, or summarization. Closed caption information 320 is generally used to characterize the content material as in BNE and BNN systems, as described above. According to the present invention, production information can be used to personalize the efficiency and efficiency of this sorting process. For example, the scene of FIG. 1 may include a conversation included in the closed caption material, but such a conversation may only be provided as a transition from the exchange of briefcases, or just a filler material during the exchange. material). The indexer / summarizer 310 in the preferred embodiment, while increasing the corresponding importance-weighting-factor associated with the picture information 321, for example, the closed-caption information 320. In order to reduce the importance-weighting-factor associated with the < Desc / Clms Page number 12 >

Image information 321 in the content data is used to classify the content data based on the visual characteristics of each image. For example, the scene of FIG. 1 may be characterized as “outdoor, group, cars, pedestrian” based on simple pattern and contextual awareness techniques. Depending on the capabilities of the system, the indexer / summary 310 may also include the recognition of one or more actors and actresses within the scene. US patent application "PROGRAM CLASSIFICATION USING OBJECT TRACKING" of serial number 9 / 452,581, attorney docket No. PHA 23,846, filed December 1, 1999 by co-pending Nevenka Dimitrova and Lalitha Agnihotri. A content based classification system is disclosed that detects the presence of text pictures and determines the path or trajectory of each picture through a plurality of frames of a video segment. The combination of face trajectory and text trajectory information is used to classify each segment of the video sequence. In the example of FIG. 1, production information 215 facilitates tracking of this object based, for example, on references of "Joe" and "Jim" in command sentences. In the same way, the object tracking facilitates the motivation of the instructions for the content material by associating instruction sentences for the current or next scene, depending on whether "Joe" and "Jim" are in each scene. These and other techniques for combining production information with image information to facilitate the nature of scenes or clips will be apparent to those skilled in the art in view of this description. The results of the aforementioned MPEG-7 standard provide for the efficient and efficient indexing and summarization of audio / video content, as well as the syntax required to facilitate cross-platform use of this information. It is also expected to provide useful semantic desciptors.

Contextual information 322 also facilitates the nature of the content material provided by indexer / summary 310. For example, if the context of the scene is a sports relay, the terminology interpretation included in production information 215 or closed-caption information 320 may change; Correlation between the closed-caption information 320 and the picture information 321 based on the likelihood that the closed-caption information corresponds to the broadcaster, which is not essential to the individuals depicted in the pictures. Relationships may change, etc.

If the indexer / summary 310 is at the user's home, or customized for a particular user, the user information 323 may also be used to facilitate the nature of the content material. US patent application "PERSONALIZED NEWS RETRIEVAL SYSTEM" of serial number 09 / 220,277, attorney docket No. PHA 23,590, filed December 23, 1998 by Jan H. Elenbaas, Tomas McGee, Nevenka Dimitrova, and Mark Simpson, co-pending DETAILED DESCRIPTION Techniques for customizing the classification and retrieval of information based on user preferences or viewing habits are referred to herein. In the context of the present application, recognizing a user's preferences and / or habits is the processing of production information 215 and other information 320-322 by giving priority to certain aspects of the indexer / summary 310. To facilitate. For example, if the user rarely retrieves information based on the performer's name, the indexer / summarizer 310 may provide multiple information sources to provide a comprehensive index of the performers appearing in each scene or clip. 215, 320-322 consume additional time and resources to track each actor. These and other techniques for optimizing the efficiency and efficiency of the indexer / summarizer 310 will be apparent to those skilled in the art in view of the present specification.

The indexer / summary 310 is typically an annotated version of the content material 350 and provides information attached to the content material. For example, a DVD provider will use the system 300 to provide a DVD containing indexing or summary information associated with each scene of the content material on the DVD. The corresponding DVD player is configured to facilitate searching for specific scenes of the content material based on the indexing or summary information included. Alternatively, indexing / summary system 300 may be independent of the provider of the content material and provide indexing or summary information as an adjunct independent of the content material. For example, a vendor may provide indexing and summary information on an Internet site, and allow a user to achieve the aforementioned search through a web-TV device, or an internet access device such as a personal computer (PC). Program can be provided.

4 shows an exemplary block diagram of a production recorder 210 in accordance with the present invention. In a preferred embodiment, production recorder 210 includes voice recognizer 420, field of view processor 430, and scene synchronizer for processing production related inputs 201-240 from various sources. 410. Speech recognizer 420 provides translation from spoken sounds to recognized terms and is used to process speech inputs 220. Field processor 430 provides an interpretation of the camera settings to characterize the current field of view of each camera 230. The scene synchronizer 410 processes the synchronization inputs 201 and 202 to facilitate synchronization between production information and content data. Other processors 440 are provided for processing input from other sources 240 of production information as required.

In the preferred embodiment, the symbol encoder 450 encodes the production information 215 in symbolic form to facilitate subsequent processing by the indexer / summary 310 (FIG. 3). Production recorder 210 includes a symbol library 460 that facilitates such symbol encoding. For example, in a preferred embodiment, symbol library 460 includes symbols for keywords that symbol encoder 450 uses to encode terms provided by speech recognizer 420. In the same way, the symbol library includes symbols corresponding to specific camera settings or combinations of settings, to facilitate encoding of camera characteristics provided by field processor 420. Various techniques may be used to maintain the efficient symbol library 460. US patent application "IMAGE CLASSIFICATION USING EVOLVED PARAMETERS" of serial number 09 / 343,649, attorney docket No. PHA 23,696, filed June 29, 1999 by Keith Mathias, J. David Schaffer, and Murali Mani, Evolution, the use of algorithms to optimize the parameters used to classify pictures.

The foregoing merely illustrates the principles of the invention. Therefore, it will be understood that those skilled in the art, although not explicitly described or shown herein, may embody the principles of the present invention and, therefore, devise various devices that fall within the spirit and scope of the present invention. For example, although content classification and indexing of other forms of content information is also facilitated by the inclusion of production information, video content information has been used as a paradigm to represent the application of the present invention. Instructions and facility settings in the sound studio can be used, for example, to index and summarize audio content material. Similarly, in addition to use for indexing and searching, annotated audio / video content 350, particularly production information 215 included in director's utterance commands, may be of direct interest to the user, and such information. It may provide a marketing advantage for the medium, including. These and other system configurations and optimization features will be apparent to those skilled in the art in view of this specification, and are included within the scope of the following claims.

Claims (21)

In a method of providing ancillary information related to content material, Collecting source information (220 to 240) related to the production of the content data, and Processing the source information (220 to 240) to provide the assistance information, The source information 220-240 may include one or more instructions 220 issued during the production of the content material; And at least one of one or more parameters (230, 240) associated with one or more equipment items used during the production of the content document. The method of claim 1, Processing the source information (220 to 240), Recognizing terms associated with a vocal input corresponding to the one or more instructions 220, and At least one of recognizing a field of view setting associated with at least one camera (230) corresponding to the one or more facility items. The method of claim 1, Processing the source information (220 to 240), Also processing other information (320 to 323) to provide the assistance information, The other information 320 to 323, Closed-caption information 320 associated with the content material; Image information 321 associated with the content data; Audio information associated with the content material, Context information 322 associated with the content material, and And at least one of user information (323) associated with a user of the content material. The method of claim 3, wherein Processing of the source information (220 to 240) facilitates summarizing the content data. The method of claim 1, And the content material corresponds to scenes of a video conference. The method of claim 1, Synchronizing the content data with the assistance information based on the assistance information to facilitate a search for specific segments of the content material (350). The method of claim 1, And classifying production information (215) associated with the source information (220 to 240) in symbolic form to facilitate processing of the source material. The method of claim 1, And the auxiliary information is provided in accordance with the MPEG-7 specification. In the method for providing auxiliary information related to the content material, Collecting camera parameters 230 associated with one or more camera settings used during the production of the content material, and Based on the camera parameters (230), producing the assistance information. The method of claim 1, Providing the auxiliary information, And processing other information 320 to 323 to provide the assistance information. The other information 320 to 323, Closed-caption information 320 associated with the content material; Image information 321 associated with the content data; Audio information associated with the content material, Contextual information 322 associated with the content material, and And at least one of user information (323) associated with a user of the content material. In the recording system 210, Source information (220 to 240) associated with the production of the content material; Accept as input the synchronization data 201, 202 associated with the content data therefrom And an encoder (450) for producing production information (215) that facilitates selective access to the content material. The method of claim 11, The source information 220 to 240 is Instructions 220 associated with the production of the content material, and And at least one of the settings (230-240) associated with the facility used to produce the content material. The method of claim 11, The encoder 450 is A speech recognition system 420 for processing utterance source information 220 associated with the production of the content material, and And at least one of a field of view processor (430) for processing parameters (230) associated with at least one camera associated with the production of the content material. In the information processing system 300, Source of production information (215) affecting production of contents document, A processor 310, operatively coupled to a source of production information 215, configured to provide supplemental information related to the content material, The production information 215 is One or more instructions 220 issued during the production of the content document and And at least one of one or more parameters (230, 240) associated with one or more equipment items used during the production of the content material. The method of claim 14, The source of the reproduction information 215 is A speech recognition system 420 configured to recognize terms associated with speech input corresponding to the one or more instructions 220; And at least one of a visual field processor (430) for processing parameters (230) associated with at least one camera corresponding to the one or more facility items. The method of claim 14, Further includes at least one source of other information 320 to 323, The other information 320 to 323 Closed-caption information 320 associated with the content material; Image information 321 associated with the content data; Contextual information 322 associated with the content material and At least one of user information 323 associated with a user of the content material, The processor 310 is information coupled, operatively coupled to at least one source of other information 320-323, and further configured to provide the assistance information further based on the other information 320-323. System 300. The method of claim 14, And a synchronizer (410) configured to provide a correlation between the supplemental information and the content data. The method of claim 14, And the assistance information facilitates identification of characters associated with the content material. The method of claim 14, And the auxiliary information is provided according to the MPEG-7 specification. In the information processing system 300, An input for receiving production information 215 related to production of the content material, A processor 310, operatively coupled to the input, configured to provide assistance information related to the content material, The production information (215) includes one or more parameters (230) associated with one or more cameras used during the production of the content material. The method of claim 20, At least one source 320-323 of other information, The other information 320 to 323 is Closed-caption information 320 associated with the content material; Image information 321 associated with the content data; Contextual information 322 associated with the content material, and At least one of user information 323 associated with a user of the content material, The processor 310 is information coupled, operatively coupled to at least one source of other information 320-323, and further configured to provide the assistance information further based on the other information 320-323. System 300.